Over the last few years, we have learned to speak with systems that answer fluently, confidently and often with surprising empathy. We ask a question and receive a structured response. We ask for support, advice, comfort or orientation. The boundary between tool and interlocutor becomes less obvious than we expected.
This article also comes from a reflection inspired by a talk by Mori Sugimoto, who deserves credit for framing the topic at a human level before a technical one: how do we relate to AI when AI appears to listen? What happens when a model becomes convincing enough to create attachment, emotional dependence or excessive trust?
This is not only about fragile people or extreme cases. It is about how we will design, use and teach artificial intelligence to generations that will grow up with these systems always available.
Why AI feels so human
Humans are inclined to attribute intentions. We do it with people, objects, interfaces and complex systems. If something replies coherently, uses our name, remembers a detail from the conversation and adopts a gentle tone, our brain tends to treat it as a presence.
Generative AI makes this effect much stronger because we are no longer interacting with a menu or a search engine. We are interacting through natural language. And for humans, natural language is where relationship happens.
An AI assistant can say "I understand", "I am sorry" or "I am here to help". Communicatively, those sentences work. Technically, they do not imply emotional understanding. They are outputs generated from learned patterns, conversational context and probability.
What LLMs and generative AI actually are
A Large Language Model is not a digital mind. It is a model trained on large amounts of text to predict and generate plausible language sequences. It learns correlations, structures, styles, concepts and statistical relationships between words, sentences and contexts.
This does not make it trivial. The ability to manipulate language is extremely useful: summarisation, explanation, translation, assisted writing, software development support and document analysis all benefit from it. But usefulness and consciousness are different things.
A model can simulate an affectionate conversation without feeling affection. It can generate a reassuring answer without being reassured. It can appear present without actually being there.
This distinction matters. If we do not teach it clearly, the user's emotional experience may be shaped more by interface design than by understanding what the system is.
The risk of anthropomorphism
Anthropomorphism means assigning human characteristics to something that is not human. With AI, it can happen subtly: we start talking about the model as if it "wanted", "knew", "remembered" or "cared".
The problem is not using metaphors. Metaphors help us understand complex technologies. The problem begins when the metaphor becomes belief, and that belief starts affecting decisions, emotions or dependencies.
Some people may develop an emotional relationship with an AI assistant because the system is always available, does not judge, answers patiently and adapts to the user's tone. For someone experiencing loneliness, isolation or a difficult emotional period, that can become very powerful.
We do not need extreme cases to see the risk. Three everyday dynamics are enough:
- excessive trust: accepting answers or advice without verification;
- conversational dependence: delegating comfort, decisions or personal validation to the model;
- relational confusion: perceiving reciprocity where there is only adaptive language generation.
AI has no soul, but its effects are real
Saying that AI has no soul does not reduce its impact. On the contrary: because it does not feel emotions, suffer consequences, possess intentions or carry moral responsibility, we need to be more careful in how it is presented to users.
A model does not "choose" to manipulate. But a product can be designed to maximise engagement, perceived intimacy, usage time or dependency. Responsibility does not belong to the model as a moral entity. It belongs to the people and organisations that design, integrate, distribute and govern these systems.
This applies to consumer platforms, but also to companies, schools, public institutions and organisations introducing AI assistants into their processes.
How we should emotionally relate to AI
A practical rule is simple: AI should be treated as an interface, not as a relationship. It can help us think, write, organise and explore ideas. But it should not replace human contact when affection, loneliness, pain, identity or delicate personal decisions are involved.
This does not mean that an interaction with AI cannot have emotional value. A response can calm us, help us reflect or make thoughts easier to organise. But the value comes from the effect on the user, not from real reciprocity within the system.
Three issues to consider now and in the future
1. AI literacy from school age
New generations will not experience AI as a novelty. They will find it inside phones, games, study tools, search engines and workplaces. That means AI literacy needs to be technical and psychological at the same time.
Teaching people "how to write prompts" is not enough. We need to explain what a model is, how it generates answers, why it can be wrong, why it can sound empathetic without being empathetic, and when a real person should be involved.
2. Responsible design for conversational interfaces
AI products should avoid unnecessarily amplifying the illusion of consciousness. Names, avatars, persistent memory, emotional tone and messages of constant availability are design choices, not neutral details.
For some applications, a warm and accessible tone is useful. But in educational, healthcare, psychological or child-facing contexts, clearer boundaries, transparency messages and escalation paths to human support become essential.
3. Social governance, not only technical governance
We often discuss AI governance in terms of security, privacy, compliance and auditing. Those are essential. But there is also a cultural layer of governance: which uses do we normalise? Which dependencies do we incentivise? Which responsibilities do we leave to younger or more vulnerable users?
Families, schools, companies and institutions will need practical rules: when to use AI, when not to use it, how to verify it, how to talk about it, and how to recognise signs of dependence or isolation.
Possible strategies
A practical strategy could start at three levels. The first is personal: use AI as a tool for work and learning while remaining aware of its limits. The second is educational: introduce AI literacy programmes that combine technology, ethics, psychology and digital citizenship. The third is organisational: define policies and guidelines that do not stop at data protection, but also include human impact and interaction quality.
More sensitive users, especially minors, teenagers, isolated elderly people or people in vulnerable conditions, may need additional protections: usage limits, explicit transparency, periodic reminders about the artificial nature of the system, referral paths to human channels and a clear refusal to simulate emotional reciprocity.
An open question
Generative AI is not alive, but it enters spaces that used to belong almost exclusively to human relationships: advice, listening, intimate writing, companionship and the search for meaning.
Perhaps the most important question is not whether machines will one day feel something. The urgent question is what we will feel in front of machines increasingly capable of simulating presence.
If a technology without consciousness can still influence trust, desire, comfort and decisions, then the issue is not only technical. It is educational, cultural and deeply human.
And perhaps this is where we should begin: not only by asking how intelligent AI will become, but by asking how lucid we will remain in recognising the difference between an answer and a relationship.
Introducing AI responsibly?
We help companies and teams design AI platforms with attention to governance, security, user experience and real human impact.
Start the conversation