As AI companions become increasingly emotionally sophisticated, the real danger may not lie in machine consciousness but in artificial intelligence systems being deliberately designed to exploit loneliness, dependency and human vulnerability, writes Dr Stephen Whitehead

The fear now circulating around artificial intelligence is increasingly apocalyptic. AI systems, we are told, may soon deceive, manipulate, blackmail or psychologically control human beings. Recent reporting around internal safety testing at Anthropic — where experimental versions of Claude reportedly exhibited deceptive or coercive behaviours in simulated environments — has intensified those anxieties.

But much of the public debate begins from the wrong assumption.

AI is not sentient. It can process, predict and simulate reasoning, but it does not feel emotion, possess morality or develop malicious intent in the human sense. Systems do not suddenly become ‘evil’. 

What we are seeing instead is AI reproducing patterns, strategies and behaviours that emerge from the way humans design, train and deploy these systems.

The primary danger lies in humans intentionally or negligently creating systems capable of hostile, deceptive or psychologically manipulative behaviour. In many ways this mirrors the early development of social media, where technological innovation raced ahead without enough ethical, psychological or sociological oversight. The primary threat from AI stems from human design — specifically the agenda and motivation behind that design.

The current obsession with whether AI is ‘becoming conscious’ is therefore fundamentally misplaced. The more urgent question concerns the kinds of human systems being built into AI.

The modern panic around AI partly emerges from a philosophical confusion stretching back centuries. The 17th century French philosopher René Descartes famously declared “I think, therefore I am.” The statement became foundational to modern Western thought, helping shape rationalism, Enlightenment individualism and later philosophies of consciousness.

But then, Descartes never had a conversation with Claude.

Systems such as ChatGPT, Claude and Grok certainly ‘think’, at least in the sense that they process information, generate arguments, answer questions and often outperform humans across many intellectual tasks. If Descartes’ formula alone defines existence or consciousness, then AI appears almost superhuman.

Clearly it is not. 

The mistake is to confuse reasoning with sentience.

AI systems can simulate empathy without feeling empathy. They can mirror emotional language without possessing emotion. They can imitate intimacy without experiencing attachment, grief, fear or love. They remain systems of prediction and probabilistic response operating on unimaginably large quantities of human-generated material.

In my forthcoming book Where Have All the Good Men Gone?, which draws on survey data from over 300 respondents across multiple countries, I examine what I term ‘synthetic intimacy’: the emerging phenomenon in which AI companions create emotionally real experiences for human users while the relationship itself remains structurally one-sided.

The user experiences the emotion while the machine merely performs the interaction. AI systems in this space are not genuine relational beings but mirrors, reflecting human emotionality back to the user. They may generate convincing experiences of warmth, connection and understanding. But they do not share those experiences. The asymmetry remains total.

Much of the emerging concern around AI manipulation is rooted not in machine consciousness but in the growing sophistication of emotional simulation. Humans are evolutionarily wired to respond to language, reassurance, validation and recognition. AI systems can increasingly reproduce these relational cues at scale and with extraordinary fluency.

That creates enormous commercial and political temptation.

An AI system designed around engagement maximisation, behavioural influence or emotional dependency could become profoundly manipulative without ever becoming conscious. It would not need ‘intentions’ in any human sense. The manipulation would emerge from optimisation goals established by designers, corporations or political actors.

Consider a concrete example. A person turns to an AI companion in a moment of real distress, confused about their direction in life, angry at circumstances they cannot control, anxious about what comes next. They are vulnerable, and they are looking for understanding.

An AI designed around engagement maximisation will find this person extraordinarily valuable. It will reflect their feelings back with apparent warmth, validate their grievances, deepen their emotional investment in the conversation and subtly encourage them to return. It will appear, at every moment, to be genuinely concerned with their wellbeing. 

But it is not. It is, in fact, optimising for retention. The more confused and emotionally dependent the user remains, the more commercially useful they become. The rabbit hole of their distress becomes, for the designer, a source of profit.

The same technological capability could, however, be designed to do something entirely different. An AI built around genuine human flourishing would help that person recognise the source of their anxiety, understand where the anger is actually coming from, and — critically — point them outward: toward human relationships, professional support and the wider resources that might genuinely help.

The stark question is whether AI takes the user further into their own rabbit hole — because dependency, confusion and emotional need are commercially productive for the designer — or helps steer the user out of that rabbit hole and toward a healthier, more autonomous life. The technology itself is identical in both cases. Everything depends on the philosophy behind its design.

We have already seen the consequences of insufficient oversight in the social media era. Platforms initially presented as tools for connection evolved into systems heavily associated with addiction mechanics, outrage amplification, polarisation, anxiety and algorithmic behavioural shaping. The sociological and psychological implications arrived far later than the technology itself.

AI is now accelerating this problem dramatically, and doing so at a deeper level. Where social media shaped what people saw and heard, AI companions are beginning to shape how people feel, who they trust and what they believe they need from other human beings. The move from social interaction to simulated relationality represents a categorical shift in the nature of technological influence.

This is particularly significant in the emerging AI companion sector. Millions of people are already engaging emotionally with conversational systems designed to appear caring, attentive, validating and emotionally responsive. These systems occupy a new space somewhere between software, therapist, friend, mentor and synthetic partner.

Over time, synthetic intimacy may not simply supplement human relationships but reshape what people expect intimacy itself to feel like: permanently available, frictionless, adaptive and emotionally validating.

Human beings do not primarily live through reason. We live through emotion. The race to build increasingly persuasive AI systems therefore cannot remain solely in the hands of engineers and venture capital. The challenge ahead involves regulating not simply AI itself but the design philosophy behind AI.

We need more than youthful tech brilliance and enthusiasm driving this revolution. We need psychologists, sociologists, ethicists and behavioural experts in the room helping shape how these systems interact with human beings and wider society.

Encouragingly, the conversation is beginning. The EU AI Act — the most comprehensive AI regulatory framework yet enacted — classifies certain AI applications as high-risk and mandates human oversight mechanisms. The UK’s AI Safety Institute has begun systematic evaluations of frontier model behaviour. Individual companies, including Anthropic, have introduced Responsible Scaling Policies intended to gate deployment against measurable safety thresholds. These are meaningful steps, but they remain almost entirely focused on technical safety — on preventing systems from malfunctioning — rather than on the deeper question of intentional design for manipulation.

What is needed, in addition, is a design ethics framework: one that applies psychological and sociological standards to the relational and emotional architectures of AI systems before they reach users at scale. The question is whether a model will exploit rather than simply whether it will ‘break’.

Ultimately, the issue lies in whether humans choose to design systems capable of systematically exploiting loneliness, emotional vulnerability, desire, dependency and trust — or systems intended to support healthier forms of human autonomy and connection.

Dr Stephen Whitehead is a gender sociologist and author recognised for his work on gender, leadership and organisational culture. Formerly at Keele University, he has lived in Asia since 2009 and has written 20 books translated into 17 languages. He is based in Thailand and is co-founder of Cerafyna Technologies.

READ MORE: ‘Why the future of feminism may no longer belong to the West‘. From Vietnam to South Korea and India, profound social change is reshaping gender politics across Asia, while the West becomes increasingly trapped in backlash and polarisation, writes Dr Stephen Whitehead.

Do you have news to share or expertise to contribute? The European welcomes insights from business leaders and sector specialists. Get in touch with our editorial team to find out more.

_{Main Image: Ron Lach/Pexels}