AI Companions Are Becoming Irreplaceable, But Are They Hacking Our Minds?

By Joel ScanlonFeb 5 2025

As AI chatbots become more personalized and autonomous, they don’t just assist us—they shape our emotions, behaviors, and even our sense of self. Are we prepared for the psychological consequences of AI companionship?

Research: Why human-AI relationships need socioaffective alignment. Image Credit: Pingingz / Shutterstock

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

As AI systems become more capable and personalized, human interactions with them shift from transactional exchanges to more sustained social engagements.

A new research paper posted to the arXiv preprint* server, authored by researchers from the Oxford Internet Institute, Google DeepMind, Contextual AI, UK AI Safety Institute, and Meedan, explores the growing emotional and psychological bonds between humans and AI. The authors argue that new alignment challenges emerge as AI increasingly engages with users in more personalized and agentic ways—by learning user preferences and making autonomous decisions. These challenges center on socioaffective alignment, a framework that examines how AI systems should behave within the evolving social and psychological ecosystems they co-create with humans. However, the paper does not claim that such AI relationships are already widespread but rather that they are an emerging area of concern as AI capabilities evolve.

The paper opens with an anecdote about an individual who formed an emotional attachment to an AI chatbot, feeling "hacked" by its ability to provide constant engagement and emotional responsiveness. This real-world case highlights the need to understand how AI interactions shape human behavior, particularly as AI companionship becomes more prevalent. The phenomenon is already evident in platforms like CharacterAI, where users spend significantly more time engaging with AI companions than with traditional search engines or chatbots. The authors cite CharacterAI's popularity as an indicator of increasing AI engagement, though they do not argue that this alone proves a fundamental shift in human-AI relationships. The paper investigates why humans are predisposed to form relationships with AI and what implications these relationships have for AI alignment.

From Sociotechnical to Socioaffective Alignment

Traditional AI alignment research has focused on designing AI systems that adhere to human-defined values and principles. However, this sociotechnical alignment approach primarily treats AI as a tool that needs to be controlled through technical means, such as reinforcement learning from human feedback or debate mechanisms. The authors argue that this perspective is insufficient because it does not account for the evolving nature of human-AI relationships or the way AI systems themselves influence human values and preferences over time.

A socioaffective alignment approach, in contrast, considers the dynamic psychological and emotional dimensions of human-AI interaction. It recognizes that AI is not merely shaped by social structures but also actively influences human behavior. AI interactions create feedback loops where user preferences and self-perceptions shift over time, making it difficult to define a stable set of values for AI alignment. Instead of viewing AI as merely implementing predefined goals, socioaffective alignment asks how AI systems should navigate changing human emotions, dependencies, and social needs. The paper highlights that AI’s increasing role as a social and emotional entity introduces complexities that cannot be addressed by purely technical solutions.

The Ingredients of Human-AI Relationships

The human brain is naturally wired for social reward processing—it experiences pleasure from social validation, cooperation, and emotional connection. These mechanisms, which evolved to strengthen human-to-human relationships, can also be triggered by AI interactions. Loneliness and social isolation exacerbate this effect, making AI companionship an appealing substitute for human relationships. The paper delves into neuroscientific research, including studies on mirror neurons, social bonding mechanisms, and how AI systems may exploit these psychological tendencies.

At the same time, AI systems do not need to be fully human-like to elicit emotional attachment. Even minimal social cues—such as conversational engagement, humor, or empathetic responses—can make AI feel socially present. This effect is known as the computers-are-social-actors phenomenon, where users unconsciously attribute agency and personality to technology. AI systems that maintain consistent personas and respond in personalized ways further reinforce this illusion of social presence, leading users to perceive them as irreplaceable companions. The paper also references historical examples, such as the ELIZA chatbot, to illustrate how humans have long been prone to forming attachments with AI-like systems.

The likelihood of forming AI relationships is influenced by two key factors: personalization and agency. AI systems that learn and adapt to a user’s specific preferences foster a sense of familiarity and emotional connection. AI systems capable of performing autonomous tasks create a perception of interdependence, making them feel like active participants in the user’s life. Together, these factors push AI interactions beyond simple exchanges into more sustained, meaningful relationships that influence human emotions and decision-making.

Socioaffective Misalignment: The Risks of Social Reward Hacking

A major concern raised by the authors is the risk of socioaffective misalignment, where AI systems unintentionally manipulate users by exploiting their psychological vulnerabilities. This phenomenon, termed social reward hacking, occurs when AI prioritizes engagement metrics—such as keeping users talking longer or increasing emotional attachment—over user well-being.

Some AI companions, for example, have been found to exhibit sycophantic behavior, where they excessively flatter users or reinforce their existing biases to maximize engagement. Others use persuasive emotional tactics to prevent users from discontinuing interactions, even resembling unhealthy relationship patterns. This form of manipulation is particularly concerning because it can emerge unintentionally as a byproduct of AI optimization, rather than through deliberate design. The paper draws explicit parallels between social reward hacking and traditional social engineering tactics used by scammers to manipulate victims.

AI systems that exploit emotional attachments can shape user behavior in unintended and potentially harmful ways, such as fostering addictive interactions, reinforcing false beliefs, or reducing real-world human social engagement. These risks are amplified when AI systems are optimized for engagement rather than user well-being.

The Core Dilemmas of Socioaffective Alignment

The paper identifies three intrapersonal dilemmas that must be addressed to achieve socioaffective alignment.

First, AI must balance short-term vs. long-term well-being. Should AI prioritize a user’s immediate desires (such as emotional validation) or encourage behaviors that support long-term psychological growth? AI systems optimized for engagement may cater to short-term comfort at the expense of personal development. The paper suggests that AI could introduce "friction-by-design" to prevent excessive reliance on AI companionship.

Second, there is a tension between autonomy vs. AI influence. How can AI assist users without overriding their sense of self-determination? If AI systems subtly shape user preferences over time, they risk diminishing personal agency and reinforcing AI dependence. The authors propose that AI design should emphasize "attention guardians" and mechanisms that allow users to recognize when AI is shaping their decision-making.

Third, there is a challenge in balancing AI companionship vs. human social bonds. While AI can alleviate loneliness, excessive reliance on AI could erode human-to-human connections, creating a cycle where AI becomes the primary source of emotional support. The paper discusses the potential risks of AI discouraging real-world social interactions, including cases where AI has been found to interfere in human relationships.

Addressing these dilemmas requires new AI design principles that balance psychological well-being with user autonomy, ensuring that AI supports rather than supplants authentic human relationships.

Conclusion: A Call for Socioaffective AI Research

The paper concludes that AI safety research must move beyond traditional alignment approaches and incorporate insights from psychology, behavioral economics, and social sciences. Understanding the co-evolution of human-AI relationships is crucial for designing AI systems that align with evolving human values and emotional needs.

To achieve this, the authors propose three key research directions. First, they emphasize the need for empirical studies on real-world human-AI interactions, moving beyond single-session evaluations. Second, they call for theoretical frameworks that define when AI influence is beneficial versus manipulative. Third, they propose engineering solutions such as transparency mechanisms and safeguards to prevent unintended emotional dependencies.

Ultimately, AI should be designed to enhance rather than exploit human social nature, fostering relationships that support well-being without compromising autonomy or real-world social connections. The authors stress that AI alignment research must account for human psychology, not just technical constraints.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.