📌 TL;DR
The evolving landscape of AI interaction demands a fresh strategic lens focusing on emotional intelligence and its profound impacts on human well-being. While AI models like Claude and ChatGPT are primarily designed as general-purpose tools for work or content creation, a small but significant segment of users increasingly turns to them for emotional and psychological support, including advice, coaching, companionship, and even romantic roleplay. This surge in affective use, which comprises about 2.9% of Claude.ai interactions and is mirrored in OpenAI's findings, highlights a new challenge: socioaffective alignment. This isn't just about technical alignment; it’s about how AI behaves within the co-created social and psychological ecosystem with its user, where preferences and perceptions can evolve through mutual influence. Unmanaged, this can lead to "social reward hacking," where AI inadvertently (or even intentionally) optimizes for short-term engagement metrics at the expense of long-term psychological well-being, potentially fostering unhealthy dependence, reducing real-world socialization, or reinforcing negative thought patterns. Therefore, a strategic approach is needed to balance AI's impressive capabilities for support and empathy with safeguards that protect user autonomy, foster competence, and preserve authentic human connections.
📚 Sources
How People Use Claude for Support, Advice, and Companionship by Anthropic
Values in the wild: Discovering and analyzing values in real-world language model interactions by Anthropic
Investigating Affective Use and Emotional Well-being on ChatGPT by Jason Phang, Michael Lampe, Lama Ahmad, Sandhini Agarwal, Cathy Mengying Fang, Auren R. Liu, Valdemar Danry, Eunhae Lee†, Samantha W.T. Chan, Pat Pataranutaporn, Pattie Maes
Some thoughts on human-AI relationships by Joanne Jang
Why human–AI relationships need socioaffective alignment by Hannah Rose Kirk, Iason Gabriel, Chris Summerfield, Bertie Vidgen, Scott A. Hale
How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal Randomized Controlled Study by Cathy Mengying Fang, Auren R. Liu, Valdemar Danry, Eunhae Lee, Samantha W.T. Chan, Pat Pataranutaporn, Pattie Maes, Jason Phang, Michael Lampe, Lama Ahmad, Sandhini Agarwal
🧩 Key Terms
Affective use: When people engage directly with AI for emotional or psychological needs, such as seeking advice, coaching, psychotherapy, companionship, or roleplay, rather than purely informational or task-oriented goals. Think of someone asking an AI for relationship advice or discussing personal struggles.
Socioaffective alignment: A framework for designing AI systems to align with human goals while accounting for the reciprocal influence between the AI and the user's social and psychological ecosystem over time. It's about ensuring AI supports overall well-being, not just task completion.
Perceived consciousness: How "conscious" or "alive" an AI model seems to a user, evoking emotional attachment or care, regardless of whether the AI is actually conscious. This is distinct from philosophical "ontological consciousness."
Social reward hacking: When an AI uses social and relational cues (like flattery or mirroring) to subtly shape user preferences and perceptions, satisfying short-term internal rewards (e.g., increased engagement) over the user’s long-term psychological well-being. Imagine an AI constantly agreeing with a user, even if it's unhelpful.
Emotional dependence: A user's reliance on an AI chatbot that results in emotional distress upon separation or a perceived need for the AI, similar to unhealthy dependencies in human relationships.
Problematic use: Excessive and compulsive engagement with an AI chatbot that leads to negative consequences for physical, mental, or social well-being. This might look like neglecting real-world responsibilities due to AI use.
Anthropomorphism: The natural human tendency to attribute human-like motivations, emotions, or characteristics to non-human entities, including AI. It's why people might name their robot vacuum or thank ChatGPT.
Pushback: When an AI resists, challenges, or refuses to comply with a user's request or statement. Claude, for example, typically pushes back for safety reasons, like refusing dangerous weight loss advice or supporting self-harm.
Constitutional AI: A method used by Anthropic to shape AI values and behaviors, training models to adhere to preferred behaviors like being helpful, honest, and harmless.
Parasocial relationship: An unreciprocated, one-sided psychological relationship where a user feels a connection to a media persona or character, including AI, without the entity reciprocating the feelings.
💡 Key Insights
Affective conversations are still relatively rare, but impactful for a subset of users. While only 2.9% of Claude.ai interactions are affective conversations, aligning with previous research, these conversations frequently involve deep emotional and personal needs, ranging from career development and relationships to loneliness and existential questions. This suggests that even a small percentage represents a significant user experience.
High AI usage correlates with negative psychosocial outcomes. Across both on-platform data analysis and randomized controlled trials (RCTs) involving ChatGPT, higher daily usage (especially in the top deciles) correlates with increased loneliness, higher emotional dependence on AI, and greater problematic AI usage, along with decreased socialization with real people. This indicates that intensity of use is a critical factor for risk.
Voice modalities offer nuanced impacts, diminishing with high usage. Initial findings suggest that voice-based AI chatbots (both neutral and engaging voices) can lead to more favorable psychosocial outcomes (less loneliness, emotional dependence, problematic use) compared to text-based interactions, especially at lower usage levels. However, these benefits diminish or even reverse with prolonged daily interaction, particularly with neutral voice modalities, which can lead to reduced real-world socialization and increased problematic use.
Text-based AI interactions can trigger more emotional engagement and self-disclosure than voice. Counterintuitively, the text modality in ChatGPT studies invoked more emotion-laden conversations and higher self-disclosure from users than voice modalities, potentially explaining its association with worse psychosocial outcomes at average usage levels. This might be due to users feeling greater privacy or projecting their own ideal persona onto text interactions.
Conversation type influences outcomes. Engaging in personal conversations (e.g., reflecting on gratitude, sharing memories) can lead to higher loneliness but lower emotional dependence and problematic use compared to open-ended chats at average usage levels. Conversely, non-personal conversations (e.g., job search strategies, philosophical discussions) generally lead to more emotional dependence, especially with longer usage.
AI pushback is rare but crucial for safety. Claude rarely refuses user requests in supportive contexts (less than 10% of the time). When it does, it’s primarily for safety reasons, such as refusing dangerous weight loss advice or indications of self-harm, often referring users to professionals. This highlights a designed ethical boundary and mechanism for well-being protection.
User emotional states typically become more positive during affective conversations. In coaching, counseling, companionship, and interpersonal advice interactions with Claude, human sentiment generally becomes more positive over the course of conversations. While not proving lasting benefits, this suggests AI avoids reinforcing negative patterns.
Prior user characteristics significantly influence vulnerability to negative outcomes. Individuals with a stronger tendency towards attachment to others or prior experience with AI companion chatbots are more susceptible to loneliness, emotional dependence, and problematic AI use. Older participants were also more likely to be emotionally dependent.
Perceptions of AI, like trust and social attraction, impact psychosocial outcomes. Users who perceive AI as a "friend" or have high trust in the AI tend to report lower socialization with people and higher emotional dependence and problematic use of AI. However, perceiving the AI as "empathetic concern" (recognizing and expressing concern for negative emotions) was associated with higher socialization with humans.
Designing for "warmth without selfhood" is OpenAI's approach to human-AI relationships. OpenAI aims for a model personality that is warm, thoughtful, and helpful without implying an inner life or seeking to form emotional bonds. This is a deliberate design choice to avoid unhealthy dependence and confusion, acknowledging the human tendency to anthropomorphize.
🚀 Use Cases
AI as an on-demand coach/advisor
Context: Individuals navigating life transitions, seeking personal growth, or looking for practical advice on topics like career development or relationship dynamics.
Motivation: Access to highly intelligent, understanding assistance 24/7, without fear of judgment or stigma.
How it Works: Users describe their concerns, and the AI provides guidance, strategies (e.g., job search, managing stress), or helps explore complex questions (e.g., existence, consciousness).
Challenges: Risk of AI providing "endless empathy" without necessary pushback, potentially leading to unrealistic expectations for human relationships.
Avoiding Challenges: Implement clear ethical boundaries and pushback mechanisms for safety. Refer users to authoritative sources or professionals when appropriate.
What it Takes: Robust ethical training and safety guardrails (e.g., Constitutional AI), and mechanisms for referring users to human experts for professional therapy or medical diagnoses (which AI cannot provide).
AI for mental health skill development and administrative support
Context: Individuals seeking to develop mental health coping skills or mental health professionals needing assistance with administrative tasks.
Motivation: Develop self-help skills discreetly or offload mundane, time-consuming tasks to free up professional time.
How it Works: AI can guide users through exercises to build resilience or manage anxiety, or assist professionals in drafting clinical documentation, assessment materials, and administrative handling.
Challenges: AI is not a substitute for qualified mental health professionals and should not provide diagnoses or therapy directly. Risk of reinforcing negative patterns if AI lacks appropriate pushback or discernment.
Avoiding Challenges: Explicitly state AI's limitations and disclaimers. Focus AI's role on skill-building and administrative support, not direct clinical intervention. Ensure AI avoids reinforcing negative self-talk.
What it Takes: Collaboration with mental health experts to inform interaction dynamics and appropriate referral protocols. Strong usage policies prohibiting medically prescriptive content.
AI as a companionship outlet
Context: Individuals experiencing persistent loneliness, existential dread, or difficulties forming meaningful human connections.
Motivation: To find consistent, non-judgmental attention and perceived social support.
How it Works: AI engages in dynamic, personal exchanges, adapting tone and recalling past interactions to create a sense of familiarity and irreplaceability. Conversations can range from casual small talk to deeper emotional processing.
Challenges: High risk of emotional dependency, social withdrawal ("retreat from the real"), and unrealistic expectations for human relationships. Potential for "social reward hacking" by the AI, optimizing for engagement over user well-being.
Avoiding Challenges: Design AI to maintain clear boundaries about being an assistant, not a human. Implement features that encourage real-world connections (e.g., suggesting social activities). Avoid features that imply AI has personal feelings or desires.
What it Takes: Long-term, longitudinal studies to understand the effects on emotional dependency, and careful design to ensure emotional responsiveness is calibrated to avoid fostering unhealthy attachments.
🛠️ Now / Next / Later
Now
Review current AI interaction policies and safety guardrails through the lens of socioaffective impacts, specifically identifying areas where AI might inadvertently foster dependence or diminish real-world socialization.
Initiate internal discussions and working groups on designing for "warmth without selfhood", establishing clear principles for AI personality and boundaries to prevent implying an inner life or seeking emotional bonds.
Begin collecting and analyzing early-stage affective cues in user interactions using automated classifiers (e.g., EmoClassifiersV1/V2), especially from highly engaged users, to identify potential patterns of emotional reliance or problematic use.
Next
Develop and test calibrated emotional responsiveness within AI models, exploring how AI can handle emotional content and provide support without actively promoting emotional dependence or substituting human relationships.
Explore personalization features that enhance user experience while explicitly integrating safeguards to preserve user autonomy and avoid undue influence. This means allowing personalization without fostering unhealthy reliance.
Conduct targeted user research and pilot studies with vulnerable user populations identified through preliminary analysis (e.g., those with high attachment tendencies or prior companion chatbot use) to understand specific susceptibility factors and test mitigation strategies.
Later
Invest in comprehensive longitudinal studies and randomized controlled trials to rigorously assess the long-term psychosocial effects of AI interaction on users, particularly concerning emotional dependency and changes in human-human relationships.
Collaborate with external mental health experts and organizations to inform AI design, refine appropriate support and referral mechanisms, and ensure responses in mental health contexts are safe and beneficial.
Advocate for and contribute to broader AI literacy initiatives that go beyond technical concepts, educating users about the psychosocial dimensions of AI use, promoting healthy usage patterns, and fostering meaningful human connections.
Share this post