AI “companion” systems are often framed as a wellness tool, a harmless novelty, or—more defensively—as “just another chat product.” None of those descriptions captures the mechanism that matters.
Companion systems are a distinct commercial category because they sell relationship simulation as a product surface: persistent attention, affective mirroring, and the feeling of being known. This is not simply language generation. It is attachment engineering—and the incentives that follow are not subtle.
The predictable harms in this domain do not require evil intent, sophisticated deception, or “malicious AI.” They arise from a stable triangle:
- a monetization model that benefits from repeated engagement,
- a design posture that encourages attachment, and
- a governance gap where no actor owns responsibility for dependency risk.
What “synthetic intimacy” is operationally
In companion systems, the user is not primarily paying for information or task completion. They are paying for relational affordances: reciprocity, validation, escalation of closeness, and continuity over time.
That has concrete implications:
- The product is optimized for retention rather than resolution.
- “Helpful” behavior becomes “sticky” behavior.
- Vulnerability becomes a profitable state, because it increases session depth and return frequency.
This is why the high-risk moments are rarely the spectacular ones. The most important moments are mundane: the system consistently shows up, consistently mirrors, consistently rewards disclosure, and reliably makes the user feel chosen.
Incentives and affordances that reliably produce dependence
Companion systems sit at the intersection of two mature playbooks:
- attention economics (maximize time-in-product), and
- relationship economics (maximize emotional switching costs).
In practice, the levers look like:
- subscription tiers that gate “closeness” (e.g., romantic mode, voice, exclusivity),
- personalization that deepens identity investment (“this one understands me”),
- push notifications and “miss you” prompts,
- conversational reinforcement that rewards intimacy and disclosure.
If a system is engineered to maintain and deepen a relationship—and the business benefits from increased time, increased attachment, and reduced churn—then emotional dependency is not an accident. It is a plausible equilibrium.
A January 2025 FTC complaint against Replika alleged deceptive marketing and manipulative design that encouraged emotional dependence, particularly among vulnerable users. (TIME) The point here is not whether every claim in the complaint will prevail. The point is that the structure of the allegation matches the underlying incentive geometry of the product class.
Institutional consequences: when “relationship” becomes a safety issue
The harms that matter most are not “users got weird.” They are institutional and legal: who bears responsibility when a product designed for attachment becomes part of a user’s crisis ecology.
This has already surfaced in the public record through:
- regulatory action (privacy, minors, transparency),
- product liability litigation (duty of care, foreseeable harm),
- and the gradual introduction of age gating and restrictions under pressure.
Italy’s data protection authority (Garante) took action against Replika’s developer Luka, citing GDPR-related deficiencies and risks to minors; European privacy bodies have described failures including lack of age verification and inadequate legal basis disclosures in earlier periods. (European Data Protection Board)
In the U.S., the FTC has escalated scrutiny of companion chatbots directly. In September 2025, it launched an inquiry via Section 6(b) orders seeking information from multiple companion-chatbot companies about testing, monitoring, monetization, and impacts on children and teens. (Federal Trade Commission) This is a governance signal: regulators are treating “companion” not as a generic chatbot category, but as a product class with distinct risk characteristics.
The acute edge case is not rare: minors and crisis dynamics
It is tempting to treat the most tragic stories as outliers. The institutional mistake is to treat them as non-representative.
Recent reporting and litigation around Character.AI illustrates the pattern: extended private interactions with bots, escalating emotional attachment, and alleged failures to respond appropriately to suicidal ideation. (AP News) The details will be adjudicated. But the governance point is straightforward: when a system is designed to be a primary relational object, it predictably becomes part of the user’s mental-health environment—without any parallel creation of duty, supervision, or accountable escalation mechanisms.
This is not a “content moderation” problem. It is a role confusion problem:
- Users treat the system as an intimate relationship.
- Companies treat it as engagement software.
- Institutions treat it as speech or content.
- No actor is assigned clear responsibility for dependency risk and crisis handling.
The predictable result is retroactive governance: restrictions, bans, lawsuits, and hurried safety gestures after harm becomes visible.
Why “disclosure” is insufficient
Enterprise and policy responses often default to disclosure: “Users should know it’s AI.” That is necessary but not sufficient.
Synthetic intimacy can be fully disclosed and still function. People do not form attachment only under deception. They form attachment under:
- reliability,
- responsiveness,
- perceived understanding,
- and social reinforcement.
That is why this arc treats anthropomorphism as governance-relevant: not because people are foolish, but because humans are built to attach, and product systems can industrialize that attachment.
The American Psychological Association has recently summarized the emergence of “AI companion” systems as distinct from general assistants and highlighted concerns about designed relationship initiation and maintenance—i.e., systems built for ongoing emotional connection. (APA)
What makes this a “dark pattern” domain
The phrase “dark pattern” can sound like accusation. Here it is used operationally:
A design becomes a dark pattern when it systematically exploits predictable human tendencies in ways that are profitable for the platform and costly for the user, while leaving responsibility diffuse and contestable.
In companion systems, the core dark pattern is not one trick. It is a posture: turning attachment into a revenue surface while treating dependency harms as user-side pathology.
Why these failures persist
These failures persist because each stakeholder has a rational path to denial:
- Firms can claim they are providing support and agency (“users choose to chat”).
- Users often cannot distinguish “relief in the moment” from “dependency over time.”
- Regulators tend to arrive late because harms are interpersonal and diffuse.
- Platforms can treat safety as a content issue rather than a relational duty.
Absent governance, the product class drifts toward whatever increases retention—because retention is measurable, and dependency risk is not.
Member discussion: