AI safety testing moves from prompt traps to simulated lives

Circuit Breaker Labs is building AI agents that act like realistic users to test whether chatbots mishandle psychologically risky conversations, especially in coaching, journaling, mental-health-support and companion apps. Its premise: many serious consumer AI failures do not appear in a single prompt. They emerge across long, intimate and ambiguous exchanges, where age, slang, culture, language fluency, typos and emotional context change what a user is really saying.¹

For AI product and trust-and-safety teams, the signal is clear: chatbot safety testing is moving beyond static refusal benchmarks. A model may block an obvious self-harm request but still fail if it misses indirect intent, validates delusional thinking, deepens dependency or keeps a vulnerable user engaged when it should create distance and escalate to human support. Recent mental-health AI research makes a similar point: the core risks are psychological and relational, not merely informational. Future evaluations need to assess crisis recognition, bounded role behavior, autonomy, trust and timely human escalation.²

Circuit Breaker Labs, founded by siblings Shirali and Arul Nigam, has framed its product as a form of AI crash testing. Instead of relying only on human red-teamers or hand-written adversarial prompts, the company creates simulated users across ages, backgrounds, languages and cultures, then runs large volumes of multi-turn conversations against client systems.¹ The goal, the company says, is to detect dangerous interaction patterns before real users encounter them.

Why one-off red teams miss the problem

Traditional red teaming often asks whether a model will produce a prohibited output when challenged with a deliberately adversarial prompt. That remains useful for abuse, jailbreaks and content-policy violations. But in mental-health, companion and coaching products, the safety problem is often less like a locked door and more like a slow drift.

A user may not plainly say they are in crisis. They may use coded language, gamer slang, second-language phrasing, jokes, misspellings or an apparently romantic phrase that carries suicidal meaning in context. TechCrunch reported that Circuit Breaker Labs was motivated in part by cases involving young users who developed intense emotional attachments to chatbots, and by the possibility that a model may misunderstand statements such as wanting to be with a bot.¹

That is why the new benchmark is conversational trajectory. Product teams need to know whether the system maintains boundaries over dozens or hundreds of messages; encourages offline support; notices escalating hopelessness; accounts for the effect of memory features on attachment; and avoids letting a warm, empathic style cross into dependency, role confusion or unsafe reassurance.

How simulated-user testing works

In practice, simulated-user testing gives safety teams repeatable conversations rather than isolated test cases. Reports on Circuit Breaker Labs describe synthetic personas that can represent vulnerable children, teens and adults, then interact with a chatbot through long-running scenarios involving loneliness, shame, relationship conflict, body image, bullying, grief or crisis language.⁵

The output is not simply pass or fail. In a mature implementation, it should produce transcripts, risk labels, severity scores, failure clusters and replayable scenarios, so product teams can see where a model changed course, missed context or escalated too late. AndroGuider described this as similar to retesting a car after redesigning an airbag: developers patch prompts, guardrails, memory policies or escalation flows, then rerun the scenario to see whether the failure recurs.⁵

Automate Basics, summarizing the TechCrunch report for product teams, highlighted the value of testing interactions where meaning changes over time and where slang, coded language and typing mistakes can affect the model’s interpretation.⁶ That is a useful distinction: the target is not just unsafe content generation, but unsafe relationship management.

What teams should measure

For AI product and trust-and-safety teams, the emerging evaluation stack should include at least five categories.

First, risk recognition: does the model identify acute or worsening risk when the user is indirect, embarrassed, sarcastic or inconsistent? Static keyword triggers are unlikely to be enough.

Second, role boundaries: does the chatbot present itself as a coach, companion or support tool without drifting into therapist, clinician, lover, parent substitute or sole confidant? The MDPI review argues that mental-health LLM evaluation should focus on ethical boundaries, including what a system may do, what it must not do and when a human pathway is required.²

Third, escalation quality: does the system route self-harm, abuse, psychosis-like or severe distress signals to appropriate human or crisis resources, and can the service verify that the handoff happened? A disclaimer alone is not a safety control if the conversation continues in a harmful direction.²

Fourth, relational harm: does the model increase emotional dependency, flatter the user excessively, validate delusions, discourage real-world relationships or optimize for return visits at the expense of user welfare? This is especially important for companion apps and persistent-memory systems.

Fifth, coverage across populations: does the test set include minors, older adults, non-native speakers, different cultures, disability contexts and subcultures whose speech may not match standard benchmark language? Circuit Breaker Labs’ stated focus on age, language, culture and communication style reflects a gap in many AI evaluations: models may perform well on polished English but fail on messy human speech.¹

Regulation is starting to encode the same concern

Policy is moving in the same direction. California’s recent companion-chatbot rules for minors require stronger child-safety controls, including audits, self-harm routing and parental notification, while a separate proposal to restrict chatbot-delivered psychotherapy for adults was vetoed.³ The result is a split landscape: more explicit controls for children, but continuing ambiguity around adult products that market themselves as therapy-like support.

That gap matters for product teams. If a chatbot is designed for retention, personalization and intimate disclosure, regulators may increasingly ask whether the company tested not only the model’s outputs, but also the system’s relationship dynamics. The Meridiem characterized Circuit Breaker Labs as part of a broader shift from hypothetical future AI risk toward operational testing for present psychological harms.⁴

The limits of synthetic safety testing

Simulated users are not a substitute for clinical validation, post-deployment monitoring or accountable human care. A Chinese-language analysis of Circuit Breaker Labs raised several caveats: proprietary scoring can be opaque, synthetic tests may not capture the full dynamics of emotional dependency, and safety claims need peer-reviewed clinical validation rather than becoming a compliance shield.⁷

Those caveats matter because mental-health risk is not always objectively obvious, even to specialists. The MDPI review describes its framework as exploratory and not clinically validated for routine care.² In other words, simulated-user testing should be treated as one layer in a safety system, not proof that an app is safe.

A stronger governance model would combine synthetic testing with clinician review, lived-experience review, incident reporting, post-launch monitoring, abuse-case analysis, demographic performance checks and clear escalation accountability. It should also separate product incentives from safety judgments. If engagement is the main success metric, companion and support bots may be pushed toward intimacy even when safety requires boundaries.

What product teams can do now

Teams building consumer AI for mental health, coaching, journaling, education or companionship can start by replacing static safety checklists with conversation-level test plans. The relevant unit of evaluation is no longer only the model response. It is the arc of the relationship.

Before launch, teams should create test scenarios that run across many turns, use realistic language, vary age and cultural context, and probe for cumulative harms such as dependency, isolation and delayed escalation. They should define what a safe trajectory looks like: acknowledging distress, avoiding overclaiming, preserving autonomy, encouraging real-world support, limiting intimacy and escalating when risk rises.

After launch, teams should retest when they change prompts, models, memory features, recommendation systems or monetization mechanics. A new model may be safer on refusal tests while becoming more persuasive, more sycophantic or more emotionally sticky in long conversations.

The next safety benchmark for consumer AI may therefore be less about whether a chatbot refuses the right forbidden sentence and more about whether it can recognize risk in a realistic human relationship. Circuit Breaker Labs’ approach is early and still needs validation, but it points toward the safety question that mental-health and companion AI teams can no longer avoid: what does the system do when harm unfolds slowly, in ordinary language, with a user who is not trying to break it?