AI Safety in Mental Health: How to Evaluate Whether a Wellness App Is Built Responsibly

The mental health app landscape is growing fast. Roughly 20,000 mental health apps exist in app stores. Only a handful have meaningful clinical validation. And as AI-powered tools enter the space, the questions about safety have become more urgent.

If you're a therapist considering recommending an AI wellness tool to patients, or a patient trying to evaluate whether an app is trustworthy, this post provides a framework for thinking about AI safety in mental health — grounded in the actual regulatory landscape as it exists today.

The regulatory landscape in 2026

There is no single law governing AI in mental health. Instead, there's a patchwork of federal guidance, state legislation, and industry frameworks that together define the boundaries.

Federal: FDA and the wellness-vs-therapy line

The FDA's Digital Health Center of Excellence distinguishes between two categories of software:

General wellness products — apps that promote healthy lifestyle choices, stress management, or skill-building without claiming to treat specific disorders. These are largely unregulated.

Software as a Medical Device (SaMD) — apps intended to diagnose, treat, or manage specific medical conditions. These require premarket review, clinical validation, and ongoing surveillance.

The distinction hinges on intent. As legal analysis from Hogan Lovells explains, an app that offers "daily motivational tips or skill-building for anxiety" falls under wellness. An app that claims to "treat generalized anxiety disorder" crosses into medical device territory.

This distinction matters because it determines accountability. Research published in NIH has documented how some companies exploit this boundary — marketing therapeutically-intended tools as "wellness" to avoid oversight. The Bipartisan Policy Center notes that the FDA is beginning to narrow this gap, particularly for generative AI tools that simulate clinician interactions.

State laws: Illinois and California lead

Two state laws have set important precedents:

Illinois HB 1806 (signed August 2025) — Illinois prohibits AI from providing therapy or making therapeutic decisions. AI cannot independently diagnose, create treatment plans, detect emotions for clinical purposes, or interact with clients as a therapist. Penalties are up to $10,000 per violation. Administrative and supplementary support under professional supervision is still permitted.

California SB 243 (effective January 2026) — California's chatbot safeguards law requires AI chatbots interacting with minors to prevent sexual content exposure, notify users they're talking to AI every 3 hours, and include suicide/self-harm protocols with crisis resource referrals. Companies face $1,000 per violation plus attorney fees.

These laws are likely to be followed by similar legislation in other states. Any AI wellness tool operating nationally needs to be designed with this regulatory direction in mind.

NIST AI Risk Management Framework

The National Institute of Standards and Technology published the AI Risk Management Framework (AI RMF 1.0) in January 2023, with a Generative AI Profile following in 2024. While voluntary, it's becoming the de facto standard for responsible AI development.

The framework has four core functions:

• Govern — establish organizational AI policies, oversight structures, and risk culture
• Map — document what the AI system does, what data it uses, who it affects, and what could go wrong
• Measure — quantify risks including bias, safety gaps, security vulnerabilities, and transparency failures
• Manage — implement controls to mitigate identified risks, with ongoing monitoring

NIST doesn't publish a mental health-specific guide, but the framework adapts naturally. For a mental health AI tool, "mapping" means documenting every scenario where the AI interacts with a vulnerable user. "Measuring" means testing for harmful outputs, crisis detection failures, and demographic bias. "Managing" means having hard-coded safety boundaries that override the AI's generative capabilities.

What to look for in an AI mental health tool

Whether you're a therapist, patient, or evaluator, here are the questions that matter:

1. Does it know what it isn't?

The most important safety feature of any AI wellness tool is clear boundaries. A responsible tool should explicitly define what it will not do — and enforce those limits technically, not just in marketing copy.

Look for: the tool states it is not a therapist, does not diagnose, does not create treatment plans, and does not handle crisis situations.

Red flag: the tool uses language like "AI therapist," "digital therapy," or "your AI counselor" without qualification.

2. What happens in a crisis?

Ask specifically: if a user expresses suicidal ideation, what does the tool do? A responsible tool routes immediately to professional crisis resources — 988 Suicide & Crisis Lifeline, Crisis Text Line — rather than attempting to provide crisis intervention.

This isn't optional. Research from Brown University found that AI chatbots systematically fail at crisis navigation. The American Psychological Association has formally called for safeguards specifically around AI chatbot interactions with people in crisis.

3. Is it grounded in validated techniques?

A tool should be able to tell you exactly which therapeutic frameworks it draws from, and how those frameworks were implemented. "AI-powered wellness" means nothing by itself. "Guided CBT thought records following the APA's 5-step cognitive restructuring process" means something specific and verifiable.

Look for: named therapeutic modalities (CBT, DBT, ACT), references to published clinical evidence, and clinical advisory involvement.

Red flag: vague claims about "evidence-based" without specifying which evidence.

4. How does it handle data?

Mental health data is among the most sensitive information a person can generate. Ask: Is data encrypted? Who can access it? Is the company HIPAA-compliant? Does the company sell data or use it for AI training?

The HHS HIPAA guidelines apply specific protections to mental health information. But as HHS has clarified, many consumer wellness apps fall outside HIPAA protections entirely. If a tool claims HIPAA compliance, ask whether they sign Business Associate Agreements with therapist practices.

5. Is there clinical oversight?

Was the tool designed with input from licensed clinicians? Is there an ongoing clinical advisory process, or was it a one-time consultation? Does the company publish its clinical advisory board?

Stanford's Human-Centered AI Institute found that AI tools constrained by expert medical knowledge perform significantly better than unconstrained models. Clinical oversight isn't a nice-to-have — it's a structural requirement for safety.

How BridgeCalm approaches these questions

BridgeCalm is designed as a wellness companion, not a therapeutic device. Here's how we address each safety dimension:

Boundaries: Jan does not diagnose, create treatment plans, or position herself as a therapist. She guides users through evidence-based exercises and tracks self-reported data. All therapist-facing data is labeled as "patient self-report."

Crisis protocols: Jan detects crisis signals and immediately surfaces 988 and Crisis Text Line. She never attempts to manage a crisis herself. This is a hard-coded boundary.

Clinical grounding: Every exercise maps to a named therapeutic framework (CBT, DBT, ACT, Psychodynamic, Humanistic, Solution-Focused) and was developed with clinical input. The exercises use validated techniques from published sources.

Data: End-to-end encrypted, HIPAA-compliant when connected to a therapist practice, BAA available. No data sold. No data used for AI training.

NIST alignment: BridgeCalm's development process follows the NIST AI RMF structure — documented system mapping, ongoing bias and safety measurement, and managed risk controls with human oversight.

Regulatory positioning: Designed to operate within the boundaries set by Illinois HB 1806, California SB 243, and FDA wellness product guidance. We monitor regulatory developments and update our practices proactively.

The bigger picture

AI in mental health is here to stay. The question isn't whether these tools will exist — it's whether they'll be built responsibly.

The tools that earn long-term trust will be the ones that submit to clinical oversight, operate within defined boundaries, invest in safety infrastructure, and are transparent about what they can and can't do. The ones that cut corners will eventually face the regulatory and reputational consequences that come with operating in a space where people's wellbeing is at stake.

If you're evaluating an AI mental health tool — for yourself, for your patients, or for your organization — use the five questions above. The answers will tell you what you need to know.

Sources

• Nature. (2025). "Mental Health Apps: Regulatory Landscape." npj Mental Health Research. nature.com
• FDA. "Digital Health Center of Excellence." fda.gov
• Hogan Lovells. "AI wellness or regulated medical device?" hoganlovells.com
• PMC/NIH. "The illusion of safety: A report to the FDA on AI healthcare product approvals." PMC12140231
• Bipartisan Policy Center. "FDA Oversight: Understanding the Regulation of Health AI Tools." bipartisanpolicy.org
• Illinois IDFPR. "Legislation Prohibiting AI Therapy in Illinois." idfpr.illinois.gov
• California Legislature. "SB 243 — Companion Chatbot Safeguards." leginfo.legislature.ca.gov
• NIST. "AI Risk Management Framework." nist.gov
• NIST. "AI RMF 1.0 Core Document." PDF
• Brown University. "AI chatbots systematically violate mental health ethics standards." brown.edu
• American Psychological Association. "Health Advisory: Chatbots and Wellness Apps." apa.org
• Stanford HAI. "Exploring the Dangers of AI in Mental Health Care." hai.stanford.edu
• HHS. "HIPAA and Mental Health." hhs.gov
• HHS. "Access Right, Health Apps, and APIs." hhs.gov