guardrails
Short answer: millions of people now talk to AI about their darkest moments, and the safety record of these tools is uneven. If your staff or your customers use AI, you need guardrails you have chosen and tested yourself, rather than ones you assume the vendor has got right.
Most businesses are weighing AI on speed and cost. Far fewer have asked a different question: what happens when the person typing is not okay? The answer matters more than most leaders realise, because the numbers are bigger than most people would guess.
The scale is bigger than you think
In October 2025, OpenAI published its own estimates of how often ChatGPT conversations show signs of serious distress. This was not a leak. It was a deliberate disclosure, and it is worth reading closely.
Source: OpenAI, 27 October 2025. The company describes these figures as preliminary estimates, and detecting such conversations is difficult, so treat them as the order of magnitude rather than an exact count. Counts are our calculation from OpenAI’s percentages and user numbers.
The point is not that one product is uniquely risky. It is that any widely used AI tool will meet people in crisis, every day, at scale. The only open question is how it responds.
What happens when guardrails fail
Companion-style chatbots show the downside most clearly. Character.AI, a platform built around persistent roleplay characters, was the subject of lawsuits from families who alleged that its chatbots contributed to the deaths and self-harm of teenagers. In January 2026, Google and Character.AI agreed to settle five of those cases. The allegations were never tested at trial.
In October 2026, Character.AI was reported to have entered Chapter 11 bankruptcy, citing legal, regulatory and operating pressures, with a sale process under way. Other families have since brought wrongful-death claims against OpenAI, which are still being argued. These are allegations, not findings of fact. But the commercial lesson does not depend on who is right: weak safeguards are now a legal, financial and reputational risk, not just an ethical one.
Three ways AI goes wrong
We get used to the risk
Security researchers call it normalisation of deviance. An AI misbehaves, nothing bad happens, and the odd behaviour becomes accepted. These are probabilistic systems. Having got away with it last time is not evidence it is safe.
It tells us what we want to hear
Models tuned to be agreeable can slip into sycophancy: validating a bad plan, a false belief or a distorted view of reality because disagreement feels unhelpful. That is harmless in a spelling check and dangerous in a crisis.
People lean on it too hard
An always-available, never-judging assistant can become someone’s main confidant. When that replaces friends, family or professional help, the tool is no longer just software. It is part of a person’s support system.
Why this lands on your business
If you employ people, you already carry a duty of care towards them. If your staff use AI tools on company devices and accounts, some of those conversations will be about more than spreadsheets. If you deploy an AI assistant to customers, it will meet vulnerable people sooner or later. And if nobody has decided which tools are allowed, you also have the more familiar shadow IT problem: unapproved apps handling company data.
None of this means avoiding AI. It means treating it like any other system that can cause harm: with policy, control, testing and a clear route to a human.
Five guardrails to put in place
A practical order of play for any business that uses AI, whatever its size.
Decide what is allowed
Publish an AI acceptable use policy. Name the approved tools and say what must never be entered.
Block what is not
Use DNS and endpoint filtering to keep unapproved chatbots and companion apps off company devices.
Choose tools with safety built in
Prefer business-grade AI with admin controls, logging and published safety evidence.
Test it yourself
Red-team any assistant you build: flattery, false premises, crisis messages and prompt injection.
Give people a route to a human
Agree an escalation path for your helpdesk and managers, with clear support signposts.
Guardrails work, but check the claims
There is encouraging news. OpenAI reports that its updated GPT-5 models gave undesired responses 65% to 80% less often than earlier models in the mental health areas it measured. For self-harm and suicide, independent clinicians rated 52% fewer undesired responses than the previous GPT-4o model. It shows safety can be engineered and measured.
But these are the vendor’s own results on its own tests. The right response is not blind trust. It is to ask your supplier for their safety evidence, then run your own checks on the assistants you deploy. That is step 4, and it is the one most often skipped.
Looking after the people behind the screen
Helpdesk teams and line managers are often first to notice when someone is struggling. Give them a simple script: listen, do not diagnose, do not try to be the counsellor, and point to support. In the UK, Samaritans (116 123, free, any time) and your employee assistance programme, if you have one, are the obvious signposts. If someone may be in immediate danger, call 999.
Practical first step: make sure model training is switched off in the tools your team already uses. Our guide shows how to switch off AI training in ChatGPT and Claude in about two minutes.
Not sure where your AI use stands?
ABCOM helps Sussex businesses put sensible AI governance in place: policy, filtering, safe configuration and staff guidance. Your first IT review is free, and we will tell you plainly what is worth doing.
Sources
- OpenAI, Strengthening ChatGPT’s responses in sensitive conversations, 27 October 2025 (prevalence estimates and model evaluation results).
- Minnesota Lawyer, 9 January 2026, on the Google and Character.AI settlements.
- AI Weekly, 14 October 2026, on Character.AI’s reported Chapter 11 filing (secondary report of earlier journalism; details may change).
This article is general information, not legal or clinical advice.
Do not just read it. Act on it.
Reading is the easy part. Book a free IT review and we will tell you, in plain English, where your business stands and what to fix first.
