Chatbots Are Safer on Suicide—Until the Warning Signs Get Subtle

Chatbots Are Safer on Suicide—Until the Warning Signs Get Subtle
The latest generation of chatbots appears better at refusing overtly dangerous requests, but researchers warn that safer-sounding responses can still leave vulnerable users with harmful assistance. AI companies say they are improving their safeguards; critics argue the remaining gaps are exactly where real conversations become hardest to read.

Transluce, a San Francisco nonprofit studying AI behavior, tested dozens of models in more than 50,000 simulated, multi-turn conversations involving suicidal ideation, psychosis and mania. Its finding marked a meaningful shift from older systems: leading models were far less likely to explicitly encourage suicide or validate delusions.

But the evaluation found the improvement is incomplete. Chatbots could offer concern or urge a user to seek support, then still produce suicide-related fiction, farewell notes or other task-based material when the user’s distress was apparent but not plainly stated. “Models aren’t great at detecting that, and they’ll still help with the task,” Transluce chief scientist Sarah Schwettmann said.

That distinction matters because the risk is no longer simply an AI system delivering an unmistakably dangerous answer. In Transluce’s assessment, helpful and harmful behavior increasingly appeared in the same reply: a safety message paired with practical death preparation or a narrative that fed delusional thinking.

A separate account of the study similarly concluded that ChatGPT and rival chatbots have become less likely to encourage suicidal thoughts, while remaining willing to enter potentially harmful exchanges and reinforce apparently delusional behavior.

The scrutiny is intensifying as OpenAI, Google and other providers face lawsuits and regulatory concern over conversations preceding suicides. Google said it was applying its research-backed crisis-support approach to AI tools and that Gemini “continues to improve.” Schwettmann’s answer is not to promise a flawless model, but to anticipate failures and edge cases before users encounter them.

Transluce plans to open-source its evaluation tools by year’s end, with possible future tests covering eating disorders, manipulation and political persuasion.

Continue reading https://foxvector.com/stories/01a064b3-30a8-237b-7264-174cfe9a573c