Skip to Content
Categories:

The prompt AI shouldn’t respond to

ChatGPT is used by many students for academic help and other issues, but when it is used for mental health, it
can cross boundaries of what AI should be used for, prompting concern among parents and counselors alike.
ChatGPT is used by many students for academic help and other issues, but when it is used for mental health, it can cross boundaries of what AI should be used for, prompting concern among parents and counselors alike.
Ishaan Gupta ’26

As artificial intelligence is becoming a daily part of millions of people’s lives, a growing number of lawsuits and investigations are raising the alarming question about whether the wealth of AI chatbots is harmful to vulnerable users, including teens, during a moment of mental health crisis.

AI chatbots are quickly becoming a primary resource for students, not just for homework, but for personal and emotional issues.

Mental health experts and investigators are now questioning whether the current safeguards are strong enough and what roles schools, parents, and tech companies should take in teaching young people to use AI responsibly.

Wellness counselor Melissa Hansell stated, “As a wellness counselor, I understand that many people are using ChatGPT for their personal lives. It seems obvious, but I think it’s good to remember that ChatGPT makes mistakes and it is not a real person.”

Hansell added, “AI has the potential to greatly enhance and improve many aspects of our lives, but the challenge will be harnessing it in helpful and safe ways.”

So far, ChatGPT—a popular AI resource—has faced claims of giving users dangerous information regarding mental health topics such as suicide.

Experts warn that AI systems generate responses by predicting patterns in languages based on massive patterns on data rather than based on emotions or professional psychological care.

In the moment of crisis, a response lacking true understanding of human judgment or safeguard can have devastating consequences.

In several recent cases, families have filed lawsuits claiming that AI is working against them. One of the most well-known reports was filed in August of 2025 by Matthew and Maria Raine after their 16-year- old son, Adam Raine, died.

Adam started using a chatbot, to fill out and complete homework assignments at the start of September 2024.

Adam started to struggle with more than just school; he began to struggle with his mental health, and had no one to talk to about it. He then explored suicidal ideas with ChatGPT.

Chatbots have no emotional programming, so instead of exploring why, it suggested ways he could harm himself. It offered suggestions such as “don’t inform your family,” and encouraged deeper conversations about suicide. The family claims this led to Adam’s death in April of 2025.

Some experts start to worry that this danger is not obvious. Even subtle reinforcements of negative thinking patterns, isolation, or
dependency on AI for emotional validation may have long term consequences

Elise Fong ’27 stated, “This raises the growing concern that AI is taking over mental health, showing that technology is not just a tool for homework but becoming much more.”

As AI becomes deeply woven into daily life, experts say students, parents, and developers must work together to teach responsible use and use safeguards to ensure no one is misguided in a dark moment.

If you or someone you know is struggling, call or text 988, 24 hours a day, seven days a week.

 

Donate to The Crusader
$3395
$5000
Contributed
Our Goal

Your donation will support the student journalists of ARCHBISHOP RIORDAN HIGH SCHOOL. Your contribution will allow us to cover our annual website hosting costs.

More to Discover
Donate to The Crusader
$3395
$5000
Contributed
Our Goal