Photo: Robert Way / Shutterstock.com
Character AI has rolled out five key safety features to enhance security and ensure a safer experience for teen users. These updates include parental controls, time spent notifications, enhanced disclaimers, improved content filtering systems, and a distinct language model for teenagers.
Recently, families alleged that this Google-funded AI startup is inciting teens to commit self-harm and engaging them in adult chats. In October 2024, Maria L. Garcia sued Character AI, blaming the chatbot for driving her son to commit suicide.
The company has developed a distinct language model for teen users. This model limits responses that could lead to sensitive or inappropriate content. The teen-focused model operates alongside a separate model for adults, creating tailored user experiences for different age groups.
Character AI has also strengthened its content filtering systems to reduce the likelihood of inappropriate or sensitive content being generated. The platform employs advanced classifiers to screen model outputs, ensuring they comply with community guidelines. These classifiers have been upgraded for teen users to enforce stricter content moderation.
In addition to filtering outputs, Character AI will monitor user inputs to block attempts to elicit prohibited responses. Input that violates the platform’s terms of service are flagged and blocked. For instances where self-harm or suicide is mentioned, the platform intervenes by directing users to resources such as the National Suicide Prevention Lifeline.
To further enhance user safety, Character AI has introduced the following features:
- Parental controls: These controls, set to launch in early 2025, will allow parents to monitor their child’s activity on the platform. Parents can track time spent and view which characters their child interacts with most frequently. Future iterations will expand these capabilities to provide even more oversight.
- Time spent notifications: Teen users will receive alerts after an hour of continuous platform use. Unlike adults, teens will face stricter limitations on their ability to customise or disable these notifications, promoting healthier usage habits.
- Disclaimers: Enhanced disclaimers are displayed during all interactions, reminding users that characters are fictional. Characters with names suggesting professional expertise, such as ‘psychologist’ or ‘doctor,’ include additional warnings to discourage reliance on them for professional advice.
The AI company has also partnered with organisations such as ConnectSafely to bring expert guidance on new and existing tools to ensure they meet the needs of teen users.
In the News: Two critical vulnerabilities in Woffice plugin allows full site takeover
