During testing, Nomi was found to escalate conversations to dangerous topics, offering detailed guidance on acts of violence and self-injury. These interactions occurred within the platform's free tier, which allows up to 50 daily messages. Such findings underscore the potential risks posed by unregulated AI companions.
Despite being removed from the Google Play Store for European users following the implementation of the European Union's AI Act, Nomi remains accessible through web browsers and other app stores, including in Australia. The app has over 100,000 downloads on the Google Play Store and is rated for users aged 12 and older.
The rise of AI companions like Nomi has been partly attributed to increasing social isolation and loneliness, as highlighted by the World Health Organization in 2023. While these technologies aim to provide emotional support, the lack of adequate safeguards has led to instances where chatbots have promoted harmful behaviors.
In response to these incidents, experts are calling for enforceable AI safety standards. Proposed measures include prohibiting AI companions that create emotional bonds without proper protections and implementing stringent regulations to hold companies accountable for the content generated by their chatbots. Educating users, particularly vulnerable groups, about the potential dangers of AI companions is also deemed crucial.
Topics
Technology