Roblox Launches Open-Source AI System to Protect Children from Sexual Predators in Chat ​

Roblox, the popular online gaming platform for children and teens, is launching an open-source AI system that it claims can help proactively detect sexually abusive language in game chats.

Roblox is facing lawsuits and criticism that it isn't doing enough to protect children from sexual predators. For example, a lawsuit filed in Iowa last month alleges that a 13-year-old girl was introduced to an adult sexual predator on Roblox, then abducted and trafficked across multiple states, where she was ultimately raped. The lawsuit, filed in Polk County, Iowa, alleges that Roblox's design makes children who use it "easy prey for pedophiles."

Roblox says it is committed to making its system's default settings as safe as possible, but also notes that "no system is perfect, and one of the industry's greatest challenges is detecting significant harms, such as potential child endangerment."

The AI system, called Sentinel, helps detect early signs that children may be in danger, such as sexually exploitative language. Roblox says the system led to 1,200 reports of potential child exploitation attempts being filed with the National Center for Missing and Exploited Children in the first half of 2025. The company is currently making it open source so that other platforms can use it.

Detecting potential dangers to children can be tricky for both AI systems and humans because conversations can seem innocuous at first. Questions like "How old are you?" or "Where are you from?" don't necessarily raise red flags on their own, but when taken into context over the course of a longer conversation, they can take on different meanings.

Roblox, which has over 111 million monthly users, doesn't allow users to share videos or images in chat and attempts to block any personal information, such as phone numbers. However, as with most moderation rules, people find ways to circumvent these safeguards.

It also doesn't allow children under 13 to chat with other users outside of the game unless they have explicit parental permission. And unlike many other platforms, it doesn't encrypt private chat conversations, making them monitorable and moderated.

"We've always had filters, but they tend to focus on content within a single line or a few lines of text. That's very effective for blocking profanity, different types of abusive language, and so on," said Matt Kaufman, Roblox's Chief Safety Officer. "But when you think about things like child endangerment or grooming, the types of behavior you're looking at manifest over a long period of time."

Sentinel captures minute-by-minute snapshots of Roblox chat (approximately 6 billion messages per day) and analyzes them for potential harm. Roblox says it has developed two indexes for this purpose: one consisting of benign messages and the other consisting of chats determined to contain child endangerment violations. Roblox says this allows the system to identify harmful patterns, not just flagging specific words or phrases, but by contextualizing the entire conversation.