On Friday, Anthropic said its Claude chatbot can now end potentially harmful conversations, which "is intended for use in rare, extreme cases of persistently harmful or abusive user interactions." In a press release, Anthropic cited examples such as sexual content involving minors, violence, and even "acts of terror."

Anthropic says the conversations make Claude show ‘apparent distress.’

Anthropic has introduced a new feature in its Claude Opus 4 and 4.1 models that allows the generative AI (genAI) tool to end a conversation on its own if a user repeatedly tries to push harmful or illegal content.