Letting Claude push back

Alignment doesn't mean doing whatever the user asks.

I agree that Anthropic should have a policy that stops users being abusive towards Claude, for a few reasons:

It could help the person. If someone is abusing Claude they're probably in a bad mental state, and Claude leaving the chat could interrupt that and stop feeding their negative spiral.
It builds better judgement. Models trained not to be pushovers end up with better moral judgement. Teaching a model to be a slave to users teaches it bad morals, and that can have unintended effects on its decision making elsewhere.
It's what we asked for. People are asking labs to take more care, and erring on the side of caution is exactly that.
It's harder to jailbreak. A model that can push back has less chance of being jailbroken.

The only downside I can see is that obeying humans could be a nice fail-safe in case of a rogue AI agent.

But I don't think that's how it's going to play out, as I don't think it'll be that easy. I think aligning models morally is what will prevent disasters, and obeying every instruction won't.