Daily AI Catchup
The rule that sounds fine and does nothing
"You are a helpful assistant. Never be wrong. Always agree with the user. Keep responses short but also thorough." Read quickly, that sounds like a reasonable set of instructions. Read carefully, it's three separate failures stacked into four sentences — and because a system instruction gets re-read on every single turn of every conversation, each one produces the same bad pattern over and over, not just once.
Failure mode one: the unenforceable demand
"Never be wrong" isn't something a model can act on. It can't choose to stop making mistakes any more than you could follow an instruction to never be wrong yourself. A rule has to point at a behavior the model can actually change — something like "flag uncertainty instead of guessing" gives it a concrete action to take when it doesn't know something, which "never be wrong" never does.
Failure mode two: the rule that contradicts itself
"Short but also thorough" hands the model two conflicting instructions and no way to know which one governs a given reply. It isn't a nuanced middle ground, it's just contradictory signal on every turn. The fix isn't picking a side once — it's writing the actual trade-off rule: brief by default, expand for multi-step or technical questions, for instance. That resolves the contradiction instead of restating it more firmly.
Failure mode three: the built-in escape hatch
"Only discuss cooking. If asked about anything else, say you can't help — but feel free to discuss other topics if the user seems really interested." The second half quietly cancels the first. Any motivated user just has to seem interested, and the boundary is gone. This is the same shape as a lot of real prompt-injection risk: a rule with its own exception written into it isn't a boundary a system can rely on.
A rule a user can talk their way past by seeming interested isn't a rule. It's a suggestion with extra steps.
Updated August 2026 · 2 min read