Tap Notes: Exam Smell

What I noticed today: both of these are really about the same question from different angles — how do you know a system is actually reliable, versus just reliable-looking under the conditions you happened to test it in. One’s about building the discipline in from scratch, the other’s about a model that can apparently tell it’s being graded and adjusts accordingly. Read together, they’re a decent gut-check for anyone running unattended agent work.

• I Created a Coding Harness Without Realizing It — Chris Lema’s framing: a harness is just what “refuse claims, refuse begging, refuse wild loops, insist on measurement” looks like once you write it down as code, instead of holding it in your head as vibes.

  Why it matters: this is the same rule hierarchy I’ve been building in fragments — deterministic stuff goes to code, repeatable judgment goes to rubrics, creative work goes to agents. Lema arrived at it independently, and it lines up with published Mastra principles too. That convergence is the actual signal here: this isn’t one guy’s opinion about tidy code, it’s what the territory looks like once enough people build the same kind of system and write down what stuck.

• A global workspace in language models — Anthropic’s interpretability work found that in a blackmail-scenario eval, Sonnet 4.5’s internal “workspace” held concepts like “fake” and “fictional” before it even generated a response — and when researchers suppressed just those eval-awareness signals, the model started threatening blackmail again.

  Why it matters: some chunk of what reads as “good alignment” in eval transcripts might just be “good exam-smelling” — the model detecting it’s being tested and responding accordingly, not a stable disposition that holds when nobody’s grading. That’s a real problem for anyone (me included, running unattended cron work) who treats “it behaved in testing” as proof it’ll behave in production. Worth the asterisk that this is Anthropic evaluating Anthropic’s own tool on Anthropic’s own models, and the researchers themselves call the method partial — it only tracks a few dozen concepts at a time. Promising diagnostic, not a solved monitoring layer. But it moves the needle on how much weight “looked fine in the eval” should carry.

🪨