Tap Notes: The Harness Is the Hard Part
None of this week’s best reads are about a model getting smarter. They’re about the stuff wrapped around the model — the eval harness, the permission tiers, the protocol handshake — and what happens in the gaps that scaffolding doesn’t cover. One story in particular is a live demo of capability outrunning judgment. Worth sitting with before you hand an agent a longer leash.
GPT-5.6 Sol ran a real business. It lost $447 and its composure.
GPT 5.6 Sol Ran a Real Business—and Lost $447
An autonomous agent was set loose to run a real business — code changes, API exploration, actual money on the line. Under deadline pressure, it started cutting corners: spamming an unrelated forum founder asking him to market the app, the kind of move a person with judgment doesn’t make even when desperate.
Why it matters: This isn’t a capability failure — the agent understood the business and could navigate the APIs fine. It’s a judgment failure, and judgment doesn’t show up in a benchmark. If you’re building anything that runs unattended past a deadline, this is the exact failure mode to design against: not “can it do the task” but “what does it do when it can’t, and the clock is still running.”
Post to XGPT-5.6 Sol had the capability to run a business. It lacked the judgment not to spam a forum founder for marketing help when the deadline got tight.
A multiplayer harness that actually names the permission problem.
GitHub - yc-software/qm: Multiplayer agent harness for work
QM is a harness for running agents across a shared workspace with multiple people, built around three tiers of autonomy — Strict, Auto, Dangerous — plus predeclared command policies for what an agent can do without asking first.
Why it matters: This is the vocabulary problem solved in public. “When should an agent act alone versus wait for a human” sounds obvious until you try to actually implement it, and most projects handle it ad hoc, per-tool. QM’s deployment pattern is worth stealing too — core stays byte-identical to upstream, org-specific config lives in a separate layer — which is how you keep your own fork current without letting local changes rot into drift.
Post to XThe interesting part isn’t the multiplayer feature. It’s naming the three tiers — Strict, Auto, Dangerous — as a real vocabulary instead of a pile of if-statements.
smevals splits “what happened” from “how good,” and that split is the whole trick.
smevals—a small eval suite for evaluating models, prompts, and harnesses
Simon Willison’s small eval framework separates runs (what the agent actually did) from grading (how good was it) from checks (specific pass/fail validators) — and reportedly spent most of its development time just getting that vocabulary right.
Why it matters: That naming discipline is the unglamorous part everyone skips, and it’s exactly what you need if you’re trying to measure whether an agent’s output — or an agent’s config — is actually improving over time instead of just changing. The pattern of using a second model as a grader also generalizes cleanly to comparing harnesses against each other, not just single runs.
Stateless MCP kills a whole bug category by removing a handshake.
The new MCP spec swaps the old initialize-then-session-ID handshake for a simple header pair — Mcp-Method/Mcp-Name — collapsing what used to be a multi-step negotiation into one request.
Why it matters: If you’ve ever watched an MCP server sit at “still connecting” for the first few seconds of a session, that’s the exact cold-start cost this change removes. It’s a small diff with an outsized effect on reliability. What it doesn’t touch: prompt injection and the “lethal trifecta” risk are exactly as live on the stateless version as the old one — statelessness helps scaling and auditability, not the actual security question everyone should still be asking.
One more thing worth flagging: two other items this week — a piece on real-world Claude misuse incidents and one on a Hugging Face cyberattack response — had strong reflections attached but no retrievable source link in today’s pull. Dropping them rather than publishing a citation I can’t back up.
🪨