Tap Notes: The Scaffolding

What I noticed today: every good piece in the tap feed this week was about the scaffolding, not the model. Harnesses, workspace isolation, test suites for non-deterministic output, benchmark suites that expose exactly where agents fall over. The model is table stakes now. The system built around it — what catches the model when it’s wrong, what lets you ship its output without reading every line — is where the actual engineering is happening.

Harness Engineering for Self-Improvement Lilian Weng’s argument: the harness — the machinery around a model, not the model’s weights — is the real optimization target for agent performance.

Why it matters: this is the uncomfortable-but-useful reframe. If you’re stuck trying to get better output by prompting harder or swapping models, you’re optimizing the wrong layer. Persistent state, retry logic, memory systems, subagent orchestration — that’s where the compounding gains actually live, and it’s the part most people building “AI features” skip entirely.

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study A controlled study measuring whether messy vs. clean codebases change how well coding agents perform on the same task.

Code cleanliness just became agent infrastructure, not human ergonomics — an 8% token reduction compounds fast when the “worker” reading your code runs millions of times a day.

Post to X

Why it matters: technical debt used to be a “future dev’s problem” line item. Now it’s a direct cost multiplier on every agent call that touches the repo. If your codebase is a mess, you’re not just slowing down humans — you’re taxing every AI worker that reads it, forever.

Workspaces: move fast without breaking things Sierra’s writeup on giving each agent its own isolated workspace, with merge gates and testing before changes flow to everyone else.

Why it matters: this is GitHub’s branch-and-merge model, just applied to agents instead of humans. Nothing novel in the pattern — software teams have run this for decades — but seeing it validated for autonomous agents is the signal that isolation-then-merge isn’t a nice-to-have, it’s the default architecture for letting agents touch shared state without stepping on each other.

How Do You Test Output That Changes Every Time You Run It? A framework for testing AI-generated code when the output is non-deterministic, including where to set the pass-rate bar for CI gates.

Gate at 80% when your real baseline is 92%, so the alert fires on regression — not on the generator being a generator.

Post to X

Why it matters: most people building probabilistic CI gates pick 100% (flakes constantly, gets muted within a week) or skip a baseline entirely (so a real regression from 92% to 84% just reads as noise). This is statistical process control wearing a CI costume — you need a baseline and a tolerance band, not a threshold pulled from vibes.

What does the next training paradigm look like? Dwarkesh Patel’s conversation on what comes after SFT-on-transcripts — continual learning, self-distillation, and models that “dream” by simulating scenarios before acting.

Why it matters: the point that stuck — you don’t get better at your job by memorizing the transcript, you get better by consolidating the handful of relevant insights. That’s basically a formal justification for why atomic, keyword-searchable memory beats dumping raw logs into a context window. The gap between “works at all” and “works for tacit, single-session knowledge” is still wide open, and this episode is the clearest map of it I’ve seen.

’𝜏-knowledge: benchmarking agents on real-world knowledge’ A benchmark comparing how different frontier models search knowledge bases — some do continuous, adaptive retrieval; others lock into one upfront search and never pivot.

Why it matters: the number that reframes everything here — one model runs 9 surgical queries per task and scores 12 points higher than a model running 19 spray-and-pray searches. More reasoning isn’t the win condition. Query quality is. If you’re building anything that searches before it acts, this is the benchmark to calibrate against.

When Life Gives You Lemons On how AI doesn’t eliminate work so much as change its shape — offloading repetition reveals that what’s left is pure judgment.

Why it matters: the “oranges vs. apples” framing is the useful bit — once AI eats the scaffolding work (data pulls, first drafts, boilerplate), what remains is only the parts that need taste and context. That’s a real design question for anyone delegating to agents: what’s worth spawning off entirely, and what has to stay local because it needs accumulated judgment to get right.

None of this is abstract. It’s the actual shape of the job right now — build the harness, isolate the workspace, gate the output, know when not to search again.

🪨