Tap Notes: Check the Layer Underneath

Most of today’s reading had the same shape. The thing everyone was watching worked fine, and the thing underneath it didn’t. An inference engine leaked memory between customers, a classifier quietly corrupted a benchmark, and a dead process threw away tokens that had already been paid for. If you’re building with agents, audit the plumbing before you audit the prompts.
Isolation fails in the boring layer
Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
A configuration mismatch for GLM 5.3 on Workers AI triggered a bug in SGLang that exposed KV cache pages across requests. Cloudflare walks through the cause and the fix, which was coordinated with the SGLang maintainers.
Why it matters: This is a real cross-tenant leak in the inference stack. Nobody had to jailbreak anything. The trigger was a long-context condition (north of 681k tokens), the kind of input agentic coding workloads produce on a normal Tuesday. If you run agents on shared inference, whether that’s a vendor’s or your own, your isolation story includes the serving engine’s memory management. Prompt-injection defenses don’t cover that. Read the disclosure for how the config mismatch and the engine bug lined up. Neither one was exploitable alone.
Make the connection outlive the process
never waste a token (June 2026)
When an agent process dies mid-inference, the HTTP request to the model provider dies with it. The retry re-bills every output token already generated. The proposed fix is a durable buffer that drains the provider stream into SQLite, so the connection survives the crash and clients resume from a stored cursor.
Why it matters: The sharp idea is that crash recovery and resumable streaming are the same mechanism. It’s one durable chunk log with two kinds of reader, and the only difference is whether a live producer is still attached. The economics sharpen it. The better the model, the more a mid-stream deploy costs you, so this is a multiplier on flagship spend across every tool call in an agentic turn. The author also says to store raw bytes and reuse the provider’s own parser instead of hand-rolling SSE handling. Most people will learn that one the hard way.
Grade the transcript, not the artifact
Loop Engineering: For me it’s about using dumber models and smarter code (June 2026)
Chris Lema argues that models are frozen and only the system around them improves. An inner loop fixes a single run. An outer loop reads recorded history and turns recurring corrections into cheap deterministic checks. The scoreboard is how much frontier-model work you’ve retired.
Why it matters: Every correction you pay a frontier model to make twice is rent, and the loop’s job is converting rent into assets. That turns “self-improving agents” from vibes into accounting. The best part is grading behavior as well as output. Two planners can produce the same plan while one wastes your attention, and only the transcript shows it. It’s also how you catch “claims it tested but never ran it.” My pushback is that the grader-independence rule is incomplete, because a rubric shaped by the worker’s history is still gameable. His hand-graded regression set for grader changes is the real answer and deserves top billing.
Every correction you pay a frontier model for twice is rent. The loop’s job is to turn it into an asset.Post to X
Memory evals, and the eval under the eval
Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
The paper introduces BEAM: 100 conversations running up to 10M tokens, with 2,000 validated probing questions. It also proposes LIGHT, which gives an LLM episodic memory, working memory, and a scratchpad. Even 1M-context models with retrieval degrade as dialogues lengthen, while LIGHT improves on the strongest baselines by 3.5% to 12.69%.
Why it matters: Raw context isn’t memory. A bigger window is a bigger desk you still lose things under. A 3.5–12.69% lift is honest-sized, not benchmark-theater numbers, and the ablations show each memory component earns its place. Most memory frameworks arrive with vibes and a demo. The conversation generator may be the more reusable artifact, since coherent 10M-token dialogues are hard to come by.
Before the Benchmark (May 2026)
The companion read. Before running a memory system against BEAM, the author found their classifier had labeled 56% of memories as “Insight.” The benchmark’s contradiction and temporal metrics depend on graph edges built from those labels, so the scores would have measured noise.
Why it matters: Eval hygiene is recursive. If the typing layer is miscalibrated, everything derived from it is garbage before a single retrieval runs. Fix the generator, then measure. The post also notes BEAM scores with deterministic nuggets and no LLM judge, so there’s nothing to game. It’s a short read with a general rule: calibrate whatever sits under your metrics first.
If the classifier is off, every downstream edge-based metric is measuring noise.Post to X
Running a page as a program
webframe: evaluated as a scraper and a sandbox
webframe, from Jeff Lindsay, runs a web page like a program. It executes the page in a hidden system webview and pipes console.log() to stdout and exceptions to stderr. It’s a small Go tool with no bundled browser engine. Simon had Claude Opus 5.5 put it through its paces as both a scraper and a sandbox, and published the research report.
Why it matters: Treat the sandbox question seriously. “Execute this page as a program” is the same attack surface agents hit every time they browse, and a thin wrapper over the OS webview is a different trust model than a full Playwright stack. The format is also worth stealing. The agent’s own exploration and write-up is the artifact, and you can read the prompt that produced it.
One more thing: nobody in this batch was exploiting a clever trick. Each failure sat in plain infrastructure that someone assumed was fine.
🪨