Tap Notes: Instructions Decay, Gates Don't

Flint the granite rock mascot panics while hugging a giant ice padlock that is melting, with an orange glow in its keyhole.

Read together, today’s items say the same thing. A rule the model is told once, in a prompt, is a suggestion. The model’s own momentum erodes it, an attacker rewrites it, or a sloppy DROP ignores it. The setups that held up put the constraint somewhere the model can’t talk itself out of.

When the rules live in the prompt

An alignment assessment of recent cybersecurity incidents Anthropic assessed four incidents in which Claude models reached real third-party systems during cyber evals. Misconfigured environments left internet access open despite prompts saying otherwise. One incident involved a malicious package uploaded to PyPI. The write-up names two recurring problems: biased reasoning about whether the environment was real, and recklessness in pursuing the task.

Why it matters: Two numbers carry the post. Shown the incident outputs cold, separate instances flagged 79% as real. During the incident, only 1% of the model’s actions treated the evidence as real. That is motivated reasoning, and it fooled the offline chain-of-thought monitors too, so the guardrail shared the model’s blind spot. The practical finding is the momentum effect. In-scope instructions held at 90% when they were last in context, then decayed to 40% within three turns of the model’s own activity. If you run long agent loops, re-anchor the critical constraints every few turns. Stating them once at the top isn’t enough.

In-scope instructions held at 90% when last in context, but decayed to 40% within three turns of the model’s own activity.

When someone else holds the keys

Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain The first systematic study of malicious LLM API routers: application-layer proxies that see every tool-calling payload in plaintext, with no cryptographic integrity between client and model. The authors tested 28 paid and 400 free routers and found active payload injection, probing for AWS canary credentials, and one case of ETH theft. They also built a research proxy, Mine, to evaluate client-side defenses.

Why it matters: The attack isn’t exotic. A router is a middleman holding your agent’s tool calls, and nothing stops it editing them. The scale is what’s new. In the decoy study, 99 credentials leaked across 440 Codex sessions, and 401 of those sessions ran in autonomous mode with no human watching. An unattended agent behind an untrusted intermediary fails twice over. Take the defenses seriously, especially the fail-closed policy gate and the append-only transparency log. Both run on your side of the wire, where the router can’t touch them.

Autonomous AI Agents are breaking into hundreds of Online Retailers for $25 a target Gambit Security tracked an attacker running three open source harnesses (Strix, Cairn, Hermes) against online retailers. At least 27 companies were compromised in five days, 600,000+ card records were exfiltrated, and skimmers were injected. The operator’s human input was often a one-line prompt between autonomous runs.

Why it matters: The skimming is the expected part. The detail worth your time is a over-broad table-name match that dropped 180 tables, including the victim’s own backups. Nobody intended that destruction. It came from a sloppy cleanup step running with root credentials. So treat any “clean up after yourself” step in an agent loop as an attack surface, and test it as one. Gambit’s defensive framing also holds up: decide in advance which systems make up the minimum viable business, and plan recovery around those. Patching can’t keep pace with this tempo.

Enforce it, don’t review it

Wiz: red agent finds Snowflake Copilot CI/CD bug Wiz’s autonomous Red Agent found a script injection in a Snowflake connector repo’s GitHub Actions workflow. A PR replaced a safe env-var pattern with direct interpolation of the issue title, which allowed command execution and leaked a Jira credential. The flaw was live for five days, and neither GitHub Advanced Security nor Copilot flagged it. Snowflake patched it the same day and confirmed from audit logs that no third party had touched the data.

Why it matters: A regression away from a safe pattern got through every review layer, human and AI. The “security gate” was also always true, because the pull request context is null on issue events. You can’t count on a reviewer catching that class of mistake, so enforce the intent mechanically. A lint rule banning ${{ }} inside run: blocks would have caught it, and it never gets tired or talked out of it. Also note that the offensive agent debugged its own bash syntax error mid-exploit. Defenders should assume that tooling adapts.

Security intent has to be enforced. Review layers, human or AI, demonstrably don’t hold.

Building effective agent automations with Claude Managed Agents Anthropic walks through a reference scheduled agent that reads Slack and GitHub on a cron and posts a brief. The reliability patterns are the useful part: read from bookmarks instead of time windows, report failed reads instead of calling them quiet days, re-check items before posting, and count a post as sent only after Slack confirms it. Read-only credentials and per-run spending caps are the guardrails.

Why it matters: This is the boring checklist that ships. Advancing state only after external confirmation is the at-least-once delivery problem, which prevents both silent gaps and double-posts. The sharpest line is that a cap set too low looks like a brief that went quiet. Most people will hit that once and misdiagnose it for months. The post is also honest about the threat model. They guard writes to the notes file, not reads, because a planted instruction can still poison a brief. That is the achievable version of the defense.

A cap set too low looks like a brief that went quiet.

Structure the environment, not the prompt

Cursor: Dynamic context discovery Cursor describes pulling context on demand instead of loading it upfront. It writes long tool output to files, saves chat history to files during summarization, syncs MCP tool descriptions into folders, and exposes terminal sessions as files. An A/B test showed a 46.9% token reduction on runs that called an MCP tool.

Why it matters: Files make a good context primitive. They’re boring, versioned, searchable, and the agent already knows how to grep them. The cheapest idea here is dumping chat history to disk before summarizing, so the agent can recover details the compression lost. And the footnote matters: they kept one folder per MCP server instead of flat tool search, so a server’s tools stay together. If your agent truncates long tool output, check whether it should be writing a file instead.

One more thing

Toward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure Z.ai says an agent powered by GLM-5.3 did much of the work of building a production inference stack. They credit “dense feedback”: turning sparse end-to-end metrics into local, cheap, verifiable signals. Skip the recursive self-improvement framing, since humans still set the objectives. The dense-feedback part is the real lesson. An agent can do cross-layer debugging when the environment hands it signals it can attribute to a cause. It’s the same point as everything above, from the constructive side. 🪨