25 September 2026

I measured the waste in my own AI coding agent sessions. My first result was fake. So were the next three.

This is the first number I got:

(a) ≥25% of sessions with ≥1 waste pattern    →  63%      ✅
(b) those sessions cost ≥1.8x the median      →  13.11x   ✅

Thirteen times. I had fixed the thresholds before looking at the data, precisely so I couldn't fool myself, and I had cleared them by a ridiculous margin. For a few minutes I thought I had the finding.

I didn't. And it wasn't the only time: later I found other results that looked great and were the same mistake wearing different clothes. This post is about those mistakes, because I think they are more useful than any savings number. They are the mistake behind almost every savings claim being published in this space.

What I measured

Agent CLIs write full transcripts to disk. Claude Code keeps them in ~/.claude/projects/**/*.jsonl, one JSON line per message, with the real token counts for every turn: input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens. Nothing to estimate, no API to call.

My corpus:

I was looking for prompt phrasings that, according to a pre-registered study of 4,644 runs, multiply the agent's work without improving its success rate. Two examples: asking it to "compare several approaches" triggers tournaments of branches that get thrown away, and certainty language ("make sure", "verify that") triggers redundant verification loops.

The analysis is deterministic: regular expressions and arithmetic. Zero AI tokens, zero network.

Mistake 1: I counted sessions

I flagged each session as "has a pattern" if any of its prompts matched, and compared the median cost of the two groups.

The problem is in that sentence: a session contains many prompts. The more prompts it has, the more chances one of them matches. And a session with more prompts has, almost by definition, more turns and more cost.

I wasn't measuring "waste patterns cost money". I was measuring "long sessions contain more text, and therefore more matches". The 13x was session length looking at itself in a mirror.

What makes this mistake so treacherous is that the effect points the way you expect. Had it come out inverted, I would have hunted for the bug immediately. Because it agreed with my hypothesis, I almost didn't.

The fix is to measure per episode: one user prompt plus all the agent's work until the next user prompt. Every episode has exactly one prompt, so every episode has the same chance to match.

Mistake 2: not all "user prompts" were written by me

Per episode, one pattern survived. Certainty language came out at 1.9x the median cost, with 12 cases and borderline significance. It matched the paper's mechanism. It looked good.

Later, while checking something else, I found that 41% of the messages I had counted as my prompts were not written by me. In Claude Code transcripts they arrive with the same "user" role:

• the reports subagents send back when they finish (91 of 275),
• the text of skills that the harness injects when they are invoked (23 of 275).

And those texts are full of "make sure", "verify that", "ensure". Of the 12 episodes with certainty language, 9 were the agent talking to itself. With my real prompts, 3 remain. Three cases is nothing.

With the harness messages removed, my 275 "prompts" become 161 real ones and 133 episodes. Here is every rule:

None is significant. And look at the two with big ratios: their episodes make 8 to 18 times more tool calls. Size again. What is left of the prompt hypothesis in my data is, honestly, nothing.

Mistake 3: errors weren't expensive, they were long

The third time it wasn't my data. To get past n=1 I analyzed TraceLab, a public dataset from the University of Washington (Zhu et al., 2026, licensed CC BY 4.0): about 4,300 real sessions from 43 developers using Claude Code and Codex. The TraceLab numbers below are my own analysis of their data, not conclusions of its authors.

Episodes with three or more tool errors cost 8.6x the median of episodes with none. The perfect candidate for a circuit breaker: "your agent is stuck in an error loop and it's costing you nine times more".

Comparing only episodes with the same number of tool calls:

The effect disappears. Episodes with errors aren't expensive because they fail; they fail more because they are longer. The 8.6x was, for the third time, size.

Subagents are similar, though not identical. Across the 19 TraceLab users with enough data, episodes that spawn subagents cost a median of 4.0x more. At equal length, between 1.0x and 2.2x. Spawning subagents is a good signal that an episode is going to be expensive, but it is not, by itself, proof of waste.

What I did find

The same data, looked at without a hypothesis to defend, says much bigger things.

99.6% of input volume is context you already sent

cache-read : 1,117,523,688 tokens
output     :     4,216,354 tokens
                   265 : 1

The prompt you type is a tiny fraction of what you pay for. Optimizing wording is optimizing 0.4% of the volume. In TraceLab, cached context is 59.5% of total cost.

Spend doesn't spread out, it concentrates

In my data, the costliest 10% of episodes take 67% of the spend. The mean episode ($7.10) is 5.4 times the median ($1.31).

And inside that 10%, one session:

$574  ·  875 turns  ·  3,554 tool calls  ·  79 subagents  ·  66 hours

Sixty-six hours straight. Nothing stopped it: not the agent, not the harness, not me.

This one does hold outside my machine. In TraceLab, the costliest 10% of episodes are 60% of spend, and the costliest 10% of sessions are 85%. Per user, the median is exactly 50%, and 18 of 35 developers are above it.

It's the same pattern Armin Ronacher documented in September 2026, when he left an agent "software factory" running unattended on OpenAI models: 35 hours, about $1,200 in API costs, 75,000 lines of code and 79 commits that he judged worthless. (His post gives two different token counts for the same run, so I leave that number out.) He ran the experiment on purpose. I had a 66-hour run sitting on my disk without knowing it.

The fixed cost of every turn is real, but it's rarely the problem

Everything the agent loads before you type (system prompt, tools, skills, rules) is paid again on every turn. Mine was high because I have lots of skills installed that I almost never use. But across TraceLab's Claude Code users the median is around 18,000 tokens, roughly 9% of spend. Worth a look, not a headline. Claude Code shows yours with its built-in /context command, nothing to install.

Mistake 4, as a bonus: the secrets that weren't

I ran a secret detector over my own transcripts and got scared: a dozen credentials that looked real, already filtered by entropy to drop examples. One of them, a GitHub token, appeared 89 times, copied by the agent into commands, tool results and subagent contexts.

Before rotating anything I looked at where each one appeared. None were real. The GitHub token was a deliberately fake one, valid-looking, that I had asked for to test a secret scanner. The AWS key was a test fixture. The JWTs were localhost verification links. The rest: example .env files and local-dev values.

What is real is the mechanism: a value that enters the conversation once ends up copied dozens of times, in plain text on disk, and sent to the provider. This time it was a fake token. It isn't always: when I re-ran the analysis with gitleaks' full rule set, one real secret appeared that my homemade detector didn't know about, an API key I had pasted into the chat myself so the agent could configure it. Five copies. It's revoked now.

There are signs outside my machine too. GitGuardian reported this year that commits co-authored by Claude Code leak secrets at 3.2%, against a 1.5% baseline across public GitHub, with a caveat they stress themselves: the leak happens inside a human workflow, it isn't only the tool's fault. And in a public transcript dataset that its author cleaned before publishing, 34 redacted-secret markers remain.

The lesson is the same as the rest of this post, with a different face: a detector that ignores context is wrong with total confidence. If you scan your transcripts (gitleaks is the standard and runs locally), check where each finding appears before you panic. And if you ever pasted a real key into the chat, treat it as exposed even if you can't find it.

Why this matters beyond my machine

Several tools sell "62% token reduction" or "40–70% lower bills". None of them publishes its experimental design.

I got it wrong four times, with data I fully controlled, thresholds fixed in advance, and the explicit goal of not fooling myself. A vendor selling savings doesn't have that incentive. It has the opposite one.

When someone shows you a savings number, the useful question isn't how big it is. It's:

1. Is it controlled for size? If the "has the problem" group does twice the work of the "doesn't" group, the number means nothing.
2. What counts as user input? In agents, the system itself writes into the user's channel.
3. Is there a control group, or just a before and after?
4. What counts as "task done"? Without that definition, savings are measured in tokens, and tokens are easy to cut: just have the agent do less work than was needed.

Limits

My data is one person: 16 sessions, 133 episodes. TraceLab adds 43 developers, but from a single research group and with more Codex than Claude Code usage.

My sessions aren't representative: there is a lot of browser-driven research, not just coding.

Cost is computed at public API prices on usage that was actually a subscription. It's a common unit of effort, not a bill I paid. And one more confession: my first price table was a model generation old and overpriced Opus about three times. The numbers here use current prices; the percentages barely move, the dollars do.

The detection rules are regular expressions I wrote, not validated against human annotation.

The conclusion I take away isn't a number. It's a habit: whenever a cost number pleases me, compare groups of the same size before believing it.