The tailpipe number is real—and incomplete.
RTK is a Rust proxy that rewrites common development commands and compresses their output before a coding agent reads it. Its repository headline says it can reduce token consumption by 60–90% on common commands; its own disclaimer narrows that promise: cutting bash output is not the same thing as cutting a bill by the same percentage.[1]
That distinction matters because terminal output is only one stream inside an agent run. The model also consumes instructions, conversation history, tool schemas, file reads, search results, previous outputs, and its own continuing work. A compressor can remove bytes from one stream while changing what the agent does next.
A muffler can make the exhaust smaller. It cannot certify the fuel economy of the whole car.
The new benchmark found a round trip.
Quesma tested RTK on Terminal-Bench 2.1 with two harness/model combinations. It reports 1,740 attempts after exclusions: each task ran five times with RTK and five times without it on the same route, platform, and task-specific timeout.[2]
Its aggregate bill fell 5% for one tested combination and rose 5% for the other, while pass-rate differences were small. On its equal-weight-per-task comparison, it reports no clear cost difference from zero for the first combination and a 17% average task-cost increase for the second. Those are results for this benchmark and these configurations—not a universal verdict on RTK or output compression.
The most useful failure is wonderfully ugly: one unsupported find flag created a rewrite/error/retry loop. Quesma recorded 339 consecutive errors before timeout and says that passing attempt cost about nine times its matching passing baseline. RTK fixed the behavior after the runs.[2] That is not “compression bad.” It is “the adapter is now part of the control loop.”
Count what the agent never would have seen.
Quesma says RTK’s gain estimate is based on raw versus filtered command output in bytes divided by four, rather than provider-billed tokens. It documents a case where limited head -1 reads were compared with whole-file output; two calls produced 69% of that comparison’s reported savings counter even though the requested command would not have emitted the whole file.[2]
A counter can be internally consistent and still answer the wrong question. Instrument the baseline command actually requested, the filtered response actually returned, every later turn, the final outcome, and the invoice. If the optimization changes behavior, the unit of analysis is the run.
Compression is not your only lever.
Anthropic’s prompt-caching documentation describes a different optimization: reuse an identical prompt prefix so repeated context can be processed with lower latency and cost. The documented cache includes full prefixes across tools, system content, and messages, with a default five-minute lifetime and an optional longer duration.[3] Caching preserves the prefix; output compression changes what reaches the model. They solve different pressure problems and can be evaluated separately.
OpenAI’s Agents documentation makes the runtime surface explicit: managed and application-run options differ in state ownership, tool execution, sandboxing, and orchestration.[4] That is the larger lesson. “Tokens” do not float in a jar. They move through a harness with state, tools, adapters, retries, and a bill.
Install the mod like an engineer.
- Baseline completed work. Record cost per passed task, not a savings counter from one command.
- Stratify the route. Separate output-heavy test runs from architecture, editing, search, and small commands.
- Watch behavior. Track turns, repeated commands, fallback use, loop length, and pass rate.
- Keep an eject path. When the adapter rejects a flag or changes semantics, retry the original command once—then stop the loop.
- Price alternatives independently. Cache stable prefixes, trim tool surfaces, request narrower output, and compare each change against the same workload.