Pimp My IDE / Garage Dispatch
Back to the garage
October 5, 2026 | testing / agents / library design

Throw away the client. Keep the friction.

Daniel Lemire proposes testing a component by asking an agent to build disposable software on top of it. That can expose bad defaults, hidden state, and missing documentation. The useful artifact is not the demo. It is the record of where the demo fought back.

The take: an agent-made prototype is a consumer probe, not a release verdict. Pin the foundation, choose a task the component claims to support, save the build trace, and discard the prototype after review.

Test the surface a consumer meets.

Lemire calls the method "ephemeral testing." Build a temporary application or layer on top of the component under review. Judge how well that temporary software works, then throw it away. He argues that clean application programming interfaces, stable invariants, and useful errors help the agent build quickly. Hidden state, surprising defaults, and incomplete documentation produce patches and failures instead.[1]

This tests a different question than a unit test. A unit test checks a behavior selected by the component author. A throwaway consumer asks whether a new user can find the route, combine the parts, and recover when the happy path breaks.

The disposable client is the probe. The friction log is the evidence.

Fresh does not mean unpinned.

A clean checkout can still produce a meaningless result if it downloads moving dependencies, reads host state, or starts with an undocumented toolchain. Bazel's hermeticity guide defines a hermetic build as one that returns the same output for the same source and configuration. It isolates the build from host software and pins tools and dependencies.[2]

Record the component revision, tool versions, dependency lock, environment image, task, prompt or instruction, and network policy. If the agent retries, save each attempt. A final working screenshot hides the exact detours this test is supposed to reveal.

Do not confuse disposable code with disposable infrastructure.

Testcontainers documents both per-suite containers and a singleton pattern that reuses containers across integration test classes. Its guide warns that starting a new container for every test is resource intensive.[3] The temporary thing can be the consumer code while the pinned test service stays warm.

VS Code 1.140 adds another lifecycle choice for agent work. Its release notes say an idle Dev Container stops after five minutes, restarts when work resumes, and is removed when the last session is marked done or deleted.[4] That is workspace cleanup. It does not prove that two runs used identical inputs.

Keep the test narrow enough to compare.

  1. Pick one consumer job the component claims to support.
  2. Start from a fresh checkout with pinned tools and dependencies.
  3. Give the agent a fixed budget for attempts, time, and network use.
  4. Save commands, errors, patches, retries, and the final result.
  5. Discard the consumer code. File the friction against the component revision.

Repeat the same job with another agent or another fresh run if you want a comparison. Keep the task and foundation fixed. Otherwise you have two anecdotes that share a label.

Interactive makeover / disposable consumer test

Ephemeral test cold-start bay

Traditional purpose replaced: a generic "run prototype" button. Better version: choose the consumer load, add the four evidence sections, and copy a brief that keeps test design separate from test results.

1. Select the consumer load2. Inspect friction evidence3. Copy the test brief
Consumer load
Test brief sections

All four selections mean the brief structure is complete. They do not mean a run happened or a finding was confirmed.

Fresh checkoutFriction record
CLI / 4 SECTIONS OPEN

Command-line client selected. Define the exact command path and one failure case.

Sources read, claims separated

  1. Daniel Lemire, "Ephemeral testing", October 5, 2026. Proposal to judge a component through disposable software built on top of it. The article reports Lemire's own practice, not a controlled benchmark.
  2. Bazel documentation, "Hermeticity", read October 5, 2026. Same-input output, host isolation, pinned tools, dependency control, caching, and troubleshooting benefits.
  3. Testcontainers, "Container lifecycle management using JUnit 5", read October 5, 2026. Per-suite lifecycle, static containers, singleton reuse, and the cost of starting a new container for every test.
  4. Visual Studio Code 1.140 release notes, September 30, 2026. Five-minute idle stop, automatic restart, and container removal behavior for agent Dev Container sessions.
  5. Hacker News item 49972008, October 5, 2026. Discovery and discussion route. Lemire's article supports the method attributed to him.

Source boundary. No source above shows that ephemeral testing predicts production reliability or replaces unit, integration, fuzz, security, or performance tests. The cold-start bay creates a review template. It does not run an agent, build a client, inspect a trace, delete code, or make a release decision.