Pimp My IDE / Garage Dispatch
Back to garage
September 22, 2026 | evaluation / compression / repository context

Before you call it smart, race it against gzip.

A compressor can continue Shakespeare without weights or training. That does not make gzip intelligent. It gives us a cheap way to catch a coding assistant that wins by copying nearby text.

THE TAKE: A repository assistant needs baselines that break easy shortcuts. Hold out files, move relevant text beyond the local window, shuffle names, and plant a tempting copy trap. Then check whether the answer still works.
Open the Compression Oracle Bench

Prediction leaves a compression receipt.

The paper "Language Modeling Is Compression" starts from an established equivalence. A probabilistic model can drive a lossless compressor, and a lossless compressor implies a prediction model. The paper also shows how a compressor such as gzip can act as a conditional generator.[1]

Nathan Barry turned that last idea into GziPT. The program primes zlib with a text corpus, tests possible byte continuations, and keeps candidates that produce shorter compressed output. Beam search helps because gzip reports whole-byte lengths. Many one-byte candidates otherwise tie.[2]

The output looks a little like Shakespeare because the corpus contains Shakespeare. It also breaks down quickly. That failure is useful. It makes the shortcut visible.

Gzip has a short rear-view mirror.

DEFLATE combines LZ77 back-references with Huffman coding. RFC 1951 limits match distance to 32 KiB and match length to 258 bytes.[3] GziPT exploits that local match machinery. A continuation costs less when it resembles bytes still visible in the window.

Local repetition is not repository understanding. A coding assistant can also look capable when the target pattern sits nearby, names stay unchanged, or the benchmark asks for a familiar patch. Move the evidence and the score may collapse.

A baseline is useful when it wins for the wrong reason. It tells you which shortcut the real system must beat.

Test the shortcut, not the brand.

Start with a task that has an executable check. Ask for a bug fix, migration, or narrow feature. Record the result with normal repository context. Then rerun matched variants that remove one cheap route at a time.

  1. Hold out the target file until the task begins.
  2. Move the relevant example beyond the assistant's immediate context.
  3. Rename local symbols without changing behavior.
  4. Add a nearby pattern that looks right but fails the test.
  5. Keep the task, budget, tools, and checker fixed.

If performance survives, you learned more than "the demo worked." If it fails, you found a dependency on proximity, naming, or copying. That is a result worth keeping.

Do not confuse compression with comprehension.

The compression paper reports strong results for large models, but it also says model parameters must count toward adjusted compression rate. Larger models can become worse compressors once their own size enters the bill.[1]

The same accounting belongs in agent evaluation. Count repository indexing, prompt assembly, retries, tool calls, review, and the checker. A polished patch with hidden setup costs is not a clean comparison. A short output that copied a nearby template is not proof of a useful internal model.

Interactive makeover / baseline rig

Compression Oracle Bench.

Traditional purpose replaced: one success rate beside a model name. Better version: one physical window rail, four shortcut-breaking cartridges, a labeled planning gauge, and a copyable test card share the same state.

Set the local-copy pressure

The window control is conceptual. It does not inspect a repository or reproduce DEFLATE. Use it to describe how much relevant material remains close to the task.

22 KiBHow much task-relevant example text remains in the local comparison window.0 KiB32 KiB
Install test cartridges
BASELINE PLAN2 / 4 cartridges
Local copying still has room to win.

The target and distance tests help. Add the rename test and copy trap before treating a passing patch as evidence that the assistant used repository structure.

Why it is better: changing one control updates the window, gauge, verdict, count, and handoff card. The result stays labeled as a planning proxy until matched runs supply evidence.
Sources read, not vibes
  1. Delétang et al., "Language Modeling Is Compression": prediction and lossless-compression equivalence, compressor-based conditional generation, model-size accounting, and experimental boundaries.
  2. Nathan Barry, "Can gzip be a language model?": GziPT implementation, corpus priming, whole-byte scoring limits, beam search, recent-tail handling, and unedited output.
  3. RFC 1951, DEFLATE Compressed Data Format Specification: LZ77 plus Huffman coding, 32 KiB distance limit, and 258-byte match-length limit.
  4. Hacker News discussion, September 22, 2026: discovery trail and live criticism about search quality, repetition, speed, and the limits of comparing gzip with neural models. It does not verify the technical claims above.

Source boundary: The paper establishes the mathematical and experimental compression connection. Barry demonstrates one generator built on zlib. RFC 1951 defines DEFLATE. The repository evaluation method and interactive score are Pimp My IDE's proposed practice, not a result reported by those sources.