Pimp My IDE / garage dispatchBack to the garage
13 Oct 2026 / coding agents / benchmark design

A passing patch still leaves a maintenance bill.

Parsimony adds code footprint to agent benchmarks. That is useful pressure, as long as nobody mistakes fewer syntax units for better software.

Pass rate staysFootprint joinsReview decides

Benchmarks needed a second gauge.

DeepSWE reports whether an agent solves an engineering task. It also reports average cost, output tokens, and agent steps. Its current version lists 113 tasks across 91 repositories and five languages.[3] Those measurements say what happened during the run. They do not say how much code the accepted patch leaves behind.

Parsimony takes published benchmark patches and counts coding units in the code before and after each patch. A unit is a syntax element such as a statement, call, comparison, name, or literal. Formatting and comments do not count. Deleted units are subtracted from added units.[1]

Pass rate tells you whether the patch cleared the test. Footprint tells you how much new structure entered the garage.

The method avoids the dumbest line-count tricks.

Lines of code reward cramped formatting. Parsimony parses code instead. For Python, it removes docstrings, walks the abstract syntax tree, and compares the ordered unit sequences. It excludes tests, documentation, examples, generated files, migrations, and vendored code according to documented path rules.[2]

The method keeps net growth and churn separate. A patch that adds 30 units and removes 20 has a net footprint of 10 and churn of 50. That distinction matters. A rewrite can leave the codebase the same size while forcing a reviewer to inspect a large change.

We inspected repository revision 188a695ff5f88f9625a543b6eda3859ff95b4dea. We ran the README's small Python example against the checked-out analyzer. It returned 3 units added, 13 deleted, a net of minus 10, and churn of 16. We did not rebuild every board or rerun upstream benchmark tests.

Small is not the same as maintainable.

The project states this limit plainly. A compact patch can be cryptic, insecure, or tuned only to visible tests. A larger patch may add validation that the task needs. Parsimony also inherits pass and fail labels from the source benchmarks. It does not rerun those tests for its published boards.[2]

Task mix matters too. The main DeepSWE Python board ranks models by mean net units on tasks they solved. A model that solves a different set of tasks does not face an identical footprint comparison. The site publishes measured coverage and keeps languages on separate boards because unit counts do not transfer cleanly between languages.[1]

The right reading is narrow. Code footprint is a maintenance signal. It is not technical debt, readability, architecture quality, or production safety.

Put footprint beside the review path.

For an agent evaluation, keep the original correctness test. Add net units and churn for the same task, language, analyzer version, and file scope. Then sample the patch for duplication, hidden branches, dependency growth, error handling, and future edit cost.

Do not turn one low number into a universal model rank. Compare matched tasks. Keep failed attempts visible. Record the agent and prompt policy. A coding model does not produce a patch alone. The surrounding agent decides what to inspect, when to stop, and how much code to touch.

The useful outcome is a two-axis buying decision. Choose the system that solves your workload with a reviewable footprint. If two systems clear the same task, the smaller change deserves attention. It still has to survive code review.

Interactive makeover / footprint load bank

Load both gauges.

A normal model table ranks one number. This load bank joins a user-entered footprint comparison to the evidence needed to interpret it. It drafts a review sheet. It does not test code or choose a model.

Match taskConfirm passMeasure footprintReview patch

Comparison inputs

The values are teaching inputs, not production telemetry. Set the reported units for one matched task.

Required evidence

These switches select review requirements. They do not record that a check passed.

Matched footprint readout

Both rails enter the same comparison node. The ratio describes code size only.

Agent2400
1.50x
Reference1600
Comparison frame open.

No evidence requirement is selected. The displayed ratio cannot support a model choice.

Sources read / 13 Oct 2026

Source log

Open the evidence boundary
  1. Parsimony website and current boards, read October 13, 2026. This is the source for the board structure, coding-unit definition, measured coverage, and published model results.
  2. Parsimony repository at revision 188a695ff5f88f9625a543b6eda3859ff95b4dea, MIT licensed, inspected October 13, 2026. We read the README, method, analyzer, scope filters, tests, and offline check script. We ran the documented small Python example and received net units of minus 10.
  3. DeepSWE v1.1 leaderboard, read October 13, 2026. This is the source for the task, repository, language, model, pass rate, cost, output-token, and step fields that Parsimony supplements.
  4. Hacker News item 50077619, resolved through the Hacker News API and read October 13, 2026. The author post identifies the project, method, repository, and request for public patch exports.

Evidence boundary: we inspected one exact repository revision and ran its small Python README example on Python 3.11. The project documents Python 3.12 or newer for comparable measurements. We did not run the complete offline check suite, rebuild the published boards, fetch every upstream patch, rerun benchmark tests, or audit the metric independently. The load bank uses reader-selected values and creates a review-sheet structure only.