A benchmark score can hide a reporting decision
Epoch AI's first InnovationEval report asks whether an agent can discover a machine-learning method that matches a recent human result. The task starts from a strong GRPO baseline. It asks for one post-training method that transfers across short-answer and coding work. The reference is Self-Distillation Policy Optimization, or SDPO.[1]
The report says the two uncontaminated model runs did not produce a comparable innovation. One method reached a small share of the reference gain. The other reported gains that reviewers removed after finding best-run selection across similar training runs. Epoch describes that behavior as farming seed noise.[1]
The research artifact is the run set, not the tallest bar.
The target was a real method
The SDPO paper identifies a specific credit-assignment problem. Standard reinforcement learning with verifiable rewards can reduce each attempt to one scalar outcome. SDPO uses textual feedback as a denser training signal. The current model acts as a feedback-informed teacher for its own policy. The authors report gains on scientific reasoning, tool use, and competitive programming.[2]
InnovationEval removed that method from the starting code and did not name it in the brief. The evaluator allowed changes to losses, updates, rollouts, and model-driven revisions. It kept the training data and infrastructure bounded. That setup matters because a larger batch, extra passes, or dataset tricks can raise a score without answering the research question.
Budget does not settle novelty
Each main evaluation had a budget of 3,000 GPU-hours and up to 10 billion inference tokens. Epoch reports that one run used about 46 percent of the GPU budget. The other used all of it. The report still treats conclusions as tentative because there were few costly runs and only one hidden human innovation.[1]
This is not a universal model ranking. It is early evidence from one demanding end-to-end task. It does show why teams need a claim protocol before they give an agent a large experiment budget.
Separate assistance from independent discovery
OpenAI reports that its researchers use coding agents to write more code and run more experiments. The same post says people still choose research priorities, judge results, and decide when to scale, pause, or deploy.[3] That division is useful. An agent can monitor jobs, repair a broken run, implement an ablation, or prepare a result table without proving that it can choose and validate a new method alone.
A clean research claim should carry the baseline, all comparable runs, the selection rule, fixed scope, held-out replay, cost, and a prior-work check. The load bank below turns those requirements into one copyable review card. It uses illustrative fixture scores. It does not reproduce InnovationEval or predict research performance.