Understanding Results
A better score establishes that one candidate performed better under a particular evaluation. It does not necessarily explain why. An experiment becomes more informative when it distinguishes possible causes and changes what the agent should try next.
For the tabular classifier, a gain after changing both encoding and the model leaves several explanations open. The model may help, the encoding may help, or their interaction may matter. A useful diagnosis asks a question whose answers lead to different actions, rather than simply adding a plausible story to the winning code.
Diagnosing Before Training Again
Iris provides a concrete example from Kuzushiji recognition, which detects historical Japanese characters and their locations. Routine tuning raised F1, a measure combining precision and recall, from 0.629 to 0.661, then stalled. The working explanation blamed model capacity. An audit of inference code instead found a default cap of 100 detections per page. Dense pages lost predictions because of the cap, not because the model failed to recognize every character. Lifting it raised F1 to 0.817 without retraining (Fu et al. 2026).
A second diagnosis found that internal downscaling prevented intended high-resolution training from taking effect. Addressing this issue led to 0.923. The two diamonds in Figure 1 precede the corresponding score jumps: the first diagnosis avoided training, while the second identified where training could help.
Iris distinguishes actions that change a solution from epistemic actions, which obtain evidence without replacing the retained solution. Error analysis, profiling, and validation checks can belong to the latter. The useful question is not whether diagnosis immediately raises the score, but whether its possible outcomes distinguish the next interventions.
Revising Explanations Instead of Accumulating Stories
Iris stores raw evidence, action summaries, and task-level claims separately. Each claim records scope, supporting and contrary evidence, and status: active, qualified, or invalidated (Fu et al. 2026).
The output-cap finding invalidated the initial explanation for low recall. Later evidence narrowed a claim about rare characters and revised the judgment that an approach was exhausted. This prevents a rejected explanation from persisting as an unquestioned reason for future work.
The mechanism does not require diagnosis before every edit. When existing evidence supports a clear next intervention, the agent proceeds. The paper reports 64.9% any-medal rate on the full MLE-bench under a 12-hour budget. Its component ablations are narrower: one run per variant on 15 small-data tasks, where removing epistemic actions reduced medal rate from 66.7% to 60.0%, and removing information management reduced it to 53.3%. The trajectory explains how the mechanism can work, while these ablations provide limited evidence about its average contribution.
Predicting Effects Under Different Conditions
WhatWorkedBench tests a different aspect of understanding: can an agent predict outcomes it has not measured? Each small experimental space has four or six binary choices. The agent receives all-off and all-on measurements, spends a budget on additional configurations, and submits utilities for the complete space. A utility is a task-performance score oriented so that higher is better. Exhaustive reference measurements allow the evaluator to check the predictions (Ning et al. 2026).
A conditional effect is the utility difference produced by toggling one choice while holding the others fixed. Interactions arise when that difference changes with another choice. In the SciFact example in Figure 2, binarizing word counts helps when sublinear term-frequency processing is off and hurts when it is on. The binary-count result stays the same, while the raw-count baseline changes.
This permits a direct separation between finding a winner and reconstructing effects. Across 22 four-factor sources, ridge regression with pairwise interactions selected the optimum in 15 cases with eight additional measurements beyond the two initial anchors. Yet 13 of those 15 had a conditional-effect error exceeding 10% of the utility range. The winning configuration was correct, but at least one conditional effect in the full experimental space was substantially mispredicted (Ning et al. 2026).
A good submission remains useful. The additional test asks whether its measurements support decisions about other configurations, which matters when transferring a change or explaining why it helped.
Extracting More from the Same Measurements
Poor predictions can arise from missing measurements or from an unsuitable method for inferring unmeasured outcomes. WhatWorkedBench compares both. For tasks whose true effects are not all zero, effect recovery is one minus the mean absolute error in predicted conditional effects divided by the mean absolute true effect, clipped below at zero. One means exact recovery, and zero means error at least as large as the mean true-effect magnitude.
Figure 3 aggregates these values across task groups. Reconstruction methods differ at the same measurement budget, particularly in the six-factor panel. Different estimators can infer different unmeasured effects from the same observations, and the ranking of measurement policies can change with the estimator (Ning et al. 2026).
For some agent cohorts, replacing the supplied ridge reconstruction with a Gaussian process improved predictions without buying another experiment. A later 12-source cohort allowed direct numerical tables as well as calls to named estimators. DeepSeek Flash submitted tables on all 12 sources and exceeded a Gaussian-process baseline using the same observations, while Pro’s all-attempt comparison did not show the same gain. These results make the measurement interface and reconstruction method part of the problem, rather than establishing either universal agent failure or universal agent superiority.
The evaluation remains deliberately bounded. It tests numerical effects in specified binary spaces, not whether a free-form explanation discovers a new scientific mechanism.
Planning Comparisons and Judging Novelty
Before measuring an effect, the agent must design a useful comparison. AblationBench evaluates whether proposed ablations recover authors’ experiments or reviewer-requested follow-ups. In Figure 4, removal plans have higher Recall@5 than modification plans for every displayed planner. This compares plans semantically with reference plans rather than executing new training experiments (Abramovich and Chechik 2026).
Removing a component and replacing it answer different questions. Removal asks how the current system depends on it. Replacement asks whether its particular implementation matters. Matching reference plans can assess planning competence, but useful unlisted plans may receive little credit and the information obtained after execution remains unmeasured.
Novelty is different again. Can LLM Agents Discover? separates novelty relative to earlier ideas in the same run, called P-creativity, from novelty relative to public human solutions, called H-creativity. Usefulness is measured separately through execution and task performance.
Its Impact metric normalizes improvement over the sample-submission baseline by the gap between that baseline and the top human leaderboard score. Zero is the baseline and one is the top human score. The cassava run in Figure 5 illustrates the separation: Episode 3 receives the highest H-creativity rating, but Episode 4 has slightly higher Impact (Bhushan et al. 2026).
Its wider trajectory study found that scores could improve while novelty relative to earlier ideas declined. The policy comparison in Figure 6 places usefulness on the left and P-creativity on the right: greedy search and MCTS generally lose novelty, while evolutionary search ends with higher novelty but no consistent usefulness advantage. No search strategy consistently improved both quantities in this smaller comparison.
The reference corpus and judges define the scope of these novelty measurements. They do not establish novelty relative to private work or all of science.
A useful experiment loop therefore keeps four achievements distinct: finding a strong candidate, predicting what changes will do, planning an informative comparison, and producing a useful novel idea. The next chapter addresses another necessary achievement: selecting the strong candidate at the end.
Figure Sources and Usage Terms
- Source: (Fu et al. 2026) Figure 4 (original figure reproduced, converted to PNG).
- Source: (Ning et al. 2026) Figure 2 (original figure reproduced, converted to PNG).
- Source: (Ning et al. 2026) Figure 4 (original figure reproduced, converted to PNG).
- Source: (Abramovich and Chechik 2026) Figure 3 (CC BY 4.0, original figure converted to PNG).
- Source: (Bhushan et al. 2026) Figure 5 (CC BY 4.0, original figure converted to PNG).
- Source: (Bhushan et al. 2026) Figure 3 (CC BY 4.0, original figure converted to PNG).