Candidate Selection

Finding a good candidate and returning it are different achievements. During search, a validation score guides the next experiment. At the end, a selection procedure chooses the artifact that will be submitted. A strong candidate somewhere in the archive does not help if the agent returns a broken file or a candidate that only looked better under an easier evaluation.

The practical goal is to preserve useful progress while keeping comparisons meaningful. The same requirements also explain why benchmark results need an explicit account of feedback and resources.

Keeping Scores Comparable

For the tabular classifier, compare candidates using a common data split, metric, and scoring implementation. Fit preprocessing only on the appropriate training data. Record which code and artifacts produced each score. Changing these conditions can change the number without improving the solution.

An agent editing its own training code may accidentally alter evaluation too. Keep the scoring protocol separate from candidate changes and record the code and artifacts behind each measurement. Preserving earlier candidates with the files needed to run them makes it possible to return to an earlier implementation and check it again if a new change fails.

AIRA² uses Hidden Consistent Evaluation (HCE) to preserve comparability. It fixes training, search, and final-selection splits before the search and measures candidates with a consistent evaluator. Agents can use search scores but cannot see the corresponding labels or redefine splits and metrics for each candidate. The separate evaluation path in Figure 1 keeps these measurements outside the agent’s editable training code. Final-selection scores are used only after search ends (Hambardzumyan et al. 2026).

Figure 1: AIRA² separates agent work from a hidden consistent evaluator. The diagram includes development validation and a separate held-out comparison used for final candidate selection. Source: (Hambardzumyan et al. 2026) Figure 2.

The comparison used 30 tasks, eight H200 GPUs, Gemini 3.0 Pro Preview, and three seeds. In Figure 2, the blue no-HCE curve flattens and declines while the green HCE curve continues to improve. Percentile rank is the share of human competitors outperformed. The vertical axis aggregates this measure across the evaluated tasks and runs, rather than tracking one candidate’s raw score. HCE was 13.0 points higher at 24 hours and 18.4 points higher at 72 hours.

Figure 2: AIRA²’s test performance under different evaluation and selection protocols. Final selection on the search split (Dsearch) and on a separate selection split (Dval) produces similar curves. Selection Oracle retrospectively picks the best candidate using test scores unavailable in normal operation. Source: (Hambardzumyan et al. 2026) Figure 4(a).

Under consistent evaluation, however, final selection on the search split performed similarly to selection on a separate held-out split. The authors interpret this as evidence that inconsistent evaluation, rather than memorization of search examples, drove much of the observed degradation. Evaluation consistency and overfitting to development data are distinct problems. This result does not rule out the latter in general.

Reserving an Independent Final Check

Development validation may be consulted repeatedly. A reserved selection split can rank a shortlist once search stops. A further test set then measures the selected artifact without another round of revision. These roles matter more than whether a file is called “validation,” “private,” or “hidden.”

MLE-bench’s grader is an external benchmark assessment. FML-bench similarly selects code using development validation and then measures it on a held-out test set (Chan et al. 2025; Zou et al. 2026). AIRA²’s additional selection split plays a different role from a final untouched assessment.

Selecting appropriately from preserved candidates also requires a mechanism. HMA reserves the final 15 minutes of its six-hour runs for another agent to review the archive. Without test scores, that agent selects a final candidate from earlier implementations and records. Final review reduced, but did not eliminate, missed historical medals. Preserving artifacts and selecting the right final candidate are not the same achievement (Li et al. 2026).

Protected scoring alone does not prevent test leakage. In AIRA²’s AIRS-Bench audit, the authors found direct answer extraction on FinQA and training on evaluation examples on SuperGLUE WSC. Other tasks raised suspicions that evaluation data had appeared in model training. Legitimate external-model reuse must be distinguished from test leakage. Evaluators need to state allowed resources and inspect solution provenance as well as scores (Hambardzumyan et al. 2026).

In practice, retain a valid fallback early so that a failed refinement does not leave the search without a submission. Before submission, verify the selected file, prediction format, evaluator compatibility, and correspondence to the recorded code.

Spending Time on Experiments Rather Than Merely Stopping

A time budget must cover planning, training, diagnosis, repairs, and final delivery. A prompt saying “finish in 15 minutes” does not give a model reliable knowledge of elapsed time. On the Clock compares budget prompts, clock observations, enforcement, and reinforcement learning (Wang et al. 2026).

In the reported MLE-bench subset with Qwen3.6-27B in OpenCode, prompt-only agents finished within a 15-minute budget in 15% of cases. Exposing clock information raised this to 62%. Refusing shell commands whose requested timeout exceeded the remaining budget raised it to 88%. Neither task quality nor valid-submission rate was thereby maximized. Quality in this comparison was measured on valid, on-time outputs and must be read with their frequency.

The separate budget sweep in Figure 3 makes the distinction visible: with clocks and enforcement but no reinforcement learning, runtime distributions shift with the allowed time, while valid-submission percentiles show no clear improvement.

Figure 3: On the Clock plots runtime distributions and leaderboard percentiles for valid submissions under 10-, 15-, 20-, and 25-minute budgets. The agents receive clocks and enforcement but no reinforcement learning. Source: (Wang et al. 2026) Figure 4.

The trained policies expose a further distinction. A policy can learn when to stop without learning how to use a larger budget. In some conditions extra time produced more attempts but no better final score. Effective budgeting therefore needs both a clock and choices about what can still finish usefully. Reserving time for a checked fallback is more robust than leaving submission until the final instant.

Comparing Compute and Search Budgets

“Twenty-four hours” describes elapsed time, not total compute. AIRA² with eight GPUs can run several training workers concurrently. In Figure 4, the horizontal axis instead counts total GPU-hours: the eight-GPU curve initially trails the one-GPU curve, then overtakes it. This compares performance at matched compute rather than matched elapsed time (Hambardzumyan et al. 2026).

Figure 4: AIRA² compares one- and eight-GPU configurations at matched total GPU-hours. Source: (Hambardzumyan et al. 2026) Figure 3(a).

Other units also hide differences. FML-bench fixes 100 validation executions but leaves token usage and execution duration variable. WhatWorkedBench counts purchased measurements, with two additional initial anchors. Population Evolution includes both iteration-matched and total-GPU-matched comparisons (Zou et al. 2026; Ning et al. 2026; Peng et al. 2026). Match the budget to the question being asked.

Independent attempts introduce another selection issue. RE-Bench found benefits from many short runs under a fixed total budget. The horizontal axis in Figure 5 counts time per run multiplied by the number of attempts. At each total budget, the curves report the best observed score@k across the tested combinations of run duration and number of attempts. They show neither one uninterrupted run nor an evaluation of a rule that chooses the allocation in advance.

Within one allocation, score@k selects the highest-scoring run using available scoring feedback. For Scaling Law Experiment, where that feedback is unavailable, selection is random. Selection among completed runs uses available feedback, but visible test feedback and noisy scores can make the reported maximum optimistic for performance on a fresh independent assessment (Wijk et al. 2025).

Figure 5: RE-Bench compares human and agent progress across time budgets and best-of-k aggregation. The selection procedure is part of the result, alongside the total time budget. Source: (Wijk et al. 2025) Figure 2.

Reporting What the Experiment Demonstrated

A compact comparison should answer five questions.

  1. What was changed? Distinguish solution code, proposal operators, search policy, memory, and the agent itself.
  2. What feedback was available? Identify development scores, final-selection scores, and retrospective test measurements.
  3. What resources were used? Report model, runtime, hardware, parallelism, and the experimental unit being counted.
  4. What was returned? Separate the best candidate discovered from the artifact actually selected and submitted.
  5. What outcome was measured? Distinguish valid delivery, predictive performance, effect prediction, and novelty.

These details let the reader judge which design helped, under what conditions, and what to reproduce.

The full loop is now connected. Generate feasible alternatives, carry forward useful experience, diagnose uncertainties that affect decisions, and evaluate candidates consistently. These functions reinforce one another. No particular tree, memory format, or score can replace all four.

Figure Sources and Usage Terms

References

Chan, Jun Shern, Neil Chowdhury, Oliver Jaffe, et al. 2025. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” International Conference on Learning Representations 2025: 50466–94. https://proceedings.iclr.cc/paper_files/paper/2025/hash/7e3767db483c942b883eb4f8cfb74e31-Abstract-Conference.html.
Hambardzumyan, Karen, Nicolas Baldwin, Edan Toledo, et al. 2026. “AIRA_2: Overcoming Bottlenecks in AI Research Agents.” arXiv Preprint arXiv:2603.26499. https://arxiv.org/abs/2603.26499.
Li, Changye, Zijian Zhang, Muheng Li, et al. 2026. “Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits.” arXiv Preprint arXiv:2610.13494. https://arxiv.org/abs/2610.13494.
Ning, Jingjie, Xueqi Li, Yibo Kong, and Dongting Li. 2026. “WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents.” arXiv Preprint arXiv:2609.27490. https://arxiv.org/abs/2609.27490.
Peng, Bo, Lizhu Zhang, Yuhang Zhou, et al. 2026. “Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery.” arXiv Preprint arXiv:2610.05950. https://arxiv.org/abs/2610.05950.
Wang, Aaron, Neelabh Madan, Vlad Sobal, et al. 2026. “On the Clock: Towards Punctual and Productive Time-Budgeted AI Agents.” arXiv Preprint arXiv:2610.10833. https://arxiv.org/abs/2610.10833v1.
Wijk, Hjalmar, Tao Roa Lin, Joel Becker, et al. 2025. “RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts.” Proceedings of the 42nd International Conference on Machine Learning, Proceedings of machine learning research, vol. 267: 66772–832. https://proceedings.mlr.press/v267/wijk25a.html.
Zou, Qiran, Hou Hei Lam, Wenhao Zhao, et al. 2026. “FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics.” arXiv Preprint arXiv:2605.17373. https://arxiv.org/abs/2605.17373.