Overview
An agent that runs machine learning (ML) experiments must do more than write a training script. It must choose an experiment, execute it, interpret the result, and decide what to retain. A large language model (LLM) supplies plans and code, while an execution system supplies data access, training, logs, and scores. The quality of the loop depends on both.
What Is Being Improved?
Consider a classifier for tabular data. Its pipeline fills missing values, encodes categories, trains a model, and produces predictions. The next experiment might replace imputation, add a feature, change the model, or investigate an unexpected error. A candidate is an executable version of that pipeline together with its relevant files and measurements. It is not a sentence in a reasoning trace.
AIDE makes the loop concrete. It drafts initial scripts, repairs scripts that fail, and improves valid ones. In Figure 1, drafting opens new branches, while fixing and improvement extend existing ones. Storing each version and its parent produces this solution tree (Jiang et al. 2025).
Two decisions occur at every step: which candidate to work from, and what change to make. A sophisticated rule for the first cannot compensate for consistently poor answers to the second. An experiment can also be useful without producing a better candidate, if it resolves an uncertainty that changes the next decision.
Which Data Can Guide the Loop?
Training data fit the predictive model. Validation data estimate how well a candidate works beyond those training examples and guide development. A held-out test set assesses the final selected solution. Its outcomes must not feed back into the same search if they are to remain an independent final check.
Repeated validation is therefore not repeated independent testing. The agent chooses later experiments in response to earlier scores. Differences in splits, preprocessing, scoring code, and random seeds that initialize training randomness can all affect the ranking of candidates. Some systems additionally reserve a separate split for final selection, keeping it out of the development loop.
For the tabular classifier, preprocessing statistics should be fitted on the training portion and applied to the validation portion. Candidates should be compared with the same metric and compatible splits. Otherwise, a higher number may reflect an easier measurement rather than better predictions.
What Do the Benchmarks Measure?
MLE-bench assembles 75 Kaggle competitions. The flow in Figure 2 separates the agent’s work from the evaluator’s work. The agent receives a task description and data, trains models, and submits predictions. The evaluator scores those predictions against historical human competition results (Chan et al. 2025). Medal rate is the fraction of attempts reaching the task’s medal threshold. It is not prediction accuracy and does not use one common difficulty threshold across tasks.
The work includes ordinary engineering as well as model design. In Figure 3, one agent discovers that a NumPy file needs binary loading, while another repairs a missing submission column after validation. These execution results change the next action, just as model scores do.
Other benchmarks answer different questions. RE-Bench compares agents and human experts on seven short AI research and development tasks. FML-bench standardizes editing and execution across 18 ML research tasks to study differences in agent strategies (Wijk et al. 2025; Zou et al. 2026). A competition medal, a research-code improvement, and a correct explanation of an experiment are distinct outcomes.
How Should the Chapters Be Read?
The book follows the decisions in an experiment loop.
- Candidate Code Search explains how to choose a starting point and generate useful modifications.
- Reusing Experience follows the information passed from earlier work into a new proposal.
- Understanding Results asks what observations justify an explanation and how to revise it.
- Candidate Selection explains how to preserve useful candidates, choose a final submission, and compare budgets.
The main subject is improving solution code for a task. Improving the experiment-running agent itself is a different level. In Figure 4, AIDE² places solution search inside an outer loop: rewritten agents are evaluated by running them on several tasks (Srikanth et al. 2026).
Harness Optimization develops that outer-loop question. Here, the organizing question is simpler: given the data, observations, and budget available now, what makes the next experiment useful?
Figure Sources and Usage Terms
- Source: (Jiang et al. 2025) Figure 1 (CC BY 4.0, figure region cropped, converted to PNG, and margins removed).
- Source: (Chan et al. 2025) Figure 1 (CC BY 4.0, original figure converted to PNG).
- Source: (Chan et al. 2025) Figure 2 (CC BY 4.0, original image).
- Source: (Srikanth et al. 2026) Figure 1 (original figure reproduced and converted to PNG).