Harness Optimization

Comparing ways to improve the systems around models through search, learning, and evaluation

Even with the same language model, instructions, tools, processing order, and the use of past experience change which problems it can solve and at what cost. The surrounding system that runs the model is called a harness. This book compares research that improves this system using execution results and evaluations.

What can change, how are the next candidates generated and selected, and what did the experiments establish? Separating these three questions reveals differences between methods that final scores alone cannot show. The book focuses on search with fixed model weights, while also covering experience reuse and learned generators. It examines when improvements carry over to unseen problems and where the evidence remains limited.