Meta-Harness: End-to-End Optimization of Model Harnesses

Introduction

Changing the harness around a fixed large language model can produce a 6× performance gap on the same benchmark. The harness—the code that determines what to store, retrieve, and show to the model—often matters as much as the model itself. Despite its importance, harness engineering remains largely manual: practitioners inspect failures, adjust heuristics, and iterate on a small number of designs.

A natural starting point is recent work on text optimization, since harness engineering also involves iteratively improving text and code artifacts using feedback from prior attempts. However, these methods are poorly matched to harness engineering because they operate with short-horizon or heavily compressed feedback: some condition only on the current candidate, others rely primarily on scalar scores, and others restrict feedback to short templates or LLM-generated summaries. Across representative text optimizers, the available context per optimization step ranges from only 100 to 30,000 tokens (Table 1)—far below the diagnostic footprint of harness search.

We address this limitation with Meta-Harness, an agentic harness for optimizing harnesses via end-to-end search. Its proposer is a coding agent—a language-model-based system that can invoke developer tools and modify code. Its key design choice is to expose full history through a filesystem, enabling selective diagnosis of raw prior code and execution traces rather than optimization from compressed per-candidate summaries. In practice, the proposer reads a median of 82 files per iteration, referencing over 20 prior candidates per step. A single evaluation can produce up to 10,000,000 tokens of diagnostic information—roughly three orders of magnitude beyond prior text optimization settings.

We evaluate Meta-Harness on online text classification, mathematical reasoning, and agentic coding—demonstrating that richer access to prior experience enables automated harness engineering across diverse domains.

Meta-Harness: A Harness for Optimizing Harnesses

The Search Loop

Meta-Harness uses a single coding-agent proposer (Claude Code with Opus-4.6) with access to a growing filesystem that serves as its feedback channel. Unlike prior systems that externalize the improvement logic in a hand-designed search loop, Meta-Harness delegates diagnosis and proposal to the coding agent itself: it decides which prior artifacts to inspect, which failure modes to address, and whether to make a local edit or a more substantial rewrite.

Each evaluated harness contributes a directory containing its source code, scores, and execution traces. The filesystem is typically far larger than the proposer's context window, so the proposer queries it through terminal tools such as grep and cat rather than ingesting it as a single prompt. In our most demanding setting, the proposer reads a median of 82 files per iteration, referencing over 20 prior candidates per step.

# Input: tasks X', LLM M, proposer P, iterations N
Initialize: population H  # initial set of valid harnesses
Initialize: filesystem D ← ∅  # stores code, scores, traces

for H ∈ H do:
    E_H ← Evaluate(H, M, X')
    D ← D ∪ {(H, E_H)}

for t = 1 ... N do:
    # Proposer P queries filesystem D
    # inspects prior harnesses and scores
    P proposes k new harnesses {H₁, ..., Hₖ}
    for H ∈ {H₁, ..., Hₖ} do:
        if H passes interface validation then:
            D ← D ∪ {(H, Evaluate(H, M, X'))}

return Pareto frontier of harnesses in D

Experiment 2: Retrieval-Augmented Math Reasoning

We study olympiad-level math solving augmented with retrieval from a corpus of 500,000+ solved problems. Naive retrieval rarely works on math benchmarks; success depends on discovering the right retrieval policy. Rather than hand-designing that policy, we give Meta-Harness a hard set of olympiad problems and allow the retrieval behavior to emerge from search.

We run 40 iterations over a 250-problem search set (OlympiadBench + Omni-MATH hard), producing 109 candidate retrieval harnesses. A single harness is selected and evaluated on 200 previously unseen IMO-level problems from IMO-AnswerBench, IMO-ProofBench, and ArXivMath—plus four held-out models never seen during search.

Inside the Discovered Harnesses

Meta-Harness discovers executable inference-time procedures—structured, domain-specific policies with nontrivial control flow. Here we examine the two text classification harness variants that represent the Pareto frontier extremes, plus generalization evidence.

References