ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

ParetoSaddler separates intermediate Pareto-aware promotion from final deployment selection, improving quality and reducing cost on ClawMark and GAIA2.

Existing automatic harness optimization methods primarily optimize response quality, while deployment must also consider execution cost. Staged evaluation screens candidates before broader development evaluation, and applying the deployment preference at this stage can prematurely reject updates valuable for later improvements. Repeated executions also vary in quality and cost, so mean comparisons can overstate weakly supported gains.

At each iteration, an optimizer updates a parent harness to produce a candidate. ParetoSaddler compares the candidate and parent over repeated executions. It uses Bayesian bootstrap Pareto support to promote candidates showing credible progress in either quality or cost. Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. Final deployment selection is applied only after the optimization budget is exhausted. The framework is evaluated on ClawMark and GAIA2 using domain-disjoint splits.

• On ClawMark, PS-Pareto Q improves quality from 70.86% to 81.35% while reducing cost from $1.17 to $1.01 per task. §5.2
• On GAIA2, PS-Pareto Q improves quality from 56.22% to 62.89% while reducing cost from $0.78 to $0.45 per task. §5.2
• PS-Pareto Q outperforms quality-only AS-Quality by 7.49 quality points on ClawMark and 5.56 on GAIA2 while also reducing cost on both benchmarks. §5.2
• PS-MinCost reaches $0.82 per task on ClawMark and $0.44 on GAIA2, lower than both development-selected PS-Pareto operating points, and dominates MH-MinCost on both benchmarks. §5.2
• For archived candidate I-12, the mean-based gate would promote it, whereas its bootstrap promotion support of 0.55 falls below gamma=0.7. §E
• Across two PS-Pareto GAIA2 runs, overall quality ranges from 61.44% to 62.89%, with costs of $0.44–$0.47 per task, compared with 56.22% and $0.78 for the default agent. §F

• With R=3 executions per task, rare outcomes may remain unobserved, and estimates of execution variability and bootstrap promotion support may be sensitive to observed runs, especially near the promotion threshold (stated in Appendix A).
• ClawMark evaluation is restricted to initial-stage tasks with explicit stage-0 rubrics, and later-stage adaptation is outside the scope of the evaluation (stated in Appendix A).
• The promotion threshold gamma=0.7 lies in a locally stable region in replay, but the paper states that a substantially different deployment domain may benefit from a small grid search over gamma when additional optimization budget is available (stated in Appendix E).
• Only one additional PS-Pareto optimization run on GAIA2 is conducted for stability because each full optimization run requires substantial computation (stated in Appendix F).

• Automatic harness optimization can improve LLM agents without modifying model weights by changing prompts, tools, middleware, and control logic. §1
• Applying the final deployment preference during intermediate screening can prematurely reject updates whose immediate quality–cost trade-off is unfavorable but whose changes enable stronger later solutions. §1
• ParetoSaddler promotes candidates showing credible progress in either quality or cost without requiring them to satisfy the final deployment preference. §1
• Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. §Abstract
• Pareto reasoning retains multiple operating points along the development quality–cost frontier rather than committing the search to a single trade-off. §5.2
• The experiments use R=3 executions per task and J=10,000 independent Bayesian bootstrap draws, with the cost safeguard disabled by setting epsilon_C to +infinity. §A
• ClawMark experiments use OpenClaw v2026.6.9 and adapt ClawMark with unique Notion roots per execution and per-execution Drive folders. §A
• Meta-Harness is run with approximately 1.37× and 1.22× the maximum task-execution budgets of the staged methods on ClawMark and GAIA2, respectively. §A