Your Benchmark Is an Empirical Measure Over Difficulty

The paper recasts a benchmark as an empirical measure over item difficulty, formalizes measure mismatch and coverage gaps as two failure modes with diagnostic metrics and theory, diagnoses 25 benchmarks, and proposes BenchFill to fill large difficulty gaps.

Benchmark evaluation averages model results over a set of questions, but the conclusion depends not only on model performance but also on how questions are distributed over difficulty. If a benchmark overweights easy questions while the difficulty distribution of interest weights harder ones, leaderboard conclusions for the same model pair can diverge; if a model improves in a difficulty region containing few or no benchmark items, the benchmark score may barely move even though real progress occurred. The paper argues these are two structural failure modes that should be studied as properties of the difficulty distribution induced by the benchmark.

The benchmark is organized along a difficulty axis and treated as an empirical measure over difficulty, with each item contributing equal evaluation mass at its difficulty level. This turns the two failure modes into a measure mismatch problem, comparing the benchmark measure to a target difficulty distribution, and a coverage gap problem, where the finite support of the benchmark leaves intervals with little or no mass. Two diagnostic metrics are proposed: the Wasserstein distance between the benchmark measure and a target difficulty distribution, and the maximum gap between adjacent sorted benchmark difficulty points. Theoretical results link the Wasserstein distance to bounded score distortion and rank stability under a Lipschitz performance-difference function, and show the maximum gap is the exact asymptotic minimax detection boundary for interval-localized improvements. Empirically, item difficulties for 25 benchmarks across 7 domains are estimated with a 1PL Item Response Theory model fitted to multi-LLM success rates, after diagnostic checks on the response matrices and posterior sampling. Finally, BenchFill iteratively rewrites parent items into children targeted at a gap's midpoint, annotates child difficulty by online calibration with pre-estimated model abilities, and keeps the valid child closest to the target, guided by a memory of nearby item-difficulty pairs.

• With only three inserted items per benchmark, BenchFill reduces the sum of the top three difficulty gaps. §1
• The empirical diagnosis covers popular benchmarks across multiple domains, all fitted with difficulty estimates before computing the two metrics. §3
• Curated gap-filled variants expand four math benchmarks to substantially more items, with all generated items manually reviewed. §4.3
• Quantile-matching benchmark calibration on eight math benchmarks improves average agreement with the leave-one-out reference ranking. §D.2
• Calibration reduces the Wasserstein distance to the uniform target on average, whereas random pruning does not. §D.2
• BIC selects the 1PL model over the 2PL model for most benchmarks, and the remaining 2PL fits show poor sampling diagnostics; on Gaokao2023en the 2PL fit has far worse convergence than 1PL. §C.5

• The paper states its framework is primarily post-hoc: difficulties are estimated and gaps reduced only after a benchmark exists, and moving the perspective into benchmark construction is left as future work (§A.2).
• The authors state their goal is not to exhaust all consequences of the measure-based viewpoint, but to show that benchmark evaluation can be studied through the geometry of the induced difficulty measure (§A.2).
• Empirical analysis relies on a scalar 1PL IRT difficulty estimate; multidimensional IRT fits showed poor mixing and did not complete reliably on larger benchmarks, so richer difficulty structure is left open (§C.5).
• The calibration experiment instantiates the target distribution as uniform in the absence of an application-specific target, a choice the paper explicitly says should not be read as assuming the reference distribution is exactly uniform (§D.1).

• A benchmark is treated as an empirical measure over difficulty rather than merely a collection of test questions, with each item contributing equal evaluation mass at its difficulty level. §1
• Measure mismatch means the benchmark difficulty distribution differs from the intended target distribution, so benchmark conclusions can diverge from conclusions under that target. §1
• Coverage gaps arise because the empirical measure is supported on finitely many difficulty points, so improvements concentrated in an empty interval produce little or no score change. §1
• If the performance difference function is L-Lipschitz, the discrepancy between benchmark and target score differences is bounded by L times the 1-Wasserstein distance, and no rank flip occurs when the target difference exceeds that bound. §B.1
• The maximum adjacent gap is the exact asymptotic minimax detection boundary: intervals of width at most the gap are undetectable, while intervals wider than the gap are detected with power tending to one as trials grow. §B.2
• Difficulty estimation collects empirical success rates from multiple LLMs, diagnoses the response matrix, fits a 1PL IRT model, performs post-fit diagnostics, and then computes the two metrics over the normalized difficulty distribution. §3
• The DDM metric is computed as the one-dimensional 1-Wasserstein distance between the benchmark empirical measure and a same-size empirical uniform reference whose points are midpoints of equal-width bins. §C.4
• On Minerva-Math, calibration raises Spearman correlation with the reference ranking substantially and Kendall correlation as well, from a much weaker starting agreement than other benchmarks. §D.2