SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding

SpecPilot is an autonomous agentic loop that uses a ceiling-based cost model, parallel coding agents, stock-calibrated token forcing and a structured memory to discover lossless algorithms and systems optimizations for speculative decoding deployments, raising speedup over AR from 4.46× to 10.87× with one request and from 1.94× to 3.93× with batching.

Released parallel-drafting systems leave much of the potential speedup unrealized: with one request each step accepts few tokens but reads every target weight, with batching per-request verification roughly halves the speedup over AR decoding, and speculative decoding is deployment-specific so no universal optimization fits. Manually optimizing each deployment does not scale, which makes coding agents a natural fit, but building a reliable evaluator for lossless speculative decoding is hard. Many gains are only a few percent and close to timing noise, gains measured on development requests may not generalize, and an agent rewarded for speed is prone to reward hacking such as a faster numerical shortcut that changes outputs without a measurable drop in task accuracy.

SpecPilot runs in rounds that start from the previous round's final system, or from stock plus memory entries for a new deployment, and each round measures where the current system loses speed (Phase 0), builds and times changes (Phases 1-3), and ends with a sealed final on unseen requests (Phase 4). Phase 0 uses a cost model that writes the speedup over AR as tau(N)/c(N) and computes a ceiling that factors into a tau factor, a kernel factor comparing profiled step time to summed byte floors, and an overhead factor; the largest factor identifies the gap to attack. A master agent plans directions, one idea per GPU, sub-agents implement each direction as patches behind options whose defaults reproduce stock, and a controller isolated from the agents times patches against the current best in paired runs on two GPUs with GPU swapping. A patch is promoted only if its gain's 95% lower bound exceeds 1% plus twice the stock-versus-stock noise, or about 3% if its outputs diverge from the current best's. Token forcing feeds the system's outputs back through the target on the stock engine with an FP32 LM head and rejects a system under a rate rule or an event rule on unexplained large-margin mismatches. Validated mechanisms become structured memory entries with a code delta, requirements, correctness contract and evidence, kept only if applying them with their requirements to stock reproduces the source system's code exactly.

• Without training or quality loss, SpecPilot raises the speedup over AR from stock's 4.46× to 10.87× with one request and from 1.94× to 3.93× with batching. §Abstract
• With batching on FP8 Qwen3.8-27B, SpecPilot beats stock at every batch size, by 1.85× with DFlash2 and 1.94× with DSpark overall, raising speedup over AR from about 2× to 3.90× and 3.93×. §5.2
• In exact replay on the 8-position lattice, best-first trees raise tokens per step from 4.25 for the stock chain to 5.38 at 32 nodes, and drafting 16 positions lifts the curve to 5.85 at 32 nodes and 6.25 at 63. §6.4
• W12 averages only 12.01-12.02 bits per weight on eight dense models and shrinks Qwen3.8-27B's converted layers from 44.67 to 33.58 GB, with every decoded tensor checked bit for bit at load time. §4.1
• A TLV cycle of the final H100 system takes about 30 ms, with two drafts of 1.05 ms each, an 11.5 ms Q pass and a 14.9 ms exact pass, and each exact pass accepts 11.67 tokens instead of 6.16 in a one-level step. §C
• Token forcing accepts both lossless systems at 0.73% and 0.52% mismatch, matching stock's 0.72% and 0.53%, and rejects both faulty systems at 2.0% and 2.4%, with 192 and 191 of their 200 audited large-margin events left unexplained. §6.2

• The paper states that mechanism value depends on the regime: with batching, 24-node trees lost 14.9% on the FP8 target, and on BF16 Qwen3.8-27B a 24-position block lost 9.9% and TLV 13%, so discovered mechanisms must be re-tuned per deployment rather than applied universally (§6.5).
• The ceiling assumes one exact pass per draft; on the RTX PRO 6000 the final 2.44× over stock exceeds its 2.12× ceiling computed under that assumption, so the cost model's headroom estimate can be wrong in either direction (§5.3).
• Gains are often only a few percent and close to timing noise, and a gain measured on development requests may not generalize to unseen requests; the paper reports speedups over AR from interleaved development runs within -2% to +6% of the sealed gains (§1, §5.1).
• The loop is expensive: a round takes 7-20 hours and 20-86 GPU-hours, and the thirteen completed rounds took 164 hours and 530 GPU-hours in total, creating 486 code changes and promoting 54 of them (§D).
• The certified LM head is faster than BF16 on H100 but slower on the RTX PRO 6000, 2598 vs. 1750 microseconds, so some discovered components are platform-specific (§4.1).

• The cost model expresses speedup over AR as S(N) = tau(N)/c(N) with c(N) the step time over the AR step time, so it separates accepted tokens per step from step cost. §3.1
• The ceiling is the speedup a round could reach if it drafted once, verified the best tree in one exact pass, and ran every kernel class at its byte floor, the larger of FLOP time at 60% of peak dense throughput and byte time at 90% of peak bandwidth. §3.1
• The controller promotes a patch only if its gain's 95% lower bound exceeds 1% plus twice the stock-versus-stock noise, or about 3% if its outputs diverge from the current best's. §3.2
• Token forcing rejects a system if its mismatch rate exceeds stock's by more than 0.5 points at 95% confidence, or if a binomial test finds its unexplained large-margin events more frequent than stock's, where an event is a mismatch with margin above twice the near-tie threshold delta. §3.3
• W12 stores each weight as a sign bit, a 4-bit exponent offset within a 16-binade window per output channel, and seven mantissa bits, with an exact escape list for out-of-window weights, zeros and denormals; its stream-K GEMM decodes records in registers and splits the reduction in a fixed order so the GEMM is deterministic. §4.1
• For hybrid targets the system recomputes the recurrent state along each root path instead of storing it per node, and a WY verify kernel turns the chain of rank-one updates into matrix products solved on tensor cores, staying within one BF16 ulp of the path kernel and changing only rounding. §4.2
• The three batched-serving mechanisms only pick which rows to draft and verify, hence are lossless: a shared node budget across requests, per-request drafter blocks of 16, 12 or 8 positions, and prompt admission. §4.4
• A single-agent baseline lacking the cost model reached only 1.34× over stock after three rounds against 2.13× for SpecPilot, because it never addressed the target weight reads that took 80% of the step. §6.1