Glance: The Generator Is Its Own Reranker

Glance equips a single resident 9B generator backbone with a lightweight decision adapter and pointer head to rerank candidate passages from one shared prefill and then generate answers with the same backbone.

Retrieval-augmented generation commonly serves separate models for reranking and answering, which duplicates model residency and passage processing. A cross-encoder reranker encodes each query-passage pair separately, and LLM-based rerankers improve quality at higher cost. Either way, an additional model must be served and candidate passages are processed before the generator reads the selected context. The paper asks whether the generator can judge all passages using computation shared across passages.

Glance prefills the candidate passages once, then evaluates short passage-specific relevance probes that branch from the shared cache, and a 256-dimensional pointer head turns each probe into a relevance score. In the main pipeline the decision adapter is active on all tokens; the top-n passages are re-prefilled with the adapter off and answered. In Glance-Activated the adapter is active only on probe tokens; candidates are prefilled with base weights and generation continues the original cache. The frozen backbone is a hybrid Gated DeltaNet/full-attention 9B model, and the only trained parameters are a rank-16 LoRA adapter and the pointer head. The paper compares packed_all, packed_q, and listwise packings, with single pass meaning one shared prefill followed by one batched continuation over all probes. Training uses 4,000 MS MARCO queries in packed_all format, each record with one positive and 7–19 hard negatives, for one epoch.

• After 39 minutes of MS MARCO fine-tuning, Glance packed_all reaches nDCG@10 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. §4.3
• On 300 HotpotQA questions, Glance top-3 with adapter off gives F1 0.718 vs 0.683 for cross-encoder top-3, and adapter-on Glance reaches 0.757; Glance context still wins by +0.055 F1 with paired bootstrap 95% CI [+0.018,+0.093]. §4.5
• In the candidate scaling study, trained-long reaches 0.697 nDCG@10 at k=100 vs 0.637 for the cross-encoder, +0.060, and 0.717 at k=200 with 43.4k tokens vs 0.659, taking 15.8 s per query on the chunked path. §4.4
• Zero-shot single-pass packed_all scores 0.715 nDCG@10 on SciFact vs 0.729 for 20 separate decision calls and 0.721 for the cross-encoder, and on NFCorpus scores 0.341 vs 0.326 for the cross-encoder and 0.323 for separate calls. §4.2
• One 9B backbone serves base chat, general decisions, and rerank modes with bit-identical probabilities max|Δp|=0 when switched per request; median switch times are 4.2–6.4 ms, 9.4 ms, and 8.8 ms, with 16.85 GiB resident and 16.86 GiB after the loop. §4.6
• The activated adapter scores 0.740/0.351 nDCG@10 on SciFact/NFCorpus vs 0.744/0.351 for a matched control, and continuing the cache gives EM/F1 0.61/0.750 vs 0.59/0.754 for re-prefilling, with lower median answer latency 138 ms vs 181 ms. §4.7

• The work is a proof of concept: evaluation sets are small (hundreds of queries), cover two BEIR datasets and one QA dataset, and test one backbone family (hybrid 9B); latencies are from an unoptimised eager path and a dedicated cross-encoder remains faster per query.
• BEIR differences are not significance-tested, and the per-passage baseline was not re-run with the trained adapter.
• The zero-shot adapter degrades beyond k=20 and collapses at k=200; batching across queries currently lowers throughput, and memory grows with k×context length because every probe branch copies the shared state cache.
• Activated-adapter and cache-reuse results are single-seed on 100 questions, the bitwise cache check used a 0.8B model, and with cache reuse the answer sees all passages rather than the reranked subset.

• A single resident 9B backbone supports generation, general decisions, and reranking, with approximately 0.1 GiB of additional memory per decision skill. §3.5
• Main Glance keeps the adapter active on all tokens and re-prefills the selected top-n passages with the adapter off; Glance-Activated keeps the adapter active only on probe tokens and continues the original candidate cache. §3.1
• In packed_all the state lists all k passages and each branch asks whether Passage i contains information that answers or supports the query; branches never attend to each other. §3.3
• On the hybrid backbone, the shared-prefix path runs the shared state once and starts every branch from a copy of each layer cache, including DeltaNet convolution window and recurrent state; the path is exact and used in training. §3.4
• Training uses 4,000 MS MARCO queries with one positive and 7–19 hard negatives per record, one epoch, 500 optimizer steps, 55,598 labelled probes, and 7.7M forward tokens on one NVIDIA B200 in 39 minutes. §3.6
• In packed_all a passage score can depend on order: nDCG@10 ranges from 0.678 in BM25 order to 0.693 across orders, and the top-ranked passage matches BM25 order in 91% of cases. §4.8
• The vLLM prototype runs 4 chat answers and 19 decision rows in one generate call in 0.76 s vs 0.43 s for chats alone; probabilities differ from Kev by 0.006 on average and 0.10 maximum, with 4 of 400 decisions flipped. §4.9
• Specialising one adapter for reranking causes moderate forgetting on JevBench: the general adapter scores 184/231 and the reranking adapter 175/231, a net loss of 9 items, 7 in the hard tier. §4.6