Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

The paper introduces ReGap-Bench, a paired benchmark that holds scene, question, and answer fixed while varying visual references, and finds that task-matched references improve multimodal reasoning by up to 25% while mismatched references give little benefit or hurt.

Multimodal large language models can recognize visual content yet fail across multi-step reasoning, and such failures are often attributed to perception. The paper argues the missing capability is a reference gap: language reasoning is not stably bound to the relevant visual entities, regions, or locations. Prior work either trains models to emit coordinates or changes image and prompt together, so it does not isolate the reference itself or test which reference a task needs. ReGap-Bench is introduced to hold scene, question, and answer fixed while varying only the reference.

ReGap-Bench pairs a plain image with one or more reference versions generated from the same scene across five tasks: shape counting, angle ranking, size ranking, clock ranking, and transformation prediction. Each task has 200 scenes. References are numbering, shading, and grid lines; each condition satisfies content preservation, answer invariance, and no answer leakage. The effect of a reference is measured as delta(c)=Acc(c)-Acc(c0). The paper evaluates five MLLMs and six Qwen3-VL Thinking models from 2B to 235B, using exact match and pairwise accuracy for rankings. It also packages the references as a training-free visual skill and tests it with a Codex GPT-5.5 agent on 100 CartoMapQA tasks under BenchFlow.

• Task-matched references improve accuracy on all five tasks; numbering yields gains of up to 17.5% on shape counting and 25% on clock ranking. §1
• In shape counting, numbering improves accuracy by 11.3% on average, compared to only 1.7% for shading, while grid lines reduce accuracy by 4.1%; the paper uses this to argue gains are not from added marks alone. §4.2
• Clock ranking mean accuracy rises as dial numbering becomes more complete: 23.9% with no numerals, 25.3% with two, 31.8% with four, and 38.0% with all twelve. §4.3
• Across six Qwen3-VL Thinking models from 2B to 235B parameters on shape counting, the numbering advantage ranges from 9.0% to 23.5% and does not decrease monotonically. §4.4
• On 100 CartoMapQA tasks, the agent draws an overlay on 57; numbering tasks rise from 63.7 to 70.1, grid-line tasks from 92.5 to 96.7, and shading tasks from 33.3 to 66.7. §5

• The benchmark covers five tasks and three reference scaffolds; other visual routines, tasks, and reference types are not tested (open from the scope of §3 and §3.1).
• All ReGap-Bench scenes are generated in Python with Pillow from a fixed seed; performance on natural or non-synthetic images is not tested (open from Appendix B).
• The model-scale analysis evaluates six Qwen3-VL Thinking models only on shape counting, so scale effects on the other four tasks are not tested (open from §4.4).
• The visual-skill evaluation uses 100 CartoMapQA tasks with Codex GPT-5.5 under BenchFlow; transfer to other agents or datasets is not tested (open from §5).

• The reference gap is defined as failure to maintain a stable link between language and the relevant visual entity, region, or location; it is distinct from a perception failure. §2.1
• Numbering is proposed to support marking and indexing, shading to support bounded activation, and grid lines to support systematic focus shifts and boundary tracing. §2.2
• The paired design holds scene, question, and answer fixed and varies only the reference, with content preservation, answer invariance, and no answer leakage. §2.3
• The five tasks and their matched scaffolds include numbering for shape counting and clock ranking, shading for angle and size ranking, and grids for size ranking and transformation prediction. §3.1
• The benchmark contains 1,000 scenes and 3,800 images, with 200 scenes per task. §B.1
• The paper evaluates five MLLMs plus six Qwen3-VL models from 2B to 235B; exact match is the main metric and pairwise accuracy is added for rankings. §3.2
• A training-free visual skill lets an agent select and draw a reference on a copy of the image and is tested on CartoMapQA. §5