PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

PreView is a trained auxiliary preprocessor that supplies center-guided cropped and enlarged views plus candidate-specific visual and textual cues to a frozen GUI grounding model, improving accuracy without retraining that model.

GUI grounding must locate small controls in dense screenshots and decide which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image can still leave the intended target ambiguous. Existing approaches use GUI-specific training, zoom-in inference, or semantic augmentation, but each can consume inference resources, discard context, or supply misleading semantics. PreView asks whether a learned auxiliary model can prepare both task-relevant views and informative cues for an existing grounder while limiting these errors.

Given a screenshot and instruction, PreView uses a trained auxiliary model to prepare inputs for a frozen grounding model. It predicts and refines a center, crops the screenshot around that center into a local view, and enlarges the view for zoomed-in grounding. It also prepares a visual cue consisting of a red-circle marker with a location hint and, when the auxiliary model accepts the candidate, a textual cue describing the candidate’s identity, function, or local relationships. The frozen grounding model receives the prepared view, the original instruction, and available cues in a single call, and its predicted click is mapped back to the original screenshot. The auxiliary model is jointly trained for center prediction and candidate verification with description generation.

• Average gains are 7.4, 13.9, and 1.5 percentage points for Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B, respectively. §5.1
• Qwen3-VL-2B with PreView scores 59.3% on SSPro and 33.7% on UI-V, exceeding Qwen3-VL-4B Full on UI-V at 25.3%. §5.1
• Qwen3-VL-4B gains 15.3 points on UI-V but loses 5.0 on MMB-GUI; on SSPro its icon accuracy rises from 32.5% to 42.5%, versus 75.8% to 76.8% for text. §5.1
• On SSPro, Refine 1 retains the target center in 80.14% of views but directly hits the target in 53.83% of cases; one refinement raises mean direct hit from 60.83% to 64.02%, while c3 yields 63.93%. §5.2
• At the same 25% pixel budget, enlarging a 12.5%-area crop beats a native 25%-area crop on all three benchmarks with identical processed visual-token counts; doubling both dimensions adds 6.5–10.8 points. §5.4
• On 8,288 target-present 2B examples, cues inside the annotated target correct 71.0% of View errors and disrupt 1.9% of correct predictions; outside the target these rates become 3.2% and 57.8%. §5.5

• Cropping may discard targets or necessary context.
• Incorrect cues may mislead the grounding model.
• Training on desktop applications limits generalization to mobile interfaces, and offline evaluation does not establish reliability in multi-step interaction.
• Auxiliary inference adds computational cost and latency; the paper reports a single run per configuration, so results may be influenced by initialization noise.

• PreView improves inputs to an existing grounding model without further training that model. §Abstract
• It prepares task-relevant views for localization and candidate-specific cues for target understanding. §Abstract
• The grounding model predicts the final click without parameter updates. §3
• One center refinement raises mean direct hit from 60.83% to 64.02% at c2; c3 yields 63.93%, so the pipeline stops at c2. §5.2
• At the same 50% visual budget, smaller crops perform better on SSPro and UI-V, while the wider crop improves OSW-G R by 0.71 points. §5.4
• Spatial accuracy remains lowest at 17.0% for 2B and 26.0% for 4B after preprocessing. §5.1
• With View, GroundNext improves from 52.9% to 59.0% on SSPro and 67.7% to 68.3% on OSW-G R despite overlapping GroundCUA training data. §5.6
• PreView requires only one call to the frozen grounding model and reports average gains of 1.5–13.9 points. §6