GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

GameVerse is a 69.7K-clip, 1,655-hour gameplay video dataset built for long-horizon interactive world modeling, pairing minute-scale clips with validated frame-level camera trajectories and four-level event-aligned captions, and fine-tuning on it improves VBench and WorldScore metrics over gameplay-domain baselines.

Interactive world models lose coherence after roughly one minute of rollout, which the authors argue is a data problem rather than a model-capacity problem. Existing gameplay corpora consist almost entirely of clips shorter than ten seconds, with whole-clip captions that conflate static scene context with dynamic events. Prior datasets optimize along single axes: large-scale exploration video without dense interaction or validated poses, engine-level poses capped at short windows, or action-conditioned interaction without reliable trajectories or event-aligned captions. The four signals — minute-scale continuous footage, validated frame-level camera trajectories, dense first- and third-person interaction, and multi-granular event-aligned captions — are never shipped jointly.

Sources are 11,002 YouTube gameplay videos at up to 1080p and 30–60 FPS spanning open-world AAA and indie titles in first- and third-person. Scene-consistent segmentation combines TransNetV2 hard-cut detection with a CLIP-based cosine semantic distance to catch gradual transitions, extracting segments of up to 120 seconds. A multi-stage quality pipeline performs basic filtering, OCR-based detection of subtitles/HUD/overlays, stability and UI detection, deduplication and diversity selection, and temporal refinement with COVER and VBench consistency scores, with a weighted-sum score plus stage-specific constraints. ViPE estimates frame-wise intrinsics and extrinsics, and every trajectory is scored by a function combining local relative-pose increments (jumps, jitter) and a global smoothness term (drift) with threshold-based filtering. Each clip is captioned by Kimi-K2.5 into four levels — global static, local static, dynamic events, and agent actions from a constrained vocabulary — aligned to event boundaries rather than fixed windows.

• GameVerse delivers minute-scale scale: total clips, hours, and two-minute continuous footage. §Abstract
• Over 40.6K clips reach the 120-second upper bound, contributing the bulk of continuous two-minute footage. §4.1
• Diversity spans at least 110 titles with a long tail, and the perspective mix is roughly one first-person clip per three third-person clips. §4.2
• Camera motion is dominated by forward translation, with complex translation-plus-rotation motions making up a large minority. §4.3
• Caption levels differ sharply in length, with local static and dynamic event captions being longest. §4.4
• Same-backbone GameVerse fine-tuning (HY-World 1.5 to HY-World-FT) improves VBench subject, background, motion-smoothness, aesthetic, imaging, and overall consistency scores. §5.2
• On WorldScore, HY-World-FT raises WS-Static and 3D consistency over HY-World 1.5, while LTX-FT reports the highest WS-Static, 3D consistency, photometric consistency, and style scores. §5.2
• Audited retained trajectories have full pose validity, low jump rates and rotational jerk, and cross-estimator agreement within a few degrees. §C.3

• The paper states that cross-estimator comparison against DROID-SLAM after Sim(3) alignment measures agreement only and 'does not establish absolute pose accuracy without ground-truth trajectories'.
• The reported 100% valid-pose ratios 'refer to the retained audit set rather than all raw source videos', so filtering loss rates are not characterized.
• The backbone diagnostic in Appendix C.4 does not isolate the causes of cross-model differences in 60-second generation, and appearance-related gaps between LingBot and LTX-FT may partly originate from the underlying backbones.
• Evaluation uses 60-second generation on 103 fixed inputs, while the dataset is built around two-minute sequences; coherence beyond one minute is not directly benchmarked (stated aim of Section 1, not measured in the reported tables).

• The authors argue the one-minute coherence ceiling of interactive world models is a data problem, not a model-capacity problem, because training corpora are dominated by clips under ten seconds with whole-clip captions. §1
• Segmentation combines TransNetV2 hard-cut detection with CLIP cosine semantic distance to also capture gradual transitions such as camera panning or UI changes. §3.2
• Quality filtering includes OCR-based removal of subtitles and HUD overlays plus COVER and VBench consistency temporal metrics, with stage-specific constraints preventing high scores from masking severe defects. §3.3
• A subset of previously filtered clips with a stationary camera but evolving scene content is recovered to preserve object and agent dynamics under fixed viewpoints. §3.3
• Trajectory validation targets three named error modes: sudden jumps from tracking failures or relocalisation, high-frequency jitter, and long-term drift. §3.4
• Captions use a four-level schema (global static, local static, dynamic events, agent actions) where agent actions come from a constrained vocabulary so the controllable channel can be modulated independently. §3.5
• The four caption levels concentrate on largely disjoint vocabularies, indicating semantically separated annotations rather than overlapping rephrasings. §4.4
• Controlled comparisons use HY-World 1.5 fine-tuned on approximately 600 hours of GameVerse for 3,000 steps with unchanged architecture, evaluated on the same 103 inputs. §5.1