Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Youtu-Parsing-Omni is a 5B omni-modal parser that uses one shared text-initialized encoder and one unified schema to convert documents, images, audio, and video into spatially and temporally grounded structured records, with state-of-the-art document parsing and leading open-weight omni-parsing results.

Current parsers cover documents, images, audio, and video only partially: document parsers are strong on standard pages but symbolic content is left largely to domain-specific models, while omni-modal models are built mainly for captioning and question answering rather than grounded structured output. Different input types are typically served by different models with different output formats, even though the target is a structured record whose units are anchored in space or time. Building one parser raises two challenges: joint perception across modalities with separate encoders that isolate modalities and index time independently, and joint optimization across ten tasks with heterogeneous output fields and criteria.

Youtu-Parsing-Omni is a decoder-only language model fed by one perception encoder, Youtu-Omni-Encoder, which uses modality-specific stems and merger-projector heads but a shared Transformer for images, audio, and video. Visual and acoustic tokens share a unified space-time positional encoding, and a few fusion layers let video frames attend to concurrent audio within short audio-visual windows inside perception. OmniSchema unifies the outputs of ten parsing tasks into elements or segments anchored in space or time, plus a global description. Training proceeds through four pre-training stages, then SFT for schema serialization, schema-routed RLVR for sequence-level optimization, and OSAD for token-adaptive on-policy distillation. OSAD trains on the model's own rollouts against token-level targets from a reference-conditioned frozen copy, which is refreshed with trained weights each round.

• On OmniDocBench v1.6, Youtu-Parsing-Omni achieves the highest Overall score while staying within 0.21 points of TeleOCR on text, formulas, and tables, and improves over Youtu-Parsing by 4.8 TEDS on tables and 3.2 CDM on formulas. §5.2
• On OmniParsingBench, it leads open-weight models with 75.08 Avg. and the best overall chart score 95.02; it surpasses Logics-Parsing-Omni at 72.26 and reaches audio 78.08 and text-rich video 75.62. §5.4
• On PureDocBench, it has the highest Avg3 among end-to-end models at 76.29 and gains approximately 8 points over Youtu-Parsing on each of the three tracks. §5.3
• On ChemOCR, it reaches exact fingerprint-match Tani@1.0 of 54.2, outperforming DeepSeek-OCR 2 at 49.6; on PDMX-Synth it reaches CER 22.73 versus LEGATO at 23.30 as reported. §5.6
• On Video-MME-v2 without thinking, it comes within 0.1 points of 4B thinking models and scores higher than Qwen3-Omni-30B-A3B-Instruct on the group score and all three levels. §5.5

• Physically recaptured document inputs remain a major challenge; most systems show a marked performance decline, indicating that distortions and artifacts from real-world acquisition remain a major challenge for document parsing.
• Adding a modality still requires a new stem and projector and training the shared weights on the new input; the architecture does not eliminate modality-specific input adaptation.
• RLVR and OSAD learn only from rollouts that pass decodability, payload, or admission checks, so a policy that rarely produces well-formed parses supplies few useful learning signals.
• OmniParsingBench evaluation excludes the camera-video subset and 136 samples with unavailable media, leaving 3,738 samples, so results do not cover those excluded samples.

• A single shared encoder reads every modality with shared weights; only stems and merger-projector heads are modality-specific, and together they hold less than 3% of encoder parameters. §B.2.1
• Visual and acoustic tokens share one space-time positional encoding and interact in fusion layers inside perception; only cross-window dependencies are left to the language model. §1
• OmniSchema covers ten parsing tasks across images, audio, and video; every element or segment is localized by an explicit spatial or temporal anchor. §3
• Pre-training uses four stages that progressively expand trainable parameters, input scale, cross-modal interaction, and input perturbation. §4.1
• SFT establishes OmniSchema serialization; schema-routed RLVR optimizes on the policy's own parses with a gated reward combining well-formedness and task-specific content accuracy. §4.2.2
• OSAD uses token-role-specific divergences and refreshes the reference-conditioned teacher with trained weights at each round boundary. §4.2.3
• The encoder block is initialized from Qwen3-0.6B-Base without its embedding layer and LM head; the decoder retains 1D rotary positions because spatial and temporal structure is already resolved in the encoder. §B.3
• Ablation uses general multimodal data and no parsing data to study the text-initialized unified encoder and shared weights. §5.7