IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

IntReal introduces a real-world interleaved dataset and a coarse-to-fine expert-model evaluation suite for multimodal interleaved generation.

Multimodal interleaved generation is underexplored because there are no well-defined generation tasks and corresponding training data. Existing interleaved datasets are mainly collected for multimodal understanding and lack generation-oriented quality control. Existing benchmarks often use VLMs for holistic assessment and do not provide fine-grained evaluation across distinct capability dimensions.

The authors identify four real-world interleaved generation scenarios and organize them into a taxonomy. They apply understanding-oriented and generation-oriented filtering to a million-scale raw pool to construct IntReal-100K. From this dataset they build a 600-sample evaluation set through automated filtering, manual curation, and stratified sampling. The suite evaluates models in two settings: holistic generation of complete interleaved responses and next-image generation conditioned on interleaved contexts. Seven metrics use LLM, VLM, IQA, and DINO-based expert evaluators to diagnose different capabilities. Post-training experiments use SFT and Flow-GRPO on IntReal-100K.

• The evaluation suite reports low metric redundancy, high agreement with human judgments, and consistency across expert evaluators. §Abstract
• For U1-8B-MoT, SFT improves Text Richness, Text Logicality, and Subject Consistency but degrades Image Aesthetic and Context Fidelity. §4.3
• U1-8B-MoT with SFT and RL achieves the highest mean score among the open-source UMMs evaluated. §4
• NanoBanana2 obtains the highest reported mean score and a relative difference versus the open-source best. §4
• IntReal-100K contains 100K samples, 538K images, and 146.6M text tokens. §A.3

• The paper states that constructing high-quality real-world interleaved data requires aggressive filtering and yields a relatively low retention rate.
• The paper states that it does not explore mixed post-training with unimodal generation data, to maintain a clean experimental setting, and leaves synthetic data augmentation and mixed-data post-training for future work.
• Due to computational constraints, fine-tuned results are reported only for the 8B-MoT variant of SenseNova-U1.
• Evaluation coverage is not complete for all models: Emu3.5 is evaluated on 592 samples and NanoBanana2 on 598 samples because of context or memory limits and API guardrails.

• The taxonomy contains four real-world categories: Planning, Knowledge, Tutorial, and Story. §2.1
• Data curation uses a multi-stage pipeline with low-cost visual quality filters followed by VLM-based text polishing while preserving factual content. §2.2
• The evaluation set contains 600 samples, with 150 samples per category after stratified sampling. §3.1
• The benchmark uses two complementary settings, holistic evaluation and fine-grained next-image evaluation, and seven capability metrics. §3.2
• RL uses 10K high-quality samples annotated with rubrics and applies Flow-GRPO with IQA, DINO, and rubric-based rewards. §4.3
• Training and evaluation are strictly disjoint: post-training uses the 99.4K samples remaining after excluding the 600 evaluation samples. §4.1
• The reliability analyses examine low metric redundancy, agreement with human annotation, and consistency across different expert evaluators. §Abstract