VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

VidUEU-Agent is an automated agent pipeline that converts raw videos into 100K multimodal instruction samples and improves understanding, editing, and unified benchmark scores after fine-tuning.

Multimodal learning is constrained by the scarcity and high annotation cost of high-quality instruction data. Raw videos are abundant and dynamic but redundant, noisy, and unstructured, and do not directly provide task-oriented instruction data. For unified multimodal models, there remains a lack of a scalable and reproducible paradigm for producing high-quality instruction data while minimizing manual annotation cost.

VidUEU-Agent follows a four-stage pipeline: global perception, keyframe mining, metadata construction, and instruction data construction. Global perception detects shots, builds shot/chapter/video summaries, and applies Gate A to remove low-quality shots. Keyframe mining samples frames, applies Gate B filtering, plans tools, scores frames, and uses weighted facility-location greedy selection. Metadata construction filters keyframe-bound signals and binds keyframes, captions, tool signals, transitions, and clip summaries. Instruction construction applies Gate C, routes metadata to understanding, editing, or unified tasks, and uses templates that require citing signal IDs. The agent autonomously iterates the data distribution using downstream training results, and the final 100K dataset is scaled from Iteration 4's target distribution.

• Fine-tuning InternVL3-8B on the understanding subset improves MME to 2369.3 and MMDU to 61.1, achieving the best results among all compared models on both benchmarks. §4.2
• For Bagel, fine-tuning improves all reported understanding scores and achieves the best MMT-Bench score, from 0.645 to 0.651. §4.2
• Fine-tuning Bagel on the editing subset improves all GEdit-Bench metrics, reaching 7.565 on G_SC, 7.328 on G_PQ, and 6.951 on G_O, the best results among all compared models. §4.3
• Fine-tuning Bagel on the unified subset improves WEAVEbench Overall from 0.446 to 0.501, reaching 0.480 on Logic and 0.419 on Game. §4.4
• Autonomous iteration shows an upward trend, with Final reaching MMDU 33.0, GEdit 7.065, and WEAVE 0.526 versus Iteration 1 values of 29.7, 6.541, and 0.448. §4.6
• The no-agent variant drops the most in ablation, to MMDU 22.0, GEdit 5.041, and WEAVE 0.393, versus Ours at 33.2, 6.951, and 0.501. §4.7

• The paper states that VidUEU-Agent does not explicitly construct pure text-to-image generation data because isolated text-image alignment is better suited to directly curated image data.
• The paper states that the dataset is scaled to 100K instruction samples, which remains modest compared with web-scale multimodal corpora.
• For unified tasks, the paper reports that long interleaved image-text chains remain challenging because early errors may propagate to later turns.
• Ablations train on 10K samples per capability while the full reference uses separately trained Bagel models on the Final 100K subsets; the paper leaves strictly budget-matched evaluation for future investigation.

• VidUEU-Agent is an automated data synthesis framework that leverages agent planning and tool use to convert raw videos into multimodal instruction data for understanding, editing, and unified tasks. §3.1
• Stage 1 uses PySceneDetect for shot detection, generates shot-level, chapter-level, and video-level summaries, and applies Gate A to remove low-quality shots. §3.1.1
• Stage 2 applies Gate B frame filtering, hierarchical tool planning, frame scoring, and weighted facility-location greedy selection to choose keyframes. §3.1.2
• Stage 3 performs specialized checks on keyframe-bound multimodal signals and constructs metadata with captions, transitions, and clip summaries. §3.1.3
• Stage 4 applies Gate C quality filtering and task routing, then matches instruction templates and requires citation of signal IDs for traceability. §3.1.4
• Autonomous iteration trains on 30K samples per iteration, uses benchmark feedback to update target distributions, and scales the final 100K from Iteration 4's distribution. §3.1.5
• VidUEU-100K contains 40K understanding samples, 30K editing samples, 30K unified samples, and 20k fine-grained metadata, built from 8,268 raw source videos. §3.2
• The tool pool includes PaddleOCR, Whisper ASR, Ultralytics YOLOv8, RAFT optical flow, and OmniParser. §B.1