Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

The paper reports Kling's experience that video generation quality is the projection of multimodal understanding across the data, conditioning, alignment, and evaluation chain.

Video generation has moved from isolated clips to coherent short-form audiovisual sequences, so the bottleneck is shifting from rendering to semantic control. General multimodal benchmarks capture only part of what generation needs, such as camera language, spatial relations, identity binding, and dense temporal change. A generation failure may originate in a caption, conditioning representation, reward model, or evaluator, so a single aggregate score is not a diagnosis.

The paper specializes Kling-MLLM, a shared multimodal foundation model, into captioners, prompt enhancement models, text encoders, reward models, and automatic evaluators. Captioning is treated as a semantic contract built through a multi-expert pipeline, distilled into a single-pass student, and validated downstream. Reference-grounded captioning adds scoped binding, residual sufficiency, and authority consistency, and uses arbitrary similar video pairs for relational supervision. Prompt enhancement combines backward construction from compelling captions, text-level reinforcement learning, DiT-in-the-loop reinforcement learning, and agentic story refinement and retrieval. The text encoder is screened at sequence level, adapted for perception, and given multi-granular stable conditioning; reward modeling pairs a general reward model with a decision reward model.

• On captioning benchmarks, Kling-Captioner-397B-A17B achieved VidCapBench accuracy 21.70, precision 60.51, coverage 96.05, and UGC-VideoCap average 95.50, leading the compared baselines. §3.5
• Kling-RGC-397B-A17B leads ViDiC difference and similarity captioning with 78.89 and 76.29, and achieves MRVBench keypoint coverage 86.12, VQA 95.18, and Ref-Bind 94.85. §4.6
• In cumulative prompt enhancement experiments, adding DiT-in-the-loop RL reached StoryEval 76.46, T2AV-Compass 75.91, AVE-Compass 59.3, and MultiRef-Compass 4.7163. §5.5
• Relative to the Qwen baseline, the Kling-MLLM text encoder with multimodal adaptation, multi-granular conditioning, and conditioning stability scored 1.27 overall, 1.14 instruction following, and 1.25 aesthetics. §6.5
• The DRM achieved 77.86 macro and 75.53 micro accuracy on the internal pairwise benchmark, and scored 52.00 on VideoScore-Bench-v2, 44.23 on VideoPhy2-test, and 57.33 on VideoGen-RewardBench. §7.3

• DiT-in-the-loop RL raises the floor but lowers the ceiling: PE learns to avoid complex choreography, large camera moves, and dense interactions, and rewarding ambition without reintroducing artifacts remains open.
• Captioning leaves open dynamic-detail capture under limited context, long-form density and referential consistency, and stronger audiovisual alignment including speaker-utterance association.
• Agentic PE extensions such as tool calls, low-resolution previews, 3D previsualization, and post-generation verification are ongoing work, with latency and cost as the main bottlenecks.
• Real requests typically give references and brief instructions without complete binding structure or reference scope, and translating them into the scoped, residual-sufficient, authority-consistent caption distribution remains a challenge.

• The central thesis is that video generation quality is the projection of multimodal understanding through the entire data-conditioning-alignment-evaluation chain, so a failure symptom is not itself a diagnosis. §1
• Kling-MLLM comprises a dense 27B variant and MoE variants 35B-A3B and 397B-A17B, and unifies images, videos, audio, synchronized audio-video, and interleaved media in one autoregressive interface. §2.2
• The caption semantic contract lists generation-relevant information families and requires conventions such as consistent entity naming, shot timestamps, faithful visible-text transcription, and temporally integrated audio. §3.1
• Supervision is produced by a multi-expert pipeline with inspection, evidence preservation, routed expert claims, conflict resolution, and conjunctive quality gating, then distilled into a single-pass student for deployment. §3.2
• Reference-grounded captioning relies on scoped binding, residual sufficiency, and authority consistency to prevent property misuse, underspecified targets, and cross-modal conflicts. §4.2
• Prompt enhancement is split into basic PE, which owns the caption dialect, and agentic PE, which performs story refinement and retrieves captions of compelling videos before basic PE writes the final condition. §5.4
• Text encoder screening should probe sequence-level prefill hidden states rather than general QA or retrieval proxies, because the DiT consumes the full sequence representation. §6.1
• Reward modeling pairs an explainable, localized GRM with a fast multi-dimensional DRM, and treats perceptual bandwidth as a first-class design parameter. §7.1