FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation

FrameForge adapts a frozen video diffusion model to unified conditional image generation by selecting input-adaptive temporal RoPE experts and inserting a latent reasoning chain between condition and target.

Real-world conditional image generation involves heterogeneous inputs such as text, reference images, and structural maps, which impose constraints at different semantic and spatial levels. Existing unified models are predominantly image-centric and inject conditions as auxiliary spatial or semantic signals without explicitly modeling how strongly the condition and target differ or how one should be transformed into the other. Video-model adaptations arrange condition and target as frames but assign fixed temporal positions, yielding an invariant temporal positional relation across tasks and samples.

FrameForge formulates conditional image generation as target-frame prediction guided by textual and visual conditions. It is instantiated with the first-stage video diffusion model of HunyuanVideo1.5, using a 3D VAE and an MM-DiT backbone, with the pretrained video-backbone weights and VAE encoder frozen. Mixture of Temporal Experts (MoTE) constructs multiple temporal RoPE experts corresponding to different virtual intervals and uses a lightweight router to select one expert per sample from text, condition, and noisy target representations. Chain of Temporal Reasoning (CoTR) inserts a noise-initialized reasoning latent between condition and target, contextualized through attention without explicit intermediate supervision. A stochastic reasoning-bottleneck attention mask blocks direct target-to-text/condition attention during an intermediate training stage, while the final stage and inference use full attention. Training inserts a single LoRA adapter into attention and feed-forward projections and optimizes the MoTE router and LoRA jointly across tasks in three stages.

• On HunyuanVideo1.0, FrameForge raises the subject-driven score from 0.699 to 0.736 over the unified baseline. §4.4
• On HunyuanVideo1.0, FrameForge raises the editing score from 4.38 to 5.00 over the unified baseline. §4.4
• On HunyuanVideo1.0, FrameForge reduces canny FID from 24.52 to 23.07 over the unified baseline. §4.4
• MoTE routing analysis reports expert selection ratios over 50 denoising steps, averaged over 500 cases. §4.2.4
• Canny-to-image generation predominantly selects the long-interval expert, especially during early denoising, while subject-driven generation and editing initially favor the medium-interval expert and later shift toward the short-interval expert. §4.2.4

• The paper reports no explicit limitations section; its evaluation covers only subject-driven generation, canny-to-image generation, and instruction-guided image editing.
• Generalization across video backbones is evaluated only on HunyuanVideo1.0 against a unified baseline, while the main backbone is HunyuanVideo1.5.
• All reported training images are resized to 512x512 and inference uses 50 denoising steps; behavior at other resolutions and step counts is not reported.

• The paper interprets heterogeneous condition-target relations through a virtual temporal interval, motivated by LPIPS variation across natural video frames and task condition-target LPIPS distributions. §1
• MoTE constructs E temporal RoPE experts over candidate latent spans; with r_t=4, default video-frame spans {17,69,121} correspond to latent spans {5,18,31}. §3.3
• MoTE routing is sample-wise Top-1 over text, condition, and noisy target representations, trained with a straight-through estimator and a batch-level load-balancing loss. §3.3
• CoTR inserts a single Gaussian-initialized reasoning latent between condition and target; it is not explicitly supervised or decoded. §3.4
• During the intermediate training stage, a stochastic reasoning-bottleneck mask with p_mask=0.5 blocks direct target-text/condition attention; the final stage and inference use full attention. §4.1
• Training freezes the pretrained video backbone and VAE encoder, inserts one LoRA adapter into attention and feed-forward projections, and uses three-stage optimization for the router and LoRA. §3.5
• Training uses 170K samples per task, 510K total, 512x512 images, 182M trainable parameters, 12,000 iterations on two NVIDIA A800 GPUs, batch size 60, and learning rate 3×10^-4, requiring approximately 6 days. §4.1
• Evaluation covers subject-driven generation on DreamBench, canny-to-image generation on COCO2017, and image editing on GEdit-Bench. §4.1