Know, then Draw: A New Learning Paradigm for Visual Generation

A two-stage paradigm where autoregressive pretraining on semantic tokens acquires visual knowledge and a frozen-backbone pixel-diffusion stage renders images, yielding a 3B text-to-image model trained in about 6,600 H100 GPU-hours that scores 0.86 on GenEval and 87.78 on DPG-Bench.

Visual generation has been framed as a competition between autoregressive models and diffusion models, asking which produces better images, although each paradigm must solve two distinct problems: acquiring high-level visual knowledge and rendering it into detailed pixels. The paper argues these two problems need not favor the same objective. A further issue is that conventional tokenizers preserve low-level appearance for reconstruction, which shapes what an autoregressive model is incentivized to learn. Separating knowledge acquisition from rendering also raises an unresolved resource-allocation question: how compute should be split between the two stages and how each stage should balance model size and data.

The framework assigns visual knowledge acquisition to autoregressive pretraining and rendering to pixel diffusion post-training. During pretraining, a frozen visual tokenizer maps images to discrete tokens, and a text-conditioned AR model (RandAR-style permuted prediction with position-instruction tokens) is trained with cross-entropy on those tokens. During post-training, the AR backbone is frozen and trainable pixel-diffusion modules (input projection, timestep embeddings, diffusion feed-forward and adaptive-normalization layers, linear pixel readout) reuse its frozen attention projections; the renderer is trained with pixel-space flow matching in velocity space while predicting the clean image. Post-training conditions on tokens extracted from the ground-truth image, whereas inference conditions on tokens generated autoregressively from the text prompt. Three tokenizers are compared: Recon (pretrained Emu3.5 reconstruction tokenizer), Semantic-Recon (IBQ codebook on frozen Qwen2.5-VL features trained with reconstruction), and Semantic-LM (same semantic representation but codebook trained by next-token prediction inside a frozen Qwen2.5-VL, with no decoder or reconstruction loss). Scaling laws are fitted separately per objective with L̂(N,D)=E+A(N/N0)^-α+B(D/D0)^-β using C=6ND, and the two stages are allocated compute via r=C_post/(C_pre+C_post).

• Autoregressive pretraining followed by pixel-diffusion post-training outperforms the evaluated LDM-pretraining pipeline and diffusion baselines at a matched data budget. §3.1
• Final generator scores for the four matched-budget routes: AR→PDM reaches 65.51 GenEval overall and 71.11 DPG-Bench, versus 61.16/62.56 for PDM, 57.34/64.33 for LDM, and 56.61/60.01 for LDM→PDM. §3.1
• Better reconstruction quality does not improve final generation: Recon has better rFID than Semantic-Recon but lower post-training scores, and Semantic-LM needs no reconstruction yet is competitive after post-training. §3.2
• AR-only generation quality does not predict final quality: Recon beats Semantic-Recon in AR-only GenEval but the ranking reverses after diffusion post-training, and Semantic-LM produces no AR-only decode yet improves by a large margin after post-training. §3.2
• Separate scaling laws were fitted for the two stages, with different model/data exponents, and autoregressive pretraining lowers the fitted asymptotic rendering loss relative to pixel diffusion trained from scratch. §3.3
• The derived compute allocation places about 60% of the budget on pretraining and 40% on rendering, and with roughly 6,600 H100 GPU-hours the resulting ~3B model reaches competitive scores. §3.4

• The paper states that cross-system comparisons in Tab. 3 remain approximate because text encoders, training data, and evaluation settings differ, and that the results do not establish that the compute allocation is optimal (§3.4).
• The comparison of fitted asymptotic losses evaluates the pipeline as a whole, including both pretrained weights and conditioning tokens extracted from the target image, and does not isolate their individual contributions (§3.3).
• The post-training scaling law describes rendering performance within the paired model family, since each renderer is paired with a correspondingly scaled frozen AR backbone, rather than diffusion capacity alone (§3.3).
• The paper notes an unresolved question from its current experiments: without explicit textual rollouts, results do not determine whether Semantic-LM or Semantic-Recon is the preferable interface when a generator already has rich language or agentic context (§A.10).

• The framework specializes autoregression for knowledge acquisition and diffusion for rendering, and the AR rollout at inference provides a structured visual representation that conditions diffusion. §1
• Post-training freezes the AR backbone and trains only pixel-diffusion modules that reuse the backbone's attention projections and representations. §2.2
• The renderer is trained with pixel-space flow matching, minimizing velocity-space MSE while the network directly predicts the clean image. §2.2
• Semantic-LM trains its IBQ codebook through next-token prediction within a frozen Qwen2.5-VL model and requires neither an image decoder nor a reconstruction loss. §2.1
• Autoregressive pretraining law was fit on 18 models spanning 55.71M–2.43B parameters, with N_opt and D_opt both growing at roughly C^0.47–0.53. §3.3
• Post-training law was fit on 14 models spanning 25.1M–2.03B parameters, with exponents varying across estimation methods: N_opt ∝ C^0.45–0.70 and D_opt ∝ C^0.30–0.55. §3.3
• Because autoregressive cross-entropy and diffusion velocity-MSE measure different tasks, the paper does not combine them into a single loss; it instead uses the fitted laws to set each stage's compute-optimal budget. §3.3
• Semantic tokens can be viewed as a compressed semantic state between inference and rendering, and semantic quality rather than reconstruction or AR-only quality governs downstream performance. §A.10