L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

L3G-World combines an online-updated latent 3D Gaussian memory with a memory-conditioned video DiT to generate spatially consistent worlds in real time.

Existing interactive world models either keep temporal context, whose memory and attention costs grow with stream length and resolution, or store explicit 3D memory. Explicit memories face a trade-off: feed-forward point-cloud or Gaussian methods accumulate dense per-pixel primitives and grow with stream length, while optimization-based methods are compact but slow. Explicit 3D memories also condition latent video generators by re-encoding RGB renderings, which precludes real-time generation.

L3G-World uses a Latent 3D Gaussian Memory of Gaussians and memory tokens that is updated online by L3G-Mem. L3G-Mem renders the previous memory into current cameras, builds a spawn mask from low opacity or high RGB error, and initializes spawning query tokens by K-means on geometry features inside the mask. It retrieves tokens for visible Gaussians as refinement tokens, processes spawn and refinement tokens with a shared transformer, decodes new Gaussians and residual updates, and leaves unselected entries unchanged. Each Gaussian also stores a VAE latent feature; memory is rendered directly into the DiT latent space with an alpha mask. L3G-Gen concatenates these rendered latents and mask with noisy target latents, keeps reference latents fixed, and is distilled into a few-step student. The two modules alternate autoregressively, so each generated chunk updates the memory and the updated memory conditions the next chunk.

• Against NeoVerse on the two datasets, L3G-World uses far fewer Gaussians and less memory while rendering faster. §4.2
• Against Lyra 2.0, L3G-World performs memory reconstruction faster with less memory and higher VAE rendering throughput. §4.2
• On DL3DV, L3G-World obtains best and second-best PSNR and LPIPS for self-consistency; L3G-World gets the best scores. §4.2
• On RealEstate10K, L3G-World achieves best PSNR and LPIPS and second-best SSIM. §4.2
• The few-step variant enables real-time world generation on one H100 GPU. §1
• The full L3G-Mem achieves the strongest reconstruction quality across most RGB and VAE metrics while retaining a compact Gaussian memory and comparable reconstruction throughput. §4.4

• Latent 3D Gaussian memory is primarily designed for static scenes and does not explicitly model independently moving objects.
• Although memory grows with explored spatial extent rather than generation length, its size can still increase under extremely large-scale exploration; adaptive pruning or hierarchical memory representations are suggested.
• The refinement mechanism can progressively correct inaccurate Gaussians, but errors introduced during long-horizon autoregressive generation may still propagate to subsequent memory updates.
• Experiments leave open generalization beyond DL3DV and RealEstate10K and beyond the single-image-to-video setting.

• Memory grows with scene extent rather than stream length; L3G-Mem spawns only in uncovered regions and refines existing Gaussians. §3.1
• The spawn mask marks pixels with low opacity or high RGB error; spawning queries are initialized by K-means on geometry features inside the mask. §3.2
• Refinement retrieves memory tokens for visible Gaussians and applies residual updates in place; unselected entries remain unchanged. §3.2
• Each Gaussian stores a VAE latent feature, so memory renders directly into the DiT latent space without VAE re-encoding. §3.2
• L3G-Gen concatenates rendered memory latents and alpha mask with noisy target latents and keeps reference latents fixed. §3.3
• The teacher is distilled into a few-step autoregressive student with on-policy distillation following AnyFlow. §3.3
• L3G-Mem is trained over K=5 consecutive chunks with stop gradients through previous memory. §B.1