Xiaomi Auto WorldModel: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving

World models have emerged as a foundational paradigm for autonomous driving, enabling data synthesis, closed-loop training, and closed-loop simulation. Recent work has converged toward a hybrid reconstruction-generation paradigm that first constructs a 3D scene representation from multi-view observations, then leverages the geometric prior to condition video generation. Despite promising results, existing methods face three persistent bottlenecks: per-scene 3D Gaussian optimization requires multi-hour training and produces ghosting artifacts at scale; causal generation models lack strong scene priors and require hundreds of denoising steps; and reconstruction and generation remain loosely coupled, making it difficult to reconcile geometric fidelity with generative diversity.

This report presents a unified framework that addresses all three challenges. WorldRec replaces costly per-scene optimization with sparse-query aggregation, completing reconstruction in ~10 seconds while eliminating ghosting artifacts. WorldGen combines bidirectional pretraining with progressive causal fine-tuning (Teacher Forcing → ODE distillation → DMD) to achieve stable generation of up to one-minute videos at 0.19s/frame. The Joint World Model tightly couples both modules through incremental scene fusion and ego-projected rendered-prior conditioning, yielding synergistic gains in temporal stability, cross-view consistency, and visual fidelity.

Most existing feedforward methods predict pixel-aligned 3D Gaussians via a DPT head, causing ghosting artifacts and Gaussian primitive explosion. WorldRec instead represents scenes as compact sparse tokens: each token aggregates features from multi-view, multi-temporal observations through a visibility-aware attention mechanism, explicitly enforcing multi-view consistency and yielding compact high-fidelity 3D Gaussian representations. Scene reconstruction from a 10-second clip is accomplished in ~10 seconds — versus ~4 hours for per-scene optimization baselines.

WorldGen adopts a Diffusion Transformer (DiT) backbone and a two-stage training framework: bidirectional pretraining → causal fine-tuning. Stage 1 trains with full bidirectional temporal attention to learn global spatiotemporal distributions. Stage 2 introduces a causal mask and progressively refines the model through Teacher Forcing (causal constraint), ODE distillation (50 steps → 4 steps, 12× speedup), and DMD (closes train-inference distribution gap) — achieving high-quality real-time causal video generation.

The Joint World Model deeply integrates WorldRec and WorldGen so that each module's strengths compensate for the other's limitations. WorldRec provides reliable 4D spatial anchors that prevent geometric drift during generation; WorldGen fills unseen regions with coherent content that reconstruction alone cannot synthesize.

To enable tight coupling, both modules are adapted: WorldRec gains incremental reconstruction via scene fusion — new observations update and extend the existing Gaussian representation, enabling larger scenes as the vehicle moves. WorldGen gains rendered-RGB conditioning — scene tokens are rasterized to target views, producing partial reference images that are injected as additional conditioning, guiding synthesis in unobserved regions.

Table 2 presents quantitative comparisons on Waymo and nuScenes. Our method achieves state-of-the-art performance across all settings. Reconstruction from a 10-second clip completes in ~10 seconds vs. ~4 hours for per-scene optimization.

Table 3 compares WorldGen against leading driving world models on nuScenes. As an autoregressive (AR) model, WorldGen achieves FID 7.04 and FVD 64.97, outperforming all listed models in FVD. It generates 81-frame sequences (far exceeding 8–16 frames of most baselines) at 0.19s/frame — 5.6× faster than the only other AR method (Epona at 1.06s/frame).

We evaluate the Joint World Model along three dimensions: long-horizon temporal consistency, multi-view spatial consistency, and multi-run stability. The geometric prior from WorldRec anchors the generative process, preventing drift and hallucination while maintaining globally coherent scene content.