← Roadmap/Learning

World Models & Sim-to-Real

Learned dynamics, simulation & crossing the sim-to-real gap.

Progress0/5 questions

hardLearningSkip to practice questions ↓

Prerequisites

Reinforcement Learning

Why it matters in robotics

World models and sim-to-real sit at the heart of modern robot learning: learning behaviors data-efficiently and then deploying them on hardware. Learning an internal model of how the world behaves and rehearsing or planning against that model buys sample efficiency over trial-and-error learning directly on real experience, at the risk of errors compounding over a long imagined rollout and a policy learning to exploit the model's inaccuracies rather than the real dynamics. The sim-to-real gap between how a simulation behaves and how the real robot behaves, arising from unmodeled friction and contact, sensor noise, delayed actuation, and visual differences, is closed by varying the simulator's physics and appearance during training, adapting a policy to real data, calibrating the simulator to match the real robot, and other transfer techniques, always trading robustness against peak performance.

What to study

✓Model-based RL and world models: latent dynamics models, learning and planning 'in imagination,' actor-critic on imagined rollouts (Dreamer/DreamerV3), and why this is more sample-efficient than model-free RL plus its failure modes (compounding error, model exploitation).
✓The sim-to-real reality gap: its sources (unmodeled contact/friction, mass/inertia errors, sensor noise, actuation latency, visual appearance) and how each degrades a transferred policy.
✓Domain randomization (Tobin et al.) for both visuals and dynamics, including the robustness-vs-optimality trade-off, and how guided/automatic randomization (e.g. SimOpt, ADR) improves on naive uniform randomization.
✓Complementary transfer techniques: domain adaptation, system identification (calibrating the simulator to the real robot), and differentiable simulation for gradient-based sim-to-real and parameter estimation.

Where to study

Cheatsheet

Symbols used below

s_t

latent state at time

t

a_t

action at time

t

\hat{s}_{t+1}

predicted next latent state

\pi

policy (actor)

World model core idea

Learn a latent dynamics model $\hat{s}_{t+1} = f(s_t, a_t)$ (plus a reward predictor) from real interaction, then train the policy mostly against rollouts imagined by $f$ instead of against the real environment — this is far more sample-efficient because imagined steps are cheap.

Dreamer/DreamerV3 train an actor-critic entirely on imagined latent rollouts, only touching the real environment to collect fresh data and correct the model.

Failure modes

Errors in $f$ compound over a long imagined rollout, same mechanism as compounding error in imitation learning, so imagined trajectories drift from what the real dynamics would produce the further out they're rolled.

The policy can also learn to exploit inaccuracies in $f$ — finding actions that look great inside the model but fail in reality — since optimization pressure has no way to distinguish a real advantage from a modeling bug.

Sim-to-real gap

A policy trained purely in simulation degrades on hardware because the simulator's dynamics and appearance don't match reality: unmodeled friction and contact, mass/inertia errors, sensor noise, delayed actuation, and visual differences (lighting, texture) each shift the effective state or observation distribution the policy sees.

Domain randomization

Randomize the simulator's physics parameters and visual appearance across training episodes so the policy is forced to work across a whole distribution of environments rather than overfitting to one nominal simulator — the real world just becomes one more sample from that distribution.

This trades peak performance in the nominal case for robustness across the range; too little randomization overfits to sim, too much can prevent the policy from learning anything precise. Guided methods (SimOpt, automatic domain randomization/ADR) adapt the randomization ranges using real-robot data or automatically expand them, rather than picking ranges by hand.

Other transfer techniques

Domain adaptation: adjust the policy or its inputs using real-world data collected after (or during) sim training, rather than relying on randomization alone.

System identification: calibrate the simulator's parameters so its dynamics better match the specific real robot, shrinking the gap directly instead of averaging over it.

Differentiable simulation: a simulator that provides gradients through the physics, enabling gradient-based policy optimization or gradient-based parameter estimation instead of black-box sampling.

	Model-based RL (world models)	Model-free RL
Learns	Latent dynamics model + policy	Policy / value function only
Sample efficiency	High — trains on imagined rollouts	Low — every update needs real/sim environment steps
Main failure mode	Compounding model error, model exploitation	High variance, slow convergence
Extra cost	Must learn and maintain an accurate model	None beyond the policy itself

Application focus

Select an application above.