Aditya Venkatesh's Personal Website

With the recent release of Genie 3, attention has once again been brought to world models. These are generative models that are trained to simulate and model the (not necessarily) real world! Most impressively, these are real-time simulations, sensitive to input. World models are able to understand the properties of the simulated world – such as forces, motion, spatial and temporal relations – and aim to apply them by simulating a coherent world.

How they work

A popular example breaks world models down into 3 components.

The vision model component allows for representing the video frame as a latent representation – done so at a lower dimensionality. This permits a compressed representation of the world. This component also ensures that the latent space representation can be decoded back into the appropriate video frame.

The memory model component uses RNNs (Recurrent Neural Networks) to process the temporal sequence of latent space representations, and predict the next one. Since RNNs are already commonly used for temporal predictions, they fit right into this use case! This component also includes the previous control action to make its prediction, which brings us to…

The control component. This unit is responsible for deciding how the video output changes. While traditionally this was automated via NNs, incorporating human input into this component allows a user to control the simulation. This is all quite the simplified explanation, but it gives a good idea of how world models work.

Why world models?

World models are highly valuable because they enable AI systems to simulate and understand complex, dynamic environments before acting in them. This capability drives applications across robotics, autonomous vehicles, and more. For example, robots can learn spatial awareness and plan multi-step tasks safely in simulations, reducing costly real-world trials. Autonomous vehicles use world models to train safely in diverse traffic, weather, and pedestrian scenarios that might be difficult to encounter consistently in reality. Beyond training, world models support better decision-making and safety by predicting future states and outcomes in real time. They also accelerate learning efficiency and task generalization, empowering AI to handle new and complex challenges with flexibility.

Genie 3

Google’s recent release of Genie 3 marks a significant step forward in world model capabilities. Genie 3 generates interactive, 3D virtual worlds in real-time, powered by a foundation model trained to simulate diverse environments accurately and responsively. What sets Genie 3 apart is its ability to maintain logical consistency and physical realism over extended interactions without relying on hard-coded physics engines. This results in an interaction horizon lasting several minutes! Through its short-term memory, it remembers past events and actions to sustain coherent experiences. Users can guide these worlds with text prompts or direct actions, creating a fluid, explorable simulation that blends imagination and grounded world understanding.