A major milestone in creating generative AI models that can interact with complex real world environments.
Simulating and interacting with real-world environments is one of the next frontiers for generative AI models. From robotics to self-driving vehicles, there are countless of scenarios that require generative AI agents to perform actions in real-world environments. Part of the challenges when building these models has to do with the size of the training corpus. Collecting data from real-world environments is particularly complex and expensive. But can we create generative AI agents that simulate entire environments with high fidelity? This is the goal of a new research from Google DeepMind that showcased GameNGen, pronounced as “game engine”, a model that can simulate a DOOM game using a diffusion model.
GameNGen operates as a generative diffusion model designed to simulate video games. Traditional video games rely on a loop that gathers player inputs, updates the game state, and renders visuals on the screen. This loop runs at high speeds, creating the immersive experience that players enjoy. While these games usually run on standard computers, some unconventional hardware has been used to emulate games like DOOM, though the underlying software remains manually crafted. Regardless of the hardware, the core of game simulation involves predefined rules set by programmers.
Before we dive into GameNGen, check out the quality of simulations we are talking about: