# Generative AI in Gaming ![[Pasted image 20241013235443.png]] _Work in progress_ I'm very excited about the impact Generative AI will have on gaming. I expect we'll enjoy two major advances in game quality thanks to two different breakthroughs in AI: Large Language Models and Diffusion Models. ### Large Language Models The impact of LLMs on gaming is quite straightforward: by implementing tools like ChatGPT within games - role playing games, strategy games, even first person shooters - we will be able to interact in a much more believable way with NPCs (non-player characters) in those games. Imagine playing civilization and being able to _actually_ negotiate with the AI. "If you attack Egypt, I promise I'll have you back in the future against Russia. Plus I'll throw in a couple of Oil Rigs in the South Pacific!". Or a Call of Duty-style shooter where you can issue voice commands to fellow soldiers in your squad. Wouldn't that be neat? The challenges are numerous, of course. To make games more interesting, AI would have to learn to take action within those games. Developers will needs to build hooks within game worlds that language-based AI could interact with. They would also need to figure out a reliable way to implement safe guards to prevent the AI from going wild. I suspect this is why we don't yet have any AAA games that rely on LLMs, almost 2 years since ChatGPT has been released. ### Diffusion Models Another fascinating impact AI might have on gaming is using diffusion models to render game worlds. Today, to play a 3D game, you need the device on which you're playing to render the 3D world, an incredibly compute-intensive process. What if AI could render that world on-the-fly for you? Researchers have already succeeded in [rendering Counter Strike: Global Offensive using diffusion models](https://diamond-wm.github.io/). These techniques might, in the future, enable any device to run AAA games on the highest graphics settings. More excitingly, perhaps, gamers might be able to change the way their games look dynamically based on personal preference. Would you like to play an anime version of Call of Duty? No problem! Ultra realistic World of Warcraft? You got it. Exciting! ### Update: Nov 3, 2024 This article was originally published a few months ago. Since then, [Sequoia announced its investment in Decart](https://www.sequoiacap.com/article/partnering-with-decart-the-future-of-ai-generated-experiences/), a company building a game engine based on diffusion models, as proposed above. While Decart's initial tech demo seems promising, it still exhibits numerous issues inherent in diffusion models: for example, rendering the HUD seems to be very difficult. This might be an issue solvable with additional training data.