How Machine Learning is Transforming Video Game Development
How Machine Learning is Transforming Video Game Development
Forget non-player characters, ML's true potential is elsewhere in gaming
AI and Games is a YouTube series made possible thanks to crowdfunding on Patreon. Support the show to have your name in video credits, vote for future episode topics, watch content in early access and receive exclusive patron-only videos and merchandise.
It is almost impossible to go a day in technology circles these days without discussing machine learning: the ability to train systems to learn about complex problems and develop nuanced and intelligent responses for them.
Well, I say almost impossible, but it's pretty easy to do if you work in the games industry. As is perhaps evident by the suite of videos I've created over the years, be it in the main AI and Games series, or here in AI 101, you'll notice that across the board machine learning isn't as explored as heavily as other techniques. But why is this? It's really a reflection of how game development has historically operated. Machine Learning has typically not been seen as a practical solution for many of the challenges that developers face, especially in non-player character design. With only a handful of exceptions to that rule.
But with the recent innovations in the field, particularly the boom of research in deep learning, this is beginning to change. But while we're seeing these new ideas creep into some familiar territory, such as your opponents in racing games like MotoGP, Forza and Gran Turismo, or even strategic adversaries in Age of Empires IV, it's arguably having an even bigger impact in other areas of video game production and is beginning to fundamentally change how video games are being made.
I'm Tommy Thompson, welcome to AI 101 here on AI and Games, and in this episode we begin our journey into the world of machine learning. In the first of many episode to come on the subject, I'm going to give you a very high level introduction to machine learning and explain what is, how it differs from traditional AI, the many different ways it can be created, and why it is now becoming increasingly practical to apply in the games industry. This means looking beyond the likes of non-player characters, and instead exploring how it's being used in player modelling, analytics, cheat detection, texture synthesis, graphical upscaling, animation blending and much much more.
What is Machine Learning?
So let's start with the absolute basics, when we say 'machine learning', what do we mean?
Well to borrow from Stuart Russell and Peter Norvig's 'Artificial Intelligence: A Modern Approach: a machine learning system is built "to adapt to new circumstances and to detect and extrapolate patterns".
As these systems adapt and extrapolate these patterns, in theory they improve at the task they're set, which we observe as it learning. You might wonder why we want to get a machine to learn to solve a problem? Especially when us humans know what the problem is, surely we can just write a program to solve it. Well, to borrow once again from Russell and Norvig there's a number of reasons for this:
"designers cannot anticipate all possible that the agent might find itself in"
"designers cannot anticipate all changes over time"
"sometimes human programmers have no idea how to program a solution themselves"
And to embellish on that latter point, sometimes even if you do know how to get a basic working solution, it might not be as effective as figuring out how to get a machine to do it for you. And yeah, that's what a machine learning system is: it learns and adapts to the information given and tries to extrapolate information from within it.
If you're keeping up with the big innovations in AI research and development, you'd be forgiven for thinking that machine learning is a new concept. But it's conceptually as old as other forms of AI. The AI techniques we've seen in AI 101 to date all conform to what is known as symbolic AI, of encoding human expert knowledge using some kind of symbols and notation, and then using it to calculate solutions. The reason you might not have heard about machine learning prior to now is that, well, to be blunt about it: machine learning was kind of a dead end and failed to work at scale or with larger, more complex problems while symbolic AI continued to deliver. However, machine learning has seen a new lease on life over the past decade, courtesy of breakthroughs in research, but critically the benefits of large-scale cloud computing platforms. This is allowing for bigger machine learning systems to crunch through large quantities of data at speeds considered impossible even a decade ago. Hence all the recent work in text and image generation tools, StarCraft 2 bots and all the other weird and wacky innovations we see in the news nowadays have benefited from this breakthrough.
Types of Machine learning
Now our definition of machine learning is a very high-level one, but there a myriad of different methods and techniques arise that we consider to be machine learning. We won't get into the specific implementation types just yet in this piece, but I wanted to break the definition down a little further. Machine Learning techniques are typically broken down into three specific categories: unsupervised learning, supervised learning, and reinforcement learning.
Unsupervised Learning
Unsupervised learning is the idea that we can feed data to a system, apply an algorithm and it will make observations from it. This is a useful technique when you have a bunch of data to feed into the system, but you don't have a particular desired output, you're just interested to see what output it can determine. Perhaps you have data and you just want to figure out if there is anything interesting to find in that dataset? How can this be applied, well back in episode 31 of AI and Games we looked at research in the Tomb Raider franchise that used unsupervised learning on player data to understand how players perform in the game. This project is one of the earliest examples of player analytics, which is now pretty much a requirement for any mobile, online or live service game to monitor the behaviour and activity of the player community.
Supervised Learning
Next up, is supervised learning. This time around you're feeding data into a system, and you have an expected outcome that you wish for it to reproduce. So your dataset doesn't just include a set of inputs to the system, but it also has a corresponding set of outputs, and we expect the system to learn the relationship between them.
So for example, imagine a list of numeric inputs (x), and then a list of numeric outputs (y), the output for the i-th number in y is always the input multiplied by four, like below.
By running a supervised learning algorithm, the system should learn that the underlying relationship is that the output is always the input multiplied by four. Now the trick is that, if trained correctly, the system should then be able to take any input that wasn't in the dataset you trained it on, and still figure out the answer.
Why is this useful? Well if we have a bunch of data on a particular phenomenon then we can reproduce it in known and unknown situations, and a great example of this in games is through player modelling. By feeding in information from the player, we can then train a system that reproduces it and you can see that behaviour in action. Some notable examples of this from previous videos include the Drivatar system of Forza in episode 60 and, as discussed in episode 48 on Google DeepMind's StarCraft player, codenamed AlphaStar, the first phase of its development is that it learns to play like professional StarCraft 2 players, by watching their replay data and trying to mimic their behaviour.
Reinforcement Learning
The third and final branch of Machine Learning is known as reinforcement learning. This is typically used where we have an environment within which a system or agent can interact within it and determine a reward for what is deemed as good or bad behaviour in the context of the objective. But the environment is often highly complex and difficult to learn simple, yet effective strategies to solve a given situation. Hence in reinforcement learning, a system learns what is called a policy through experimentation, it conducts an educated exploration of the problem and over time it learns that certain actions are better than others, and learns to use these more frequently.
Now you might be wondering, hey, what about Deep Learning? Well, deep learning is really a process of training what's known as a neural network, using a combination of supervised learning and reinforcement learning. That's a whole topic in and of itself. But think of deep learning as a subset of machine learning, in the same way that machine learning, is simply a subset of AI as a whole.
Why no ML for Game AI?
So having covered the absolute basics, it's worth highlighting the main differences and strengths and weaknesses of machine learning versus more traditional AI methods. And some of this will help give context as to how it is now being used in the industry, as well as why it has not received much love until recently.
The Benefits
Machine learning as we've already described is about taking existing data and establishing the correct answers from that data. At the start, the system will make all the wrong decisions and over time it will begin to improve, and from the outside in, we can pretend that it's learning like a human does. Though ultimately all that's really happening is the algorithm is performing better at the task we told it to do.
Now, this doesn't mean humans are out of the loop, we define the function the algorithm will optimise against, we provide the data, and we design the structure and configuration of the algorithm. But the system is then left to understand and build its own understanding of the problem, and how to solve it.
This often results in these systems developing a much more nuanced understanding of the data it is provided, than what a human could achieve in a similar timeframe. As discussed in my previous examinations of the Forza Drivatars and Gran Turismo Sophy, a machine-learning-powered race car begins to recognise how to handle thousands of similar, but slightly different configurations of turn type quite easily. But also they can adjust their strategy for different road and weather conditions and even the vehicle they're driving.
The Drawbacks
All that said, these strengths - or rather what is required of them to be achieved - inherently result in two big weaknesses.
First in order to generalise against that dataset, it needs a significant amount of data and time to calculate the outcome, and if it doesn't do quite what you want. You need to hit the drawing board again; rewrite the algorithm and train it again, or gather even more data given it's not got enough to learn from. Now further research in the field has led to a lot of ML algorithms learning and converging on smart behaviours much faster than before, but it's also compounded by the second weakness.
After training an ML system, depending on the approach it's nigh impossible to edit it quickly or add specific flourishes that, in the context of games, would enhance the experience. Designers can't crack it open to add unique behavioural flourishes with ease or remove a behaviour they don't like. In each case, you'd need to change the restrictions being applied to the AI to capture these nuances (which of course incurs the expense of training all over again). This presented three distinct failings that, for the games industry, weren't compatible:
First of all, it doesn't support game development’s iterative nature. NPCs in games go through a lot of refinement. They get playtested for bugs, focus-tested for engagement and iterated upon to bring a high level of polish. Machine learning didn't really fit that given it’s incredibly expensive and time-consuming to iterate on your systems.
Secondly, it doesn't support the game development pipeline. It's fine for a programmer experienced in ML to mess around with these tools. But tweaking learning rate parameters or neural network configurations is a whole bunch of technical gibberish that game designers typically don't understand, and frankly shouldn't have to in order to get on with their job.
Lastly, the resources required to train each system were a dealbreaker. Whether it was the time taken, the data required, or the sheer financial cost of running these systems. It wasn't worth the effort.
Now, this isn't to say there hasn't been any effort to put it into practice. We already mentioned the likes of Forza's Drivatar, which is one of the longest-running machine learning applications in the industry, but prior to that, there were the likes of Lionhead's Black & White and the AI critters in Creatures which are notable examples of machine learning for games. But these were anomalies rather than a reflection of an ongoing trend in the industry.
Given the more recent advancements in AI research, we're now seeing a change in the tides, as more games utilise machine learning for NPCs and opponents. Ranging from the likes of the AI drivers in the MotoGP series to the enemy AI in Hello Neighbor 2, and the campaign AI in Age of Empires IV. But for the most part? Games studios still rely on tried and true symbolic AI techniques when designing characters for games.
And this brings us to the big secret: the real AI revolution happening in the games industry is happening elsewhere. We're seeing significant investment in machine learning technologies for games, but it's all happening in others areas of video game production.
The ML Revolution
Right now we're seeing an increasing number of machine learning applications in games for areas that aren't NPCs and enemy AI. Instead, it's designed to support the future of the industry and seek to improve game production. And in each case it plays to machine learning’s strengths: it's capturing interesting patterns in datasets and using that recognition to yield intelligent responses.
For example, way back in episode 45 of AI and Games on the AI of Hitman, I discussed an approach that computed optimal blend points for animations. This concept has expanded beyond Hitman into a more complex and nuanced approach typically referred to as Motion Matching and is now used in everything from The Last of Us to FIFA. And this is being expanded to enable animations to be increasingly more flexible thanks to machine learning to capture movements across rocky terrain, handle unique geometric features, respond to obstacles and compensate for the character’s direction and rate of motion. We'll be returning to this in a future AI 101 article to discuss it in more detail.
Another obvious one to pull from earlier episodes is Texture Upscaling, we talked about this at length in episode 61 of AI and Games, where the idea is to train a system to learn how to generate a copy of an existing image, but at a higher resolution in a way that still looks crisp. As detailed in that video, this concept was originally advocated for by modding communities, but the more recent work in the Mass Effect: Legendary Edition highlights that this is a concept that can be applied at scale in AAA studios.
And of course, we can't ignore the work in upscaling graphics in real-time as well. Deep Learning is powering the likes of NVIDIA's DLSS superresolution technology. What this means is that machine learning is being used to take the output of a game and increase the resolution of the final image. Allowing users to play games at 4K resolutions without melting their hard drives along the way. This is a topic I discussed both in episode 62 of AI and Games, as well as in my dedicated DLSS 3 video that explains how this technology has evolved in recent years.
Perhaps one of the most exciting ideas that did make some public traction is quality assurance. Having AI systems playtest games isn't just a fairly useful tool given the ever-increasing size of games, it can help alleviate a lot of the pressures being placed on QA teams - who let’s face it, are often not treated very well by certain companies. In fact, we've talked about this already back in my videos on the AI of Division 2, in which they used symbolic AI methods to have bots run around and test missions and the open world.
This has since been expanded upon by a variety of companies to have bots trained with machine learning that can learn to play a game, and in doing so, find issues in it as they play. We saw Electronic Art's early work make headlines as they had ML bots playing Battlefield One back in 2018 and they're still researching new systems that conduct more rigorous coverage of issues in a game world. In fact, this is a big priority for EA given they started using symbolic AI bots to test Battlefield V as far back as 2018, and have continued updating their testing tools for everything from Star Wars Battlefront 2 to Apex Legends and Battlefield 2042. And it's not just EA, companies like modl.ai are now developing tools that run in game engines to facilitate this for smaller dev teams. You can find out more about the Battlefield V bots below, and a follow-up talk recently appeared at GDC in March of 2023
Another fun direction is in cheat detection. Recognising and identifying erroneous behaviour by players is often very hard to do, and if they're running cheat systems on their PC it's very hard to catch it happening lest you run some sort of spyware on their device, which is of course, not a popular approach. There has been some work in this area that is yielding positive results. Why many people will have heard more recently about Richochet, which is the ML-powered cheat detection in Call of Duty and Warzone, the longest-running system is in Overwatch. No, not that Blizzard shooter. Overwatch is the cheat reporting system that runs in Counter-Strike: Global Offensive or CS:GO, and players are incentivised to catch cheaters and report them to a team of human assessors who will then act on the data provided.
However, since early 2017 there has been a deep learning system called VACnet that is trained on human-reported instances in the Overwatch system to identify trends of behaviour cheaters. Then VACnet can report cheaters to the human moderators in the same infrastructure as humans do via Overwatch. The big difference is that the AI has been found to be significantly more accurate, depending on a myriad of factors relating to the games ecosystem, a human reporting a player via Overwatch may lead to a conviction by the moderation team only 15-30% of the time. Meanwhile, with VACnet, the number of successful convictions increases drastically to 80-95% of all reports filed via Overwatch by the AI system. You can find out more in the GDC video below.
These are just some of the many areas that machine learning is being employed, and to go into even more detail would result in a non-exhaustive list. That said, I refrained from mentioning the likes of text and image generation, given at the time of this video in 2023, these techniques have not proven themselves to be valuable in regular video game productions. Time of course will tell whether the recent boom in these techniques may result in something fruitful.
Closing
And so with this episode of AI 101, we're painting a picture of what machine learning is, and how it can be used in games. But ultimately this is all at a very high level. In future episodes, we'll dig deeper, and look at specific techniques and applications, ranging from neural networks to reinforcement learning, clustering techniques and diffusion models. We have a lot of ground to cover, I'm hoping you'll stick around to find out more with each new episode.