AI in games is not a new idea. Opponents that use traditional AI have existed for decades, in every genre from fighting games to shooters. However, with the idea of machine learning getting more popular, the possibility that you might one day face an opponent that is specifically trained against you does not seem so futuristic anymore.
I really liked this concept, so I decided to try it out. My project is a game, made in Unreal Engine 5, where you race a reinforcement learning algorithm on a very small track. The plan is for it to be released later, once everything else is done (main menu, more tracks, etc.)
For the algorithm, I created a DQN Reinforcement Learning algorithm using PyTorch. I used this tutorial from PyTorch to help me, but I had to adapt many parts of it because my environment (a racetrack) is a lot more complex than the tutorial environment, which was 2D and had only 2 actions. I used this plugin for Unreal Engine which allowed me to communicate between the game and my program. I built on my previous program, which I talked about here.
The track I used was very simple, and looked like this.
The size of the track was chosen because of the amount of time I had, and a larger track would take a lot more time to train for the model. I tried training on a larger, circular track, but the model would end up going backwards too many times, and the training was too slow. In the finished game, the track will be bigger.
I used the same method for camera images as I did for my last project, which was using the “Export to Disk” function to save the images that the cameras saw as a file every frame.
I chose a resolution of 256x256 per image, and it ended up using a lot of my system’s resources to save 4 images per 0.3 seconds.
For the reward function, I chose to measure the distance between the car and the finish line, and subtract that from 2200 as that is a bit more than the maximum distance between the car and the finish line. I had to make sure to remove the vertical axis from this, to make sure the reward function works as intended.
Another difference between my project now and the one before is making the program wait out any errors, so the model can keep training even when I’m not at my computer. This made the training a lot more efficient, and made my model a lot better at racing. I also implemented better model saving, which made it much easier for me to train the model. It would save the model every time it detected an error, which was a lot easier on my computer than saving every time an action is picked, but also is a lot better than saving every time the program ends, which doesn’t get triggered when there’s an error. Finally, I started using my GPU for the algorithm. It didn’t help the speed, as the algorithm still only receives images every 0.3 seconds, but it did help lower my CPU temperatures, and also the noise coming from my computer fans.
I also made some changes to the car itself. I used the Chaos vehicle system in Unreal Engine 5, which allowed me to fine tune the settings for the car more. I could add things like thrusters, aerofoils, and other things that could be fun to play with. This would also allow me to add other types of vehicles, such as motorcycles, to the game later. It also allows for better vehicle collision, which is something I plan to take advantage of in my game. In short, the Chaos vehicle system is great, and it is a lot better for me and my game.
Instead of using the default car, I decided to use a Lamborghini Aventador model I made in Blender. This made the simulation and the game look a lot better than the project I made last time. Combine that with actual materials on the track, and it will end up looking like a real game.
This is how it ends up looking in a simulation by itself. It hasn’t trained for long enough, but by the time that the whole game is finished, it will be good enough to challenge humans.
One last challenge that I haven’t figured out yet is to find out how to keep the algorithm running while other players are playing. I could rent out a server, but that might cost too much money. Running my computer 24/7 wouldn’t work either, as the power bill would be too high. I would also have to figure out how to keep each program separated, so the actions only correspond to one player’s situation, but also close enough that it can learn from each player.
Working with Reinforcement Learning has been a challenging yet rewarding experience. I believe that so much is to come from this technology, and I cannot wait to see the impacts it has on the world.
In the meantime, check out my newsletter for any updates on my next project.
Want to learn more about this technology? Check out my video here.
