Self-driving cars are the future. There are so many reasons why they are better than humans driving cars. But you already know that. The question is, how do we create this future?
The method I chose for my simulation was to use PyTorch and Unreal Engine 4. Specifically, I used reinforcement learning for the algorithm, and a TCP connection to send data between my simulation and the model. There are other ways to do this, but this was the way I was most comfortable with because I had already used Unreal Engine many times before this.
The setup that I used was a 3D model of an intersection and the default car template that comes with UE4. The car has to go forward in the correct lane (at least in North America). Pretty simple, right?
Absolutely not.
I spent lots of time getting new ideas and iterating on those ideas, so progress was pretty slow for the first few months. I started by using Tensorflow (using all of these tutorials) and running Python in the Engine. I realized that this was not optimal for performance, so I used a plugin to create a TCP connection between the Engine and my Python program, which would run separately. Tensorflow’s documentation, at least for this use case, is outdated and terrible. So, after months of trying to get Tensorflow to work, I gave up and tried PyTorch.
PyTorch was much easier to get working. I used this tutorial, and it was much better. The code was a lot shorter, and the documentation was a lot clearer. I was able to understand what each part of the program did, and was able to change that to fit my use. Using PyTorch was so much easier, and I don’t think I will go back to Tensorflow for a while.
There were a few more minor problems, such as trying to get the camera images from the Engine to the program (which I was able to solve by using the Export To Disk function in Unreal Engine), and figuring out which direction the car was supposed to go (which I was able to solve by making the car only need to go in one direction), but I was mostly done!
The end product used collision boxes to figure out where the car was, and then adjust the reward accordingly. While in the starting lane, the reward would increase the farther away from the starting point. This would encourage the program to move away from the back, instead of staying there forever. I had a system where the car should only enter the intersection if the traffic light is yellow or green, but that was probably too much for the model, and it would just result in more time spent training.
Now, all I had to do was train the model.
This proved harder than it seemed. I was expecting to just let the simulation play, and for my model to train by itself. However, this took a lot of time, which was a problem, as I didn’t have an unlimited amount of time. I needed to optimize my environment to make the training faster.
The first change that I made was to make the environment physically smaller. The only benefit to having a long road was to train the car to go forward as fast as possible. I decided that it was a sacrifice I was willing to take to get my project finished before the end of April. Obviously, I wasn’t able to meet this goal, but I believe I have finished a lot faster than if I didn’t optimize my environment.
The second change that I made was to delay the image capturing. My system consists of 4 cameras, one on each side of the car. Before I made this change, my computer would try to take pictures as fast as possible, which generated a lot of strain. This also made it harder for me to do other things as my model was training. I changed it so the pictures were only taken every 2 seconds, making it much easier for me to actually use my computer while training.
The third change that I made was to implement the saving of my model so I could resume my training even if I restart my computer. I used this tutorial, which seems like it was more geared towards normal ML models, not reinforcement learning models, but it still works as far as I can tell. This was a bit more complicated than saving a model in Tensorflow, but I think that sacrifice is still one I’m willing to take for the better tutorials, much shorter and cleaner code, and better documentation.
I made some smaller changes to help with performance, such as limiting the engine FPS to 30 and removing all print statements (this helped a lot). Finally, I was done!
So, what’s the final result?
Pretty weird, I know.
It seems weird that the car is in reverse in the middle of the turn, but it wouldn’t actually matter for a car that has sensors on all sides. It can always see perfectly on each side, so being in reverse isn’t really a problem.
However, this was just a proof of concept. This type of simulation can work, and that is what I aimed to show with this project. My next project will build on top of this and hopefully will create a realistic simulation. I am expecting that to take a long time, but it will be awesome. I’m planning to have a realistic road, with other cars on the road as well. However, I’m expecting that to take a lot of time, so be patient.
In the meantime, check out my newsletter for any updates on my next project.
Want to learn more? Check out my video here.
Resources I used:
Tensorflow:
PyTorch:
Unreal Engine 4:
