OpenAI is making heads turn in Dota 2. Last year, they had a Shadow Fiend bot take down the best players at TI7. This year, a team of five bots defeated Team Human comprising of Merlini, Blitz, Fogged, Cap and Moonmeander. The team of five were also brought to TI8 to take on professional teams, but fell short. However, they did give the pros a run of their money. I had the chance to talk with Szymon Sidor, who is the distributed systems lead of the OpenAI Dota 2 team. We spoke about why they chose this project and whether or not the OpenAI bots will be available for the entire community to play against, among other things.
Hello Szymon! Thank you for talking the time to talk with us. What position do you hold in OpenAI?
I am the distributed systems lead of the Dota Team.
Intriguing! Let’s start from the beginning. How did this concept of developing an OpenAI team, or even hero start? At TI 7, you guys had a Shadow Fiend bot that defeated a lot of professional players. This year, you are bringing a team of OpenAI to TI. Why did you think of doing it?
There were a lot of independent reasons that kept accumulating. Initially, we just wanted to see if we could get good at any popular video game. So we went through Twitch, looking at all the streams to find out the most popular game. I think at that time, League of Legends was at the top and Dota 2 was second. But Dota 2 is easier to integrate, easier to work with software wise. So we decided to go with Dota 2! After the decision, we found many more reasons to support it as well. Valve is a very nice company to work with and people say there is a more strategic aspect to Dota 2 than to League of Legends.
Was Valve being easy to work with a major reason you continued Dota 2 and not go to League of Legends?
That was a major reason we went ahead with the project. I’m sure Riot Games would have been very helpful to work with as well, but we never went to them.
You said it yourself that Dota 2 has more strategic aspects than League of Legends. Considering you were doing something life this for the first time, wouldn’t it have been a be a bit easier to work with LoL? Or did the OpenAI Dota 2 team want the challenge?
Potentially, LoL might have been easier. But we did want to be more ambitious and wanted the task to be more challenging. The reason we chose Dota was because there was no obvious way to do it. So at the start of the project, we had no idea of how we would solve the problem.
Did any of the OpenAI team for this project play Dota 2 before or did you guys start after the project was decided?
Yes, Jakub who was on the panel with me for TI 8, played both Dota 2 and League of Legends. Many others at the company have thousands of Dota 2 hours clocked. I started playing after the project started and have clocked about a thousand hours. I am particularly bad at it! But I feel like a learn a lot about the game just by watching the bots play.
Aren’t we all bad?! So I remember the OpenAI team saying the bots learn from their previous encounters and get better. Did you need to teach them anything before the first encounter?
Not really. The technology that is used is reinforcement learning. It is based on observations, actions and rewards. The bots have to see something, they have actions to chose from and then they get rewards for good behavior. Initially when they start out, they wander around the base randomly. Then maybe they get close to the lane and get a last hit which gives them a reward in the form of gold. So they will be happy and will do this again. But then a bot might go too far, get killed by the enemy tower and learn to keep a distance and this way, the bots slowly evolve. But it starts out completely random. There is no bootstrapping of any sorts.
But you still need to tell the bot what the final target is? Like say in a 1 v 1, the goal is to get the tower down. In a 5 v 5, the goal is to get the Ancient down…
Yeah of course. That’s the reward. We tell the bots that if you win the game, it’s good. We also give them some other indications like last hitting is good. But how to last hit, we don’t tell them. That they have to figure it out themselves. If they manage to do that, it’s good!
Really interesting. I cannot imagine the efforts and the thought process that goers behind it. In a year, the OpenAI team has come from 1 v 1 to OpenAI team taking on a team of professional Dota 2 players. How difficult was that?
It was pretty difficult. It required a lot of new innovations. There is a long list of things that had to be incorporated. To cite an example, the team spirit parameter which we cited on the TI 8 panel. Initially we teach the bots to be selfish, like the typical pub! Eventually we make them more and more like team mates, until at the very end they are completely selfless.
Must be some process. How many heroes can the OpenAI program control for now? And are the item choices pre-decided?
OpenAI can chose between 18 heroes for now. Yes, the item choices are pre-decided, but we are working on making the item choices dependent on the opposition and situation.
You said this is for research purposes. Who is OpenAI research funded by?
We have a lot of contributors and investors. One of the initial contributors is Elon Musk…
This year, the OpenAI team gave Pain Gaming a very tough time. Maybe next year they will beat Pain Gaming. So the bots are going to keep on improving. The learnings that you have here, in Dota 2, are you implementing them in any other field?
Oh yes, we are already using this technology in different places. One things that we did from the findings of Dota, where we used much of the same code, was robotic manipulation.
Now the important question from the point of view of the community – Will the OpenAI software be available to the community to play against?
We are considering the integration of OpenAI with the Dota 2 client. But it’s a tough task that will require ongoing maintenance. We are not sure if we will have the resources, so we are not making any promises. If we do go on with it, the nice thing is we can have different levels of OpenAI ability which all feel like human play. That way, even 2k-3k MMR players can also enjoy playing against the bots.
I ask this to a lot of people, but you are probably the best one to answer it. DO you think that the work that is being with AI has the potential to do something like in the movie ‘The Matrix’? Like developing a mind of its own, and maybe not right away, but a few years down the road takes over humankind or has the power to do that…
Artificial Intelligence has many dangerous scenarios. The scenario that you described is considered the most likely one. We also need to worry about malicious users. So there are a few things to worry about and like you said, there are no immediate dangers. In case of OpenAI, we have a dedicated safety team where we ponder over these problems.
Thank you for taking the time, Szymon!