The video games industry has developed and changed much over the last forty odd years, but few could have foreseen professional competitive gaming tournaments that fill large arenas with spectators. Nowadays, the gaming industry all included is worth approximately $140 billion; yet, even with the best foresight, could one have predicted a competitive scene in which professional players and teams compete for large cash prizes that run into the millions of dollars? That is where we are today.
One might ask what the difference is between a professional gamer and an amateur. The answer is that there is a great deal of difference in the same way that there is a great deal of difference between Lionel Messi and an amateur football player. Counter-Strike: Global Offensive (CS:GO), a first person shooter video game that draws approximately 600k concurrent players (unique players playing at a given moment), is now considered a sport (e-sport) and is one of many that runs competitions with multi-million dollar prize pools. What does it take to attain the skill required to get into such tournaments? Again, a great deal. Like professional footballers, e-sports athletes require practice and lots of it. This is where Skoltech Assistant Professor Andrey Somov and Associate Professor Evgeny Burnaev come in.
Professional e-sports is a budding industry that is barely ten years old and growing every year. Two years ago, Andrey Somov and Evgeny Burnaev began to ponder over ideas that would eventually lead to the startup, Head Kraken, which, as part of Skoltech’s E-sports Academy, gathers and processes data that coaches can use to create personalized training programs for e-sports athletes, honing their skills and tackling their weaknesses. This involves using a combination of gaze tracking, facial recognition, physiological state sensors, and motion detectors to gather data, which Burnaev and his team processes using machine learning, predicting a player’s achievements and failures.
Machine learning has come a long way in recent years, and Evgeny Burnaev stated during a recent interview that it can be applied in almost any sphere. Who would have thought to create a startup that focuses on developing personalized training programs for professional e-sports players using machine-learning techniques to process gathered data? Through Head Kraken, Somov and Burnaev aim to meet this need of the e-sports market.
We caught up with Andrey Somov, who recently gave a talk at Park Zaryadye about this project, and Evgeny Burnaev to hear their thoughts firsthand.
What drew Somov and Burnaev into the area of e-sports?
Two years ago, the E-sports Academy was created at Skolkovo and with its advent, the two professors saw an opportunity to apply their knowledge that would contribute to the new project. Having applied for an internal grant by way of the STRIP program, they began to work on understanding the psycho-emotional side of e-sports. At the time, Professor Somov was already working on wearable technologies related to Parkinson’s Disease. Professor Burnaev had already had experience running a behavioral economics course at the Moscow Institute of Physics and Technology.
“It was an interesting approach analyzing physiological data to understand how people make economic decisions while playing an economic game,” said Burnaev. “People play economic games to buy and sell shares. I thought, ‘Why not apply this to games and analyze the behavior of cyber-sportsmen?’ It is an interesting source and there are many challenges but you can gather a lot of multi-model data.”
Professor Somov was the expert that Head Kraken needed because of his expertise with sensor technology. He would oversee the activities with the goal of developing a system of sensors. Somov’s and Burnaev’s joint efforts in the field of e-sports have already produced results.
“My colleague, Evgeny Burnaev, works in artificial intelligence and machine learning. We recently published a number of papers and one of these got a Best Paper Award at the IEEE IoP a few weeks ago. What I am proud of about that paper is that our MSc student, Anton Smerdov, who was attending my course introduction to the Internet of Things, did most of the preparation. Anton was interested in the topic and so dug deeper and began building a prototype with his classmates. The result was that paper,” said Somov.
The prototype in question is a smart chair platform, which collects data on a seated person’s behavior, using integrated sensors and giving a feed of data that identifies a player’s physical reactions to important game events.
Using data collection at a live event
Counter Strike: Global Offensive (CS:GO) serves as Head Kraken’s primary source of data because, as mentioned above, it is one of the most popular e-sports and the first professional team to join the Head Kraken gaming house was comprised of players whose specialty is CS:GO.
Last August, Head Kraken collected data from a real e-sports tournament, though not a CS:GO event. The e-sport in question, which is also a popular first person shooter, is a fast-paced game called Team Fortress 2 (TF2). Professor Somov had given a talk at Nike in Moscow in December 2018 and exchanged contact details with an e-sport events organizer. That is how Head Kraken came to be a research partner in the TF2 tournament.
“We worked on data collection and data visualization during the live broadcast. Both a Russian-speaking and an English-speaking team broadcast the event and we added the information from our sensors during these broadcasts. Seeing an e-sport athlete’s heart-rate during a live stream makes an impression on the audience, because viewers can see a kind of digital person through something real. Displaying live physiological data at a tournament is very simple, and you know what they say: ‘Simplicity is also genius.’ This encouraged us to create new valuable metrics to visualize during the live broadcast,” said Somov.
Gaming tournaments already use hard data to calculate a team’s win chances during a match, but it is the first time that a tournament has used live data based on players’ physiological data. Such details add to the hype of a tournament, because viewers can see which player is more nervous. For example, during the TF2 tournament, one of the captains’ heart-rate was 160bpm, while the other captain’s was 130bpm; the team whose captain had the lower heart-rate won the final. Interestingly, the team with the more nervous captain had already won the tournament last year.
“We provided a sensor to measure the heartbeat – in terms of the data analysis for this application it was almost zero, but in terms of the fun for the viewers, providing a visualization of the heart-rate created a sense of excitement in the audience. If we have a system of sensors, then we can provide live online analytics at tournaments as well. In football, you can quickly calculate the number of moves, the distance that each player runs, and provide analytics between sections of the game; this is useful for a coach to make an immediate analysis of the situation. We can do the same with e-sports,” said Burnaev.
Application of data-gathering technology
Using specific technologies such as eye tracking, you can build different heat maps that show where a player’s gaze is focused on the screen and for how long.
The gaze tracker shows whether a player is wasting time looking at the radar box or is focused on the game – the cross-hairs, in other words. Heart-rate sensors monitor stress and keyboard-to-mouse – “game telemetry” – offers yet another source of data from which you can generate patterns. Based on this, one can tell whether or not the data set is that of an amateur or a professional.
Professional players all behave differently, and this can be visualized; such visualization is crucial to coaches, because in using these analytics they can show in concrete form what teams and individual players are doing wrong and what requires improvement. Also, by monitoring and understanding communication between players during a game and seeing how it affects the game results, then analyzing the audio, it is possible to find out who is the real team leader.
According to the coaches, they need access to hard data because the average pro-player is between sixteen and twenty-six. Younger players require more explanation as to why they are not good in particular situations during a game. Head Kraken gives coaches the ability to back up their coaching advice with graphs, heat maps, analytics and statistics.
“Being young and confident in their own abilities as pro-players, they are more likely to listen to a coach’s advice if it is accompanied by a graph and heat map based on hard data. That is far more persuasive. The point is to show them that it isn’t simply about getting lots of head-shots,” said Professor Somov.
The Head Kraken team is in the process of developing the final indicators that coaches will be able to use, but according to Professor Burnaev, this isn’t an easy task.
“Every time, we need to generate a hypothesis and develop a prototype for how to estimate the indicator for that given hypothesis. We then test it on a coach,” he said.
Nowadays, it’s possible to measure different types of data from Smart Chair, the eye tracking device, and the Head Kraken team shows that it is possible to calculate different characteristics that are well correlated with the performance of an individual player; these characteristics are also there for the coach to interpret.
However, it’s not just about improving training for pro-players and strategies for the team as a whole. Machine learning can be applied during training sessions to analyze demo files. A demo file is a recording of a match that is stored on a server and available for download. By using the analytics of demo files, not only can you improve your own skills and strategies, but you can also understand the opponent’s strategy and, perhaps, weaknesses.
As an example of using a demo file to spot an unusual strategy, Professor Somov mentioned his team’s analysis of a match involving the pro-team OG:
“Until recently, they have always made themselves appear to be the underdogs, yet they seem to understand their game on a very deep level. We discussed them in our research group and concluded that they lose minor tournaments so that the big teams do not consider them formidable opponents. Yet when it comes to the big tournaments like The International and the Majors, they fight and win. We think that might be their strategy based on what we have seen.”
While this technology is not new, it is new to e-sports research and Somov’s team could find no information on such research being conducted elsewhere in the e-sports sector.
What is the future of Head Kraken in the up-and-coming e-sports market?
The e-sports market has been growing steadily over the last decade and is projected to continue for many years to come – a promising prospect for developing analytics and training. The market is not only growing because e-sport is another avenue for competitive tournaments, but also because it is accessible to a broad audience who might not necessarily be active gamers, but simply enjoy watching professionals play competitively.
“I think what attracts people is that e-sports athletes have to make very quick decisions, even quicker than regular sports such as basketball, volleyball, or football. It makes for very entertaining commentary during live broadcasts,” said Professor Somov.
Professor Somov also indicated that there are other potential uses for Head Kraken’s personalized training programs, such as pilot training. The airline industry has already expressed interest in developing personalized flight training programs.
“What we need to do for sure is develop some prototypes of different applications for coaches, for end-users, and for end-gamers. With those things, we might be able to get funding from a bigger entity or a large company. That is the ultimate goal for the next year or so.” said Professor Burnaev.
With the e-sports market projected to reach approximately $3 billion by 2025, the prospects seem promising for Head Kraken. Will its live data analytics become a regular feature in other e-sports tournaments such as for CS:GO, Dota 2, League of Legends, and Fortnight? Will its methods for developing personalized training programs become the go-to for e-sports athletes? That remains to be seen.
+7 (495) 280 14 81