Deep reinforcement learning for the design and validation of modern computer games

Modern video games are one of the most important forms of entertainment today. Their importance in popular culture is demonstrated by the ever-increasing number of gamers and video games. Today’s games are complex environments, with photo-realistic graphics, advanced avatar animations, and they are full of disparate interactions. One of the crucial aspects of the quality of a video game is non-player character behavior: the interactions between players and characters in the game is a vital factor, with the potential to elevate or ruin the player experience. However, the technological progress in game AI techniques has not followed that of other game design aspects - for instance advancements in game graphics. For this reason modern non-player character behaviors suffer from problems inherited from the use of stale techniques. At the same time, recent advances in deep reinforcement learning have had significant impact in training super- human autonomous agents. As a result, now it is possible to train agents that can beat professional players in modern and complex video games. Deep reinforcement learning offers the promise of creating non-player characters that are alive, smart, adaptive and challenging. From a game developer point of view, a fair question that arises looking at these successes is: could I use these techniques to bring my game characters to life? We argue that the answer is no, and for many reasons. This thesis discusses the issues that arise from an inappropriate, if direct, application of deep reinforcement learning techniques as game design tools. In order to understand these problems, we first provide an extensive and detailed survey of the state-of-the-art in both game AI and deep reinforcement learning. This analysis defines the foundation on which this thesis builds. Then we propose a list of desiderata that each machine learning system should satisfy in order to create enjoyable non-player characters. Alongside these desiderata, we introduce a new environment and free-to-play game that serves as one of our main testbeds for this domain. Based on the proposed requirements, we design several improvements and novel algorithms that can help video game designers in the use of deep reinforcement learning as an effective design tool. These improvements focus on: the adaptability of trained agents, the problem of faulty reward functions and how to replace them, the low-level usability of current reinforcement learning algorithms, the poor quality of trained behaviors and the problem of model interpretation. For each of the proposed algorithms, we provide a detailed experimental analysis showing that these methods are indeed useful for solving the cited issues. Finally, a list of open challenges illustrates the problems that currently still exist even after the improvements proposed in this manuscript. We strongly believe that solving these challenges will represent a huge leap forward in the creation of better quality video games.