Learning Backward-Compatible Representations via Stationarity

Machine learning with Deep Neural Networks (``deep learning'') allows for learning complex features directly from raw input data. This leads to state-of-the-art performance being achieved across several---previously disconnected---domains, including computer vision, natural language processing, reinforcement learning and generative modeling. However, learning this representation requires increasingly significant computational costs and resources. Additionally, to leverage new network architectures, more datasets, and evolving training strategies frequent updates to these models are necessary. These updates often alter the models' internal feature representations. End users, who develop a mental model of the systems' behavior, are therefore forced to continually adjust to changes in functionality and capabilities. This adjustment is not only challenging and potentially dissatisfying but also dangerous. This is especially critical in applications such as autonomous driving, image retrieval systems, and generative systems based on language models. This issue can be addressed by learning backward-compatible representations. The aim of these representations is to align models trained with additional data, newer network architectures, or advanced training techniques so that their outputs remain interchangeable across model updates. I hypothesize that focusing on methods which provide stationary representations---where the geometric configuration of the representation space does not change over time---is key. In this thesis, I directly validate this hypothesis through theoretical analysis and empirical support, showing that stationary representations not only represent a solid foundation for future works in this line of research but also presents implications that can be exploited in practical learning scenarios, such as lifelong learning.