Learning from Video Streams: Virtual Environments and Parallel Computation

Researchers have always been fascinated by the idea of developing computer programs that could replicate innate human abilities such as language or vision. Recently, the Machine Learning community has increased its efforts towards Continual and Lifelong Learning, pursuing the ambitious objective of developing autonomous learning agents that learn similarly to humans. Research in this direction highlights the strikingly artificial approach that has been followed until now in Machine Learning, where the learning procedures dictates the use of huge datasets and that the learner is shown shuffled samples with no particular correlation among them and sampled from a static dataset. Clearly, this is significantly different from what humans and animals experience in the real world, that is a continuous multi-sensory stream extracted from a dynamical environment, with correlations among each modality of the stream, but also between consecutive samples in each stream, where the flow of time has a central importance in the way the environment is experienced by the learning agents. The need of suitable environments in which an artificial learning agent can live and learn has driven the Artificial Intelligence community to design and implement 3D physical simulations, called Virtual Environments, that straightforwardly offer a dynamical environment with the capability of creating agents that interface with learning algorithms and can experience their surroundings and interact with it. However, currently available solutions are not mature enough to fully implement Lifelong Learning agents, with environments that remain fundamentally static with the exception of interactions by the learning agent. Furthermore, up until now there has been little research on the safety and security of such Virtual Environments with respect to malicious users that wish to poison or undermine the integrity of Virtual Environments to damage the learning of agents living inside them. Finally, while there has been abundant research on accelerating traditional batch-mode offline learning, little research has been produced on the matter of accelerating real-time online learning, which is needed by a learning agent perceiving a real-time online sensory stream. In this thesis, we address these open problems on three fronts, i.\,e.~real-time stream generation, safety and security to Adversarial Attacks, acceleration of real-time online learning. We introduce SAILenv, a platform specifically designed to allow real-time generation and perception of visual streams, with powerful features of parametrical generation of scenarios aimed at creating incrementally complex streams of data, specifically considering Continual Learning tasks; we study the safety and security of the graphical generation engines of the available Virtual Environments, showing that it is possible to implant Adversarial 3D Objects able to poison all scenarios in which such objects are integrated; finally, we introduce PARTIME, a library specifically designed for online real-time learners, that must complete processing of a sample from a stream before the next sample is made available, to mantain real-time performances.