Guaranteed Inference for Probabilistic Programs: A Parallelisable, Small-Step Operational Approach

In the context of probabilistic programming languages, we put forward an approach to formal semantics and sampling-based inference with guarantees, centred on an action-based language equipped with a small-step operational semantics. We argue that this choice offers benefits in terms of clarity and effective, vectorised implementations. In measure-theoretic terms, a product of Markov kernels is used to formalise the small-step operational semantics. A trace semantics is also introduced based on a probability space of infinite sequences, along with a finite approximation theorem, relating the exact semantics to a truncated-execution semantics. This result directly leads to a sampling algorithm with guarantees that can be efficiently SIMD-parallelized. Experiments conducted with an implementation based on TensorFlow show that our approach compares very favourably to state-of-the-art tools for probabilistic programming and inference.