Optimization and machine learning in support of statistical modeling

A crucial step in data analysis is to formulate the most appropriate model for reliable inference or prediction. Both optimization and machine learning assist the modeler towards this task. Whenever the inference is the focus, a linear regression model represents a suitable tool for an initial understanding of the reality that the model aims to describe. An automatic and objective procedure to select the predictors of the regression model is fundamental to achieve this target. On this matter, as a first contribution, we propose an algorithm, based on Mixed Integer Optimization (MIO), for best subset selection problem in Gaussian linear regression scenario. The algorithm, with simple modifications, is also suitable for the order selection problem in Gaussian ARMA models. The proposed approach has the advantage of considering both model selection as well as parameter estimation as a single optimization problem. The core of the algorithm is based on a two-step Gauss-Seidel decomposition scheme which favors the computational efficiency of the procedure. The performed experiments show that the algorithm is fast and reliable although not guaranteed to deliver the optimal solution. As a second contribution, we consider the maximum likelihood estimation problem of causal and invertible Gaussian ARMA models of a given order (p,q). We highlight the convenience of fitting these models directly in the space of partial autocorrelations (autoregressive component) and in the space of partial moving average coefficients (moving average component) without having to exploit the additional Jones reparametrization. In our method, causality and invertibility constraints are handled by formulating the estimation problem as a bound constrained optimization problem. Our approach is compared to the classical estimation method based on the Jones reparametrization which leads to an unconstrained formulation of the problem. The convenience of our approach is assessed by the results of several computational experiments which reveal a significant reduction of fitting times and an improvement in terms of numerical stability. We also propose a regularization term in the model and we show how this addition improves the out of sample quality of the fitted model. As a final contribution, the problem of forecasting univariate temporal data is considered. When the purpose of the model is prediction, combining forecasting models is a well known successful strategy leading to an improvement of the accuracy of prediction. Usually, knowledge of experts is needed to combine forecasting models in an appropriate way. However, especially in real-time applications, the need of automatic procedures, which replace the knowledge of experts, is evident. By learning from past forecasting episodes, a meta learning model can be properly trained to learn the combination task. On this matter, we introduce two meta-learning systems which recommend a weighting schema for the combination of forecasting models based on time series features. We focus on sparse convex combinations. Zero weighted forecasting models do not contribute to the computation of the final forecast and their fit can be avoided. Therefore, the more the degree of sparsity increases, the more the computational time for producing final forecasts decreases. The methodology is tested on the M4 competition dataset. Obtained results highlight that it is possible to reduce significantly the number of models in the combination without affecting the quality of prediction.