Bringing Machine Learning Classifiers Into Critical Cyber-Physical Systems: a Matter of Design

Domain experts are striving to solve classification problems through the application of Machine Learning (ML) models, aiming for the highest accuracy. However, classifiers are inherently prone to misclassifications, especially when encountering unknown input data. This complicates their deployment in critical Cyber-Physical Systems (CPSs), where misclassifications can have detrimental effects on people, the environment, or infrastructure. This paper argues that ML classifiers should not be designed or evaluated in isolation but rather conceptualized as Critical System Classifiers (CSCs), which can reject uncertain predictions and trigger mitigation strategies in accordance to the encompassing CPS. We present an high-level architecture for CSCs that supports black-box integration, preprocessing for unknown detection, post-hoc calibration, and cost-sensitive thresholding, emphasizing the importance of evaluating classifiers using cost-aware metrics that account for rejections. We validate our proposal through experiments on tabular datasets related to failure prediction, intrusion and error detection, which are common applications of classifiers within CPSs. Key findings are: i) cost-sensitive evaluation promotes the usage of different classifiers compared to standard performance metrics, ii) tree-based models outperform statistical ones, classification-wise, iii) prediction calibration and rejection strategies provide a robust notion of confidence, and iv) combining multiple uncertainty-based rejectors achieves a favorable balance between high accuracy, low rejection rate, and cost. The analyses above were conducted using our publicly available CINNABAR GitHub repository for experimentations with CSCs. Overall, our study offers a system-level perspective, software architecture and implementation on classifier deployment in critical CPSs domains, paving the way for a trustworthy ML integration into real-world infrastructures.