Software Architectures for Trustworthy Classifiers

Modern critical systems increasingly rely on autonomous decision-making and data classification, requiring that such operations be performed safely i.e., without negative consequences for people, infrastructure, or the environment. These tasks are often designed for high accuracy, yet remain vulnerable to misclassifications caused by noise, ambiguity, distribution shifts, or unknown inputs. This thesis proposes a paradigm shift: rather than treating classifiers as isolated components to be perfected individually, they should be complemented with mechanism that enable runtime monitoring triggering coordinated responses and mitigations to uncertain or potentially harmful predictions. This means enabling the rejection of outputs suspected to be incorrect, thereby triggering appropriate mitigation strategies. To address these challenges, we introduce the concept of Fail-Controlled Classifiers (FCCs), Software Architectures that are capable of correctly classifying, misclassifying, or explicitly rejecting uncertain predictions. FCCs are designed to reject only those predictions that are likely to be erroneous, minimizing the system’s exposure to incorrect decisions. This approach is particularly relevant in safety-critical applications such as autonomous driving, robotic surgery, and industrial inspection, where rejection or omitting an output is a proven-in-use design pattern for many existing applications. After discussing background, we introduce FCCs, relevant evaluation metrics, and several architectural patterns built on the FCC concept, including the Input Processor (IP), Output Processor (OP), and Safety Wrapper (SW). To further strengthen robustness, these single architectures are extended into ensemble solutions through Voting, Stacking, and Recovery Blocks, enabling richer diversity and broader coverage of the input space. Metrics such as Double Fault, Double Reject, and Disagreement guide the selection and combination of FCCs by quantifying how components differ in their predictions. Building on these ideas, we introduce SPROUT (Safety wraPper thROugh ensembles of UncertainTy measures), a safety monitor that continuously computes multiple uncertainty indicators to identify and reject classifier outputs that cannot be trusted. SPROUT is model-agnostic, fully black-box, and applicable to binary and multiclass tasks over tabular or image data. To ensure that findings of this thesis extend beyond standalone systems, we embed FCCs and SPROUT within a collaborative learning paradigm, specifically Federated Learning (FL). While traditional FL requires clients to adopt the same model architecture and share gradients or weights which impose 3 limitation in the privacy, and industrial deployment. Our Trustworthy Black-Box Federated Learning (TBB-FL) allows clients to train any model they choose, sharing only executable versions treated as black- box components. Extensive experimentation across a wide range of datasets, classifiers, and parameter configurations confirms that the proposed architectures consistently reject a significant proportion of misclassifications and, in some scenarios, detect all incorrect predictions. TBB-FL further demonstrates substantial reductions in misclassifications compared to traditional FL, while retaining strong predictive performance and enhanced privacy. To promote reproducibility and real-world applicability, we provide an open-source library equipped with pre-trained models and ready-to-run case studies, offering a practical pathway for trustworthy ML integration in both centralized and federated safety-critical systems.