Machine Learning components in safety-critical applications can perform some complex tasks that would be unfeasible otherwise. However, they are also a weak point concerning safety assurance. An aspect requiring study is how the interactions between machine-learning components and other non-ML components evolve with training of the former. It is theoretically possible that learning by Neural Networks may reduce the effectiveness of error checkers or safety monitors, creating a major complication for safety assurance. We present an initial exploration of this problem focused on automated driving, where machine learning is heavily used. We simulated operational testing of a standard vehicle architecture, where a machine learning-based Controller is responsible for driving the vehicle and a separate Safety Monitor is provided to detect hazardous situations and trigger emergency action to avoid accidents. Among the results, we observed that indeed improving the Controller could make the Safety Monitor less effective; it is even possible for a training increment to make the Controller’s own behaviour safer but the vehicle’s less safe. We discuss implications for practice and for research.

Impact of Machine Learning on Safety Monitors / Francesco Terrosi,Andrea Bondavalli,Lorenzo Strigini. - ELETTRONICO. - 13414:(2022), pp. 129-143. (Intervento presentato al convegno International Conference on Computer Safety, Reliability, and Security.) [10.1007/978-3-031-14835-4_9].

Impact of Machine Learning on Safety Monitors

Francesco Terrosi
;
Andrea Bondavalli;
2022

Abstract

Machine Learning components in safety-critical applications can perform some complex tasks that would be unfeasible otherwise. However, they are also a weak point concerning safety assurance. An aspect requiring study is how the interactions between machine-learning components and other non-ML components evolve with training of the former. It is theoretically possible that learning by Neural Networks may reduce the effectiveness of error checkers or safety monitors, creating a major complication for safety assurance. We present an initial exploration of this problem focused on automated driving, where machine learning is heavily used. We simulated operational testing of a standard vehicle architecture, where a machine learning-based Controller is responsible for driving the vehicle and a separate Safety Monitor is provided to detect hazardous situations and trigger emergency action to avoid accidents. Among the results, we observed that indeed improving the Controller could make the Safety Monitor less effective; it is even possible for a training increment to make the Controller’s own behaviour safer but the vehicle’s less safe. We discuss implications for practice and for research.
2022
LNCS,volume 13414
International Conference on Computer Safety, Reliability, and Security.
Francesco Terrosi,Andrea Bondavalli,Lorenzo Strigini
File in questo prodotto:
File Dimensione Formato  
impact_of_machine_learning_on_safety_monitors.pdf

accesso aperto

Licenza: Open Access
Dimensione 667.58 kB
Formato Adobe PDF
667.58 kB Adobe PDF

I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificatore per citare o creare un link a questa risorsa: https://hdl.handle.net/2158/1283239
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? 1
social impact