Validation of Safety Metrics for Object Detectors in Autonomous Driving

Object detection consists in perceiving and locating instances of objects in multi-dimensional data, such as images or lidar scans. While object detection is a fundamental step in autonomous vehicles applications, it is typically evaluated with generic metrics like precision and recall. Recently, metrics that take into account safety have been proposed in the literature. In this paper we compare two recently proposed safety metrics models for object detectors, "Planning KL divergence"and "Object Criticality Model", validating to what extent they actually measure the safety of an object detector when employed in an autonomous driving application. We base our experiments on the nuScenes dataset, and we compare the two metrics in different scenarios, both nominal ones and with the deliberate injection of detection faults. We conclude that both metrics serve as an indicator of the safety of an object detector, but they also provide different perspectives, and should therefore be used complementarily. As a by-product of this work, we also release a library for the injection of faults in experiments based on the nuScenes object detection task.