Color Is Not Enough: Dataset and Method for Identifying Relevant Traffic Lights in Driving Scenes

Accurate localization and classification of traffic lights in driving scenes are crucial for enhancing road scene understanding in various intelligent vehicles applications. However, determining which traffic lights are relevant for the ego-vehicle remains an under-explored challenge. In this paper, we address both the local task of identifying the state and relevance of each traffic light in an image and the strictly related global task of recommending the correct course of action for the ego-vehicle (should it stop?). We propose a novel architecture, which not only localizes each traffic light and identifies its relevance with respect to the ego-vehicle, but also generates a global recommendation. To address the scarcity of datasets with these types of annotations, we introduce the Verizon Connect Traffic Light Dataset (VZC-TLD), the first U.S. dataset that provides 3,000 images annotated with traffic light boxes, states, and relevance. Experimental results on both VZC-TLD and the DriveU Traffic Light Dataset (DTLD) show that our unified approach is indeed effective, and leads to significant improvements over approaches that do not exploit the synergies between the local and global tasks.