Towards a Better Understanding of Human Emotions: Challenges of Dataset Labeling

A major challenge in automatic human emotion recognition is that of categorizing the very broad and complex spectrum of human emotions. In this regard, a critical bottleneck is represented by the difficulty in obtaining annotated data to build such models. Indeed, all the publicly available datasets collected to this aim are either annotated with (i) the six prototypical emotions, or (ii) continuous valence/arousal (VA) values. On the one hand, the six basic emotions represent a coarse approximation of the vast spectrum of human emotions, and are of limited utility to understand a person’s emotional state. Oppositely, performing dimensional emotion recognition using VA can cover the full range of human emotions, yet it lacks a clear interpretation. Moreover, data annotation with VA is challenging as it requires expert annotators, and there is no guarantee that annotations are consistent with the six prototypical emotions. In this paper, we present an investigation aiming to bridge the gap between the two modalities. We propose to leverage VA values to obtain a fine-grained taxonomy of emotions, interpreting emotional states as probability distributions over the VA space. This has the potential for enabling automatic annotation of existing datasets with this new taxonomy, avoiding the need for expensive data collection and labeling. However, our preliminary results disclose two major problems: first, continuous VA values and the six standard emotion labels are often inconsistent, raising concerns about the validity of existing datasets; second, datasets claimed to be balanced in terms of emotion labels become instead severely unbalanced if provided with a fine-grained emotion annotation. We conclude that efforts are needed in terms of data collection to further push forward the research in this field.