THE ANNOTATION OF GESTURE AND GESTURE / PROSODY SYNCHRONIZATION IN MULTIMODAL SPEECH CORPORA

This paper was written with the aim of highlighting the functional and structural correlations between gesticulation and prosody, focusing on gesture / prosody synchronization in spontaneous spoken Italian. The gesture annotation follows the LASG model (Bressem et al., 2013), while the prosodic annotation relies on the identification of terminal and non-terminal prosodic breaks which, according to L-AcT (Cresti, 2000; Moneglia and Raso, 2014), determine speech act boundaries and the information structure, respectively. Gesticulation co-occurs with speech in about 90% of the speech flow examined and gestural arcs are synchronous with prosodic boundaries. Gesture Phrases, which contain the expressive phase (Stroke) never cross terminal prosodic boundaries, finding in the Utterance the maximum unit for gesture / speech correlation. Strokes may correlate with all information unit types, however is infrequent with Dialogic Units (i.e. those functional to the management of the communication). The identification of linguistic units via the marking of prosodic boundaries allows us to understand the linguistic scope of the gesture, supporting its interpretation. Gestures may be linked to information belonging to different linguistic levels, namely: a) the word level; b) the information unit phrase; c) the information unit function; d) the illocutionary value.