Visual modulation of speech perception in sentence context

Dans le cadre du séminaire de maîtrise de l’Orientation psychologie cognitive et psychologie du développement, conférence de

Salvador SOTO-FARACO
Universitat Pompeu Fabra & ICREA

Jeudi 22 octobre 2020 | 10h15 | Zoom

Rejoindre la salle Zoom
https://unige.zoom.us/j/98847668396?pwd=M05WUy8zaHd2OGZrbWVicFJFUWU1dz09

ID de réunion : 988 4766 8396
Code d'accès : 032867

Résumé

Audio-visual interactions have been a classic subject of research in speech perception, in part owing to dramatic phenomena such as the McGurk effect. Most of the studies in this area have considered smaller speech units, such as phonemes, syllables, or words, often devoid of any context. However, when we talk to each other in everyday life, the interplay between visual and auditory speech cues unfolds in larger units which provide prosodic, syntactic, semantic, and pragmatic context.

I will present experiments conducted in our lab trying to get at the impact of visual cues on speech processing in th context of sentences, or small utterances. Theses studies brought about several findings: (1) Visual speech cues contain sufficient information for language discrimination in infants as well as adults. We suggest that the basis for this discrimination is amodal prosodic information. (2) Speaker's hand gestures, a visual expression of prosody, have an impact on early neural responses in the listener. In particular, gestures tend to synchronize low-frequency brain oscillations in anticipation of the affiliate word. Based on this we suspected that gestures summon the listener's attention to specific words in the sentence. However, we failed to find any behavioral correlates of this putative attention-grabbing role of gestures in phonological, syntactic, or memory processes. (3) Finally, I will present results from studies that show how visual speech information fron the face and mouth of the speaker helps anticipate speech content at the phonological level during sentence comprehension. In all cases, linguistic experience (native vs. unfamiliar language) is a limiting factor.