Language modality and temporal structure impact processing: Sign and speech have different windows of integration

Abstract

Language comprehension depends on the ability to temporally process the periodic structure of the language signal. In this study we investigate temporal processing of Spanish Sign Language (LSE), isolating the specific contribution of signal modality through a comparison with spoken Spanish and the contribution of linguistic status by comparing language with a non-linguistic temporally structured visual signal. Twenty-three highly proficient hearing users of LSE performed an intelligibility task with these three types of materials, manipulated with different levels of temporal distortion. The results show that the distortion differently affects the intelligibility of these signals. Spanish is characterized by a threshold of temporal distortion, beyond which intelligibility rapidly decreases and is almost completely lost. Conversely, in LSE and the visual non-linguistic task greater temporal distortion led to a gradual and constant reduction in intelligibility with no clear threshold. LSE is more resilient to temporal manipulation compared to the visual non-linguistic signal: participants’ performance never drops below 50% even with the most severe distortion. Overall, these findings suggest that the temporal processing of language arises from the complex interaction between the properties of the sensory system and the special characteristics of the language signal.