Where imagination meets sensation : mental imagery, perception and multisensory integration

What happens if we imagine seeing something while we are listening to something? Will it change what we hear? What happens if we imagine hearing something while we are looking at something? Will it change what we see? In everyday perception, our brains integrate the information provided to us by our different senses in order to form a coherent perception of the world around us in a process referred to as multisensory integration. However, sometimes the information provided to our senses arises from within, as is the case when we imagine a sensory stimulus; for example, when you picture in your mind the face of a loved one, or imagine how they sound when they say your name. The term mental imagery is used to refer to these willed simulations of sensory stimuli in our minds. Empirical research on mental imagery has demonstrated that there is a great deal of similarity in how we consciously experience, and in how our brains process the sensory stimuli we imagine and the sensory stimuli we perceive from the external world. However, whether our brains integrate stimuli that are imagined in one sense and perceived in the other has never before been explored. The main aim of this thesis was to investigate this possibility.

There were two main goals of the work comprising this thesis. First, to examine whether mental imagery is integrated with incoming sensory stimuli from a different sensory modality to change perception, and second, to examine the neural correlates of these mental imageryinduced changes in perception. Multisensory illusions have come to be a hallmark of multisensory integration as they are an easy and demonstrable way of measuring the integration of cross-modal sensory stimuli. Here, we have made use of classic multisensory illusions, and adapted them to investigate whether mental imagery in one sensory modality can integrate with veridical sensation in another sensory modality to produced fused multisensory percepts. We also used functional magnetic resonance imaging (fMRI) to examine whether brain areas related to multisensory integration of real stimuli were involved in multisensory integration of real and imagined stimuli.

By supplanting a real sensory stimulus with an imagined one in three different classic multisensory illusions, we found that imagined sensory stimuli were integrated with real sensory stimuli from a different sensory modality to change perception. Moreover, we found that these imagery-induced multisensory illusions followed the same spatial and temporal rules as classic multisensory illusions (Study I), as well as the unity-assumption rule of multisensory integration (Study II). Furthermore, we found that the neural correlates of a mental imagery-induced multisensory illusion were closely related to those known to be involved in integration of real multisensory stimuli (Study III). Lastly, consistent with what is known about adaptation to real multisensory stimuli, we found that repeated pairings of imagined and real stimuli from different sensory modalities lead to changes in future perception of the latter (Study IV). Together, these findings suggest that, indeed, what we imagine hearing can change what we see, and what we imagine seeing can change what we hear, affording mental imagery a larger role in multisensory perception than has ever before been considered.

List of scientific papers

I. Berger, C. C. & Ehrsson, H. H. (2013). Mental Imagery Changes Multisensory Perception. Current Biology 23(14), 1367-72.
https://doi.org/10.1016/j.cub.2013.06.012

II. Berger, C. C. & Ehrsson, H. H. What We Imagine Hearing Changes What We See. [Submitted]

III. Berger, C. C. & Ehrsson, H. H. (2014). The Fusion of Mental Imagery and Sensation in the Temporal Association Cortex. The Journal of Neuroscience 34(41), 13684-13692.
https://doi.org/10.1523/JNEUROSCI.0943-14.2014

IV. Berger, C. C. & Ehrsson, H. H. Rapidly Induced Auditory Plasticity by Imagined Visual Stimuli. [Manuscript]