Shape conveyed by visual-to-auditory sensory substitution
activates the lateral occipital complex

A. Amedi, W. Stern, J. A. Camprodon, F. Bermpohl, L. Merabet, S. Rotman, C. Hemond, P. Meijer and A. Pascual-Leone,  ``Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex,'' Nature Neuroscience, Vol. 10, No. 6, pp. 687 - 689, June 2007 (doi:10.1038/nn1912). This neuroscience research was also featured in the New Scientist print edition of May 26, 2007, with an article titled  "How the brain can hear shapes", p. 22. Related research at Harvard Medical School was covered on Canadian CBC Radio One in the Quirks & Quarks broadcast of June 16, 2007, in a report titled  ``Blinded by Science,'' by Alison Motluk.

vOICeObj: object recognition using sensory substitution device (SSD) mapping AudObj: auditory object recognition of "natural" sounds made by objects vOICeScr: vOICe scrambled images control AudScr: control matched for basic auditory characteristics Figure 2C. Percent signal change analysis of LOtv, for magnitude of activation in left LOtv across subjects in the nontactile conditions. Robust activation was found for vOICeObj (red) in all vOICe experts, whereas no such activation was evident in the five sighted controls (percent signal change in each subject and average percent signal change ± s.d. for all sighted vOICe experts and controls).

Images, and a soundscape, are also included in the Powerpoint slides  "nn2007.ppt".

The fact that the brain area LOtv is driven by the presence of shape, indiscriminate of whether the input is visual, tactile or auditory, is consistent with the notion of a metamodal brain organization put forward by Pascual-Leone and Hamilton ( "The metamodal organization of the brain", PDF file).

Related to the Nature Neuroscience publication is the conference presentation at the 8th Annual Meeting of the International Multisensory Research Forum (IMRF 2007) in Sydney, Australia, July 5-7, 2007, titled "Extracting shape and location information conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex and dorsal visual stream respectively in blind and sighted individuals", the conference presentation at the 12th Annual Meeting of the Organization for Human Brain Mapping (HBM 2006) in Florence, Italy, June 11-15, 2006, titled "LO is a meta-modal operator for shape: an fMRI study using auditory-to-visual sensory substitution", and the conference presentation at the 11th Annual Meeting of the Organization for Human Brain Mapping (HBM 2005) in Toronto, Canada, June 12-16, 2005, titled "Neural correlates of visual-to-auditory sensory substitution in proficient blind users". Also related is the independent research presented at the 14th Annual Meeting of the Cognitive Neuroscience Society (CNS 2007) in New York, USA, May 5-8, 2007, titled "Visual substitution learning of abstract shapes via auditory input". For other useful neuroscience related literature, see recent publications on cross-modal plasticity, neuroplasticity (neural plasticity), brain plasticity (cortical plasticity), and sensory substitution.

Example spectrogram of a one-second sound generated by The vOICe. Amedi et al. show that in trained subjects such shape-preserving sounds ("soundscapes") activate part of LOtv, a brain region normally associated with shape processing by vision or touch. Since other types of sounds do not activate LOtv, this indicates that LOtv is mostly driven by shape information regardless of sensory input modality.

Note: The vOICe technology is being explored and developed under the Open Innovation paradigm together with R&D partners around the world.