« The vOICe Home PageSelected scientific publications based on use of or benchmarking of The vOICe sensory substitution technology for the blind, with a focus on neuroscience:
| A. Kucinkas, C. Retsa, P. B. L. Meijer, M. T. Wallace, M. Gori and M. M. Murray, ``Learning visual-to-auditory sensory substitution reveals flexibility in image-to-sound mapping,'' npj Science of Learning. (Lausanne University Hospital and University of Lausanne, Metamodal BV, The Netherlands, Sense Innovation and Research Center, Switzerland, Vanderbilt University, USA, Italian Institute of Technology, Genoa, Italy) |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| M. J. Kim, M. Kim, H. Yi and C. Y. Jeong, ``Seeing with sound in safe virtual environments: A walk-in-place VR training system for users with visual impairment using the vOICe algorithm,'' Proceedings of the 2025 31st ACM Symposium on Virtual Reality Software and Technology (VRST '25), Article No. 102, pp. 1-2. (Electronics Telecommunications Research Institute - ETRI, Korea) |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| B. F. Händel, X. Chen, M. Inbar, F. Kusnir and A. N. Landau, ``A quantitative assessment of EOG eye tracking during free viewing in sighted and in congenitally blind,'' Brain Research, 149794, ISSN 0006-8993, June 2025. (University Hospital Würzburg, Germany, Zhejiang University, China, Hebrew University of Jerusalem, Israel, University College London, UK) |
|
Y. Park, K. Moon, M. Kim, S. J. Park and C. Y. Jeong, ``Effectiveness analysis
of structured training program for sensory substitution systems,'' 2025 27th
International Conference on Advanced Communications Technology (ICACT), Korea,
February 2025, pp. 1-5. (Electronics Telecommunications Research Institute -
ETRI, Korea)
|
|
P. J. J. Matusz, L. Reich, L. A. Stolz, C. Retsa, D. A. Tovar, E. R. Striem-Amit,
E. Aggius-Vella, A. Amedi and M. M. Murray, ``The brain dynamics of congenitally
blind people seeing faces and letters via sound,'' bioRxiv 2025.01.22.634358,
January 2025. (Institute of Health Sciences, Switzerland, The Sense Innovation
and Research Center, Switzerland, Vanderbilt University, USA, Lausanne University
Hospital, Switzerland, University of California, USA, Hebrew University Jerusalem,
Israel, Reichman University, Israel, Georgetown University Medical Center, USA)
|
|
R. Jin, M. A. Petoe, C. D. McCarthy, J. R. Serra, S. Starkey, J. McGinley and
L. N. Ayton, ``Functional performance comparison of long cane and secondary
electronic travel aids for mobility enhancement,'' British Journal of Visual
Impairment, October 2024. (University of Melbourne, Australia)
|
|
A. S. Butorova, E. A. Koryukin, N. M. Khomenko and A. P. Sergeev, ``Assessment
of accuracy of spatial object localization by means of mono and stereo modes of
visual-to-auditory sensory substitution in people with visual impairments (a
pilot study),'' Modern Technologies in Medicine, Volume 16, Issue 4, pp. 29 ff.,
August 2024. (Ural Federal University, Russia)
|
|
L. Ruan, G. Hamilton-Fletcher, M. Beheshti, T. E. Hudson, M. Porfiri and J. R. Rizzo,
``Multi-faceted sensory substitution for curb alerting: A pilot investigation in
persons with blindness and low vision,'' arXiv, 2024. (New York University, New York,
USA)
|
|
M. Kim, Y. Park, K. Moon and C. Y. Jeong, ``Impact of device and environment on
visual-auditory sensory substitution: A comprehensive behavioral analysis using
the vOICe algorithm,'' IEEE Access, June 2024. (Electronics Telecommunications
Research Institute - ETRI, Korea)
|
|
E. A. Koryukin, A. S. Butorova, V. S. Bobakov and A. P. Sergeev, ``Development
of an algorithm for audiovisual sensory substitution with YOLO network
preprocessing to reduce the user's sensory overload,'' 2024 IEEE Ural-Siberian
Conference on Biomedical Engineering, Radioelectronics and Information Technology
(USBEREIT), May 2024, pp. 122-125. (Ural Federal University, Russia)
|
|
H. J. Kim, J. S. Kim, C. K. Chung, ``Visual mental imagery and neural dynamics of
sensory substitution in the blindfolded subjects,'' NeuroImage, 120621, May 2024.
(Seoul National University, Seoul, Korea)
|
|
J. Ruiz-Serra, J. White, S. Petrie, T. Kameneva and C. McCarthy, ``Learning scene
representations for human-assistive displays using self-attention networks,''
ACM Trans. Multimedia Comput. Commun. Appl., March 2024. (Swinburne University of
Technology, Hawthorn, Australia)
|
|
L. Amaral, P. Thomas, Amir Amedi and Ella Striem-Amit, ``Longitudinal stability
of individual brain plasticity patterns in blindness,'' bioRxiv 2023.11.01.565196,
November 2023. (Georgetown University Medical Center, Washington, USA, and
Reichman University, Herzliya, Israel)
|
|
M. Kim, Y. Park, K. Moon and C. Y. Jeong, ``Deep learning-based optimization of
visual-auditory sensory substitution,'' IEEE Access, February 2023. (Electronics
Telecommunications Research Institute - ETRI, Korea)
|
|
A. S. Butorova, A. A. Naizagarinova, D. A. Tarasov and A. P. Sergeev,
``The vOICe visual-auditory sensory substitution technology in the depth perception
task,'' 6th Scientific School Dynamics of Complex Networks and their Applications
(DCNA), Kaliningrad, Russia, September 2022. (Ural Federal University, Ekaterinburg,
Russia)
|
|
G. Hamilton-Fletcher and K. C. Chan, "Auditory scene analysis principles improve
image reconstruction abilities of novice vision-to-audio sensory substitution users,"
2021 43rd Annual International Conference of the IEEE Engineering in Medicine and
Biology Society (EMBC), pp. 5868-5871, November 2021. (New York University, New York,
USA)
|
|
S. Real and A. Araujo, ``VES: A mixed-reality development platform of navigation
systems for blind and visually impaired ,'' Sensors, 21(18):6275, September 2021.
(Polytechnic University of Madrid, Madrid, Spain)
|
|
J. Pesnot Lerousseau, G. Arnold and M. Auvray, ``Training-induced plasticity
enables visualizing sounds with a visual-to-auditory conversion device,''
Nature Scientific Reports, Vol. 11:14762, July 2021. (Aix-Marseille University,
France, Caylar, France, and Sorbonne University, France)
|
|
M. Kim, Y. Park, K. Moon and C. Y. Jeong, ``Analysis and validation of
cross-modal generative adversarial network for sensory substitution,''
International Journal of Environmental Research and Public Health,
18(12), 6216, June 2021. (Electronics Telecommunications Research
Institute - ETRI, Korea)
|
|
R. P. Kruger, F. de Wet, T. R. Niesler, ``Interactive image exploration
for visually impaired readers using audio-augmented touch gestures,'' 24th
International Conference Information Visualisation (IV2020), September
2020, (Stellenbosch University, South Africa, and CSIR, South Africa)
|
|
C. Jicol, T. Lloyd-Esenkaya, M. J. Proulx, S. Lange-Smith, M. Scheller,
E. O'Neill and K. Petrini, ``Efficiency of sensory substitution devices
alone and in combination with self-motion for spatial navigation in
sighted and visually impaired,'' Frontiers in Psychology, Vol. 11:1443,
July 2020. (University of Bath, UK, and Liverpool John Moores University,
UK)
|
|
J. Kvansakul, L. Hamilton, L. N. Ayton, C. McCarthy and M. A. Petoe,
``Sensory augmentation to aid training with retinal prostheses,''
Journal of Neural Engineering, June 2020. (University of Melbourne,
Australia, Bionics Institute, Australia, Swinburne University of Technology,
Australia)
|
|
M. J. Proulx, D. J. Brown, T. Esenkaya, J. B. Leveson, O. S. Todorov,
S. Watson and A. A. de Sousa, ``Visual-to-auditory sensory substitution
alters language asymmetry in both sighted novices and experienced visually
impaired users,'' Applied Ergonomics, Volume 85, May 2020, 103072 and
PsyArXiv, November 7 2019. (University of Bath, UK, and University of
Queensland, Australia)
|
|
M. Uesaki, H. Ashida, A. Kitaoka and A. Pasqualotto, ``Cross-modal
size-contrast illusion: Acoustic increases in intensity and bandwidth
modulate haptic representation of object size,'' Nature Scientific
Reports, Vol. 9:14440, October 2019. (Nanyang Technological University,
Singapore, Kyoto University, Japan, and Ritsumeikan University, Osaka,
Japan)
|
|
D. Hu, D. Wang, X. Li, F. Nie, Q. Wang, ``Listen to the Image,''
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019,
pp. 7972-7981. (Northwestern Polytechnical University, China)
|
|
K. C. Chan, M. C. Murphy, J. W. Bang, J. Sims, J. Kashkoush and A. C. Nau,
``Functional MRI of sensory substitution in the blind,'' 40th Annual
International Conference of the IEEE Engineering in Medicine and Biology
Society (EMBC 2018), Honolulu, USA. (New York University, USA, Mayo Clinic,
USA, University of South Florida, USA and Korb and Associates, USA)
|
|
C. Graulty, O. Papaioannou, P. Bauer, M. Pitts and E. Canseco-Gonzalez,
``Hearing shapes: event-related potentials reveal the time course of
auditory-visual sensory substitution,'' Journal of Cognitive Neuroscience,
December 2017. (Reed College and University of California, USA)
|
| PowerPoint at
S. Spagnol, S. Baldan and R. Unnthorsson, ``Auditory depth map representations
with a sensory substitution scheme based on synthetic fluid sounds,'' 2017 IEEE
19th International Workshop on Multimedia Signal Processing (MMSP), Luton, UK,
October 2017, pp. 1-6. (University of Iceland, Iceland and University of Venice,
Italy)
|
|
K. C. Chan, M. C. Murphy, J. Kashkoush and A. C. Nau, ``Training-induced visual
cortex activity correlates with duration of blindness and speed of performance
during sensory substitution,'' Investigative Ophthalmology & Visual Science
June 2017, Vol. 58 No. 8, 5627. ARVO Annual Meeting abstract, Baltimore, USA.
(University of Pittsburgh, USA)
|
|
N. Stiles, V. Chib and S. Shimojo, ``Auditory crossmodal plasticity can activate
visual regions automatically and mildly deactivate natural vision,'' Meeting
abstract presented at VSS 2016, Journal of Vision, September 2016, Vol. 16, 539.
(California Institute of Technology, Pasadena, USA, and Johns Hopkins University,
Baltimore, USA)
|
|
U. Hertz, D. Zoran, Y. Weiss and A. Amedi, ``fMRI dependent components analysis
reveals dynamic relations between functional large scale cortical networks,''
bioRxiv Neuroscience, July 2016. (UCL Institute of Cognitive Neuroscience, London,
UK, Google DeepMind, London, UK, Hebrew University of Jerusalem, Israel)
|
|
D. J. Brown and M. J. Proulx, ``Audio-vision substitution for blind individuals:
Addressing human information processing capacity limitations,'' IEEE Journal of
Selected Topics in Signal Processing, Vol. 10, No. 5, pp. 924-931, August 2016.
(University of Bath, UK)
|
|
A. Pasqualotto and T. Esenkaya, ``Sensory substitution: the spatial updating
of auditory scenes 'mimics' the spatial updating of visual scenes,'' Frontiers
in Behavioral Neuroscience, Vol. 10:3389, 2016. (Sabanci University, Turkey,
and University of Bath, UK)
|
|
M. C. Murphy, A. C. Nau, C. Fisher, S.-G. Kim, J. S. Schuman and K. C. Chan,
``Top-down influence on the visual cortex of the blind during sensory substitution,''
NeuroImage, Vol. 125, pp. 932-940, January 2016. (University of Pittsburgh, USA, ...)
|
|
A. C. Nau, M. C. Murphy and K. C. Chan, ``Use of sensory substitution devices
as a model system for investigating cross-modal neuroplasticity in humans,''
Neural Regeneration Research, Vol. 10, No. 11, pp. 1717-1719, November 2015.
(University of Pittsburgh, USA)
|
|
N. Sigalov, S. Maidenbaum and A. Amedi, ``Reading in the dark: Neural
correlates and cross-modal plasticity for learning to read entire words
without visual experience,'' Neuropsychologia, Vol. 83, 2016. (Hebrew
University of Jerusalem, Israel)
|
|
M. J. Proulx, J. Gwinnutt, S. Dell’Erba, S. Levy-Tzedek, A. de Sousa,
D. J. Brown, ``Other ways of seeing: From behavior to neural mechanisms
in the online "visual" control of action with sensory substitution,''
Restorative Neurology and Neuroscience, Vol. 34, No. 1, pp. 29-44,
November 2015. (University of Bath, UK, ...)
|
|
N. Stiles and S. Shimojo, ``Auditory sensory substitution is intuitive
and automatic with texture stimuli,'' Nature Scientific Reports,
Vol. 5:15628, October 2015. (California Institute of Technology,
Pasadena, USA)
|
|
N. Stiles, ``Behavioral and fMRI measures of crossmodal plasticity
induced by auditory sensory substitution,'' PhD thesis, California
Institute of Technology, 2015.
|
|
L. Reich and A. Amedi, ``'Visual' parsing can be taught quickly without
visual experience during critical periods,'' Nature Scientific Reports,
Vol. 5:15359, October 2015. (Hebrew University of Jerusalem, Israel)
|
|
D. J. Brown, A. J. R. Simpson and M. J. Proulx, ``Auditory scene analysis
and sonified visual images. Does consonance negatively impact on object
formation when using complex sonified stimuli?'' Frontiers in Psychology,
Vol. 6:1522, October 2015. (University of Bath, UK, ...)
|
|
D. J. Brown, ``Complexity, the auditory system, and perceptual learning
in naïve users of a visual-to-auditory sensory substitution device,''
PhD thesis, Queen Mary University of London, 2015.
|
|
N. Stiles, Y. Zheng and S. Shimojo, ``Length and orientation constancy
learning in 2-dimensions with auditory sensory substitution: The importance
of self-initiated movement,'' Frontiers in Psychology, Vol. 6:842, June 2015.
(California Institute of Technology, Pasadena, USA)
|
|
F. Bermejo, E. A. Di Paolo, M. X. Hüg and C. Arias, ``Sensorimotor
strategies for recognizing geometrical shapes: a comparative study with
different sensory substitution devices,'' Frontiers in Psychology,
Vol. 6:679, June 2015. (National Technological University – Córdoba
Regional Faculty, Argentina, ...)
|
|
C. Graulty, O. Papaioannou, P. Bauer, M. Stephens, J. Sheiman, M. Pitts and
E. Canseco-Gonzalez, ``Hearing shapes: ERPs reveal changes in perceptual
processing as a result of sensory substitution training,'' Cognitive
Neuroscience Society (CNS) poster presentation, March 2015. (Reed College,
USA)
|
|
D. J. Brown, A. J. R. Simpson and M. J. Proulx, ``Visual objects in the
auditory system in sensory substitution: How much information do we need?,''
Multisensory Research Vol. 27, 2014, pp. 337-357. (Queen Mary University
of London, UK, ...)
|
|
C. Graulty, O. Papaioannou, P. Bauer, M. Pitts and E. Canseco-Gonzalez,
``Electrophysiological dynamics of auditory-visual sensory substitution,''
Vision Sciences Society (VSS) poster presentation, May 2014. (Reed College,
USA)
|
|
E. Striem-Amit and A. Amedi, ``Visual Cortex extrastriate body-selective
area activation in congenitally blind people "seeing" by using sounds,''
Current Biology, Vol. 24, No. 6, pp. 687-692, March 2014. (Hebrew University
of Jerusalem, Israel)
|
|
U. Hertz and A. Amedi, ``Flexibility and stability in sensory processing
revealed using visual-to-auditory sensory substitution,'' Cerebral Cortex,
Vol. 25, No. 8, pp. 2049-2064, 2014. (Hebrew University of Jerusalem,
Israel)
|
|
N. R. B. Stiles and S. Shimojo, ``Sensory substitution: A new perceptual
experience,'' Oxford Handbook of Perceptual Organization, Oxford University
Press, 2014. (California Institute of Technology, Pasadena, USA)
|
|
D. J. Brown and M. J. Proulx, ``Increased signal complexity improves
the breadth of generalization in auditory perceptual learning,''
Neural Plasticity, 879047, November 2013. (University of Bath, UK)
|
|
E. Striem-Amit, ``Neuroplasticity in the blind and sensory substitution
for vision,'' PhD thesis, Hebrew University of Jerusalem, November 2013.
|
|
A. Haigh, D. J. Brown, P. Meijer and M. J. Proulx, ``How well do you see
what you hear? The acuity of visual-to-auditory sensory substitution,''
Frontiers in Cognitive Science, Vol. 4:330, June 2013. (Queen Mary
University of London, UK, ...)
|
|
M. J. Proulx, D. J. Brown, A. Pasqualotto and P. Meijer, ``Multisensory
perceptual learning and sensory substitution,'' Neuroscience and Biobehavioral
Reviews, Vol. 41, April 2014, pp. 16-25. (University of Bath, UK, ...)
|
|
E. Striem-Amit, L. Cohen, S. Dehaene and A. Amedi, ``Reading with sounds:
sensory substitution selectively activates the visual word form area in the blind,''
Neuron, Vol. 76, No. 3, pp. 640-652. (Hebrew University of Jerusalem, Israel, ...)
|
|
N. Stiles, V. Chib and S. Shimojo, ``Behavioral and fMRI measures of "visual"
processing with a sensory substitution device,'' Meeting abstract presented at
VSS 2012, Journal of Vision, August 2012, Vol. 12, 703. (California Institute
of Technology, Pasadena, USA)
|
|
E. Striem-Amit, O. Dakwar, L. Reich and A. Amedi, ``The large-Scale organization
of "visual" streams emerges without visual experience,'' Cerebral Cortex,
Vol. 22, No. 7, pp. 1698–1709, July 2012. (Hebrew University of Jerusalem,
Israel)
|
|
E. Striem-Amit, M. Guendelman and A. Amedi, ``'Visual' Acuity of the
Congenitally Blind Using Visual-to-Auditory Sensory Substitution,''
PLoS ONE, Vol. 7, No. 3, March 2012, e33136. (Hebrew University of
Jerusalem, Israel)
|
|
J.-K. Kim and R. J. Zatorre, ``Tactile-auditory shape learning engages the
Lateral Occipital Complex,'' Journal of Neuroscience, Vol. 31, No. 21, pp.
7848-7856, May 2011. (Montreal Neurological Institute, McGill University,
Canada)
|
|
E. Striem-Amit, O. Dakwar, U. Hertz, P. Meijer, W. Stern, A. Pascual-Leone
and A. Amedi, ``The Neural Network of Sensory-Substitution Object Shape
Recognition,'' Functional Neurology, Rehabilitation, and Ergonomics, Vol. 1,
No. 2, pp. 271-278. Nova Science Publishers, 2011. (Hebrew University of
Jerusalem, Israel, ...)
|
|
D. Brown, T. Macpherson and J. Ward, ``Seeing with sound? Exploring
different characteristics of a visual-to-auditory sensory substitution device,''
Perception, Vol. 40, No. 9, pp. 1120-1135, 2011. (Queen Mary University of
London, UK, ...)
|
|
J.-K. Kim, ``Sensory substitution learning using auditory input:
Behavioral and neural correlates'', PhD thesis, Montreal Neurological
Institute, McGill University, 2010.
|
|
J. Ward and P. Meijer, ``Visual Experiences in the Blind induced by an
Auditory Sensory Substitution Device,'' Consciousness and Cognition,
March 2010. (University of Sussex, Brighton, UK, ...)
|
|
J.-K. Kim and R. J. Zatorre, ``Can you hear shapes you touch?'',
Experimental Brain Research, Vol. 202, No. 4, pp. 747-754, February 2010.
(Montreal Neurological Institute, McGill University, Canada)
|
|
L. B. Merabet, L. Battelli, S. Obretenova, S. Maguire, P. Meijer and A.
Pascual-Leone, ``Functional recruitment of visual cortex for sound encoded
object identification in the blind,'' Neuroreport, Vol. 20, No. 2, pp.
132-138, January 2009. (Beth Israel Deaconess Medical Center, Harvard
Medical School, Boston, USA, ...)
|
|
L. B. Merabet, D. Poggel, W. Stern, E. Bhatt, C. Hemond, S. Maguire,
P. Meijer and A. Pascual-Leone, ``Retinotopic visual cortex mapping using
a visual-to-auditory sensory-substitution device,'' Conference abstract:
10th International Conference on Cognitive Neuroscience, September 2008.
(Harvard Medical School and Boston University, Boston, USA)
|
|
A. Amedi, W. Stern, E. Striem, U. Hertz, P. Meijer and A. Pascual-Leone,
``A what/where visual-to-auditory sensory substitution fMRI study:
can blind and sighted hear shapes and locations in the visual cortex?,''
31st European Conference on Visual Perception (ECVP 2008), August 2008.
(Hebrew University of Jerusalem, Israel, Beth Israel Deaconess Medical
Center, Harvard Medical School, Boston, USA, ...)
|
|
M. J. Proulx, P. Stoerig, E. Ludowig and I. Knoll, ``Seeing 'Where' through
the Ears: Effects of Learning-by-Doing and Long-Term Sensory Deprivation on
Localization Based on Image-to-Sound Substitution,'' PLoS ONE, Vol. 3,
No. 3, March 2008, e1840. (University Düsseldorf, Germany)
|
|
J.-K. Kim and R. J. Zatorre, ``Generalized learning of visual-to-auditory
substitution in sighted individuals,'' Brain Research, Vol. 242, pp.
263-275, 2008. (Montreal Neurological Institute, McGill University, Canada)
|
|
A. Amedi, W. Stern, J. A. Camprodon, F. Bermpohl, L. Merabet, S. Rotman, C.
Hemond, P. Meijer and A. Pascual-Leone, ``Shape conveyed by visual-
to-auditory sensory substitution activates the lateral occipital
complex,'' Nature Neuroscience, Vol. 10, No. 6, pp. 687 - 689, June 2007.
(Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, USA,
...)
|
|
M. Auvray, S. Hanneton and J. K. O'Regan, ``Learning to perceive with a
visuo-auditory substitution system: Localisation and object recognition
with 'The vOICe','' Perception, Vol. 36, No. 3, pp. 416-430, 2007.
(University of Oxford, UK, ...)
|
|
P. B. L. Meijer, ``An experimental system for auditory image representations,''
IEEE Trans. Biomedical Engineering Vol. 39, No. 2, pp. 112-121, Feb 1992.
(Philips Research, Eindhoven, The Netherlands)
|
| |
Note that there are many more scientific publications that discuss or refer to The vOICe in some way, but generally without adding new experiments based on use of The vOICe. For instance on
PubMed and
Google Scholar
you can find a list of scientific publications that cite the original 1992 paper "An experimental system for auditory image representations" (
PMID 1612614). What we still need is more neuroscience research that contributes to mental imagery engineering for sound-guided visual mental imagery. We welcome independent studies from all angles, without strings, non-disparagement clauses or NDAs attached.