Seeing With Your Ears

By ALISON MOTLUK
Published: December 11, 2005

Seeing is something that most of us expect to do with our eyes. But what if you are born blind or lose your sight later in life? Peter Meijer suggests you consider seeing with your ears instead.

Illustration by +ISM A camera scans the visual field. The images are converted into a soundscape by the computer. How the sounds might "look": a brightness represented by volume; elevation is represented by pitch.

Meijer, a research scientist in the Netherlands, has developed a technology called The vOICe, which allows you to represent visual information - to "see" - with sounds. The device is a tiny camera, a laptop and headphones. The camera is mounted on your head and the laptop takes the video input and converts it into auditory information, or soundscapes. The scene in front of you is scanned in stereo: you hear objects on your left through your left ear and objects on your right through your right ear. Brightness is translated as volume: bright things are louder. Pitch tells you what's up and what's down. The image refreshes once a second.

With practice, Meijer says, you can learn to sense instinctively how the features of a soundscape correspond to objects in the physical world. Pat Fletcher, for instance, a proficient user of The vOICe who could see until age 21, describes the grayscale images in her head as "ghostly" but real. At a meeting of the Cognitive Neuroscience Society in New York in April, researchers from Harvard Medical School announced that when they viewed the activity in the brains of two vOICe users (one blind at birth, the other who went blind later in life), it was in many respects like that of a sighted person while seeing.

Not everyone has the inclination to kit themselves out with a head-mounted camera and a laptop. Fortunately, with the help of an enterprising Bulgarian software company, Meijer has rejiggered his setup to work using one of the most ubiquitous gadgets of our day: the camera phone. Now, after downloading a simplified version of the software, practically anyone can point her camera phone at what she wants to see and have a listen to what it looks like.