Walk towards Fence

This page contains a number of orientation and navigation related examples. A set of photographs was created while crossing a street and moving towards a fence with a door. Continuously getting visual feedback from distant landmarks and skylines may help in orientation and mobility, wayfinding, and at the same time help avoid veering.

The vOICe mobile technology targets real-time synthetic vision for use in "blind navigation" by mapping live camera views from a wearable system into corresponding complex sounds that help improve 3D situation awareness and allow the blind user to mentally visualize the surrounding terrain, complementing any optional use of talking GPS navigation systems and/or sonar devices. The soundscapes were created with The vOICe for Windows and afterwards the saved .wav files were converted into MP3 to reduce the file size for downloading. Click on any image link to load the corresponding stereo soundscape into your media player. Each MP3 sound sample file is only a 18K download.

1. Distant view with a fence on the other side of the street. A tree is at the center of the view.	2. View after turning to the right, making a few strides forward, and starting to cross the road.	3. The tree trunk has dropped out of the view on the far left. The lamp post has become very noticeable in the sky.	4. The door in the fence is at the center of the view. On the far right a small part of a parked car enters the view.
5. Moving still closer to the dark door in the wooden fence, with the lamp post audible in front of the fence.	6. The lamp post has dropped out of the view on the far left. Visual perspective causes changing fence rhythm rate.	7. Close to the door in the fence, with the dark door bordered by two bright door posts.	8. Finally very close to the door in the fence, with the door's grip just audible.

The following example discusses the eight photographs of crossing the street at an angle while approaching a wooden fence on the other side of the road. For reasons of safety you should not try any of this yourself unless you have someone to guard and protect you. We will not be discussing safety precautions and listening for traffic sounds in this example.

OK, let's first start with some brief thoughts about moving around.

``Called vOICe, the software runs on a regular notebook PC and is being designed to integrate with hand-held GPS systems when they come on the market. Dr. Meijer envisions the user of his device guided by GPS and speech software to the door of a building, then prompted about the location of the knob by the finely tuned camera view. He says the technology will allow a user to accurately grasp a cup of coffee just set down on a table.'' From the July 10, 2003 edition of The Christian Science Monitor, Computers & Technology section, in an article by Lakshmi Sandhana titled  "Seeing-eye and navigation technologies mean more freedom for the blind, A hand-held device that reads GPS signals, and one with a mini-camera, promise big advances".

Generally, it helps a lot when you are in a familiar environment, and only need orientation to figure out where you are exactly and where you are heading. Then the context and characteristic sounds of unique local visual structures, such as the wooden fence boarding photographed in my neighbourhood, can help with both orientation and mobility. From the soundscapes, you say to yourself "Ah, this must be that fence that I came across yesterday, let's head for the door now!" and you get the relevant information to correct your position and heading from the soundscapes. In a completely new environment you would have to really understand the sound patterns to know what they represent, for instance that a particular sound rithm and perspective probably corresponds to some sort of fence, and that is much harder than recognizing things that you have heard before in a neighbourhood that is somewhat familiar to you.

The eight photographs are in a zip file that you can download as a 210K zip file fence.zip, and after unzipping you will find a number of GIF images with names starting in "fence". The images are best viewed in a 176 by 144 video format (Edit | A/V Preferences | Video Format). You can load images via the File menu of The vOICe for Windows and then selecting the Open and Sonify Image File(s) option, or apply the keyboard shortcut Control O.

• Now load the first photograph, named fence1.gif. You will notice two high-pitched noise bursts. The one just left of the middle is the sound of a white window cover, part of the house on the other side of the street. The one on the far right is a tiny piece of bright sky that is just within the camera view. Next listen for the low-pitched sounds. These seem to go up in pitch, don't they? These sounds belong to the road surface, and the fact that the pitch changes means that we are not quite facing the road in a perpendicular manner. We are actually looking across the road, but facing slightly to the right, and visual perspective then makes that the road sounds like going up in our left-to-right scans. When facing to the left, we would have heard the opposite effect of pitch going down, but here the pitch goes up so we are facing to the right. Visual perspective thus tells you that the road and pavement run away from you on your right side: you don't need to "shoreline" in order to know where the road is heading. You can also hear a certain roughness in the road sounds, and this corresponds to the bricks of which this road is made.

  In the middle of the soundscape you can hear a very brief sound interruption. This is the trunk of a tree on the other side of the street. On the far right you hear a rapid rithm, which is caused by the main poles of a fence that we will walk towards. Try pressing F4 for zoom, to enlarge the middle section of the photograph. This makes it easier to hear the tree trunk, but you will now also have a better chance of hearing two additional beeps of medium pitch. These are two short poles of a low-height white fence just in front of the house. Now turn off the zoom by again pressing F4.

• Next load the second photograph fence2.gif. Here we have turned a little bit more the right, and made a few strides forward, and starting to cross the road. The window of the house has moved outside our view, but the tree trunk is still audible on the left. On the far right the high-pitched sound has a longer duration now because we have a larger piece of the bright sky in view. Moreover, there is a brief interruption in the sky sound, caused by a lamp post. Again we hear the low pitched sound going up because we are actually, and in this case deliberately, veering a bit to the right while crossing the street. The rithms on the right correspond to the fence that we are approaching. On the far right there is a short interruption in this sound pattern too, which is a bit hard to pinpoint at this distance unless you know what to listen for. It belongs to the dark door in the fence that we will walk towards.

• After loading the next photograph fence3.gif, you can basically hear the same items, except for the tree trunk which has dropped out of our view on the far left. The interruption of the high-pitched sky area by the lamp post is very clear now.

• In loading the next photograph fence4.gif, we have the interruption of the fence rithm exactly in the middle of the soundscape. On the far right a small part of a parked car enters out view. Above that car, there is an extremely short higher pitched beep of the window in another house. That house borders the fence on the right, and the side of the house is what causes the relatively high pitched but less loud sound to the right of the loud and high-pitched sky area. Again the lamp post intersects the sky area.

• In the next photograph fence5.gif, we walked still closer to the dark door in the wooden fence, and the lamp post is not only audible as intersecting the high-pitched sky area, but also at lower heights, hence lower pitch, at the same horizontal position, hence at the same time point.

• In the next photograph fence6.gif, the lamp post has also dropped out of our view on the far left, and we clearly hear the interruption in the fence caused by the door in the middle of our view. If you listen carefully, you can hear that the fence rithm is slower on the left than it is on the right. Again this is caused by visual perspective, because the fence part on the left is nearby while the part on the right is further away. Things that are further away in generally sound "smaller and faster" because they fill a smaller part of the soundscape, both in pitch and duration. An accelerating rithm is thus an indication for a regular visual structure that is further away on the right than on the left, while a slowing rithm typically indicates that the right part of this visual structure is closer to us. These are not hard facts, because there could be physical differences in spacing in a visual structure, but that is rare. Certain visual illusions meant to fool the sighted are based on this, and these illusions work because the sighted don't expect a changing "optical rithm" other than caused by the laws of visual perspective.

  The dark door in our scene is bordered by two bright door posts, causing brief sound bursts immediately before and after the interruption caused by the door itself. Try zoom F4 to hear this more clearly. The effect of zoom is similar to moving closer to items in the middle of the scene, which is what we will do in the next few photographs. Turn off zoom again for the sake of this discussion, or the following description will not match what you hear.

• Load the next photograph fence7.gif, where we have walked close to the door, and the result indeed sounds similar to what we got with zoom in the previous photograph.

• Finally load photograph fence8.gif, where even a different wooden texture can be heard in the dark door with its soft noise. The loud sound bursts on the left and right are the white door posts. Now switch to slow motion with F3. If you listen carefully, you can hear a medium-pitched short tone just before the right door post. This is actually the grip with which we can open the door. In addition to slow motion, add zoom with F4. You will hear the individual pieces of wood that make up the door, sounding as a soft and slow rithm. Moreover, you can hear the grip now as a low-pitched short rising tone just before hearing the door post on the right. The door post on the left has dropped out of view in the current zoom mode. Press F4 again to turn off zoom, and F3 again to go back to normal speed, and explore what amount of detail you can still hear in the default mode of operation.

Make sure that your media player runs in autorepeat mode such that you can listen to each soundscape a number of times. With Microsoft Windows Media Player 9 and 10 you can toggle the repeat mode via Control T or via the menu Play | Repeat. With the older Microsoft Windows Media Player 6.4 you can go to the menu View | Options, select the Playback tab and then select "Repeat forever".

Other MP3 sound samples can be found on the Visual Orientation for the Blind page, the Blindsight of a Parked Car page, the Hearing a Printed Graph page, the The vOICe of America page, the Planet Saturn page, the Access Symbol page and the Television for the Blind page.

As a further illustration of The vOICe soundscapes in an orientation and mobility context, you can watch a single-shot streaming video (about twenty two minutes) video clip of a walk along the streets of Eindhoven in The Netherlands. Note that the video quality is deliberately visually degraded as it crudely indicates the visual information that gets encoded in the one-second soundscape scan for each video frame.