The vOICe on BBC News, October 2003
This web page is obsolete and will no longer be maintained.
|
Media coverage:
Other: The Globe and Mail, June 2006: Handhelds give blind insight |
|
The vOICe MIDlet adds a sonic augmented reality overlay to the live camera view,
thereby giving even totally blind people detailed information about their visual environment
that they would otherwise not get.
The vOICe camera-based soundscape technology for the totally blind is now available for most Java-enabled camera phones and PDAs! Free!
The vOICe MIDlet software runs on both Symbian and non-Symbian devices, although in practice it is mostly Nokia Symbian phones that meet the requirements for running The vOICe MIDlet. Among the phones that run The vOICe MIDlet are the Nokia 6600 and Sony Ericsson K700i, the Nokia 3230, 5500, 6131, 6620, 6630, 6670, 6680, 6682, 7610, 9500, E50, E51, N70, N73, N75, N82, N86, N93 and N95, and likely also on the Nokia 6681, 5230, 6710 Navigator, 6730 classic, 7710, E52, E55, E72, E61i, N90, N91, N92, N96, N97 (mini), Nokia N8, C6, C7, E7, and other MIDP-2.0 & MMAPI (JSR-135) compliant devices of various brands. The software even includes a talking color identifier, such that you can point the camera of your Java-enabled smartphone or PDA to any item of interest and hear the color name spoken. Moreover, the software is available free of charge for non-commercial personal and academic use!
|
The vOICe is now available for multiple hardware platforms to suit different user preferences.
Apart from The vOICe for Windows for an immersive "backpack"
system with a netbook, PC camera (webcam or video sunglasses)
and stereo headphones, some users prefer using the much simplified but highly portable "magic wand"
version that runs on mass-market modern camera phones and PDAs. For continuous use and maximum
context-awareness, a head-mounted camera would be much preferred for best sensory feedback, but
for occasional orientation and wayfinding purposes, for reading signs, or to have a look at graphs
or other graphical material in print, on the blackboard or on displays, or for simple use as a
light probe and light locator, the mobile camera
phone version of The vOICe can form an attractive alternative. Depending on your personal interests,
you may consider it a mobile augmented reality game - the game of sight (disclaimer).
You can read a few accounts written by blind users of this software.
Within a decade, second-hand camera phones could become an affordable platform for use of The vOICe
by blind people living in developing countries! These phones can at the same time serve many general
communication, Internet access and pervasive computing needs for the sighted poor, while doubling as a
digital camera. The vOICe camera phone software can thus form an example of targetted
| |||||||
|
| |||||||
Now if you are blind, you may need initial sighted assistance with the installation, but after that you should be able to work with the software using fixed keystroke sequences. The details of that are not specified here, because these may vary with the brand and type of device. Beware that with most camera phones the lens is on the backside of the phone, so the visual object of interest should then be behind the phone when you are pointing the phone to the object while you are facing the keyboard just like you would when operating the keys.
What color is it?
The "*" (star, asterisk) key toggles the talking color identifier on and off. This mobile color
recognizer for blind and color-blind people speaks the color of whatever shows at the center of
the camera view. Note that
results of color recognition depend on ambient light and camera quality. Recognized colors
include (dark, normal, and light) red, green, blue, cyan, yellow, orange and magenta, as well
as combination colors such as red-orange. Black, grey and white are also identified, bringing
the total number of identified colors and shades to 47.
Further options exist for color filters, even including a filter for detecting exposed skin,
e.g., for face detection, for detecting people or to help find empty seats. The color filters
can also be used by blind and color-blind people to pick wires of a user-selected color, e.g.,
to distinguish red or yellow wires in electronics. Beware that the choice of color names can
be culturally biased:
cyan is a color in between green and blue, while magenta is basically the same as the color
purple. Also, light-magenta and light-red make for the color pink or very similar colors,
while dark-orange appears as a shade of brown. Dark yellow-green makes for olive-green.
Need wide-angle view?
Especially in mobility applications, it is important to detect objects and obstacles to the side,
as well as at floor level in front of you. This may require a wider field of view (FOV) than most
phone cameras provide. For this purpose a convenient accessory is the USBfever 180°
magnetic detachable fish-eye lens,
which does not require removal of the original lens but is simply magnetically mounted in front of
the phone lens, and is compatible with multiple phones.
Note that the 180° lens - despite the name - in general does not give a 180° view but
a view that is 2 to 3 times wider than without the add-on lens, up to a maximum of about
180°, e.g., one gets a 80° to 120° view in case the view was 40°.
A wide-angle view can also help blind users of The vOICe to more quickly build a mental overview of their visual environment.
Blind photography
Some blind people like to take pictures to document situations much like other people do,
or they apply it as an art form. On camera phones with file read/write support (JSR-75),
The vOICe will save a high resolution camera snapshot of the current view to the memory card
when pressing the "p" key. On Nokia phones, the saved images will be found in the image gallery.
The image filenames get a timestamp.
The soundscapes of The vOICe can thus be used as a kind of auditory viewfinder to let
blind photographers frame their subsequent high resolution photographs.
More information on blind photography can be found at the
Blind with Camera School of Photography.
Reconfigurable
A number of other key commands are available for changing settings. The "0" key toggles the muted state. The "1" key toggles the negative video mode, which can help to see/hear small or thin dark items on a bright background. The "7" key toggles a mode that helps prevent soundscape stuttering and buzzing on devices that cannot handle simultaneous soundscape rendering and playing. The "9" key toggles between different contrast enhancement modes. The "#" (pound, hash) key toggles between different sound volume levels when not muted. The default sample rate is 16 kHz, but lower sample rates can be selected by using the "DOWN" key (joystick down), and higher sample rates can be selected by using the "UP" key (joystick up). Available sample rates are 8 kHz, 11 kHz, 16 kHz and 22 kHz, but phones need not support all of these sample rates. Lower sample rates give lower sound quality, but may offer somewhat less sluggish responses. The "RIGHT" key doubles the soundscape duration to at most two seconds, while the "LEFT" key halves the soundscape duration to at least half a second. Note that on some devices the "UP", "DOWN", "LEFT" and "RIGHT" keys may be mapped through the "2", "8", "4" and "6" keys, respectively.
More information about various features is available in the The vOICe MIDlet manual (Microsoft Word format). Enjoy!
| The vOICe Key | Action | Default |
|---|---|---|
| 0 | Toggles muted state | Off |
| 1 | Toggles negative video | Off |
| 3 | Toggles speech feedback | On |
| 7 | Toggles "anti-stutter/buzzing" mode | Off |
| 9 | Cycles contrast enhancement | 100% |
| * | Toggles talking color identifier | Off |
| # | Cycles sound volume levels | 25% |
| UP | Higher sample rate, up to 22 kHz | 16 kHz |
| DOWN | Lower sample rate, down to 8 kHz | 16 kHz |
| LEFT | 0.5 or 1 second soundscape | 1 second |
| RIGHT | 1 or 2 second soundscape | 1 second |
| FIRE | [ Flash and] say color |
Off |
| 00 | Mute and pause (low CPU load) | Off |
| ** | Color identifier, no soundscapes | Off |
| ## | Toggles blinders (narrow view) | Off |
| r | Red-only color filter | Off |
| g | Green-only color filter | Off |
| b | Blue-only color filter | Off |
| c | Cyan-only color filter | Off |
| y | Yellow-only color filter | Off |
| o | Orange-only color filter | Off |
| m | Magenta-only color filter | Off |
| 11 or s | Skin-only color filter | Off |
| a | Analyze colors by cycling filters | Off |
| l | Toggles moving scanline | On |
| p | Save snapshot picture to memory card |
If you seek a screen reader for your mobile phone, you may wish to check out products like
Mobile Speak
(Pocket for SmartPhones, MSP) plus MCR from Code Factory,
Pocket Hal
from Dolphin, ADnota speaker from
adnota GmbH,
VSpeak from
VoiceSignal
and Text to Speech for Pocket PC from
Digital Future.
Compatibility with The vOICe for mobile phones is not guaranteed, although both Nuance TALKS
and Mobile Speak have been successfully applied in combination with The vOICe. Beware though
that Mobile Speak versions 1.125 and 2.03 reportedly have serious compatibility problems with
Java MIDlets.
Whenever possible, try before you buy. The user's choice of text-to-speech engine for the
screen reader may affect compatibility with The vOICe. The vOICe supports two menu styles:
the "Textual" style for the submenus is only advised for old versions of Nuance TALKS,
while the "Normal" style is advised in all other cases.
|
eyePilot for
camera phones, which performs functions that are similar to The vOICe color identifier.
Mobile TV?
For phones that include RealOne/RealPlayer, such as the Nokia 6600, there is a video clip about the PC version
of The vOICe, namely the
 2002 G4 TechTV television broadcast,
downsized to 176 × 128 pixels (735 KB non-streaming file, tinyurl.com/s832f). With Nokia phones, PC Suite
can be used to put the downloaded video clip on the phone.
Similarly, if your phone supports the AMR format, you can use your phone to listen to the
In case you seek an accessible MP3 player for use with a screen reader, check out
|
Stereo MIDlet with 3D audio
The vOICe MIDlet supports stereo panning and 3D audio spatialization on suitable stereo-enabled
camera phones, such as the Nokia 6620, 6630 (tested OK), 6680 (tested OK), N82 (tested OK) and
many others. The stereo panning makes perception of lateral position in image scans more intuitive,
while binaural cues offered by 3D audio may further help with object segregation in complex scenes.
Note that many Symbian based phones will only give stereo with wired stereo headsets
and not with bluetooth wireless stereo headsets, due to operating system limitations:
Symbian OS 9 or later is needed for stereo via bluetooth.
Options for Mono (default), Stereo, 3D audio and
4D audio are available in the Channels submenu under phone's
Options key. Phones lacking stereo capabilities may give distorted mono sound
when set to the Stereo or higher audio modes. The 4D audio is meant
only for advanced users, and encodes visual motion. The View rotation
entry in the Options menu can be used to accomodate camera phones with built-in
stereo speakers, such as the Nokia N82 and Nokia N97, allowing the two speakers
to be aligned horizontally with the camera facing forward and the sound view
matching this device orientation. Also note that while using a screen reader
it may be necessary to first mute The vOICe (with key "0") to hear the screen
reader speak all menu and submenu items under the Options key. After changing
settings, one can then unmute The vOICe again with key "0".
The vOICe's stereo and 3D audio support is completely independent of the microQ 3D audio engine from
|
"Bat call" quirks mode
A "bat call" quirks mode is toggled by a long-press of the FIRE button (holding down the joystick button,
or center d-pad button: the button name varies with the type of phone). This gives you two loud but
very brief high-pitched chirps in rapid succession, much like a human-audible version of the clicks
or sound flashes emitted by bats for echolocation purposes. When sounding these over the phone's
built-in speaker, you may detect nearby obstacles from any returned echoes. The sound flash patterns
are repeated with the same interval used for the soundscapes. If you prefer, you can toggle use of
single sound flashes by pressing the "1" key. You can also independently cycle the audio volume of
the bat calls by pressing the "#" (pound, hash) key while in the bat call mode.
|
Barcodes and Semacodes? Free OCR?
For a barcode reader for online price comparisons using a mobile camera phone, check out the
public domain Barcr
UPC code reader project,
or Scanbuy.
Rob Fugina created the Internet UPC Database
for obtaining product information from barcode numbers.
Other potentially useful developments for blind camera phone users include the
Visual tags could be applied at for instance the entrance of bus stations, railway stations and airports to easily bring the blind or sighted phone user to automatically updated web pages with the latest (dynamic) traffic information, train or bus platform for a given destination, flight schedules with arrivals and departures (including any delays), route descriptions, tourist information, local service and support phone numbers, virtual instructions and other information that can aid in navigating specific unfamiliar and complex environments. In combination with a phone screen reader these web pages are readily accessible to blind people. The web page format also easily complies with existing worldwide standards for presenting information: no problems with incompatible proprietary or localized data formats, while different sites may of course offer very different targetted information much like different and independently developed homepages on the Internet do. Associating sounds with selected objects or places has also been coined the delivery of "
The distinctive visual textures of visual tagging codes like barcodes, semacodes, shotcodes, QR codes and SP codes
give equally distinctive sound patterns with The vOICe MIDlet that may help in properly orienting the camera for
subsequent barcode, semacode, shotcode, QR code or SP code reading. The vOICe MIDlet may similarly be used to locate
and orient print for subsequent OCR analysis (optical chacacter recognition) from camera views
once sufficiently reliable OCR engines become available for handheld devices. The combination of OCR and a talking
screen reader could also serve people with severe dyslexia once the OCR recognition quality becomes good enough.
Camera snapshots that are saved in JPEG format through The vOICe's "p" key command may be used as input for
third-party OCR engines. OCR from camera snapshots may also be done through communication with remote servers
and linked to search engines, as is envisioned with
23half's
Blind users are already applying The vOICe MIDlet to determine
which side of a page has print on it. The soundscapes may also assist in aligning the view when taking pictures of
scenes or objects for remote sighted identification or guidance after submitting the resulting pictures through MMS.
Supplementary mobile phone based face and object recognition, such as with Neven Vision's
|
 For GPS-based navigation with your mobile phone, check out 
Vodafone Wayfinder OSS (free),
Loadstone GPS (free),
Wayviewer GPS (free),
amAze GPS (free),
Mobile Geo,
TomTom MOBILE and
"Seeing-eye
and navigation technologies mean more freedom for the blind", complementing GPS with
"camera-based vision technology that would mean centimeter-scale accuracy within
arm's length of its user... Dr. Meijer envisions the user of his device guided by GPS
and speech software to the door of a building, then prompted about the location of the knob
by the finely tuned camera view. He says the technology will allow a user to accurately
grasp a cup of coffee just set down on a table"). Nokia offers
Ovi Maps
for free turn-by-turn GPS navigation, but it is not yet known if this will include compatible
speech output and be accessible to blind users.
|
Java 3D and VRML/X3D?
Training for seeing with sound could benefit from mobile phone access to purely visual 3D virtual environments
encoded in VRML or X3D (XML). This future option will depend on the further development of compatible parsers
and Java 3D rendering support for mobile phones. The topic is related to The vOICe's existing PC-based support
for Accessible VRML/X3D for the Blind.
Development in this direction is currently taking place through the
See also the interactive online 3D Maze example. |
| Continuous backlight? Many Nokia phones do not provide MIDlet support for disabling the backlight timeout. Workaround: get FExplorer
and in the menu Options | Tools select the option "Turn light ON". Leave FExplorer
running, go to the application menu and start The vOICe.
|
| Is my phone MIDP-2.0 and (Camera) MMAPI compliant? It is often difficult to determine this with certainty from the available phone specifications. |
Now what if it is raining?
In order to be able to use The vOICe MIDlet in adverse weather conditions such as heavy rain,
you may consider using a
waterproof MP3 player bag
(eBay shop link).
Don't get mugged
In all uses, please stay aware that pointing the phone's camera at people who do not know you
or The vOICe, in public places or elsewhere, might trigger hostile reactions, for instance
because people may think that you are taking their photograph without their permission or
otherwise invading their privacy. Similar issues may apply when pointing the camera at certain
properties.
Credits
Hosting on ApGap
of The vOICe MIDlet files was kindly provided by
William Frantz.
Speech samples were kindly provided by
License restrictions
You may
Please read the disclaimer with respect to any damages or harm that use of The vOICe software might cause through inappropriate use. Getting adequate O&M training support for mobile orientation and mobility use is therefore considered extremely important, but even with that, 100% safety can never be guaranteed.
|
|
Common acronyms: Java ME (formerly J2ME) stands for Java Micro Edition, MIDP stands for
Mobile Information Device Profile, MMAPI stands for Mobile Media API, CLDC stands for
Connected Limited Device Configuration,
WMA stands for Wireless Messaging API, AMMS stands for Advanced Multimedia Supplements (JSR-234),
JTWI stands for Java Technology for the Wireless Industry, OTA stands for Over-the-Air.
Disclaimer: THIS PRODUCT (The vOICe MIDlet) IS PROVIDED AS IS WITHOUT ANY WARRANTY
OF ANY KIND. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, PETER B.L. MEIJER FURTHER
DISCLAIMS ALL WARRANTIES, INCLUDING WITHOUT LIMITATION ANY IMPLIED OR STATED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.
THE ENTIRE RISK ARISING OUT OF THE USE OR PERFORMANCE OF THIS PRODUCT AND DOCUMENTATION
REMAINS WITH RECIPIENT. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT
SHALL PETER B.L. MEIJER OR HIS SUPPLIERS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL,
DIRECT, INDIRECT, SPECIAL, PUNITIVE, RECURSIVE, OR OTHER DAMAGES WHATSOEVER (INCLUDING,
WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF
BUSINESS INFORMATION, PERSONAL INJURY, DISRUPTION OF FAMILY LIFE, OR OTHER PECUNIARY LOSS)
ARISING OUT OF THIS AGREEMENT OR THE USE OF OR INABILITY TO USE THE PRODUCT, EVEN IF
PETER B.L. MEIJER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. BECAUSE
SOME STATES/JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR
CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT APPLY TO THE RECIPIENT.
LET IT BE KNOWN THAT THE USER OF THIS PROGRAM HAS PAID PETER B.L. MEIJER THE SUM TOTAL
OF $0.00 FOR THE USE OF THIS PROGRAM. ANY DAMAGES AWARDED SHALL NOT BE IN EXCESS OF
SAID AMOUNT.
Supplemental information associated with disclaimer:
Concerning privacy of the observed: The vOICe MIDlet by default (i.e. in normal use)
does not store or transmit any video or snapshots; it converts live video frames to sound
and next discards both the video data and the generated sound data.
The vOICe MIDlet is intended for educational uses, while its effectiveness
or benefit has not been independently demonstrated and formally validated. It is not
intended to diagnose, treat, cure, or prevent any disease or condition. It does not
replace long cane and/or guide dog. Depending on national law and regulations, usage of
The vOICe technology may be subject to approval by an institutional review board. Also,
as with any audio-based system, permanent hearing loss may result when sounds are played
at high volume (> 80 dB, OSHA) for prolonged periods of time, while environmental sounds
can be partially masked even at a modest volume.
Also note that this is still a research project, while the software may contain bugs, so
there has to be a disclaimer with respect to any damages or harm
that use of this educational software might cause through inappropriate use, just like you
shouldn't attempt to dry your cat in the microwave oven. Never rely on The vOICe for your
safety. Getting professional rehabilitation support, including adequate O&M training
support for mobile orientation and mobility use, is considered extremely important when
moving beyond mere educational uses, but even with that, 100% safety can never be guaranteed.
Finally, some of the third-party product information may prove inaccurate, incomplete
or outdated as it reflects a best effort to describe selected developments in a complex
and highly dynamic market where adequate and reliable documentation is often missing or
hard to track down.
Common blind misspellings: v.o.i.c., vioce, voic, voyce, voyc.
Once started, The vOICe MIDlet will continuously grab and sound snapshots from the camera.
There are no connection costs while using it, because The vOICe MIDlet runs off-line.
Each camera snapshot is sounded via a polyphonic left-to-right scan through the snapshot while
associating height with pitch and brightness with loudness. For example, a bright
rising line on a dark background sounds as a rising pitch sweep, and a small bright
spot sounds as a short beep.
The vOICe MIDlet manual
"
