|
Use stereo headphones for best results! You'll perceive better sound quality and intuitive left-to-right stereo panning. The vOICe web app turns live camera views into corresponding soundscapes. It is an HTML5 web app that runs inside your browser, for use on any device or platform including Android, iOS, Windows and Linux. It could therefore also be called a universal app, and it implements general visual-to-auditory sensory substitution for the totally blind. It requires a browser with WebRTC support, such as Firefox, Google Chrome or Microsoft Edge, or Safari on Apple devices. The vOICe web app is a so-called progressive web app (PWA), such that it can even be started and used when offline, and can be installed and run as a home screen app with suitable browsers. This web page offers usage notes and describes accessibility issues for blind users.
At startup, seconds after loading The vOICe web app web page, a popup dialog may appear that asks for your permission to access the camera. The dialog may disappear if you accidentally click outside this dialog or move focus to another window, and then it returns only after reloading The vOICe web app web page. To access the popup in Firefox for Windows, press Alt-S, followed by the Tab and Enter key to grant access to the camera. Alternatively, you can repeatedly press F6 until the dialog gets focus, and then Tab around. In Google Chrome for Windows, after pressing the Enter key for the URL of The vOICe web app in the browser address bar, press the Tab and Enter key to grant access to the camera. Bandwidth requirements in running The vOICe web app are extremely low, because almost all processing including soundscape generation is done locally in your browser and not "in the cloud", such that communication overhead is negligible. In fact you can disconnect your Internet connection after starting The vOICe web app and it will continue to sound live camera images. Moreover, by virtue of being a progressive web app, you can even use The vOICe web app as an offline app in places where you have no Internet connection.
The vOICe web app has been tested or reported to run on Windows in at least Firefox,
Google Chrome, Opera and Microsoft Edge, on Android in at least Firefox, Chrome and
Opera, on iOS in at least Safari, on ubuntu Linux in at least Firefox, on Raspbian
(Raspberry Pi) in IceWeasel (a Firefox port), and on OS X (MacBook Pro) in Google Chrome.
The vOICe web app has even been shown to run in the Helio browser on the Magic Leap One
augmented reality glasses. The vOICe web app supports several query parameters that be appended to the app URL in order to automatically set a special operating mode at startup. Some of these modes mostly target sighted users, by simulating forms of blindness on the visual display while maintaining the full soundscape. The URL format is to first append a question mark '?' followed by one or more query parameters separated by ampersands '&'. For example,
sets a maximized screen view, while applies the (left field) hemianopia simulator, in VR mode, with a Russian interface. Supported query parameters include "hl=ru" (without the double quotes) to set the host language to Russian, and "hl=en", "hl=fr", "hl=cn", "hl=jp", "hl=de", "hl=nl" for English, French, Chinese, Japanese, German and Dutch, respectively. The "hemianopia" or "hemianopiaL" query parameter activates a left hemifield hemianopia simulator, and "hemianopiaR" activates a right hemifield hemianopia simulator. The "cataracts" query parameter activates a severe cataracts simulator, and "vr" sets a maximized virtual reality view with separate left eye and right eye pane for use with a virtual reality headset. Note that you may need to toggle the "large" checkbox off and on manually for a truly full-screen view, because most browsers require user interaction to enable full-screen viewing. In case you have multiple cameras, it is also possible to select another camera at startup using query parameters like "camera=1" or "camera=2" (the default camera being "camera=0"). Other query parameters are "mute", "edges", "negative", "zoom", "mono", "volume", "viewport" and "preview". The "volume" query parameter sets the audio volume for soundscapes and speech through an integer in the range between 0 and 100 (default is 100). For example,
Especially in VR mode when using a large smartphone in a VR headset, the landscape width of your smartphone screen may be too wide for your eyes to adjust to binocular viewing, because human eyes can converge but usually not diverge. If your VR headset lacks adequate adjustment options, you can set the horizontal viewport of The vOICe web app to a percentage value between 10 and 100 (default is 100). For example, you can set it to 70% of the maximum width using
This downscales the horizontal view and adds black borders to the left and right side.
You may need to experiment a bit to find which value best matches your smartphone screen
size, VR headset, and your interpupillary distance (the distance between your eyes). which behaves very similar to the "large" query parameter, but instead of a coarse large grayscale view it shows a higher resolution large color preview on the screen for interpretation and description by an AI model while at the same time hearing The vOICe soundscapes, such that you get the best of both worlds. You need to give the AI model permission to share and stream the entire screen and permission to use the microphone. You can then give the AI model spoken instructions such as "describe the view", and it will respond in a few seconds. Note that large AI models typically run in the cloud and therefore fail when there is no Internet connection, whereas The vOICe web app continues without any Internet connection. The preview mode can be turned off by clicking or tapping the camera view, but it can also be toggled by using the "AI" checkbox. The main reason for having this "AI" checkbox alternative is that stand-alone installed web apps cannot read URL query parameters because there is no address bar, but they can respond to checkboxes. Despite the "AI" name of the checkbox it performs no AI function of its own but merely offers a higher quality view for better interpretation by external AI models.
Beware that query parameters are case sensitive.
Depending on your browser and platform, keyboard shortcuts may be available for some
common functions. In Google Chrome you'd press the Alt key in combination with the
shortcut key, while in Firefox or Microsoft Edge you'd press Alt+Shift in combination
with the shortcut key. For example, Alt+m toggles mute in Chrome while Alt+Shift+m
does the same in Firefox and Edge.
In this manner there are shortcut keys 'm' for pressing the mute button, 'e' for edges,
'n' for negative, 'z' for zoom, 's' for mono, 't' for torch, 'r' for reset, 'p' for
preview, and 'q' for the record button, as well as '+' for volume up and '-' for volume down. If you want a sighted person to give verbal feedback on your live camera view, and provided that your platform supports it, you may after installing a video conferencing app such as Microsoft Teams and making a connection to this sighted person activate the app's screen sharing option before starting The vOICe web app. This should let the sighted person see The vOICe web app window with the live camera view and let him or her talk with you about what is showing in the live camera view. Screen sharing provides a good workaround for the fact that only one app at a time can access the camera. Known issues
Web app updates:
A new version of The vOICe web app may not immediately show up in your browser after
you open its web page. This is due to caching mechanisms where a web app gets updated
in the background while running. After visiting its web page, close and open your browser,
or the tab that the web app is running in, and visit the web page again. Then you should
have the latest version. Some browsers such as Google Chrome need an extra nudge to
update upon request: for Chrome for Android visit chrome://webapks and press the Update
button, if present, or try visiting chrome://restart, or else it may take
Display auto-rotate, screen rotation lock: On devices such as phones and tablets that can be used in multiple orientations like portrait and landscape, make sure that in the device Settings for the display you have the auto-rotate option turned on, such that the camera view is always scanned from left to right. The feature goes under different names, and for instance on the iPhone or iPad it is known as the "screen rotation lock", which in that case needs to be turned off. This is particularly important when you start making your phone head-mounted in a VR headset for hands-free immersive use, because then your phone will be in landscape orientation and the web app should follow that orientation. Visual scanline: If the soundscapes appear choppy, it may help to uncheck the scanline checkbox, in order to maximize performance and minimize audible artifacts. The moving scanline is automatically turned off when the fast scan rate is selected or when the full-screen preview mode for AI models is active. Apple iPhone and iPad: The vOICe web app requires iOS 11 or later. Moreover, it may appear muted until you press the Play button in the The vOICe web app, or press "Allow" in a popup reading "seeingwithsound.com would like to access the camera". This is needed because Apple WebKit requires user interaction before starting video with sound. In case you still get no sound, check if you need to grant permission elsewhere. To avoid that you have to grant permission for camera access upon every run or refresh of The vOICe web app, go to the Safari address bar, "Website Settings", "Camera", and select "Allow" instead of "Ask". To prevent a screen timeout you may also wish to temporarily set your screen lock (Auto-Lock) time to "Never" in your device Settings | Display & Brightness. This "Never" option may be missing on managed devices (MDM - Mobile Device Management). Although The vOICe web app is designed to work as a progressive web app (PWA), iOS 11 does not yet support camera access in home screen web apps. It is not known if or when Apple will fix this. You must therefore load The vOICe web app in Safari. It does not work as a stand-alone home screen web app under iOS. Turn off Low Power Mode if it is on: The vOICe web app may not work if your iOS device is in Low Power Mode. If The vOICe web app then still does not work, you may need to install and use Firefox Nightly or Chrome Canary. Apple Vision Pro, Meta Quest 3, Meta Quest 2, Oculus Quest and Rift: The vOICe web app unfortunately does not work on these devices because they do not support (i.e. allow) pixel-level access to individual camera frames, often in the name of privacy, thereby also preventing use by computer vision applications for blind accessibility purposes. "This site can't be reached": In rare circumstances, The vOICe web app can get corrupted during installation, for instance due to a server glitch, resulting in the aforementioned message popping up repeatedly even after any server problems are long gone. The remedy is then to uninstall and re-install The vOICe web app. Flashlight: Some browsers support toggling a camera's flashlight (torch) on and off. If so, The vOICe web app will display a "torch" button. However, even if the browser supports it, the camera need not, in which case use of the torch button will have no effect. Video recording: The video recording feature toggled by the "record" button is highly experimental, because it depends on browser support that is still rather immature. In particular, video and audio in the resulting vOICe*.webm video files may be poorly synchronized. You can open .webm files in your browser for playback. A UNIX-style timestamp number is automatically added to the filenames, and you can visit the timestamp web page to extract a human-readable date and time from the filename. Video recording is only intended for taking short video clips of up to a minute or so, or else the browser may run out of memory. Upon stopping the recording, you may either get a file requester for saving the video file or you may find the video file already saved in your browser's download folder. This depends on browser security settings and policies. Video recording is not currently supported by the Microsoft Edge browser nor by the Apple Safari browser, and the "record" button will remain hidden in these browsers. License restrictions You may freely and without cost apply The vOICe web app for personal or academic use. You will not charge a fee or request donations for The vOICe web app, nor distribute or include The vOICe web app in or with commercial products, nor modify or reverse engineer The vOICe web app. You will not use The vOICe web app for commercial purposes. You may save and privately store a local zipped copy of The vOICe web app webapp.zip to ensure that in case of political turmoil it can be revived at a later stage even if all its websites or even the whole Internet Archive would go down.
Disclaimer: THIS PRODUCT (The vOICe web app) IS PROVIDED AS IS WITHOUT ANY WARRANTY
OF ANY KIND. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, PETER B.L. MEIJER FURTHER
DISCLAIMS ALL WARRANTIES, INCLUDING WITHOUT LIMITATION ANY IMPLIED OR STATED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.
THE ENTIRE RISK ARISING OUT OF THE USE OR PERFORMANCE OF THIS PRODUCT AND DOCUMENTATION
REMAINS WITH RECIPIENT. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT
SHALL PETER B.L. MEIJER OR HIS SUPPLIERS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL,
DIRECT, INDIRECT, SPECIAL, PUNITIVE, RECURSIVE, OR OTHER DAMAGES WHATSOEVER (INCLUDING,
WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF
BUSINESS INFORMATION, PERSONAL INJURY, DISRUPTION OF FAMILY LIFE, OR OTHER PECUNIARY LOSS)
ARISING OUT OF THIS AGREEMENT OR THE USE OF OR INABILITY TO USE THE PRODUCT, EVEN IF
PETER B.L. MEIJER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. BECAUSE
SOME STATES/JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR
CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT APPLY TO THE RECIPIENT.
LET IT BE KNOWN THAT THE USER OF THIS PROGRAM HAS PAID PETER B.L. MEIJER THE SUM TOTAL
OF $0.00 FOR THE USE OF THIS PROGRAM. ANY DAMAGES AWARDED SHALL NOT BE IN EXCESS OF
SAID AMOUNT.
Supplemental information associated with disclaimer:
Concerning privacy of the observed: The vOICe web app by default (i.e. in normal use)
does not store or transmit any video or snapshots; it converts live video frames to sound
and next discards both the video data and the generated sound data.
The vOICe web app is intended for educational uses, while its effectiveness
or benefit has not been independently demonstrated and formally validated. It is not
intended to diagnose, treat, cure, or prevent any disease or condition. It does not
replace long cane and/or guide dog. Depending on national law and regulations, usage of
The vOICe technology may be subject to approval by an institutional review board. Also,
as with any audio-based system, permanent hearing loss may result when sounds are played
at high volume (> 80 dB, OSHA) for prolonged periods of time, while environmental sounds
can be partially masked even at a modest volume.
Also note that this is still a research project, while the software may contain bugs, so
there has to be a disclaimer with respect to any damages or harm
that use of this educational software might cause through inappropriate use, just like you
shouldn't attempt to dry your cat in the microwave oven. Never rely on The vOICe for your
safety. Getting professional rehabilitation support, including adequate O&M training
support for mobile orientation and mobility use, is considered extremely important when
moving beyond mere educational uses, but even with that, 100% safety can never be guaranteed.
Finally, some of the third-party product information may prove inaccurate, incomplete
or outdated as it reflects a best effort to describe selected developments in a complex
and highly dynamic market where adequate and reliable documentation is often missing or
hard to track down.
|
