Toward 3D scene understanding via audio-description : kinect-iPad fusion for the visually impaired

Gomez, Juan Diego (Computer Science Department, University of Geneva, Carouge, Switzerland) ; Mohammed, Sinan (Computer Science Department, University of Geneva, Carouge, Switzerland) ; Bologna, Guido (Computer Science Department, University of Geneva, Carouge, Switzerland) ; Pun, Thierry (Computer Science Department, University of Geneva, Carouge, Switzerland)

Microsoft's Kinect 3-D motion sensor is a low cost 3D camera that provides color and depth information of indoor environments. In this demonstration, the functionality of this fun-only camera accompanied by an iPad's tangible interface is targeted to the benefit of the visually impaired. A computer-vision-based framework for real time objects localization and for their audio description is introduced. Firstly, objects are extracted from the scene and recognized using feature descriptors and machine-learning. Secondly, the recognized objects are labeled by instruments sounds, whereas their position in 3D space is described by virtual space sources of sound. As a result, the scene can be heard and explored while finger-triggering the sounds within the iPad, on which a top-view of the objects is mapped. This enables blindfolded users to build a mental occupancy grid of the environment. The approach presented here brings the promise of efficient assistance and could be adapted as an electronic travel aid for the visually-impaired in the near future.


Note: BOLOGNA, Guido est un chercheur à la HES-SO, HEPIA, depuis 2011.


Keywords:
Conference Type:
short paper
Faculty:
Ingénierie et Architecture
School:
HEPIA - Genève
Institute:
inIT - Institut d'Ingénierie Informatique et des Télécommunications
Publisher:
Dundee, Scotland, UK, 24-26 October 2011
Date:
2011-10
Dundee, Scotland, UK
24-26 October 2011
Pagination:
2 p.
Published in:
Proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (ASSESTS'11), 24-26 October 2011, Dundee, Scotland, UK
DOI:
ISBN:
9781450309202
Appears in Collection:

Note: The status of this file is: restricted


 Record created 2020-06-26, last modified 2020-07-17

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)