Open Access Open Access  Restricted Access Subscription Access

Speaking Louder than Words with Pictures Across Languages

Andrew Finch, Wei Song, Kumiko Tanaka-Ishii, Eiichiro Sumita

Abstract


In this article, we investigate the possibility of cross-language communication using a synergy of words and pictures on mobile devices. Communicating with only pictures is in itself a very powerful strategy, but is limited in expressiveness. On the other hand, words can express everything you could wish to say, but they are cumbersome to work with on mobile devices, and need to be translated in order for their meaning to be understood. Automatic translations can contain errors that pervert the communication process, and this may undermine the users’ confidence when expressing themselves across language barriers. Our idea is to create a user interface for cross-language communication that uses pictures as the primary mode of input, and words to express the detailed meaning. This interface creates a visual process of communication that occurs on two heterogeneous channels that can support each other. We implemented this user interface as application on the Apple iPad tablet, and performed a set of experiments to determine its usefulness as a translation aid for travellers.

Full Text:

PDF


DOI: http://dx.doi.org/10.1609/aimag.v34i2.2471

Copyright © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.