Web-Based Personal Digital Photo Collections: Multimodal Retrieval

When personal photo collections get large, retrieval of specific photos or sets of photos becomes difficult mainly due to the fairly primitive means by which they are organized. Commercial photo handling systems help but often have only elementary searching features. For many years, graphical user interfaces (GUls) have become the user interface of choice in web applications. Moving beyond mouse and keyboard, multimodal interfaces are expected to be more transparent, flexible, efficient and powerful for human-computer interaction. In this paper, we describe an interactive webbased photo retrieval system that enables personal digital photo users to accomplish photo browsing by using multimodal interaction. This system not only enables users to use mouse click input modalities but also speech input modality to browse their personal digital photos in the World Wide Web (WWW) environment. The prototype system and it architecture utilize web technology which was built using web programming scripting (JavaScript, XHTML, ASP, XML based mark-up language) and image database in order to achieve its objective. All prototype programs and data files including the user s photo repository, profiles, dialogues, grammars, prompt, and retrieval engine are stored and located in the web server. Our approach consists of human-computer speech dialogue based on photo browsing of image content by four main categories (Who? What? When? and Where?). Our user study with 20 digital photo users showed that the participants reacted positively to their experience with the system interactions.