On cognitive relevance in automatic multimodal systems

This paper advances a formal approach about how to encode cognitive relevance into the design of interfaces of multimodal systems, with a focus on the automatic multimedia generation systems. We make a proposal that the optimality of the encoding of cognitive relevance depends on a proper formalization of the processes of modality allocation and modality combination. We describe how this can be done by adopting and encoding the principle of relevance based modality combination which evolved out of our formal analysis of modality allocations and modality combinations.