Automatic indexing of cooking video by using caption-recipe alignment

In recent years, the internet use has led to rapid increases in multimedia resources. In this regard, handling multimedia data has become an important issue in information retrieval fields, and multimedia indexing plays an essential role in handling multimedia data. Textual metadata plays an important role in describing and analyzing multimedia resources. However, because only few multimedia resources provide textual metadata, there is a need for methods to generate the metadata. This paper proposes an automatic method for indexing cooking videos using textual data. The proposed method consists of web crawling for cooking videos and food recipes, the extraction of time-stamped text captions, and caption-recipe alignment using a dynamic programming technique. With information on caption-recipe alignment, automatic indexing is performed for cooking videos. The experimental results show that the proposed approach achieves highly accurate indexing. The user interface allows users to easily move to specific pinpoints with indexed videos.