Real-time eating action recognition system on a smartphone

Recently, many mobile applications to record everyday meals for dieting have been popular. Some of them can recognize names of food items in meals by only taking photos. However, such image-recognition-based food recording systems requires taking meal photos before eating, which are not applicable for the meals in which the amount of food to be eaten is not decided before eating such as large platter for sharing and barbecue-style dishes. Then in this paper, we propose a mobile real-time eating action recognition system. It continuously recognizes user´s eating action and estimates the categories of eaten food items during mealtime. With this system, we can get to know total amount of eaten food items, and can calculate total calories of eaten foods even for the meals where the amount of foods to be eaten is not decided before starting eating. The system implemented on a smartphone continuously monitor eating actions during mealtime. It detects the moment when a user eats foods, extract food regions near the user´s mouth and classify them. In the experiments, we implemented a mobile system the target of which is Japanesestyle “Yakiniku” where people eat meats and vegetables while grilling. It can recognize five different kinds of ingredients for “Yakiniku” such as beef, carrot and pumpkin in the real-time way. It has achieved 74.8% classification rate, and was evaluated as being superior to the baseline system which employed no eating action recognition by user study.