An approach towards benchmarking of table structure recognition results

After we developed a model free table recognition system we had the desire to automatically register the effect of minor changes to parameters upon the overall performance quality of our system in order to tune parameters. Therefore we developed a complete benchmarking environment, containing a user front-end to acquire ground truth data as well as mechanisms to evaluate the quality of the recognition results. The tasks involved in the analysis systems were the locating of table regions, identification of cells and mapping of cells to rows and columns. This paper presents our approach towards the comparison of recognition results with the ground truth. The established definitions of recall and precision did not meet our requirements, as we wanted to register even smallest improvements (or changes in general) in the results, even when both results were imperfect. We therefore extended the measures recall and precision in order to deal with recognition probabilities of objects rather than just with Boolean values.