An online benchmark system for image processing algorithms

When we have to solve a specific problem, we can find various approaches and recommendations provided by individual partners. A fair competition among the partners can be reached only if there are possibilities to compare their results in objective ways. When the task relates to data processing, such an objective evaluation can be reached using ground truth through annotated publicly available datasets. Moreover, in case of available ground truth, efficient learning-based algorithms could enter the specific field. However, in several application fields such ground truth data are missing or incomplete making the participants report their results on own datasets. In this paper, we present a platform that will aid researchers in developing functional components (image processing algorithms) for specific (detection and classification) tasks by providing training data sets with ground truth/gold standard and with providing evaluation of the outputs of the algorithms in an objective manner under identical conditions on separate test data sets using standardized measures. This quantitative evaluation will allow the comparison and ranking of the algorithms with the same functionalities. Different participants will contribute datasets and data validation to the platform.