A monitoring and testing framework for critical off-the-shelf applications and services

One of the biggest verification and validation challenges is the definition of approaches and tools to support system assessment while minimizing costs and delivery time. This includes the integration of OTS software components in critical systems that must undergo proper certification or approval processes. In the particular case of testing, due to the differences and peculiarities of components, developers often build ad-hoc and poorly-reusable testing tools, which results in increased time and costs. This paper introduces a framework for testing and monitoring of critical OTS applications and services. The framework includes i) a box that is instrumented for monitoring OS and application level variables, ii) an adaptable toolset for testing the target components, and iii) tools for data storing, retrieval and analyzes. A prototype of the framework is under development, and future testing scenarios are designed to show the applicability and effectiveness of the framework.