Experience report: Assessing the reliability of an industrial avionics software: Results, insights and recommendations

Real Time Operating System for Avionics (RTOS4A) is responsible for providing an operating environment for avionics application software. Avionics software being safety-critical in nature poses several safety and reliability requirements on RTOS4A in addition to the requirements imposed by standards, for instance, DO-178B. Due to this reason, reliability assessment of RTOS4A is very critical to demonstrate confidence about its reliability to its relevant stakeholders. One common way of assessing reliability is by systematic analyses of testing data such as number of tests, number of failures, and coverage using appropriate statistical tests. In this paper, we report our experience of assessing the reliability of an industrial RTOS4A based on testing data collected for 17 months on eight continuous releases. We studied correlation among various measures including: Testing Effort Measures (e.g., complexity of test cases), Testing Effectiveness Measures (e.g., number of failures), and Complexity Measures (e.g., number of functions in a release) and provide in this paper a set of recommendations to assess the reliability of RTOS4A, which serve as guidelines to practitioners in the domain of RTOS4A.