Assessing the Correctness of JVM Implementations

We present a study on a significantly large and varied set of freely available Java virtual machines implementations. The aim of the study is to assess their level of conformity to the structural constraints of the Java language specification. In order to do that we set up a refined framework for the automated generation of a conformance checking test suite. In the framework, the test cases are generated by model checking the Java language specifications in order to target each a specific constraint. Complete coverage of all the modeled constraints within a limited number of tests is obtained by means of a combinatorial covering technique. Results obtained from the test suite application are presented and discussed and proved the efficacy and validity of the underlying approach to validation.