A comparison of database fault detection capabilities using mutation testing

Mutation testing involves systematically generating and introducing faults into an application to improve testing. A quasi-experimental study is reported comparing the fault-detection capabilities of realworld database application test suites to those of an SQL vendor test suite (NIST SQL) based on mutation scores. The higher the mutation score the more successful the test suite will be at detecting faults. The SQLMutation tool was used to generate query mutants from beginner-level sample schemas obtained from three popular real-world database test suite vendors - MySQL, SQL Server, and Oracle. Four SQLMutation operators were applied to both realworld and NIST SQL vendor compliance test suites - SQL Clause (SC), Operator Replacement (OR), NULL (NL) and Identifier Replacement (IR). Two mutation operators, SC and NL generated significantly lower mutation scores in real-world test suites than for those in the vendor test suite. The IR operator generated significantly higher mutation scores in real-world test suites than for those in the vendor test suite. The OR operator produced roughly the same mutation scores in both the real-world and vendor test suites.