Test your tests in CI

DevOps puts a huge emphasis on test quality, and statement coverage is the most popular metric to assess this quality. Yet, this metric is known to be a poor indicator of a test suite’s capacity to detect bugs.

This talk will introduce and illustrate the benefits of another technique for test suite quality: the mutation analysis. The intuition is simple: to determine if a test suite can detect bugs in a program, simply inject bugs in this program and check if the tests can detect them.

In this talk, I will cover :

  • the key principles of mutation analysis.

  • the Descartes tool for mutation analysis on large Java projects

  • concrete examples of weaknesses in the test suite that are missed by statement coverage and revealed by mutation analysis

  • the integration of mutation analysis in the CI



Benoit Baudry

Benoit Baudry is a Professor of software technology at the KTH Royal Institute of Technology in Stockholm, Sweden. His research focuses on novel techniques for automatic test amplification and build ...