Mutation testing; Fault detection; Tool comparison; Human study; Real faults
Abstract :
[en] Mutation analysis is a well-studied, fault-based testing technique. It requires
testers to design tests based on a set of artificial defects. The defects help
in performing testing activities by measuring the ratio that is revealed by the
candidate tests. Unfortunately, applying mutation to real-world programs
requires automated tools due to the vast number of defects involved. In such a
case, the effectiveness of the method strongly depends on the peculiarities of
the employed tools. Thus, when using automated tools, their implementation
inadequacies can lead to inaccurate results. To deal with this issue, we
cross-evaluate four mutation testing tools for Java, namely PIT, muJava, Major
and the research version of PIT, PITRV, with respect to their fault-detection
capabilities. We investigate the strengths of the tools based on: a) a set of
real faults and b) manual analysis of the mutants they introduce. We find that
there are large differences between the tools’ effectiveness and demonstrate
that no tool is able to subsume the others. We also provide results indicating
the application cost of the method. Overall, we find that PITRV achieves the
best results. In particular, PITRV outperforms the other tools by finding 6%
more faults than the other tools combined.
Disciplines :
Computer science
Author, co-author :
Kintis, Marinos ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Papadakis, Mike ; University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
Papadopoulos, Andreas; Athens University of Economics and Business > Department of Informatics
Valvis, Evangelos; Athens University of Economics and Business > Department of Informatics
Malevris, Nicos; Athens University of Economics and Business > Department of Informatics
Le Traon, Yves ; University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
External co-authors :
yes
Language :
English
Title :
How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults