Article (Scientific journals)
How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults
Kintis, Marinos; Papadakis, Mike; Papadopoulos, Andreas et al.
2018In Empirical Software Engineering
Peer Reviewed verified by ORBi
 

Files


Full Text
Kintis_EMSE_2017.pdf
Publisher postprint (6.19 MB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
Mutation testing; Fault detection; Tool comparison; Human study; Real faults
Abstract :
[en] Mutation analysis is a well-studied, fault-based testing technique. It requires testers to design tests based on a set of artificial defects. The defects help in performing testing activities by measuring the ratio that is revealed by the candidate tests. Unfortunately, applying mutation to real-world programs requires automated tools due to the vast number of defects involved. In such a case, the effectiveness of the method strongly depends on the peculiarities of the employed tools. Thus, when using automated tools, their implementation inadequacies can lead to inaccurate results. To deal with this issue, we cross-evaluate four mutation testing tools for Java, namely PIT, muJava, Major and the research version of PIT, PITRV, with respect to their fault-detection capabilities. We investigate the strengths of the tools based on: a) a set of real faults and b) manual analysis of the mutants they introduce. We find that there are large differences between the tools’ effectiveness and demonstrate that no tool is able to subsume the others. We also provide results indicating the application cost of the method. Overall, we find that PITRV achieves the best results. In particular, PITRV outperforms the other tools by finding 6% more faults than the other tools combined.
Disciplines :
Computer science
Author, co-author :
Kintis, Marinos ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Papadakis, Mike ;  University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
Papadopoulos, Andreas;  Athens University of Economics and Business > Department of Informatics
Valvis, Evangelos;  Athens University of Economics and Business > Department of Informatics
Malevris, Nicos;  Athens University of Economics and Business > Department of Informatics
Le Traon, Yves ;  University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
External co-authors :
yes
Language :
English
Title :
How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults
Publication date :
2018
Journal title :
Empirical Software Engineering
ISSN :
1573-7616
Publisher :
Springer Science & Business Media B.V.
Peer reviewed :
Peer Reviewed verified by ORBi
Available on ORBilu :
since 24 March 2018

Statistics


Number of views
188 (9 by Unilu)
Number of downloads
624 (5 by Unilu)

Scopus citations®
 
44
Scopus citations®
without self-citations
37
OpenCitations
 
37
WoS citations
 
40

Bibliography


Similar publications



Contact ORBilu