Reference : What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

	Document type :	Scientific congresses, symposiums and conference proceedings : Paper published in a book
	Discipline(s) :	Engineering, computing & technology : Computer science
	To cite this reference:	http://hdl.handle.net/10993/55849

Title :	What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness
Language :	English
Author, co-author :	Habchi, Sarra []
	Haben, Guillaume [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal >]
	Sohn, Jeongju [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal >]
	Franci, Adriano []
	Papadakis, Mike [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
	Cordy, Maxime [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal >]
	Le Traon, Yves [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal >]
Publication date :	Oct-2022
Main document title :	What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness
Peer reviewed :	Yes
Event name :	38th International Conference on Software Maintenance and Evolution
Event date :	from 02-10-2022 to 07-10-2022
Keywords :	[en] Software Testing ; Flaky Tests ; Fault Localisation
Abstract :	[en] Flaky tests are defined as tests that manifest non-deterministic behaviour by passing and failing intermittently for the same version of the code. These tests cripple continuous integration with false alerts that waste developers' time and break their trust in regression testing. To mitigate the effects of flakiness, both researchers and industrial experts proposed strategies and tools to detect and isolate flaky tests. However, flaky tests are rarely fixed as developers struggle to localise and understand their causes. Additionally, developers working with large codebases often need to know the sources of non-determinism to preserve code quality, i.e. avoid introducing technical debt linked with non-deterministic behaviour, and to avoid introducing new flaky tests. To aid with these tasks, we propose re-targeting Fault Localisation techniques to the flaky component localisation problem, i.e. pinpointing program classes that cause the non-deterministic behaviour of flaky tests. In particular, we employ Spectrum-Based Fault Localisation (SBFL), a coverage-based fault localisation technique commonly adopted for its simplicity and effectiveness. We also utilise other data sources, such as change history and static code metrics, to further improve the localisation. Our results show that augmenting SBFL with change and code metrics ranks flaky classes in the top-1 and top-5 suggestions, in 26% and 47% of the cases. Overall, we successfully reduced the average number of classes inspected to locate the first flaky class to 19% of the total number of classes covered by flaky tests. Our results also show that localisation methods are effective in major flakiness categories, such as concurrency and asynchronous waits, indicating their general ability to identify flaky components.
Research centres :	Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SerVal - Security, Reasoning & Validation
Permalink :	http://hdl.handle.net/10993/55849
DOI :	10.1109/ICSME55016.2022.00039

	File(s) associated to this reference
	Fulltext file(s): File Commentary Version Size Access Open access ICSME_2022.pdf Author preprint 308.18 kB View/Open

All documents in ORBi^lu are protected by a user license.

 

University of Luxembourg Library |  Feedback |  Legal notices Site Map