References of "Arcuri, Andrea 4000A815"
     in
Bookmark and Share    
Full Text
Peer Reviewed
See detailSeeding Strategies in Search-Based Unit Test Generation
Rojas, Jose Miguel; Fraser, Gordon; Arcuri, Andrea UL

in SOFTWARE TESTING, VERIFICATION AND RELIABILITY (STVR) (2016)

Search-based techniques have been applied successfully to the task of generating unit tests for object-oriented software. However, as for any meta-heuristic search, the efficiency heavily depends on many ... [more ▼]

Search-based techniques have been applied successfully to the task of generating unit tests for object-oriented software. However, as for any meta-heuristic search, the efficiency heavily depends on many factors; seeding, which refers to the use of previous related knowledge to help solve the testing problem at hand, is one such factor that may strongly influence this efficiency. This paper investigates different seeding strategies for unit test generation, in particular seeding of numerical and string constants derived statically and dynamically, seeding of type information, and seeding of previously generated tests. To understand the effects of these seeding strategies, the results of a large empirical analysis carried out on a large collection of open source projects from the SF110 corpus and the Apache Commons repository are reported. These experiments show with strong statistical confidence that, even for a testing tool already able to achieve high coverage, the use of appropriate seeding strategies can further improve performance. [less ▲]

Detailed reference viewed: 124 (32 UL)
Full Text
Peer Reviewed
See detailJava Enterprise Edition Support in Search-Based JUnit Test Generation
Arcuri, Andrea UL; Fraser, Gordon

in Java Enterprise Edition Support in Search-Based JUnit Test Generation (2016)

Many di fferent techniques and tools for automated unit test generation target the Java programming languages due to its popular- ity. However, a lot of Java's popularity is due to its usage to develop ... [more ▼]

Many di fferent techniques and tools for automated unit test generation target the Java programming languages due to its popular- ity. However, a lot of Java's popularity is due to its usage to develop enterprise applications with frameworks such as Java Enterprise Edition (JEE) or Spring. These frameworks pose challenges to the automatic gen- eration of JUnit tests. In particular, code units (\beans") are handled by external web containers (e.g., WildFly and GlassFish). Without consid- ering how web containers initialize these beans, automatically generated unit tests would not represent valid scenarios and would be of little use. For example, common issues of bean initialization are dependency in- jection, database connection, and JNDI bean lookup. In this paper, we extend the EvoSuite search-based JUnit test generation tool to provide initial support for JEE applications. Experiments on 247 classes (the JBoss EAP tutorial examples) reveal an increase in code coverage, and demonstrate that our techniques prevent the generation of useless tests (e.g., tests where dependencies are not injected). [less ▲]

Detailed reference viewed: 104 (8 UL)
Full Text
Peer Reviewed
See detailEvolutionary Robustness Testing of Data Processing Systems using Models and Data Mutation
Di Nardo, Daniel UL; Pastore, Fabrizio UL; Arcuri, Andrea UL et al

in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (2015, November)

System level testing of industrial data processing software poses several challenges. Input data can be very large, even in the order of gigabytes, and with complex constraints that define when an input ... [more ▼]

System level testing of industrial data processing software poses several challenges. Input data can be very large, even in the order of gigabytes, and with complex constraints that define when an input is valid. Generating the right input data to stress the system for robustness properties (e.g. to test how faulty data is handled) is hence very complex, tedious and error prone when done manually. Unfortunately, this is the current practice in industry. In previous work, we defined a methodology to model the structure and the constraints of input data by using UML class diagrams and OCL constraints. Tests were automatically derived to cover predefined fault types in a fault model. In this paper, to obtain more effective system level test cases, we developed a novel search-based test generation tool. Experiments on a real-world, large industrial data processing system show that our automated approach can not only achieve better code coverage, but also accomplishes this using significantly smaller test suites. [less ▲]

Detailed reference viewed: 291 (32 UL)
Full Text
Peer Reviewed
See detailImproving the Performance of OCL Constraint Solving with Novel Heuristics for Logical Operations: A SearchBased Approach
Ali, Shaukat; Iqbal, Zohaib; Khalid, Maham et al

in Empirical Software Engineering (2015)

A common practice to specify constraints on the Unified Modeling Language (UML) models is using the Object Constraint Language (OCL). Such constraints serve various purposes, ranging from simply providing ... [more ▼]

A common practice to specify constraints on the Unified Modeling Language (UML) models is using the Object Constraint Language (OCL). Such constraints serve various purposes, ranging from simply providing precise meaning to the models to supporting complex verification and validation activities. In many applications, these constraints have to be solved to obtain values satisfying the constraints, for example, in the case of modelbased testing (MBT) to generate test data for the purpose of generating executable test cases. In our previous work, we proposed novel heuristics for various OCL constructs to efficiently solve them using search algorithms. These heuristics are enhanced in this paper to further improve the performance of OCL constraint solving. We performed an empirical evaluation comprising of three case studies using three search algorithms: Alternating Variable Method (AVM), (1+1) Evolutionary Algorithm (EA), and a Genetic Algorithm (GA) and in addition Random Search (RS) was used as a comparison baseline. In the first case study, we evaluated each heuristics using carefully designed artificial problems. In the second case study, we evaluated the heuristics on various constraints of Cisco’s Video Conferencing Systems defined to support MBT. Finally, the third case study is about EURent Car Rental specification and is obtained from the literature. The results of the empirical evaluation showed that (1+1) EA and AVM with the improved heuristics significantly outperform the rest of the algorithms. [less ▲]

Detailed reference viewed: 90 (19 UL)
Full Text
Peer Reviewed
See detailDo Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges
Shamshiri, Sina; Just, Rene; Rojas, Jose Miguel et al

in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2015)

Rather than tediously writing unit tests manually, tools can be used to generate them automatically — sometimes even resulting in higher code coverage than manual testing. But how good are these tests at ... [more ▼]

Rather than tediously writing unit tests manually, tools can be used to generate them automatically — sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting faults. Although 55.7% of the faults were found by automatically generated tests overall, only 19.9% of the test suites generated in our experiments actually detected a fault. By studying the performance and the problems of the individual tools and their tests, we derive insights to support the development of automated unit test generators, in order to increase the fault detection rate in the future. These include 1) improving coverage obtained so that defective statements are actually executed in the first instance, 2) techniques for propagating faults to the output, coupled with the generation of more sensitive assertions for detecting them, and 3) better simulation of the execution environment to detecting faults that are dependent on external factors, for example the date and time. [less ▲]

Detailed reference viewed: 132 (7 UL)
Full Text
Peer Reviewed
See detailCombining Multiple Coverage Criteria in Search-Based Unit Test Generation
Rojas, Miguel; Campos, Jose; Vivanti, Mattia et al

in Symposium on Search-Based Software Engineering (SSBSE) (2015)

Automated test generation techniques typically aim at max- imising coverage of well-established structural criteria such as statement or branch coverage. In practice, generating tests only for one speci c ... [more ▼]

Automated test generation techniques typically aim at max- imising coverage of well-established structural criteria such as statement or branch coverage. In practice, generating tests only for one speci c criterion may not be su cient when testing object oriented classes, as standard structural coverage criteria do not fully capture the properties developers may desire of their unit test suites. For example, covering a large number of statements could be easily achieved by just calling the main method of a class; yet, a good unit test suite would consist of smaller unit tests invoking individual methods, and checking return values and states with test assertions. There are several di erent properties that test suites should exhibit, and a search-based test generator could easily be extended with additional tness functions to capture these properties. However, does search-based testing scale to combinations of multiple cri- teria, and what is the e ect on the size and coverage of the resulting test suites? To answer these questions, we extended the EvoSuite unit test generation tool to support combinations of multiple test criteria, de ned and implemented several di erent criteria, and applied combinations of criteria to a sample of 650 open source Java classes. Our experiments suggest that optimising for several criteria at the same time is feasible without increasing computational costs: When combining nine di erent criteria, we observed an average decrease of only 0.4% for the constituent coverage criteria, while the test suites may grow up to 70%. [less ▲]

Detailed reference viewed: 85 (8 UL)
Full Text
Peer Reviewed
See detailGenerating TCP/UDP Network Data for Automated Unit Test Generation
Arcuri, Andrea UL; Fraser, Gordon; Galeotti, Juan Pablo

in 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE) (2015)

Although automated unit test generation techniques can in principle generate test suites that achieve high code coverage, in practice this is often inhibited by the dependence of the code under test on ... [more ▼]

Although automated unit test generation techniques can in principle generate test suites that achieve high code coverage, in practice this is often inhibited by the dependence of the code under test on external resources. In particular, a common problem in modern programming languages is posed by code that involves networking (e.g., opening a TCP listening port). In order to generate tests for such code, we describe an approach where we mock (simulate) the networking interfaces of the Java standard library, such that a search-based test generator can treat the network as part of the test input space. This not only has the benefit that it overcomes many limitations of testing networking code (e.g., different tests binding to the same local ports, and deterministic resolution of hostnames and ephemeral ports), it also substantially increases code coverage. An evaluation on 23,886 classes from 110 open source projects, totalling more than 6.6 million lines of Java code, reveals that network access happens in 2,642 classes (11%). Our implementation of the proposed technique as part of the EVOSUITE testing tool addresses the networking code contained in 1,672 (63%) of these classes, and leads to an increase of the average line coverage from 29.1% to 50.8%. On a manual selection of 42 Java classes heavily depending on networking, line coverage with EVOSUITE more than doubled with the use of network mocking, increasing from 31.8% to 76.6%. [less ▲]

Detailed reference viewed: 99 (6 UL)
Full Text
Peer Reviewed
See detailContinuous Test Generation on Guava
Campos, Jose; Fraser, Gordon; Arcuri, Andrea UL et al

in Symposium on Search-Based Software Engineering (SSBSE) (2015)

Search-based testing can be applied to automatically gener- ate unit tests that achieve high levels of code coverage on object-oriented classes. However, test generation takes time, in particular if ... [more ▼]

Search-based testing can be applied to automatically gener- ate unit tests that achieve high levels of code coverage on object-oriented classes. However, test generation takes time, in particular if projects consist of many classes, like in the case of the Guava library. To allow search-based test generation to scale up and to integrate it better into software development, continuous test generation applies test generation incrementally during continuous integration. In this paper, we report on the application of continuous test generation with EvoSuite at the SS- BSE'15 challenge on the Guava library. Our results show that continuous test generation reduces the time spent on automated test generation by 96%, while increasing code coverage by 13.9% on average. [less ▲]

Detailed reference viewed: 55 (9 UL)
Full Text
Peer Reviewed
See detailAutomated Unit Test Generation during Software Development: A Controlled Experiment and Think-Aloud Observations
Rojas, José Miguel; Fraser, Gordon; Arcuri, Andrea UL

in ACM International Symposium on Software Testing and Analysis (ISSTA), 2015 (2015)

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A ... [more ▼]

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A developer would first need to know when and how to apply such a tool, and would then need to understand the resulting tests in order to provide test oracles and to diagnose and fix any faults that the tests reveal. Considering all this, does automatically generating unit tests provide any benefit over simply writing unit tests manually? We empirically investigated the effects of using an automated unit test generation tool (EVOSUITE) during development. A controlled experiment with 41 students shows that using EVOSUITE leads to an average branch coverage increase of +13%, and 36% less time is spent on testing compared to writing unit tests manually. However, there is no clear effect on the quality of the implementations, as it depends on how the test generation tool and the generated tests are used. In-depth analysis, using five think-aloud observations with professional programmers, confirms the necessity to increase the usability of automated unit test generation tools, to integrate them better during software development, and to educate software developers on how to best use those tools. [less ▲]

Detailed reference viewed: 91 (19 UL)
Full Text
Peer Reviewed
See detailEvoSuite at the SBST 2015 Tool Competition
Fraser, Gordon; Arcuri, Andrea UL

in 8th International Workshop on Search-Based Software Testing (SBST) 2015 (2015)

EVOSUITE is a mature research prototype that automatically generates unit tests for Java code. This paper summarizes the results and experiences of EVOSUITE’s participation at the third unit testing ... [more ▼]

EVOSUITE is a mature research prototype that automatically generates unit tests for Java code. This paper summarizes the results and experiences of EVOSUITE’s participation at the third unit testing competition at SBST 2015. An unfortunate issue of conflicting dependency versions in two out of the nine benchmark projects reduced EVOSUITE’s overall score to 190:6, leading to the overall second rank. [less ▲]

Detailed reference viewed: 62 (5 UL)
Full Text
Peer Reviewed
See detailEnvironment Modeling and Simulation for Automated Testing of Soft Real-Time Embedded Software
Iqbal, Muhammad Zohaib; Arcuri, Andrea UL; Briand, Lionel UL

in Software & Systems Modeling (2013)

Detailed reference viewed: 206 (22 UL)
Full Text
Peer Reviewed
See detailGenerating Test Data from OCL Constraints with Search Techniques
Ali, Shaukat; Iqbal, Zohaib; Arcuri, Andrea UL et al

in IEEE Transactions on Software Engineering (2013), 39(10),

Detailed reference viewed: 447 (24 UL)
Full Text
Peer Reviewed
See detailDoes automated white-box test generation really help software testers?
Fraser, Gordon; Staats, Matthew UL; McMinn, Phil et al

in ISSTA (2013)

Detailed reference viewed: 53 (14 UL)
Full Text
Peer Reviewed
See detailA Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering
Arcuri, Andrea UL; Briand, Lionel UL

in Software Testing : Verification & Reliability (2012)

Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation ... [more ▼]

Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation. However, randomized algorithms are affected by chance and so require the use of appropriate statistical tests to be properly analysed in a sound manner. This paper features a systematic review regarding recent publications in 2009 and 2010 showing that, overall, empirical analyses involving randomized algorithms in software engineering tend to not properly account for the random nature of these algorithms. Many of the novel techniques presented clearly appear promising, but the lack of soundness in their empirical evaluations casts unfortunate doubts on their actual usefulness. In software engineering, although there are guidelines on how to carry out empirical analyses involving human subjects, those guidelines are not directly and fully applicable to randomized algorithms. Furthermore, many of the textbooks on statistical analysis are written from the viewpoints of social and natural sciences, which present different challenges from randomized algorithms. To address the questionable overall quality of the empirical analyses reported in the systematic review, this paper provides guidelines on how to carry out and properly analyse randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing, which is by far the most frequent application area of randomized algorithms within software engineering. [less ▲]

Detailed reference viewed: 187 (17 UL)
Full Text
Peer Reviewed
See detailFormal Analysis of the Probability of Interaction Fault Detection Using Random Testing
Arcuri, Andrea UL; Briand, Lionel UL

in IEEE Transactions on Software Engineering (2012), 38(5), 1088-1099

Modern systems are becoming highly configurable to satisfy the varying needs of customers and users. Software product lines are hence becoming a common trend in software development to reduce cost by ... [more ▼]

Modern systems are becoming highly configurable to satisfy the varying needs of customers and users. Software product lines are hence becoming a common trend in software development to reduce cost by enabling systematic, large-scale reuse. However, high levels of configurability entail new challenges. Some faults might be revealed only if a particular combination of features is selected in the delivered products. But testing all combinations is usually not feasible in practice, due to their extremely large numbers. Combinatorial testing is a technique to generate smaller test suites for which all combinations of t features are guaranteed to be tested. In this paper, we present several theorems describing the probability of random testing to detect interaction faults and compare the results to combinatorial testing when there are no constraints among the features that can be part of a product. For example, random testing becomes even more effective as the number of features increases and converges toward equal effectiveness with combinatorial testing. Given that combinatorial testing entails significant computational overhead in the presence of hundreds or thousands of features, the results suggest that there are realistic scenarios in which random testing may outperform combinatorial testing in large systems. Furthermore, in common situations where test budgets are constrained and unlike combinatorial testing, random testing can still provide minimum guarantees on the probability of fault detection at any interaction level. However, when constraints are present among features, then random testing can fare arbitrarily worse than combinatorial testing. As a result, in order to have a practical impact, future research should focus on better understanding the decision process to choose between random testing and combinatorial testing, and improve combinatorial testing in the presence of feature constraints. [less ▲]

Detailed reference viewed: 165 (9 UL)
Full Text
Peer Reviewed
See detailCombining Search-based and Adaptive Random Testing Strategies for Environment Model-based Testing of Real-time Embedded Systems
Iqbal, Muhammad Zohaib; Arcuri, Andrea UL; Briand, Lionel UL

in 4th Symposium on Search Based Software Engineering (2012)

Detailed reference viewed: 118 (5 UL)