Automated Unit Test Generation during Software Development: A Controlled Experiment and Think-Aloud Observations

Automated unit test generation tools can produce tests that are
superior to manually written ones in terms of code coverage, but
are these tests helpful to developers while they are writing code? A
developer would first need to know when and how to apply such a
tool, and would then need to understand the resulting tests in order
to provide test oracles and to diagnose and fix any faults that the
tests reveal. Considering all this, does automatically generating unit
tests provide any benefit over simply writing unit tests manually?
We empirically investigated the effects of using an automated unit
test generation tool (EVOSUITE) during development. A controlled
experiment with 41 students shows that using EVOSUITE leads to
an average branch coverage increase of +13%, and 36% less time is
spent on testing compared to writing unit tests manually. However,
there is no clear effect on the quality of the implementations, as
it depends on how the test generation tool and the generated tests
are used. In-depth analysis, using five think-aloud observations
with professional programmers, confirms the necessity to increase
the usability of automated unit test generation tools, to integrate
them better during software development, and to educate software
developers on how to best use those tools.