Online Seminar: Mitigating Flaky Tests

Abstract: Software is an important part of our society, and we depend on software
developers to implement and maintain the quality of software. In turn,
developers rely on regression testing, the practice of running tests after
every software change, to check that they do not introduce bugs as they make
changes. However, regression testing is plagued by flaky tests that
non-deterministically pass and fail independently of software changes. The
outcomes of flaky tests often mislead developers about the validity of their
changes. Flaky tests are also quite prevalent, e.g., Google once reported 16%
of their 4.2 million tests are flaky and that 84% of their changes with test
failures actually involve a flaky test.

To mitigate the negative effects of flaky tests, I have developed several
techniques to automatically fix and detect different types of flaky tests. My
techniques have fixed and detected hundreds of flaky tests in open-source
software. This talk will focus on my work in fixing and detecting a specific
type of flaky tests, namely order-dependent flaky tests, which pass when run in
one order of the test suite but fail when run in a different order. First, I
will present my technique, iFixFlakies, for automatically fixing
order-dependent flaky tests. Next, I will present my technique, PolDet, for
proactively detecting root causes that lead to order-dependent flaky tests
before such tests can lead to flaky test failures.

Bio: August Shi is a PhD candidate in Computer Science at the University of Illinois
at Urbana-Champaign. August works with Professor Darko Marinov in the area of
Software Engineering, with a focus on Software Testing. His research aims to
improve developer productivity by making regression testing (1) more reliable
with respect to flaky tests and (2) faster without loss in quality of testing.
August has published in ICSE, ESEC/FSE, ASE, ISSTA, OOPSLA, ICST, and ISSRE,
and his work on improving regression testing received an ACM SIGSOFT
Distinguished Paper Award at ICSE 2017. More information is available on his
web page: http://mir.cs.illinois.edu/awshi2.