Reks: Towards Refactoring-Aware Regression Test Selection

Kaiyuan Wang, Chenguang Zhu, Ahmet Celik, Jongwook Kim, Don Batory, and Milos Gligoric

Welcome

Welcome to Reks home page. The Reks project targets at supporting refactoring-aware regression test selection during software development process.

Abstract

Regression testing checks that recent project changes do not break previously working functionality. Although important, regression testing is a costly when changes are frequent. Regression test selection (RTS) optimizes regression testing by running only tests whose results might be affected by a change. Traditionally, RTS collects dependencies (e.g., files) for each test and skips the tests, at a new project revision, whose dependencies did not change. Existing RTS techniques do not differentiate behavior-preserving transformations (i.e., refactorings) from other code changes. As a result, tests are run more frequently than necessary.

We present the first step toward a refactoring-aware RTS technique, dubbed Reks, which skips tests affected only by behavior-preserving changes. Reks defines rules to update the test dependencies without running the tests. To ensure that Reks does not hide any bug introduced by the refactoring engines, we integrate Reks only in the pre-submit testing phase, which happens on the developers' machines. We evaluate Reks by measuring the savings in the testing effort. Specifically, we reproduce 100 refactoring tasks performed by developers of 37 projects on GitHub. Our results show that Reks would not run, on average, 33% of available tests (that would be run by a refactoring-unaware RTS technique). Additionally, we run 27 refactoring types on ten open-source projects. The results, based on 74,160 refactoring tasks, show that Reks would not run, on average, 16% of tests (max: 97% and SD: 24%). Finally, our results show that the Reks update rules take negligible time.

Refactorings

We evaluated Reks by measuring savings in the testing effort. Specifically, we reproduced 100 refactoring tasks performed by open-source developers of 37 projects on GitHub. Our results show that Reks would not run, on average, 33% (max: 100% and SD: 33%) of available tests; these tests would be run by a refactoring-unaware RTS technique.

Table 1. Refactorings From Open-Source Projects

Project Commit Refactoring Type
Pure Refactorings (the commit only contains refactoring changes)
truth 53f457b7 Rename Method
truth 95018e20 Rename Method
truth 23d88f9d Rename Class
truth da634db4 Rename Method
truth 95938102 Rename Class
truth 742cbeb4 Encapsulate Field
errorprone 8774176e Rename Class + Move Class
auto 75a9cee0 Rename Class
oldauto 6aedf2fd Rename Package
jimfs 1e39f6a5 Rename Method
jimfs 118fd4ec Rename Method
re2j cbf6cce7 Rename Class
re2j 1cd2113d Remove Unused Method
openrtb-doubleclick a0a02631 Rename Class
openrtb fb85f7c1 Rename Class + Rename Method
mug 6c68aa81 Rename Method
mug 94e1e0bf Rename Local + Inline Method
mug fd5acc53 Rename Method
mug 646f5ee0 Rename Method
MOE c59435b7 Rename Package + Rename Class + Move class
MOE 56e5463d Rename Class
closure-templates 40cf35b2 Rename Method
closure-templates c2b2169e Rename Package + Rename Class + Rename Method
closure-templates 0ed335c3 Rename Class
closure-templates e01bd363 Rename Class + Rename Method + Rename Local
closure-compiler dbd00b98 Rename Class
closure-compiler b9f92fc1 Rename Method
closure-compiler b70f1d80 Rename Class
closure-compiler 13d3f477 Rename Class
closure-compiler 4a7d2609 Rename Method
closure-compiler 58b1dea7 Rename Method
closure-compiler 9e9565d5 Rename Local
closure-compiler f4a119be Rename Field + Rename Method
compile-testing 3c8eb886 Rename Package
compile-testing 224c3266 Rename Package
commons-codec 63687e70 Rename Field
commons-codec f282e951 Rename Local
commons-codec 89de3df8 Rename Field + Rename Method
commons-codec d61d1cf0 Rename Field + Rename Method
commons-compress 3e3a1388 Rename Class
commons-compress 2b5ba89b Rename Field
commons-compress 9f418190 Rename Method
commons-compress ce13559c Move Method
commons-compress 3e45dc86 Rename Field
commons-compress f3a5bf68 Move Method
commons-crypto ad81d236 Rename Class + Rename Method
commons-crypto 1d854e99 Rename Method
commons-crypto 69889c4a Rename Class
commons-crypto 1a33867f Move Field
commons-crypto afa3d61e Rename Class
commons-dbcp 8c1a7f37 Rename Class
commons-io 24330e77 Rename Field + Rename Local
commons-io cf971eb0 Rename Method
commons-io 508f4664 Rename Method
commons-io 94afbe46 Move Class
commons-csv a72c71f5 Rename Class + Rename Field + Rename Local
commons-csv aa0762d5 Rename Field + Rename Local
commons-csv 040c2606 Rename Method + Rename Local
commons-csv 7ac5dd3e Rename Method
commons-configuration 25227140 Rename Class
commons-configuration 3410aed6 Rename Class
commons-configuration 60e0cbe8 Move Class
wicket 9ec235f9 Rename Class
wicket 431fba86 Rename Class
commons-math 346a81d7 Rename Interface
commons-math 3fd9cf17 Rename Method
commons-pool a6246167 Move Field
commons-pool b954d153 Rename Field
commons-pool 15067e86 Rename Field
commons-pool 6d9eadeb Rename Field
byte-buddy f1dfb66a Pull Up Method
bitcoinj a6601066 Move Attribute + Move Method
bitcoinj 95bfa406 Extract Method + Introduce Parameter
bitcoinj 2fd96c77 Extract Method
camel 9f319029 Extract Interface
checkstyle a07cae0a Inline Method
docx4j 1ba36143 Rename Package
graylog2-server f05e86c4 Move Attribute
metrics 4c6ab3d7 Extract Method
robovm 1ef86e69 Rename Package
sonarqube 06fa5718 Move Class
commons-collections 80b08f8c Move Method + Rename Method
commons-collections ed085d06 Rename Class
commons-lang a49ea7a6 Rename Local
commons-lang bcb33ec1 Rename Class
commons-lang d9bd12dc Rename Class
commons-lang 3efd6ccd Extract Constant
commons-lang 8a82d7b7 Rename Class
commons-lang f5a83bb9 Extract Method
commons-lang f416eb20 Rename Field
commons-lang b168f291 Move Class
commons-lang a7530eae Rename Method
Mixed Changes (the commit contains both refactoring changes and non-refactoring changes)
singularity 46f65138 Pull Up Attribute
camel 5e08a9e8 Extract Method
jline2 80d3ffb5 Extract Method
robovm 7837d0ba Extract Method
druid 87f3f814 Extract Method
sonarqube 091ec857 Move Class
truth d4f564ba Rename Method
jimfs 5ad28d5e Rename Class

Contact

Please send comments or questions to Kaiyuan Wang.