azkaban-aplcache

Optimize gradle build time (#2113) #1231 configured gradle …

3/5/2019 3:47:15 PM

to parallelize test runs. However, in practice enabling parallel tests actually makes the full azkaban test run slower on some machines. There hasn't been any noticeable speed-up on Travis. Hence, changing back to no parallel tests.

Instead of removing entirely, keeping the setting with value 1 & adding a comment to warn about it, so that it will be seen instead of enabling it again in the future without proper validation.

./gradlew cleanTest test took 3m 42s on my machine. That seemed way too much. I was running tests of each module in IDEA and for sure the total time wouldn't be that long. It lead to think that the command-line gradle runner must be doing something that's not optimal.

I knew that idea runs all tests sequentially, so I tried disabling parallelism entirely also for the command-line test runner. It was ~4 times faster:

maxParallelForks = 1
-> ./gradlew cleanTest test 49s

I have 4 cores on my machine so I also gave this a try, but not much help:
maxParallelForks = 4
-> ./gradlew cleanTest test 2m 50s

To make sure it's not just random fluctuation I tested the "fix" once more:
maxParallelForks = 1
-> ./gradlew cleanTest test 1m 4s

And one more go with the original setting:
maxParallelForks = 12
-> ./gradlew cleanTest test 2m 0s

It could be that the forked execution was even slower when my machine had other heavy workloads going on (hence the big variation from 2m to almost 4m).

Maybe the tests interfere with each other.. Maybe the additional forking is just expensive because every test needs to load some heavy classes / static members again.. Whatever the reason is, there's no point in having parallelism on if makes the total build time longer.

Juho Autio

Commit: 774cfaa

Tree: 6a4c4f0

Parents: e37d11c

Changes

build.gradle 4(+3 -1)

Details

build.gradle 4(+3 -1)

diff --git a/build.gradle b/build.gradle
index e825fdc..5a20bf0 100644
--- a/build.gradle
+++ b/build.gradle
@@ -179,7 +179,9 @@ subprojects {
         test {
             // Set maxParallelForks to a large number and let gradle to force it to a the
             // max-workers number when needed.
-            maxParallelForks = 12
+            // NOTE! For some reason anything more than 1 doesn't work well for azkaban build:
+            // any maxParallelForks > 1 seems to make `./gradlew cleanTest test` ~4 times slower
+            maxParallelForks = 1
         }
     }