8/18/2017 8:00:52 PM
docs that gradle daemon should not be run in CI environments.
https://docs.gradle.org/3.3/userguide/gradle_daemon.html#when_should_i_not_use_the_gradle_daemon
|
8/18/2017 5:16:33 PM
the job to appear for 10 seconds instead of 1 second. This should help avoid test failures when they're running slower than usual for some reason.
If wait is interrupted, have another round (i--) of trying, so that the method always waits _at least_ 10 seconds before it gives up.
|
8/18/2017 5:07:23 PM
a race condition which causes Job status to be first set to some other finished status and then subsequently to KILLED. The practical implication is that when a job is killed, it's possible that it sends a JOB_FINISHED event with some other status, although it is eventually marked as KILLED.
Obviously, the final status of a job should nevertheless be "final" instead of "flapping" after a finished status has been set.
This race condition is fixed by synchronizing in both run & kill methods so that the combination of these operations are wrapped into "a single atomic operation":
checking the status and/or killed flag
based on the previous check result(s), setting the status and/or killed flag
Kill method already had the synchronization in place, but the run method was missing it.
|
8/18/2017 4:59:03 PM
(#1361)
## Issues:
The test data files are under the resource directory in the test
submodule.
Gradle tests will use the jar file that contains the data files. This
makes it harder to use these files as files directly. As a result the
build script of other submodules has to use the build directory as
dependencies instead of using the standard project dependency.
Recent changes in intellij makes the matter worse by insisting on using
its own build output directory. When compiling from intellij along
without building the test submodule first in gradle, the compilation
will fail because intellij can't find the dependent classes.
When navigating to the dependent classes defined in the test submodule,
intellij will jump to the decompiled class instead of the correct source files.
## Fix:
- Move the data files to a top level directory in the test submodule.
- Provide static methods to locate these files instead of having the
other submodules to directly hardcode the directory paths.
- Change gradle build files to use the project dependency.
- Move the java code in the test submodule to the main directory since
this code is not a test of itself but rather code to be used by other
modules.
- Add unit tests for the test module itself.
|
8/17/2017 9:47:59 PM
only (#1360)
|
8/17/2017 8:47:52 PM
class (#1357)
This code patch is a follow-up of #1345. Similar to that PR, I start moving executor related operations. The first step is to create a delegate class and move all related methods to the new class.
Note that I didn't change anything, but only move methods from one place to another place. After this PR, I will change the implementation inside, and add unit test. The benefits of this PR is to help team do review.
|
8/17/2017 8:13:19 PM
code patch does two refactor works:
* remove FetchExecutableFlows, as new ExecutionFlowDBManager already takes over this work.
* consolidate set up test DB in a common place, such that every test is able to call it.
|
8/17/2017 7:03:33 PM
is old code where an actual concrete class was written to mock `ProjectLoader`. This can easily be replaced by creating a mock object using `Mockito`. This change precisely does that.
|
8/17/2017 6:14:33 PM
azkaban db module (#1345)
The goal of this code patch is to gradually migrate to new azkaban DB module. I construct a new class ExecutorFlowDBManager to store Table execution_flow related DB operations. Basically, I move a few methods out from JdbcExecutorLoader to new class and leverage new DB API. There are 3 benefits for the new class:
* Split #1291 into a few commits, and will make reviews easier.
* Put Query Handler class and its caller together. In this way, make code clean and easy to read.
* Also easy to manage unit Tests.
I'm considering using ExecutorFlowDBManager directly to replace JdbcExecutorLoader, when it comes to execution_flows related operations. However, Since some classes like FlowRunner has not been guificied, it is not easy to add another dependency. We will do the follow-up refactor after executor loader is completed migrated.
|
8/17/2017 2:08:50 PM
now we're seeing noise in the number of errors being generated in our logs in part because the current working directory and the output properties file are both being recorded with System.err.println().
I changed these two messages to use this.info() instead. I also refactored an error message in createOutputPropsFiles to leverage the logger error method instead of using System.err.println().
|
