6/4/2017 3:21:28 PM
three flaky FlowRunner related tests
azkaban.execapp.FlowRunnerTest2.testPauseFailFinishAll
and
azkaban.execapp.FlowRunnerTest2#testRetryOnFailure
testCancelOnFailure
Issue: #1155
* ignore testCancelOnFailure too
see #1157
|
6/4/2017 2:29:11 PM
(#1152)
Using hamcrest matcher `is` to compare arrays, because it gives an informative error message, like:
java.lang.AssertionError:
Expected: is [<FLOW_STARTED>, <FLOW_FINISHED>] but: was [<FLOW_STARTED>]
|
6/4/2017 2:05:25 PM
(#1151)
* Consolidate the duplicate code into one prepareProject method
Issue:
Three test classes in execapp have the same duplicate prepareProject
logic.
Fix:
Consolidated into one method in a shared test utility class.
With this change, the FlowRunnerTest2 has no intellij warnings left
except warnings about typos.
* Update Copyright notices
|
6/4/2017 1:25:57 PM
cache from web server
* Fix one test case failure
* Added unit test for fetching recentlyFinishedFlows
* Modified recently finished lifetime
* Some improvement on recently finished flows cache change.
* Quick fix
* Fix merge and comments
|
6/4/2017 11:44:03 AM
to the intellij's guide.
* Users should be able to use import code style features instead.
* Advised against turning on "Remove unused suppress warning
annotation" option in the save actions plugin since it seems to be
unreliable.
|
6/3/2017 7:40:55 PM
in MockExecutorLoader: it shouldn't modify the state of actual flows, so make a clone of the flows, too.
Also log job logs in MockExecutorLoader instead of discarding them, so that it's possible to debug failures in tests.
The problem was this (originally revealed when FlowRunnerTest still used this mock class instead of a Mockito mock):
MockExecutorLoader kept a handle to the actual ExecutableFlow instance
MockExecutorLoader.updateExecutableFlow set the job statuses based on the current flow status, which is not synchronized with the job runner threads
So this wrote an old job status over the newer one in ExecutableNode
Stacktrace for that:
job4: setStatus SUCCEEDED -> QUEUED
at azkaban.executor.ExecutableNode.setStatus(ExecutableNode.java:141)
at azkaban.executor.ExecutableNode.applyUpdateObject(ExecutableNode.java:398)
at azkaban.executor.ExecutableFlowBase.applyUpdateObject(ExecutableFlowBase.java:352)
at azkaban.executor.ExecutableFlowBase.applyUpdateObject(ExecutableFlowBase.java:369)
at azkaban.executor.MockExecutorLoader.updateExecutableFlow(MockExecutorLoader.java:120)
at azkaban.execapp.FlowRunner.updateFlow(FlowRunner.java:322)
at azkaban.execapp.FlowRunner.updateFlow(FlowRunner.java:316)
at azkaban.execapp.FlowRunner.progressGraph(FlowRunner.java:552)
at azkaban.execapp.FlowRunner.runFlow(FlowRunner.java:413)
at azkaban.execapp.FlowRunner.doRun(FlowRunner.java:238)
at azkaban.execapp.FlowRunner.run(FlowRunner.java:215)
Overlapping updates by 2 different threads could look something like this:
2017/05/24 12:20:54.695 +0300 INFO [JobRunner-job3-1] [ExecutableNode] job4: applyUpdateObject: RUNNING-> QUEUED
2017/05/24 12:20:55.700 +0300 INFO [JobRunner-job4-1] [JobRunner] job4: changeStatus: QUEUED -> SUCCEEDED
2017/05/24 12:20:55.701 +0300 INFO [FlowRunner-exec-1] [ExecutableNode] job4: applyUpdateObject: SUCCEEDED-> QUEUED
Background discussion at #1105.
|
6/3/2017 7:27:46 PM
warnings reported by the intellij IDE
* Switch to FileUtils.forceMkdir
This will do the error checking.
* Inline the flowName parameter instead of suppress the warning
|
6/3/2017 6:29:25 PM
(#1146)
Issue: The javadoc of the FlowRunnerTest2 class uses ascii art to describe the DAG. When intellij auto format runs, it renders the art incorrectly. Fix: Wrapped the parts affected in <pre> </pre> blocks.
|
|
|
|
|
