11/9/2018 10:05:21 PM
3.61.0
(#2022)
|
11/6/2018 7:29:42 PM
execution if executor doesn't exist
* Fix according to review comments
* Swap missing executor warn messages according to review comment
|
11/5/2018 5:21:16 PM
commit shows the problem:
When trying to execute a cleaned project version, an exception is thrown that says that hash code comparison failed.
2nd commit improves the error message in case of trying to read a cleaned (deleted) version. It also fails faster (don't try to generate a hashcode from 0 chunks).
Move project.version.retention to Constants.java.
|
11/2/2018 6:03:38 PM
a flow is scheduled with concurrentOption=skip, it's perfectly normal that triggering of a schedule is skipped. This PR changes such ERROR lines in the server log to INFO level.
On a general level, in my opinion ERROR level should be only used for platform errors, ie. when Azkaban fails to do something that it promises to be able to do. If this rule holds, it will be easier to monitor that Azkaban is working correctly by checking that server logs don't contain errors.
Before:
2018/10/23 13:41:14.337 +0300 INFO [ExecuteFlowAction] Invoking flow test-project.test-flow
2018/10/23 13:41:14.338 +0300 ERROR [TriggerManager] Failed to do action Execute flow test-flow from project test-project for Trigger Id: 0, Description: Trigger from triggerLoader with trigger condition of ThresholdChecker.eval() and expire condition of EndTimeCheck_1.eval(), Execute flow test-flow from project test-project
java.lang.RuntimeException: azkaban.executor.ExecutorManagerException: Flow is already running. Skipping execution.
at azkaban.trigger.builtin.ExecuteFlowAction.doAction(ExecuteFlowAction.java:232)
at azkaban.trigger.TriggerManager$TriggerScannerThread.onTriggerTrigger(TriggerManager.java:363)
at azkaban.trigger.TriggerManager$TriggerScannerThread.checkAllTriggers(TriggerManager.java:343)
at azkaban.trigger.TriggerManager$TriggerScannerThread.run(TriggerManager.java:297)
Caused by: azkaban.executor.ExecutorManagerException: Flow is already running. Skipping execution.
at azkaban.trigger.builtin.ExecuteFlowAction.doAction(ExecuteFlowAction.java:229)
... 3 more
After:
2018/10/23 13:41:51.778 +0300 INFO [ExecuteFlowAction] Invoking flow test-project.test-flow
2018/10/23 13:41:51.779 +0300 INFO [TriggerManager] Skipped action [Execute flow test-flow from project test-project] for [Trigger Id: 0, Description: Trigger from triggerLoader with trigger condition of ThresholdChecker.eval() and expire condition of EndTimeCheck_1.eval(), Execute flow test-flow from project test-project] because: Flow is already running. Skipping execution.
|
10/31/2018 9:25:46 PM
(#2012)
To avoid possible race condition caused by two threads (FlowRunnerManager#Cleaner and FlowPreparer#setup), and to make project directory clean up logic simpler, cleaning up projects of old version method is removed. Project dir deletion will be handled in a single thread in FlowPreparer#setup.
a race condition example:
Old Executor(initial state: active) New Executor(initial state: inactive) start to delete the project of old version
set to inactive set to active, start to load the project dirs into in-memory list load project dir 1.1 into memory
delete project dir 1.1
deletion completes loading completes then new executor's in-memory active project list could contain a project dir deleted by the old executor.
|
10/31/2018 12:43:52 AM
the assumptions
|
10/31/2018 12:41:52 AM
but not both (#2006)
We ran into a bug in the production environment. Users' jars cannot override default pig libraries. It was found that AZ always adds pig additional jar if pig job type defines it under plugin.properties. The fix is straightforward. If a user defines the property, we use it directly by not including system default settings.
|
|
|
10/25/2018 11:40:03 PM
3.60.0
in shutting down mode when its shutdown API is called. Executor will wait for all the ongoing flows to complete then shut itself down. Web server talks to executor periodically to fetch the status of each flow in its running flow cache. This PR accommodates a grace period between completion of all flows and killing the executor process, in order to allow web server to finalize flow status updating.
|
|
|
