3/21/2018 5:36:53 PM
better support for very large flows, we need to optimize the flow traversals that occur when loading a new project and verifying it. The included unit test shows the issue occurring, and without the fixes takes ~40s (on my laptop) to complete, but with the fixes takes ~2s to complete.
The first optimization is to prevent reprocessing nodes when executing constructFlow(). This is acheieved by keeping track of all previously visited nodes and then checking if we have previously visited this node or not before processing.
The second optimization is in azkaban.flow.Flow#initialize. This code also does a traversal of the DAG in order to set the edges and the levels of the nodes, and again the solution here was optimize the traversal to ensure that we dont reprocess nodes which have previously been processed. This was done using a breadth first ordering so that the nodes from each level are only visited once. A node could exist at multiple levels and will be traversed multiple times in this case, but that is expected behaviour since we want the node's level to be the MAX level it exists at in the flow.
|
3/20/2018 4:22:47 PM
NPM, and moment version number updates
This change is because github flagged the version of moment
used by Azkaban as having a CVE. While updating moment, I also
updated Node & NPM as we now have a package-lock.json file for
helping manage dependencies.
|
3/18/2018 5:03:39 PM
over the execution graph pane"
|
3/13/2018 1:49:42 AM
Constants class (#1626)
* session.ttl config key moved to Constants class
* Add unit tests
|
3/12/2018 6:45:25 PM
instance context
Dependency name and trigger instance id should be passed into dependency instance config and runtime props when creating dependency instance context so that trigger instance id and dependency name is obtainable by dependency instance context to identify itself.
E.x.
Dependency instance context could put trigger instance id and dependency name in the log, thus user can easily scope all logging messages produced by a particular dependency instance.
|
3/9/2018 8:22:31 PM
by @HappyRay: #1654 (comment).
Refactoring only - no functional changes.
|
3/9/2018 8:13:04 PM
description: #1681
Since webServer.scheduler and webServer.flowTriggerService are injected and always non null, even if quartz is disabled, in old code webServer.scheduler.shutdown() will be still executed throwing NPE if quart is disabled.
|
3/8/2018 2:54:21 PM
project cache to case insensitive.
* Explicitly set Mysql DB to case insensitive.
* update sql
* Change h2 DB for unit tests to case insensitive.
* Add test case for project cache case sensitivity.
* Added proper comments and java docs.
* fix project cache synchronization issue.
* Address comments.
* Added unit tests for CaseInsensitiveConcurrentHashMap. Updated h2 db settings.
* Add comments for project cache.
* Address comments.
|
3/2/2018 9:08:36 PM
3.44.0
the name consistent with the old names to minimize changes in other places that still reference the old names.
|
3/1/2018 6:48:59 PM
service (#1670)
Flow trigger service currently maintains an in memory list of running trigger instances for fast retrieval of the list of currently running trigger instances. However, it also introduces some issues like race condition and threads being created per request of getting running trigger instances. Removing the dependency on running trigger instances and retrieving directly from DB is a simpler and less error prone option. If retrieval of running triggers starts to have performance concern, we will reconsider it.
But this PR doesn't remove the running trigger entirely from flow trigger service since the mapping between running trigger instance and and its DependencyInstanceContext still needs to be kept in memory in case where users want to cancel a running trigger instance.
follow up:
Add index to execution_dependencies table for faster retrieval.
As execution_dependencies table keeps growing, we will need to implement purging mechanism to keep reasonable amount of historical executions.
|
