7/17/2018 3:03:52 PM
to solve is detailed in #1803
Drawback of previous design is detailed in #1841
New design:
Create a file in each project directory and write the size of the project to the file when the project is created. The project files are not supposed to change after creation. Touch this file each time the project is used. This way, we can have a more efficient LRU algorithm based on last access time, not creation time.
Maintain the total size of the project cache in memory to avoid the overhead of re-calculating it. The size shouldn't change too often. This way we can afford to run the check more frequently.
This PR implements part 1.
|
7/16/2018 11:57:47 PM
more complete cleanup: don't leave executions whose executor doesn't exist any more in some interim state like RUNNING - finalize them as FAILED.
To explain the problem scenario, it goes like this:
if an executor is stopped gracefully (so that it has time to finish at least some shutdown hooks), it removes itself from the executors table
however, it's still possible that not all executions on that executor were properly finalized ie. status of each interrupted execution is set to one of FAILED/KILLED/SUCCESS
when an executor is started, it adds itself into the executors table with an executor id that is the next running number
As a result, there can be executions with status RUNNING (for example) that would never be marked as FAILED, because the executor id of those executions is not found in executors table any more. This PR fixes that: the status of such executions are finalized.
This is only a concern in multi-executor mode. To be noted, it would be good to support only multi-exec mode in the future. See also description of #1831.
|
7/16/2018 11:53:42 PM
set is empty (#1855)
When no dependency is defined for a flow trigger, it just works like a regular schedule - when the scheduled time comes, flow will be kicked off without waiting for any dependency. In this case, max wait min is not required.
|
7/16/2018 10:57:25 PM
coverage banner in README page as well as travis ci one.
|
7/12/2018 6:02:38 PM
3.50.1
has a conflict against our internal gradle system. So replacing a different one.
|
7/11/2018 6:30:43 PM
improvements (#1835)
As described in #1822, CRON expression allows year to be specified. This pull request will expose year textbox in schedule panel, allow one time execution, and properly update the next 10 executions based on the CRON expression with a slightly improved descriptions.
|
7/11/2018 5:48:26 PM
active (#1824)
FlowRunnerManager has a variable named "installedProjects" to keep the projects currently residing in the shared project dir. It gets initialized when FlowRunnerManager is constructed and used to look up the project directory by project ID and version. It's maintained by a cleaner thread inside FlowRunnerManager and FlowPreparer - when new project is downloaded or old project is deleted, its corresponding in-memory mapping is populated or kicked out from installedProjects.
However current logic suffers from risk of cache out of sync. E.g, when FlowRunnerManager populates installedProjects upon construction, sometimes executor is still in inactive mode, bearing some verification work like running integration test. However, another executor at same host which is active is performing project cleaning up. Some projects might be deleted by the active executor but not reflected by installedProjects in the inactive executor.
The resolution is to load the installedProjects when executor becomes active. An inactive executor doesn't use installedProjects for any reason.
|
7/10/2018 3:28:16 PM
(#1847)
|
7/10/2018 3:09:24 PM
will pave the way for adding more python based tools.
Future work:
* Run the python tests automatically as other unit tests.
* Separate test code and production code into separate directories.
|
7/10/2018 2:55:17 PM
3.50.0
trigger (#1813)
Upon project uploading, azkaban should be able to schedule the associated flow trigger. This is already achieved when project upload is done from ajax API call(#1631). This PR enables scheduling flow trigger when uploading project from restli endpoint.
|
