11/29/2018 8:17:40 PM
in CommonJobProperties.
* Fix mismatch in name of job log files when trying to kill Hadoop/Spark jobs. Issue: when a user tries to kill a Hadoop/Spark job within an embedded flow the application looks for a log file using the following name pattern "_job.<exec_id>.<job_id>.log" no matter whether the job was embedded or not. But actually if a job is inside an embedded flow the name format used to create its log file is: "job.<exec_id>.<parent_flow>..<job_id>.log". So trying to kill a job inside an embedded flow before would fail because of the log file name mismatch.
* Fix according to review comments
|
|
|
11/27/2018 11:36:05 PM
provide the ability to call HTTP resources as described in 'HTTP Job Callback' https://azkaban.github.io/azkaban/docs/latest/#common-configurations
job.notification.started.1.url=.. and etc
In the implementation https://github.com/azkaban/azkaban/blob/master/azkaban-exec-server/src/main/java/azkaban/execapp/event/JobCallbackManager.java#L263 'azkabanHostName' used as the value of 'Host' header. But the value should be the hostname of requested HTTP resource, not Azkaban's host.
For example for https://hooks.slack.com/services should be Host:hooks.slack.com
|
|
|
|
|
|
|
11/16/2018 11:27:44 PM
execution (#2023)
* Extract method submitFlowRunner
* Extract method createFlowRunner
* Improve dispatch request handling of a previously submitted execution
- If the execution is indeed running, return OK so that dispatcher knows that it was successfully dispatched
- If the execution was left in some intermediate state, return an error so that dispatcher knows to retry or finalize the execution as failed
|
11/15/2018 11:45:28 PM
PR:
1. adds more logging and standardize existing logging for each token prefetching methods so that we know prefetching from which service is stuck.
2. removes "synchronized" for doPrefetch(HadoopSecurityManager_H_2_0#doPrefetch). Current design makes it hard to debug which token service the job is stuck with fetching token. Since HadoopSecurityManager_H_2_0 is shared by all jobs in the executor, if one job is stuck with fetching token with a problematic token service, all other jobs will be blocked from entering into this synchronized method. It's impossible to infer which token service jobs are stuck with from job logs as they are just waiting for one job to finish fetching token.
|
11/14/2018 1:26:42 AM
actions" plugin -- format changes only.
2. separate different token prefetching procedures into individual methods (this is done with intellij's method extract: https://www.jetbrains.com/help/idea/extract-method.html).
Follow-up -- to be done in next PR:
1. more logging for each token prefetching methods so that we know prefetching from which service is stuck.
2. see if removing "synchronized" for doPrefetch(HadoopSecurityManager_H_2_0#doPrefetch) is feasible.
|
11/13/2018 10:15:19 PM
could happen when multiple azkaban executor process are running:
It's possible when two azkaban executor process perform deletion in the same azkaban project dir even when one executor is inactive. E.g, flow run on any executor by using useExecutor label.
If so, the list of azkaban project dir in memory(installedProjects) kept by azkaban executor process, will be out of sync from what's on the disk.
Another case is race condition between one executor process is deleting a project dir while another executor process is creating execution dir based on the project dir.
This PR
removes the installedProjects from executor. So every time a project needs to be downloaded, a scan of every project dir and calculation of total disk usage sum will be done to decide whether purging is needed. This could takes tens of seconds when number of project dir is >= 5000 but a few seconds with inode cache.
make project dir cleanup(deleteProjectDirsIfNecessary) synchronized. Since the method is a check-then-act process which is vulnerable to race condition when multiple threads are doing deletion. An alternative is to synchronize on an interned string of project id+project version(https://stackoverflow.com/questions/133988/synchronizing-on-string-objects-in-java), however this is not that elegant as the linked post points out. Synchronization on the object level makes sense given flow setup is low frequency operation in most cases(<= 5 ops/mins in our production environment).
when project dir is created, another metadata file keeping the file count is created. The purpose of it is is to address the race condition between one executor process is deleting a project dir while another executor process is creating execution dir based on the project dir. A sanity check on the file count will be conducted against created execution dir. If execution dir's file count is not same as base project dir, then fail the flow setup and let azkaban web server dispatch it again.
Note even with this fix, there still could be race conditions. E.g, when two executor process are calling ProjectCacheDirCleaner#deleteProjectDirsIfNecessary, one might delete a dir while the other is loading the same dir.
A potential long term fix: #2020
Follow-up
add file count sanity check mentioned above.
|
