8/7/2017 3:24:17 PM
Change API's concurrent default option to skip
Today, given a schedule, the default option for concurrent setting is ignore, that is, allowing concurrent executions whatever schedule triggers. This could be regarded as a bug in current AZ. The reason is that, In UI, the default schedule option is "skip" while API's default option is "ignore". The inconsistency between API and UI should be resolved.
This commit fixes the option to skip. When next schedule triggers, as long as the last execution hasn't finished, it won't trigger the next execution. As a consequence, this commit changes API's default option to "skip" and UI's default behavior is not affected. Tested on our staging cluster, and working.
|
8/6/2017 5:34:24 PM
clean build took 22 seconds vs > 1 minute before this change.
output:
BUILD SUCCESSFUL
Total time: 22.587 secs
120 tasks in build, out of which 81 (68%) were executed 13 (11%) up-to-date 13 (11%) no-source 13 (11%) loaded from cache 6 (5%) cache miss 75 (63%) not cacheable
|
8/3/2017 6:04:49 PM
dep: httpcore (#1306)
Updating httpclient dependency to the latest
https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient
Removing httpcore since it is a dependency of httpclient and will be pulled in automatically.
|
|
|
8/2/2017 7:54:11 PM
login warning (#1303)
This commit reverts the XMLUserManager code since it is found that we don't need to customize XML error login message, since only headless users access from XMLUserManager. Standard users go through LDAPUserManager, which we already added this feature.
The original PR is at: #1277
|
8/2/2017 7:14:11 PM
class (#1294)" (#1304)
This reverts commit 9d079ec.
This change removed an API (getSessionData) relied upon by the hdfs viewer.
|
8/1/2017 3:14:21 PM
3.32.0
commit d1b836a.
Conflicts:
azkaban-common/src/main/java/azkaban/executor/Status.java
azkaban-exec-server/src/main/java/azkaban/execapp/JobRunner.java
From @jamiesjc: When we kill the flow immediately after it starts in our integration test, it couldn't be killed due to some race condition. Users will see on the execution page that the job is in KILLING status but it actually never gets killed. And users cannot click kill button again during the KILLING period.
At the time when we kill, the process might not have started yet or the jobRunner has not yet been added to the activeJobRunners.
You can check more details in the PR description: #1289. We are still working on the fix.
The intention is to reintroduce this commit once the underlying bug is fixed.
|
7/31/2017 6:38:27 PM
logs every hour
We observed the long deletion transaction in our production Azkaban cluster. In mysql, if delete transaction is large (The data going to be deleted is too big), it occupies too much undo space (buffer), which affect other transactions and other databases in this host. So we change every day one time's deletion to every hour.
|
7/28/2017 9:46:55 PM
disk space (#1295)
The execution directory can bloat to be relatively large, which could lead to problems if the executor is shutdown and stale executions remain sitting on the executor occupying disk space.
This change deletes the execution directory on executor shutdown. The directory itself is created on executor initialization, so the directory isn't necessary to reset the executor later.
Note that the execution directory will not be deleted until AFTER all flows are done if the executor is brought down via shutdown() - so it will not delete partially-written logs before they are uploaded to the database.
|
7/28/2017 1:38:48 PM
execution cleanup (#1292)
Setting the gid bit on the group for a directory causes all items created within that directory to inherit the group of the directory. So all items users create in their /execution/<exec_id> directory will automatically be a part of the azkaban group. This allows the azkaban cleanup thread to properly remove user-generated files/directories.
The gid bit is system-specific, so there is no java standard library api for setting it. The solution I proposed spawns a subprocess that performs the chmod command. I don't think this is very clean, but I haven't been able to find a better way. Anybody have any ideas?
Another option would be to build this executionDirectory as part of our build process, but that isn't how we do things right now (tested by running deploy on holdem4 jenkins job without any of the other steps and verifying no /executions directory is created until system start).
Note that this change is not covered by testing due to the difficulty of dealing simultaneously with filesystems and subprocesses within a testing environment. If disk space usage increases in the future, it has possible that this change has been regressed. In order to confirm that this change is working in production on particularly large clusters (where problems have been seen), I'll be keeping an eye on it when it's released.
|
