10/2/2017 9:01:10 PM
to Metastores in HA mode (#1491)
The logic that Azkaban uses to get the delegation token from ‘other_hcat_locations’ is trying to connect to all hcat servers in that field, and getting the delegation token from each hcat server. This is could cause delegation token confusion problems. Thus, we decide to introduce a new property field called “other_hcat_clusters”, which can also be set in the “properties” field in the workflow file. This field requires users to group the hcat servers to different clusters, in the form as “cluster1hcat01, cluster1cat02;cluster2hcat01,cluster2hcat02”. In our implementation, this string will be split by semicolon, and each group(cluster) will be traversed until we can get the delegation token from one of the machines in that cluster.
|
9/29/2017 6:49:41 PM
3.36.0
not run at all, but it has been marked as killed, the right thing to do is set the status to KILLED. Otherwise the flow would get stuck because this job would send the JOB_FINISHED event with the unfinished status: KILLING.
|
9/29/2017 1:38:21 PM
back "New status: KILLING (#1172)"
This reverts commit 92aff731f4e852674154c2a2d2ebf7bb1d9c03ec.
After `git revert 92aff731f4e852674154c2a2d2ebf7bb1d9c03ec`, resolved conflicts in:
azkaban-common/src/main/java/azkaban/executor/ExecutorManager.java
azkaban-common/src/test/java/azkaban/executor/InteractiveTestJob.java
azkaban-exec-server/src/main/java/azkaban/execapp/FlowRunner.java
azkaban-exec-server/src/main/java/azkaban/execapp/JobRunner.java
azkaban-web-server/src/main/resources/azkaban/webapp/servlet/velocity/historypage.vm
azkaban-web-server/src/web/js/azkaban/view/exflow.js
* Fix test compilation
The method signature of assertFlowStatus had changed in the meanwhile.
|
9/29/2017 1:23:33 PM
(#1510)
* Update dev guide with intellij inserting new line configuration
* Address comments
|
9/29/2017 1:23:09 PM
in #1466 (#1500)
* Create directory flow loader for yaml files.
* Added test case for loading multiple flow yaml files.
* Address comments
* Fix potential inconsistency between project cache and DB during uploading.
|
9/29/2017 2:13:37 AM
reverts commit be6972a146ab769fa535a1788f6cc82c0c7b87ce.
Reason: Unable to release within LinkedIn due to licensing issues emerged from using gobblin-library.
|
9/28/2017 9:36:47 PM
and Dependency are two immutable classes holding the trigger/dependency configuration and parameters. They will be constructed from trigger config files. Once created, they couldn't be modified.
Class constructors will also perform following validations:
check all required parameters are there.
no duplicate dependency name across same trigger's dependencies.
no duplicate dependency parameters across same trigger's dependencies(different name same other parameters).
|
9/26/2017 8:30:36 PM
writable by user process (#1504)
JOB_OUTPUT_PROP_FILE is a file used to pass properties between jobs. You can read about it here: http://azkaban.github.io/azkaban/docs/latest/#job-configuration
#1325 introduced a regression where JOB_OUTPUT_PROP_FILE became no longer writable due to an upgraded execute-as-user executable that stopped user processes from having the azkaban group permission. This file needs to be written to so properties can be passed from one job to another.
This fix assigns ownership of JOB_OUTPUT_PROP_FILE and JOB_PROP_FILE to <username>:azkaban to guarantee user ability to write to this file. It is uncertain whether or not JOB_PROP_FILE is intended to be written to, but I believe it's a good idea to do allow it to be in order to prevent regressions.
This is covered by an integration test that simply tries to write to the JOB_OUTPUT_PROP_FILE to confirm that it is doable. This is already in master of azkaban-auto-test.
This change will more drastically take effect when the execute-as-user executable is upgraded to its newest version.
This was tested on solo-server and holdem4.
I'll continue to explore JOB_OUTPUT_PROP_FILE and JOB_PROP_FILE usage and implementation so we can make sure these features are fully covered by tests and can be leveraged successfully by our users.
|
9/22/2017 8:42:25 PM
3.35.0
in build.gradle (#1503)
|
9/21/2017 9:36:00 PM
build: change hadoop gradle dependencies to compileOnly instead of incuding them as runtime dependencies
* Fix tests broken due to moving to compileOnly for hadoop dependencies.
Fix was to use testCompile to include hadoop dependencies.
Also fixed transitive dependencies from gobblinKafka dependency
|
