6/13/2017 8:34:04 PM
3.29.0
authorizing prior to any HDFS call, the HDFS storage code does not seem to work without an existing Kerberos session. The problem is that the `FileSystem.get()` API needs to be called after a successful UGI login. Currently that is not the case. The Hadoop code caches the logged in user at the time of creating the File System object and this is not reflected in the stack traces making it difficult to debug.
Fix: ensure UGI auth prior to creation of file system object.
|
6/13/2017 1:58:54 AM
time (PR #1207) I only did this for waitForStatus(), while I should've caught these all.
- Additional stability even is test runner is slow (like supposedly on Travis sometimes)
- This doesn't make the tests any slower than before, as long as they pass
|
6/12/2017 9:48:50 PM
suggests "@SuppressWarnings("unused")" when compiling, but this did not appear to remove the warning.
Following the recommendation suggested in the link above and adding "@SuppressWarnings("FutureReturnValueIgnored")" seemed to do the trick.
|
6/12/2017 9:35:31 PM
(#1197)
* Fix deprecated api compiler warnings in HadoopSecurityManager_H_2_0
Warning:(446, 20) java: cancelDelegationToken(org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.delegation.DelegationTokenIdentifier>) in org.apache.hadoop.mapred.JobClient has been deprecated
Fixed by removing the now unused private methods.
Also corrected the log messages.
Todo: I can't find the history that explains why these token cancellation requests were ignored. It appears that that code change was made at least 2-3 years ago. We haven't seen reports of related issues yet. There is no good reason to take the risk to re-introduce the logic now.
|
6/12/2017 9:27:26 PM
consistency and make updating dependencies easier.
see
https://discuss.gradle.org/t/what-is-the-proper-way-to-make-dependencies-consistent-in-a-multi-project-setup/17383/12
Also:
Define the common dependencies in the root build.gradle file. compile deps.log4j compile deps.guice compile deps.slf4j runtime deps.slf4jLog4j
Introduce common versions in the ext.versions map.
|
6/12/2017 9:10:11 PM
guide
"By default, Gradle uses the name of the directory it finds the settings.gradle as the name of the root project. This usually doesn't cause problems since all developers check out the same directory name when working on a project. On Continuous Integration servers, like Jenkins, the directory name may be auto-generated and not match the name in your VCS. For that reason, it's recommended that you always set the root project name to something predictable, even in single project builds. You can configure the root project name by setting rootProject.name.
"
https://docs.gradle.org/current/userguide/intro_multi_project_builds.html
|
|
|
6/12/2017 6:41:01 PM
azkaban prefix is redundant and requires unnecessary
typing.
Keep the azkaban prefix in the jar file name.
|
6/12/2017 5:01:58 PM
3.28.0
reverts commit ffd9ebd.
The problem is not in the "fails fast" strategy per se, but there's another bug that would cause a job in a subflow to fail instead of running it, when two subflows contain jobs of the same name. It's because current DB record key for executable node is (execution id, job id, attempt). So the problem only happens when two subflows contain two jobs with the same name.
|
6/12/2017 2:45:52 PM
to consider allowsOnDuplicateKey since the original Azkaban database modules has this.
|
