9/26/2016 5:42:53 PM
3.3.0
(#751)
|
9/26/2016 2:48:27 PM
(#750)
The distribution tasks for azkaban-test were pointing to an invalid directory and were not resulting in any artifacts. Removing this dead code. The zip / tar archives are not used anywhere else in the project.
|
9/22/2016 6:04:44 PM
Java 8 (#747)
This patch will disable travis build tests for Java 7 and retain only java 8 tests. The motivation is to reduce support load and also incorporate a bunch of syntax goodies and features supported by Java 8.
Since build already succeeds / is deployed for Java 8. No further testing is required.
Also, updated Gradle IntelliJ configuration to use Java 8 language level for the project.
|
9/22/2016 5:57:13 PM
(#746)
Hadoop Tokens retrieved by Azkaban for running user processes were left world readable in the temp directory. This poses a major security threat since anyone can fish tokens from the tmp directory and use them to authenticate with hadoop using Azkaban.
This has been fixed by ensuring that the tokens created by Azkaban are readable only to the specific user. They are left readable/writable by the azkaban group which is helpful for performing cleanup tasks later on.
Code Changes: The Hadoop 2 Security Manager after fetching the token now, restricts permissions on the file and changes the file ownership to the specific user executing the job.
Testing: Build succeeds. This has been manually tested on a Kerberos enabled Hadoop Cluster. Most Acceptance tests pass. Those which fail are due to time outs for unavailable services.
No Automated tests were added due to the nature of the change - running shell based commands on file creation. Mocking this API doesn't help with any testing either.
Note: This has been only fixed in Hadoop 2. For deployments, azkaban-hadoopsecuritymanageryarn jar has been deprecated and should not be used. please use the azkaban-hadoop-security-plugin jar instead.
|
9/19/2016 9:24:37 PM
3.2.0
|
9/19/2016 3:24:15 PM
(#733)
* Throw exception on deploy action fail. * Added unit tests to make sure this behaviour happens (#725)
* Made deploy action code raise RestLi exceptions on failure. (#729)
This allows us to choose the specific HTTP status code when operations
fail. The status codes for this action have been updated to return 400 on failure.
* Made checkReport package-private for encapsulation. (#729)
Also moved test file for this into a different package. Condensed some
of its logic for readability, and added a new test to ensure messages
are getting through on error.
* Resolved file name change problem (#725)
* Made DeployProjectTest more compact, and renamed checkReport to
checkReports. (#725)
|
|
|
9/16/2016 3:59:15 PM
from 1-7 in quartz to 0-6 for Unix Cron (#738)
|
9/15/2016 8:28:20 PM
and azkaban-soloserver modules (#737)
Renamed azkaban-webserver, azkaban-execserver and azkaban-soloserver modules to azkaban-web-server, azkaban-exec-server and azkaban-solo-server respectively.
Motivation: Previously, the names of Azkaban artifacts were manually fed. Thus, despite the name of the project being azkaban-webserver, it would still create artifacts with name azkaban-web-server which makes it inconsistent with the project name. This renaming would keep the name consistent with the previous artifacts.
Updated build scripts to point to the new names of these modules.
The build passes successfully with all tests. Artifacts are generated with the correct names.
|
9/15/2016 5:51:09 PM
to this repo (#734)
The current changes consolidates azkaban-plugins/hadoopsecuritymanager-yarn and azkaban-plugins/hadoopsecuritymanager-common into a single submodule azkaban-hadoop-security-plugin in this repo. Following this commit, the Azkaban hadoop security code in the azkaban-plugins will be deprecated but will remain as is. However, future deployments of Azkaban will use this module as opposed to the one in azkaban-plugins.
Motivation: Moving the plugin code into the main repo helps in consolidation. The azkaban-plugins repo is not maintained well. It currently uses hardcoded and uploaded dependencies in the repo which is not a good model for managing artifacts. The new module works with gradle and pulls dependencies using ivys. This allows easier management of the hadoop security code and also makes making changes much easier by removing cross project dependencies.
azkaban-hadoop-security-plugin: The plugins hadoopsecuritymanager-commmon and hadoopsecuritymanager-yarn code has been merged into this single project. The Hadoop 1 management code has not been imported. We will need to rethink the structure a bit if we do have a requirement to maintain multiple hadoop branches. However, since that isn't the case now. I'm suggesting a simplified flat structure for the project.
Code Changes: Very minimal code changes have been made to make the code compile. No changes in code logic. The only changes are wrapping toString() calls on byte arrays to Arrays.toString(). This is flagged as an error by the errorprone library.
Testing: The build succeeds with the required artifacts. There are almost zero code changes which makes the risk of this plugin failing to be extremely low.
Downside of this change: Since the hadoop-security-plugin depends on hadoop and hive. (Yes ! it depends on hive too!) Every time we change hadoop versions we may need new artifacts. I still don't consider these to be a big problem because Hadoop interfaces rarely change. And even if they do, firing a new version of Azkaban is not that big a deal. However, if we want to maintain compatibility with different versions of hadoop at the same time, then it could be tricky.
|
