|
|
9/16/2016 3:59:15 PM
from 1-7 in quartz to 0-6 for Unix Cron (#738)
|
9/15/2016 8:28:20 PM
and azkaban-soloserver modules (#737)
Renamed azkaban-webserver, azkaban-execserver and azkaban-soloserver modules to azkaban-web-server, azkaban-exec-server and azkaban-solo-server respectively.
Motivation: Previously, the names of Azkaban artifacts were manually fed. Thus, despite the name of the project being azkaban-webserver, it would still create artifacts with name azkaban-web-server which makes it inconsistent with the project name. This renaming would keep the name consistent with the previous artifacts.
Updated build scripts to point to the new names of these modules.
The build passes successfully with all tests. Artifacts are generated with the correct names.
|
9/15/2016 5:51:09 PM
to this repo (#734)
The current changes consolidates azkaban-plugins/hadoopsecuritymanager-yarn and azkaban-plugins/hadoopsecuritymanager-common into a single submodule azkaban-hadoop-security-plugin in this repo. Following this commit, the Azkaban hadoop security code in the azkaban-plugins will be deprecated but will remain as is. However, future deployments of Azkaban will use this module as opposed to the one in azkaban-plugins.
Motivation: Moving the plugin code into the main repo helps in consolidation. The azkaban-plugins repo is not maintained well. It currently uses hardcoded and uploaded dependencies in the repo which is not a good model for managing artifacts. The new module works with gradle and pulls dependencies using ivys. This allows easier management of the hadoop security code and also makes making changes much easier by removing cross project dependencies.
azkaban-hadoop-security-plugin: The plugins hadoopsecuritymanager-commmon and hadoopsecuritymanager-yarn code has been merged into this single project. The Hadoop 1 management code has not been imported. We will need to rethink the structure a bit if we do have a requirement to maintain multiple hadoop branches. However, since that isn't the case now. I'm suggesting a simplified flat structure for the project.
Code Changes: Very minimal code changes have been made to make the code compile. No changes in code logic. The only changes are wrapping toString() calls on byte arrays to Arrays.toString(). This is flagged as an error by the errorprone library.
Testing: The build succeeds with the required artifacts. There are almost zero code changes which makes the risk of this plugin failing to be extremely low.
Downside of this change: Since the hadoop-security-plugin depends on hadoop and hive. (Yes ! it depends on hive too!) Every time we change hadoop versions we may need new artifacts. I still don't consider these to be a big problem because Hadoop interfaces rarely change. And even if they do, firing a new version of Azkaban is not that big a deal. However, if we want to maintain compatibility with different versions of hadoop at the same time, then it could be tricky.
|
9/13/2016 3:47:44 PM
gradle distribution code
Currently, the gradle code for building distributions of azkaban modules is hosted in the root project build script. Although this works fine in terms of building artifacts, it is not modular / easier to maintain since the expected way of using the gradle distribution plugin is to have the build code inside the submodule which it is trying to build.
For example:
The distribution code for azkaban-webserver should reside in ./azkaban-webserver/build.gradle instead of build.gradle.
The distribution code logic has been moved into each submodule. Each sub-project is more self contained.
Also the copying of files etc are being handled manually which is nicely supported by the DSL. This has been refactored so that the distribution plugin configuration is able to take care of the copy logic
Removal of azkaban.version file. The azkaban.version file used to store the version number of the tar ball according to the git tag. Since we store the version in the file name itself as well as in manifest attributes of jar files, the version file provides a second source of truth. Thus, the code to create this file has been removed.
All bash scripts have now been moved to src/bash instead of package/bin.
All conf scripts have been moved to src/main/resources instead of package/conf. This change is motivated by the general java project structure where properties files and configuration reside in resource directories.
* Adding task dependency
The problem with Gradle distribution plugin is that there is no root task which is depended upon by installDist, distTar and distZip. Thus, if we need to add a dependent task to the distribution, it needs to be added to all the 3 tasks.
* Removing errorprone configuration
Since the errorprone plugin is applied to all subprojects, we do not need any custom configurations for plugin integration. This was required for previous versions of the plugin / gradle
|
9/12/2016 4:57:13 PM
(#728)
1). #727
2). #693
This commit brings cron utilities to Azkaban. Users are able to define a cron job now through Azkaban UI or Rest Call to Azkaban. Besides, this commit fixed DST change bug (#727) by enforcing cron running at server timezone (e.g. Pacific time). To enable the new schedule panel UI/utilities:
1). specify azkaban.schedulePanelPageName=schedulepanel.vm in .conf file
2). restart webserver
|
9/8/2016 3:04:01 PM
copyJSLibs must dependon bowerInstall (bowerInstall is not completed before copyJSLibs)
|
|
|
|
|
|
|
