4/26/2017 7:15:31 PM
Remove StorageConfig and move everything to AzkabanCommonModuleConfig. This creates a config container which can be used for all classes in azkaban-common. If the config grows too large, we can always refactor later. Besides currently there is no config class for the modules.
Added HDFS storage implementation stub. Following changes will implement the APIs.
|
4/26/2017 2:16:14 PM
delegates all get calls via the `StorageManager`. By `get` I mean the process of fetching a project file (blob) from storage. By default, this storage is the database. There are some hacks in the code which can be fixed later as we go.
There are some refactors in the executor codebase. This was done to easily fetch the `StorageManager` class via Guice.
The changes have been verified on solo server locally and also on executors separately.
Others - Changed catch block as per previous review comment - Updated todos with more detail
|
4/26/2017 1:10:52 PM
to have returned also the cron schedule at /schedule?ajax=loadFlow endpoint.
|
4/25/2017 8:08:14 PM
(#1015)
Root level Guice injection has been added to web server, executor and
solo server. This change ensures that the main server classes are
injected as opposed to manually constructed. This can help us refactor
the code further and introduce the major classes as Guice dependencies.
The code has been tested on solo, web and executor instances both
locally and on the grid. Unit testing these modules are currently not
easy due to the code structure. However, Guicification will help us a
lot in moving towards that direction.
|
4/25/2017 2:44:46 PM
a reporter which is subject to the multi-thread
related timing issues, verify that the metrics registry has the correct information. We don't have to test that reporter works here.
Redesigned the tests to make the tests more explicit and more closely
reflect the actual usage pattern of the metrics related methods.
The existing shared test routines make strong assumptions about the
behavior of the functional methods passed to them. IMHO they make the
test code harder to read.
The meter related tests are a bit trickier to test since they
return double type.
The existing meter related tests only test the count not the actual
value reported in the AutoMetrics anyway.
Remove these tests for now until we have a better solution.
Removed the empty Executor Metrics test class.
|
4/24/2017 10:39:33 PM
Azkaban doesn't ship the hadoop-security-plugin with the executor.
Although this is clean since the executor doesn't depend on the plugin it
doesn't make Azkaban any usable since most use cases of Azkaban requires
hadoop. For folks who aren't using Azkaban for hadoop, this would be sitting
idle.
However, the strong assumption to use this plugin is that Hadoop classes must
be on the classpath. Else those jobs using the plugin will fail.
Tested this on the solo server and this works fine.
|
4/24/2017 10:30:22 PM
code (#1013)
This change refactors the initialization of the server instances. The current code in these classes is quite entangled. Breaking this up into multiple commits. This is a prerequisite to the upcoming Guice changes. - removed duplicate methods from Azkaban*Server classes - moved some of the references constants to `Constants` class - config was getting updated during server initialization. Fixed that. The same change is now applied when we create the config object `Props` - refactored IntelliJ suggested warnings which simplify the code - extracted methods and created smaller methods to make it easier to navigate and isolate code
Tested on solo server as well as web and executor separately on dev instances.
|
4/24/2017 10:16:27 PM
error (logging inside CpuStatusFilter should use CPUSTATUS_FILTER_NAME, not MINIMUMFREEMEMORY_FILTER_NAME)
|
4/24/2017 6:41:05 PM
simply puts the DB calls behind the newly created storage layer. Currently the flow of the code is as follows
WEB API (REST) -> ProjectManager -> ProjectLoader (JdbcProjectLoader) -> DB
This has been modified to WEB API (REST) -> ProjectManager -> StorageManager -> Storage(DatabaseStorage) -> ProjectLoader (JdbcProjectLoader) -> DB
This has only been done for the put call. The get implementation is untouched yet.
|
4/24/2017 5:25:14 PM
update resolved a build warning:
"The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead."
see
https://github.com/cinnober/semver-git/releases
|
