4/7/2018 12:36:16 AM
3.46.0
should be able to use a custom job log format such as GMT timestamps
|
4/6/2018 6:49:03 PM
of following fix/enhancement around flow trigger service:
1. Cancel non-recoverable trigger instances after web server restarts:
the following situation would cause trigger instances unrecoverable:
1.1. project A with flow A associated with flow trigger A is uploaded
1.2. flow trigger A starts to run
1.3. project A with flow B without any flow trigger is uploaded
1.4. web server restarts
In this case, flow trigger instance of flow trigger A will be equipped with latest project, thus failing to find the flow trigger since new project doesn't contain flow trigger at all.
2. fix edge cases in getIncompleteTriggerInstances
3. shutdown executor service of TriggerInstanceProcessor when shutting down FlowTriggerService.
|
4/6/2018 5:38:02 PM
3.45.2
back azkaban/azkaban-plugins#278. People are still calling out for this feature. Will make it configurable in a follow-up PR, so that reportal have the flexibility to enable or disable this feature.
|
4/6/2018 5:27:25 PM
jar first (#1706)
Flow Trigger plugin manager currently prefers class provided by azkaban platform even if there exists same class in plugin jar. This is designated behavior of URL class loader - when loading a class, it searches parent class loader first. (https://docs.oracle.com/javase/7/docs/api/java/net/URLClassLoader.html).
But our Linkedin's internal dependency plugin jar(Dali: https://engineering.linkedin.com/blog/2017/11/dali-views--functions-as-a-service-for-big-data) has a class conflicting with a dependent class with downgraded version from hive library by Azkaban platform, causing dependency initialization failure. As for now the quick workaround is to implement custom class loader prefering child classloader first.
Also there could be some other alternatives:
1. make the conflicting class same by upgrading jar from AZ(short term resolution)
2. let plugin run in isolated process and manage its own classpath.
|
4/6/2018 3:47:36 PM
followup of #1712. We move jobtype module from azkaban-plugins to the main AZ repo, including their original tests.
|
3/29/2018 10:09:44 PM
unit tests
* Optimize wait in SleepJavaJob
- no extra wait
- code that better highlights the intention
* Document testing of createDeepHardlink
* Remove sleep in MetricManagerTest
|
3/29/2018 3:20:42 PM
reportal is at azkaban-plugin repo, an old and seldom maintained development environment. There exists much inconvenience to do development there:
* Ant is old and not industry Standard. It requires much debugging if something goes wrong.
* Can not set up Ant in Intellij. Because of lacking IDE support, we walk through/ View code extremely inefficiently.
This PR moves reportal codebase to main AZ repo and will be able to facilitate reportal development a lot. Fromat/Minor refactor is done through intellij plugin. More refactor are expected in future PRs
|
3/28/2018 10:07:06 PM
3.45.1
was changed to use POST for all requests, but I missed that azkaban-web also makes calls to /serverStatistics, /jmx & /stats via the gateway, not just /executor.
The problem wasn't seen in manual tests with AzkabanSingleServer because it doesn't use multi-executor mode.
|
3/23/2018 4:18:43 PM
3.45.0
to using POST with form params instead of GET with URL params, so that the number of execution ids passed can be longer.
Also deleting some unused code.
Tested manually that it works like this:
run AzkabanSingleServer in IDEA with debugger
start a flow via the UI (http://localhost:8081/)
set breakpoint to check that requests come to azkaban.execapp.ExecutorServlet and are handled successfully
set breakpoint to check that ExecutorManager can get the execution updates
(initial PR & discussion in #1655)
Builds on #1707 to validate the fix.
|
3/22/2018 9:24:51 PM
(#1705)
|
