3/12/2018 6:45:25 PM
instance context
Dependency name and trigger instance id should be passed into dependency instance config and runtime props when creating dependency instance context so that trigger instance id and dependency name is obtainable by dependency instance context to identify itself.
E.x.
Dependency instance context could put trigger instance id and dependency name in the log, thus user can easily scope all logging messages produced by a particular dependency instance.
|
3/9/2018 8:22:31 PM
by @HappyRay: #1654 (comment).
Refactoring only - no functional changes.
|
3/9/2018 8:13:04 PM
description: #1681
Since webServer.scheduler and webServer.flowTriggerService are injected and always non null, even if quartz is disabled, in old code webServer.scheduler.shutdown() will be still executed throwing NPE if quart is disabled.
|
3/8/2018 2:54:21 PM
project cache to case insensitive.
* Explicitly set Mysql DB to case insensitive.
* update sql
* Change h2 DB for unit tests to case insensitive.
* Add test case for project cache case sensitivity.
* Added proper comments and java docs.
* fix project cache synchronization issue.
* Address comments.
* Added unit tests for CaseInsensitiveConcurrentHashMap. Updated h2 db settings.
* Add comments for project cache.
* Address comments.
|
3/2/2018 9:08:36 PM
3.44.0
the name consistent with the old names to minimize changes in other places that still reference the old names.
|
3/1/2018 6:48:59 PM
service (#1670)
Flow trigger service currently maintains an in memory list of running trigger instances for fast retrieval of the list of currently running trigger instances. However, it also introduces some issues like race condition and threads being created per request of getting running trigger instances. Removing the dependency on running trigger instances and retrieving directly from DB is a simpler and less error prone option. If retrieval of running triggers starts to have performance concern, we will reconsider it.
But this PR doesn't remove the running trigger entirely from flow trigger service since the mapping between running trigger instance and and its DependencyInstanceContext still needs to be kept in memory in case where users want to cancel a running trigger instance.
follow up:
Add index to execution_dependencies table for faster retrieval.
As execution_dependencies table keeps growing, we will need to implement purging mechanism to keep reasonable amount of historical executions.
|
3/1/2018 12:59:24 AM
(#1669)
Issue:
It's confusing which script to use.
If the wrong script is used, AZ admins are confused as to why the
servers are not running.
This type of mistakes have happened to multiple team members.
Solution:
Move the script that is designed to be called by the user facing scripts
to a new directory called internal.
Renamed scripts.
The console logs will now go to the current working directory instead of assuming that there is a logs directory under the current working directory.
Use the same startup/shutdown approach for the solo server.
Did some minor code clean up as well to improve readability and confirm
to the Google coding style.
Reduced code duplication in the shutdown scripts.
Todo: need to update docs to reflect the new script names.
Todo: Remove assumptions about where the AZ installation root directory
is and allow the start up and shutdown scripts to be invoked from a
directory other than the installation root. Also makes the conf file
configurations more portable i.e. in dependent of the working directory.
Testing:
Manually tested starting and stopping web, executor and solo servers.
e.g.
[~/oss/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.43.0-9-g9545a072 (move-start-stop-scripts)]$ ./bin/start-web.sh
[~/oss/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.43.0-9-g9545a072 (move-start-stop-scripts)]$ ./bin/shutdown-web.sh
Killing web-server. [pid: 32249], attempt: 1
shutdown succeeded
Testing executor:
start both an executor and a web server locally.
copy a test conf directory to the extracted binary directory.
Hit
http://localhost:8081/status
{
version: "3.43.0-9-g9545a072",
pid: "32950",
installationPath: "/Users/ruyang/oss/azkaban/azkaban-web-server/build/distributions/azkaban-web-server-3.43.0-9-g9545a072/lib/azkaban-web-server-3.43.0-9-g9545a072.jar",
usedMemory: 122591464,
xmx: 3817865216,
isDatabaseUp: true,
executorStatusMap: {
1: {
id: 1,
host: "localhost",
port: 12321,
isActive: true
}
}
}
[~/oss/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.43.0-9-g9545a072 (move-start-stop-scripts)]$ ./bin/shutdown-executor.sh
Killing executor. [pid: 32739], attempt: 1
shutdown succeeded
Made sure the process is no longer running.
---
shell script unit tests passed
[~/oss/azkaban/azkaban-common/src/test/bash (move-start-stop-scripts)]$ ./test_util.shunit2
test_is_process_running
test_kill_process_with_retry
Ran 2 tests.
OK
|
2/27/2018 11:10:42 PM
instance in db. (#1666)
This PR switched column mysql data type from datetime to bigint to keep the start/end time of dependency instance. The reason is datetime is not timezone agnostic(https://dev.mysql.com/doc/refman/5.5/en/datetime.html). It's less error-prone to keep UTC timestamp(System.currentTimeMillis()) in db.
|
2/27/2018 5:51:37 PM
3.43.0
(#1667)
changed error message from "max wait min must be longer than X min"
to "max wait min must be at least X min(s)"
|
2/22/2018 10:40:29 PM
finished trigger instances should be retrieved based on end time. Corrected SQL statement and added associated unit test.
|
