2/6/2018 3:06:33 AM
a singleton class in the AZ web server to process all flow trigger-related operations. Externally it provides following operations -
Create a trigger instance based on trigger definition.
Cancel a trigger instance.
Query running and historic trigger instances.
Recover incomplete trigger instances.
Internally, it
maintains the list of running trigger instance in memory.
updates status, starttime/endtime of trigger instance.
persists trigger instance to DB.
FlowTriggerService will be leveraged by Quartz scheduler, our new AZ scheduler to schedule starting triggers.
Flow Trigger Service internally contains a single-threaded executor service to process all trigger instance related operation to avoid complicated cases of race condition.
|
1/31/2018 10:44:39 PM
always get confused when they read
>See logs at: https://blahblahXXX://
This PR adds another log message before this message, in order to instruct users how to set job ELK logger.
|
1/31/2018 9:15:44 PM
of the plugin management module for flow trigger dependency type.
We implemented data dependency interface and keep interface code inside main Azkaban open source repo and allow both internal and OSS developers to implement the interface for their own dependency type as plugin.
Plugin management module loads plugin property files from plugin directory and created corresponding DependencyCheck objects based on dependency plugin properties upon plugin management initialization or upon request(when doing plugin deployment).
|
1/29/2018 10:24:23 PM
3.41.0
(#1619)
This PR added two classes to handle status change of trigger instance and dependency instance:
DependencyInstanceProcess/TriggerInstanceProcessor, handling the work post status update. E.x, execute a flow when trigger instance becomes successful, shoot an email when trigger instance is cancelled, persist status update into database.
|
1/25/2018 10:34:38 PM
for flow trigger. The loader provides all DB related operations for flow trigger instance and dependency instance(#1611) to interact with dependency instance table(#1612).
The PR also added toString(), hashCode() and equals() to several existing classes for complementary purpose.
|
1/23/2018 10:01:23 PM
to create table to keep execution of all dependency instances. There will be no table specifically for trigger instance since trigger instance is nothing but the collection of dependency instances. So trigger instance can be easily constructed from dependency instances. The table will be used to keep the historic executions for users to query in the UI and to recover the running triggers from web server restart.
|
1/23/2018 9:18:32 PM
classes for flow trigger feature: TriggerInstance and DependencyInstance.
Conceptually trigger Instance represents an execution of a flow trigger. It holds execution context such as a historically unique trigger execution id, execution status, start/end time, and a list of dependency instances.
Similarly dependency Instance denotes an execution of a dependency performing availability check on a particular data dependency. It holds execution context such as execution status and start/end time, all dependency Instances within the same trigger instance share the same execution id as trigger instance.
A trigger instance and its dependency instances will be created on scheduled trigger start time. The trigger instance will wait until all dependency instances are available, at which point a new flow instance is created, or the maximum allowed wait time is exceeded. Once successfully created, trigger instance will equipped with an UUID to identify itself, initial status as running, and a list of dependency instances whose status are running as well.
Trigger instance can be cancelled manually by user, by timeout when max waiting time is exceeded, or by internal dependency failure.
Trigger/Dependency instance has following status(Running, Cancelling, Cancelled, Succeeded).
Since trigger instance is nothing but a collection of dependency instances, it doesn’t have variables to keep status, start or endtime, all of which can be quickly inferred from its belonging dependencies’ status, start/endtime, which unburdens us from maintaining extra variables.
Cancellation cause will be attached to dependency instance for better user experience when cancelled for whatever reason.
Valid cancellation cause includes:
TIMEOUT, // cancellation is issued due to exceeding max wait time
MANUAL, // cancellation is issued by user
FAILURE, // cancellation is issued by internal dependency instance failure
CASCADING // cancelled by cascading failure(peer dependency is cancelled)
|
|
|
1/23/2018 5:28:15 PM
PR made several changes to FlowTrigger/FlowTriggerDependency class
Made FlowTrigger and FlowTriggerDependency Serializable. This is to required by Quartz scheduler to persist scheduled job.
Added error message when doing argument validation. This is for better user experience when misconfigured flow trigger is uploaded.
Converted dependencies type from list to a hashmap with name as key for convenient lookup.
|
1/23/2018 5:26:42 PM
updates to Flow Trigger Interface set.
added runtimeProps to DependencyInstanceContext#run argument list. runtimeProps might be populated with runtime properties like starttime by azkaban if needed.
added onCancel method to DependencyInstanceCallback.
added hashmap backed implementations to key/value property holder interfaces: DependencyInstanceConfig, DependencyInstanceRuntimeProps and DependencyPluginConfig.
removed kill method DependencyCheck and added cancel method to DependencyInstanceContext to achieve the same purpose.
|
