flow_trigger.flow

---
# Flow trigger
trigger:
  maxWaitMins: 2880
  schedule:
    type: cron
    value: 0 0/2 * * * ?

  triggerDependencies:
    - name: dep1 # an unique name to identify the dependency
      type: kafka
      params:
        match: .*
        topic: AzEvent_Topic4
    - name: dep2 # an unique name to identify the dependency
      type: kafka
      params:
        match: hadoop?.*
        topic: AzEvent_Topic1
    - name: dep3 # an unique name to identify the dependency
      type: kafka
      params:
        match: .*Partition[A-Z]....Event
        topic: AzEvent_Topic4


 A map data structure that enables efficient lookup by topic and adding/removing topic event pairs.
 Structure looks like:
 {
  Topic1:{
    Rule1:[List of dependencies]
    Rule2:[List of dependencies]
  }
  Topic2:{
    Rule1:[List of dependencies]
    Rule2:[List of dependencies]
  }
  Topic3:{
    Rule1:[List of dependencies]
    Rule2:[List of dependencies]
  }
}

# All flow level properties here
config:
  #failure.emails: example@example.com

# This section defines the list of jobs
# A node can be a job or a flow
# In this example, all nodes are jobs
nodes:
  # Job definition
  # The job definition is like a YAMLified version of properties file
  # with one major difference. All custom properties are now clubbed together
  # in a config section in the definition.
  # The first line describes the name of the job
  - name: shell_end
    # Describe the type of the job
    type: noop

    # List the dependencies of the job
    dependsOn:
      - shell_pwd
      - shell_echo

  - name: shell_echo
    # Describe the type of the job
    type: command
    config:
      command: echo "This is an echoed text."

  - name: shell_pwd
    # Describe the type of the job
    type: command
    config:
      command: sleep 10