azkaban-aplcache

Details

diff --git a/az-examples/flow20-projects/basic.flow b/az-examples/flow20-projects/basic.flow
new file mode 100644
index 0000000..089a97a
--- /dev/null
+++ b/az-examples/flow20-projects/basic.flow
@@ -0,0 +1,22 @@
+---
+config:
+  failure.emails: noreply@foo.com
+
+nodes:
+  - name: jobC
+    type: noop
+    # jobC depends on jobA and jobB
+    dependsOn:
+      - jobA
+      - jobB
+
+  - name: jobA
+    type: command
+    config:
+      command: echo "This is an echoed text."
+
+  - name: jobB
+    type: command
+    config:
+      command: pwd
+
diff --git a/az-examples/flow20-projects/basicFlow20Project.zip b/az-examples/flow20-projects/basicFlow20Project.zip
new file mode 100644
index 0000000..b643e8a
Binary files /dev/null and b/az-examples/flow20-projects/basicFlow20Project.zip differ
diff --git a/az-examples/flow20-projects/embedded.flow b/az-examples/flow20-projects/embedded.flow
new file mode 100644
index 0000000..0a80efc
--- /dev/null
+++ b/az-examples/flow20-projects/embedded.flow
@@ -0,0 +1,25 @@
+---
+config:
+  failure.emails: noreply@foo.com
+
+nodes:
+  - name: jobC
+    type: noop
+    dependsOn:
+      - embedded_flow
+
+  - name: embedded_flow
+    type: flow
+    config:
+      prop: value
+    nodes:
+      - name: jobB
+        type: noop
+        dependsOn:
+          - jobA
+
+      - name: jobA
+        type: command
+        config:
+          command: pwd
+
diff --git a/az-examples/flow20-projects/embeddedFlow20Project.zip b/az-examples/flow20-projects/embeddedFlow20Project.zip
new file mode 100644
index 0000000..0263537
Binary files /dev/null and b/az-examples/flow20-projects/embeddedFlow20Project.zip differ
diff --git a/az-examples/flow20-projects/flow20.project b/az-examples/flow20-projects/flow20.project
new file mode 100644
index 0000000..4929753
--- /dev/null
+++ b/az-examples/flow20-projects/flow20.project
@@ -0,0 +1 @@
+azkaban-flow-version: 2.0

docs/createFlows.rst 134(+134 -0)

diff --git a/docs/createFlows.rst b/docs/createFlows.rst
new file mode 100644
index 0000000..e7aaac5
--- /dev/null
+++ b/docs/createFlows.rst
@@ -0,0 +1,134 @@
+.. _CreatingFlows:
+
+Creating Flows
+=============
+
+This section covers how to create your Azkaban flows using Azkaban Flow 2.0.
+Flow 1.0 will be deprecated in the future.
+
+
+*****
+Flow 2.0 Basics
+*****
+
+########
+Using Azkaban Flow 2.0 can be simple and straightforward. Just start with below severals steps:
+
+Step 1:
+######## 
+Create a simple file called ``flow20.project``. Add ``azkaban-flow-version`` to indicate this is a Flow 2.0 Azkaban project:
+::
+  azkaban-flow-version: 2.0
+ 
+Step 2:
+######## 
+Create another file called ``basic.flow``. Add a section called ``nodes``, which will contain all the jobs you want to run. You need to specify ``name`` and ``type`` for all the jobs. Most jobs will require the ``config`` section as well. We will talk more about it later. Below is a simple example of a command job.
+:: 
+  nodes:
+    - name: jobA
+      type: command
+      config:
+        command: echo "This is an echoed text."
+
+
+Step 3:
+########
+Select the two files you've already created and right click to compress them into a zip file called ``Archive.zip``. You can also create a new directory with these two files and then ``cd`` into the new directory and compress: ``zip -r Archive.zip .`` Please do not zip the new directory directly.
+
+Make sure you have already created a project on Azkaban ( See  :ref:`createProjects` ).
+You can then upload Archive.zip to your project through Web UI ( See :ref:`uploadProjects` ).
+
+Now you can click ``Execute Flow`` to test your first Flow 2.0 Azkaban project!
+
+*****
+Job Dependencies
+*****
+Jobs can have dependencies on each other.
+You can use ``dependsOn`` section to list all the parent jobs. In the below example, after jobA and jobB run successfully, jobC will start to run. 
+::
+
+  nodes:
+    - name: jobC
+      type: noop
+      # jobC depends on jobA and jobB
+      dependsOn:
+        - jobA
+        - jobB
+
+    - name: jobA
+      type: command
+      config:
+        command: echo "This is an echoed text."
+
+    - name: jobB
+      type: command
+      config:
+        command: pwd
+
+You can zip the new ``basic.flow`` and ``flow20.project`` again and then upload to Azkaban. Try to execute the flow and see the difference.
+
+*****
+Job Config
+*****
+Azkaban supports many job types. You just need to specify it in ``type``, and other job-related info goes to ``config`` section in the format of ``key: value`` pairs. Here is an example for a Pig job:
+::
+  nodes:
+    - name: pigJob
+      type: pig
+      config:
+        pig.script: sql/pig/script.pig
+
+You need to write your own pig script and put it in your project zip and then specify the path for the pig.script in the config section.
+
+*****
+Flow Config
+*****
+Not only can you configure individual jobs, you can also config the flow parameters for the entire flow.
+Simply add a ``config`` section at the beginning of the ``basic.flow`` file. For example:
+::
+  ---
+  config:
+    user.to.proxy: foo
+    failure.emails: noreply@foo.com
+
+  nodes:
+    - name: jobA
+      type: command
+      config:
+        command: echo "This is an echoed text."
+
+When you execute the flow, the ``user.to.proxy`` and ``failure.emails`` flow parameters will apply to all jobs inside the flow.
+
+*****
+Embedded Flows
+*****
+Flows can have subflows inside the flow just like job nodes. To create embedded flows, specify the type of the node as ``flow``. For example:
+::
+  nodes:
+    - name: embedded_flow
+      type: flow
+      config:
+        prop: value
+      nodes:
+        - name: jobB 
+          type: noop
+          dependsOn:
+            - jobA
+
+        - name: jobA
+          type: command
+          config:
+            command: pwd
+
+*****
+Download Examples
+*****
+You can download the simple Flow 2.0 project zip examples to start playing with Azkaban:
+
+* `basicFlow20Project.zip <https://github.com/azkaban/azkaban/blob/master/az-examples/flow20-projects/basicFlow20Project.zip>`_
+
+* `embeddedFlow20Project.zip <https://github.com/azkaban/azkaban/blob/master/az-examples/flow20-projects/embeddedFlow20Project.zip>`_
+
+
+
+

docs/index.rst 1(+1 -0)

diff --git a/docs/index.rst b/docs/index.rst
index a6b4374..92429d1 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -35,6 +35,7 @@ Features
    getStarted
    configuration
    userManager
+   createFlows
    useAzkaban
    eventTrigger
    ajaxApi
diff --git a/docs/useAzkaban.rst b/docs/useAzkaban.rst
index 18d0b50..eb01d05 100644
--- a/docs/useAzkaban.rst
+++ b/docs/useAzkaban.rst
@@ -7,6 +7,7 @@ This section covers how to use Azkaban Web UI to create, view and
 execution your flows.
 
 
+.. _createProjects:
 
 
 Create Projects
@@ -43,7 +44,7 @@ the project), you can delete the project, update the description,
 upload files and view the project logs from this page.
 
 
-
+.. _uploadProjects:
 
 
 Upload Projects