azkaban-aplcache
Changes
az-examples/flow20-projects/basic.flow 22(+22 -0)
az-examples/flow20-projects/embedded.flow 25(+25 -0)
docs/createFlows.rst 134(+134 -0)
docs/index.rst 1(+1 -0)
docs/useAzkaban.rst 3(+2 -1)
Details
az-examples/flow20-projects/basic.flow 22(+22 -0)
diff --git a/az-examples/flow20-projects/basic.flow b/az-examples/flow20-projects/basic.flow
new file mode 100644
index 0000000..089a97a
--- /dev/null
+++ b/az-examples/flow20-projects/basic.flow
@@ -0,0 +1,22 @@
+---
+config:
+ failure.emails: noreply@foo.com
+
+nodes:
+ - name: jobC
+ type: noop
+ # jobC depends on jobA and jobB
+ dependsOn:
+ - jobA
+ - jobB
+
+ - name: jobA
+ type: command
+ config:
+ command: echo "This is an echoed text."
+
+ - name: jobB
+ type: command
+ config:
+ command: pwd
+
diff --git a/az-examples/flow20-projects/basicFlow20Project.zip b/az-examples/flow20-projects/basicFlow20Project.zip
new file mode 100644
index 0000000..b643e8a
Binary files /dev/null and b/az-examples/flow20-projects/basicFlow20Project.zip differ
az-examples/flow20-projects/embedded.flow 25(+25 -0)
diff --git a/az-examples/flow20-projects/embedded.flow b/az-examples/flow20-projects/embedded.flow
new file mode 100644
index 0000000..0a80efc
--- /dev/null
+++ b/az-examples/flow20-projects/embedded.flow
@@ -0,0 +1,25 @@
+---
+config:
+ failure.emails: noreply@foo.com
+
+nodes:
+ - name: jobC
+ type: noop
+ dependsOn:
+ - embedded_flow
+
+ - name: embedded_flow
+ type: flow
+ config:
+ prop: value
+ nodes:
+ - name: jobB
+ type: noop
+ dependsOn:
+ - jobA
+
+ - name: jobA
+ type: command
+ config:
+ command: pwd
+
diff --git a/az-examples/flow20-projects/embeddedFlow20Project.zip b/az-examples/flow20-projects/embeddedFlow20Project.zip
new file mode 100644
index 0000000..0263537
Binary files /dev/null and b/az-examples/flow20-projects/embeddedFlow20Project.zip differ
diff --git a/az-examples/flow20-projects/flow20.project b/az-examples/flow20-projects/flow20.project
new file mode 100644
index 0000000..4929753
--- /dev/null
+++ b/az-examples/flow20-projects/flow20.project
@@ -0,0 +1 @@
+azkaban-flow-version: 2.0
docs/createFlows.rst 134(+134 -0)
diff --git a/docs/createFlows.rst b/docs/createFlows.rst
new file mode 100644
index 0000000..e7aaac5
--- /dev/null
+++ b/docs/createFlows.rst
@@ -0,0 +1,134 @@
+.. _CreatingFlows:
+
+Creating Flows
+=============
+
+This section covers how to create your Azkaban flows using Azkaban Flow 2.0.
+Flow 1.0 will be deprecated in the future.
+
+
+*****
+Flow 2.0 Basics
+*****
+
+########
+Using Azkaban Flow 2.0 can be simple and straightforward. Just start with below severals steps:
+
+Step 1:
+########
+Create a simple file called ``flow20.project``. Add ``azkaban-flow-version`` to indicate this is a Flow 2.0 Azkaban project:
+::
+ azkaban-flow-version: 2.0
+
+Step 2:
+########
+Create another file called ``basic.flow``. Add a section called ``nodes``, which will contain all the jobs you want to run. You need to specify ``name`` and ``type`` for all the jobs. Most jobs will require the ``config`` section as well. We will talk more about it later. Below is a simple example of a command job.
+::
+ nodes:
+ - name: jobA
+ type: command
+ config:
+ command: echo "This is an echoed text."
+
+
+Step 3:
+########
+Select the two files you've already created and right click to compress them into a zip file called ``Archive.zip``. You can also create a new directory with these two files and then ``cd`` into the new directory and compress: ``zip -r Archive.zip .`` Please do not zip the new directory directly.
+
+Make sure you have already created a project on Azkaban ( See :ref:`createProjects` ).
+You can then upload Archive.zip to your project through Web UI ( See :ref:`uploadProjects` ).
+
+Now you can click ``Execute Flow`` to test your first Flow 2.0 Azkaban project!
+
+*****
+Job Dependencies
+*****
+Jobs can have dependencies on each other.
+You can use ``dependsOn`` section to list all the parent jobs. In the below example, after jobA and jobB run successfully, jobC will start to run.
+::
+
+ nodes:
+ - name: jobC
+ type: noop
+ # jobC depends on jobA and jobB
+ dependsOn:
+ - jobA
+ - jobB
+
+ - name: jobA
+ type: command
+ config:
+ command: echo "This is an echoed text."
+
+ - name: jobB
+ type: command
+ config:
+ command: pwd
+
+You can zip the new ``basic.flow`` and ``flow20.project`` again and then upload to Azkaban. Try to execute the flow and see the difference.
+
+*****
+Job Config
+*****
+Azkaban supports many job types. You just need to specify it in ``type``, and other job-related info goes to ``config`` section in the format of ``key: value`` pairs. Here is an example for a Pig job:
+::
+ nodes:
+ - name: pigJob
+ type: pig
+ config:
+ pig.script: sql/pig/script.pig
+
+You need to write your own pig script and put it in your project zip and then specify the path for the pig.script in the config section.
+
+*****
+Flow Config
+*****
+Not only can you configure individual jobs, you can also config the flow parameters for the entire flow.
+Simply add a ``config`` section at the beginning of the ``basic.flow`` file. For example:
+::
+ ---
+ config:
+ user.to.proxy: foo
+ failure.emails: noreply@foo.com
+
+ nodes:
+ - name: jobA
+ type: command
+ config:
+ command: echo "This is an echoed text."
+
+When you execute the flow, the ``user.to.proxy`` and ``failure.emails`` flow parameters will apply to all jobs inside the flow.
+
+*****
+Embedded Flows
+*****
+Flows can have subflows inside the flow just like job nodes. To create embedded flows, specify the type of the node as ``flow``. For example:
+::
+ nodes:
+ - name: embedded_flow
+ type: flow
+ config:
+ prop: value
+ nodes:
+ - name: jobB
+ type: noop
+ dependsOn:
+ - jobA
+
+ - name: jobA
+ type: command
+ config:
+ command: pwd
+
+*****
+Download Examples
+*****
+You can download the simple Flow 2.0 project zip examples to start playing with Azkaban:
+
+* `basicFlow20Project.zip <https://github.com/azkaban/azkaban/blob/master/az-examples/flow20-projects/basicFlow20Project.zip>`_
+
+* `embeddedFlow20Project.zip <https://github.com/azkaban/azkaban/blob/master/az-examples/flow20-projects/embeddedFlow20Project.zip>`_
+
+
+
+
docs/index.rst 1(+1 -0)
diff --git a/docs/index.rst b/docs/index.rst
index a6b4374..92429d1 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -35,6 +35,7 @@ Features
getStarted
configuration
userManager
+ createFlows
useAzkaban
eventTrigger
ajaxApi
docs/useAzkaban.rst 3(+2 -1)
diff --git a/docs/useAzkaban.rst b/docs/useAzkaban.rst
index 18d0b50..eb01d05 100644
--- a/docs/useAzkaban.rst
+++ b/docs/useAzkaban.rst
@@ -7,6 +7,7 @@ This section covers how to use Azkaban Web UI to create, view and
execution your flows.
+.. _createProjects:
Create Projects
@@ -43,7 +44,7 @@ the project), you can delete the project, update the description,
upload files and view the project logs from this page.
-
+.. _uploadProjects:
Upload Projects