<html><head><title>Azkaban Setup Manual</title><style type="text/css">ol{margin:0;padding:0}.c18{vertical-align:top;width:468pt;border-style:solid;background-color:#efefef;border-color:#cccccc;border-width:1pt;padding:5pt 5pt 5pt 5pt}.c7{vertical-align:top;width:156pt;border-style:solid;border-color:#000000;border-width:1pt;padding:5pt 5pt 5pt 5pt}.c2{vertical-align:top;width:234pt;border-style:solid;border-color:#000000;border-width:1pt;padding:5pt 5pt 5pt 5pt}.c20{list-style-type:decimal;margin:0;padding:0}.c22{max-width:468pt;background-color:#ffffff;padding:72pt 72pt 72pt 72pt}.c23{list-style-type:disc;margin:0;padding:0}.c0{padding-left:0pt;direction:ltr;margin-left:36pt}.c14{color:inherit;text-decoration:inherit}.c10{font-size:9pt;font-family:"Courier New"}.c6{color:#1155cc;text-decoration:underline}.c1{line-height:1.0;direction:ltr}.c19{height:18pt}.c24{font-size:18pt}.c21{background-color:#d9d9d9}.c13{background-color:#cccccc}.c9{height:11pt}.c5{direction:ltr}.c15{font-weight:bold}.c17{font-size:24pt}.c8{font-style:italic}.c16{text-indent:36pt}.c4{height:0pt}.c11{background-color:#b7b7b7}.c12{border-collapse:collapse}.c3{font-size:10pt}.title{padding-top:24pt;line-height:1.15;text-align:left;color:#000000;font-size:36pt;font-family:"Arial";font-weight:bold;padding-bottom:6pt}.subtitle{padding-top:18pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:24pt;font-family:"Georgia";padding-bottom:4pt}li{color:#000000;font-size:11pt;font-family:"Arial"}p{color:#000000;font-size:11pt;margin:0;font-family:"Arial"}h1{padding-top:24pt;line-height:1.15;text-align:left;color:#000000;font-size:18pt;font-family:"Arial";font-weight:bold;padding-bottom:6pt}h2{padding-top:18pt;line-height:1.15;text-align:left;color:#000000;font-size:14pt;font-family:"Arial";font-weight:bold;padding-bottom:4pt}h3{padding-top:14pt;line-height:1.15;text-align:left;color:#666666;font-size:12pt;font-family:"Arial";font-weight:bold;padding-bottom:4pt}h4{padding-top:12pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:11pt;font-family:"Arial";padding-bottom:2pt}h5{padding-top:11pt;line-height:1.15;text-align:left;color:#666666;font-size:10pt;font-family:"Arial";font-weight:bold;padding-bottom:2pt}h6{padding-top:10pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:10pt;font-family:"Arial";padding-bottom:2pt}</style></head><body class="c22"><p class="c5 title"><a name="h.m5e41j34gnpn"></a><span class="c17">Azkaban Quickstart Manual</span></p><p class="c5 title"><a name="h.wy1awy8gek9i"></a><span class="c15 c24">Introduction</span></p><p class="c5"><span>Azkaban2 is a batch workflow job scheduler. It is the successor of the original Azkaban. Often times there is a need to run a set of jobs and processes in a particular order within a workflow. Azkaban will resolve the ordering through job dependencies and provide an easy to use web user interface to maintain and track your workflows.</span></p><p class="c5"><span>Here are a few features:</span></p><ol class="c23" start="1"><li class="c0"><span>Web UI</span></li><li class="c0"><span>Easy workflow uploads</span></li><li class="c0"><span>Easy to set up job dependencies</span></li><li class="c0"><span>Schedule workflows</span></li><li class="c0"><span>Authentication/Authorization (permissions on jobs)</span></li><li class="c0"><span>Ability to kill and restart workflows</span></li><li class="c0"><span>Modular and pluginable</span></li><li class="c0"><span>Project workspaces</span></li><li class="c0"><span>Logging and auditing of workflow and jobs</span></li></ol><h1 class="c5"><a name="h.iwjl2qhjh5k1"></a><span>Getting Started</span></h1><p class="c5 c16"><span>Azkaban2 is fairly easy to set up, although has more moving pieces than its predecessor. There are twho servers and a database that need to be setup:</span></p><ol class="c23" start="1"><li class="c0"><span class="c15">MySQL instance</span><span> - Azkaban uses MySQL to store projects and executions</span></li><li class="c0"><span class="c15">Azkaban Web Server</span><span> - Azkaban Web Server is a Jetty server which acts as the controller as well as the web interface</span></li><li class="c0"><span class="c15">Azkaban Executor Server</span><span> - Azkaban Executor Server executes submitted workflow.</span></li></ol><h2 class="c5"><a name="h.2p30s839mbas"></a><span>Download</span></h2><p class="c5 c16"><span>There are three download packages for Azkaban: the web server, the executor server and the MySQL setup scripts.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>The package download page will be maintained on the github wiki:</span></p><p class="c5"><span class="c6"><a class="c14" href="https://github.com/azkaban/azkaban2/wiki/Download-Packages">https://github.com/azkaban/azkaban2/wiki/Download-Packages</a></span></p><h2 class="c5"><a name="h.b86p3lvidzbb"></a><span>Setting up the DB</span></h2><p class="c5 c16"><span>Currently, Azkaban2 only uses MySQL as its data store. Installation of MySQL DB is not covered in this guide.</span></p><ol class="c23" start="1"><li class="c0"><span>Download the </span><span class="c8">azkaban-sql-script</span><span> tar. Contained in this archive are table creation scripts.</span></li><li class="c0"><span>Run the scripts on the MySQL instance to create your tables.</span></li></ol><h3 class="c5"><a name="h.corrtfl57k7"></a><span>Getting the JDBC Connector jar</span></h3><p class="c5 c16"><span>For various licensing reasons, Azkaban does not distribute the MySQL JDBC connector jar. You can download the jar from this link:</span><span><a class="c14" href="http://www.mysql.com/downloads/connector/j/"> </a></span><span class="c6"><a class="c14" href="http://www.mysql.com/downloads/connector/j/">http://www.mysql.com/downloads/connector/j/</a></span><span>. This jar will be needed for both the web server and the executor server.</span></p><p class="c9 c5"><span></span></p><h2 class="c5"><span>Setup the Web Server</span></h2><h3 class="c5"><span>Download and Install</span></h3><ol class="c20" start="1"><li class="c0"><span>Download the </span><span class="c8">azkaban-web-server</span><span> tar. Extract it into your azkaban web install directory.</span></li><li class="c0"><span>Copy the jdbc jar into </span><span class="c8">./extlib</span><span> directory. Azkaban will automatically look to this directory for external (non-distributed) jars.</span></li></ol><p class="c5"><span>In the conf dir, there should be several files:</span></p><p class="c9 c5"><span></span></p><a href="#" name="2bb1fe00b42e18c113b61b3f93628ffee64d79b7"></a><a href="#" name="0"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr><td class="c2 c13"><p class="c1"><span class="c15">File</span></p></td><td class="c2 c13"><p class="c1"><span class="c15">Description</span></p></td></tr><tr><td class="c2"><p class="c1"><span class="c3">azkaban.properties</span></p></td><td class="c2"><p class="c1"><span class="c3">Used by Azkaban for runtime properties</span></p></td></tr><tr><td class="c2"><p class="c1"><span class="c3">global.properties</span></p></td><td class="c2"><p class="c1"><span class="c3">Global static properties that are passed as shared properties to every workflow and job.</span></p></td></tr><tr><td class="c2"><p class="c1"><span class="c3">azkaban-users.xml</span></p></td><td class="c2"><p class="c1"><span class="c3">Used to add users and roles for authentication. This file is not used if the XmLUserManager is not set up to use it.</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.os05ke3hsw8e"></a><span>Setting up SSL</span></h3><p class="c5 c16"><span>Azkaban uses SSL socket connectors, which means a keystore will have to be available. You call follow the steps provided at this link (</span><span class="c6"><a class="c14" href="http://docs.codehaus.org/display/JETTY/How+to+configure+SSL">http://docs.codehaus.org/display/JETTY/How+to+configure+SSL</a></span><span>) to create one. Once a keystore file has been created, Azkaban must be given its location and password. Within </span><span class="c8">azkaban.properties</span><span>, the following properties should be overridden.</span></p><p class="c9 c5"><span></span></p><p class="c9 c5"><span></span></p><a href="#" name="c84ad6cecdf22bc8c85585fda0e371d780d42779"></a><a href="#" name="1"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr><td class="c18"><p class="c1"><span class="c10"> jetty.keystore=keystore<br> jetty.password=password<br> jetty.keypassword=password<br> jetty.truststore=keystore<br> jetty.trustpassword=password</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.bam7vnf9xml1"></a><span>Setting up the UserManager</span></h3><p class="c5"><span>Azkaban uses the UserManager to provide authentication and user roles. By default, Azkaban includes and uses the </span><span class="c8">XmlUserManager</span><span> which gets username/passwords and roles from the </span><span class="c8">azkaban-users.xml</span><span> as can be seen in the </span><span class="c8">azkaban.properties</span><span> file.</span></p><ol class="c23" start="1"><li class="c0"><span>user.manager.class=azkaban.user.XmlUserManager</span></li><li class="c0"><span>user.manager.xml.file=conf/azkaban-users.xml</span></li></ol><p class="c5"><span>The following is an example of the contents of the </span><span class="c8">azkaban-users.xml</span><span> file.</span></p><p class="c9 c5"><span></span></p><a href="#" name="2843b9afb96f9c3ad9b242b676fc033f472bd806"></a><a href="#" name="2"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c18"><p class="c5"><span class="c10"><azkaban-users> <br> <user username="azkaban" password="azkaban" roles="admin" groups="azkaban"/> <br> <role name="admin" permissions="ADMIN" /> <br></azkaban-user></span></p></td></tr></tbody></table><p class="c5"><span><br>It is possible to override the UserManager to use other methods of authentication (i.e. DB, JNDI, LDAP etc) by including your own implementation of the </span><span class="c8">azkaban.user.UserManager</span><span> interface and changing the </span><span class="c8">user.manager.class</span><span> property.</span></p><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.39wublce8e5i"></a><span>Setting up the DB</span></h3><p class="c5"><span>To point Azkaban web client to the MySQL instance, you will need to configure the following properties in </span><span class="c8">azkaban.properties</span><span>.</span></p><p class="c9 c5"><span></span></p><a href="#" name="45445316232cbad8025c1bb18bfc4c942f8334ff"></a><a href="#" name="3"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c18"><p class="c5"><span class="c10"> database.type=mysql<br> mysql.port=3306<br> mysql.host=localhost<br> mysql.database=azkaban2<br> mysql.user=azkaban<br> mysql.password=azkaban<br> mysql.numconnections=100</span></p></td></tr></tbody></table><p class="c5"><span><br></span><span>Currently MySQL is the only data store type supported in Azkaban. So </span><span class="c8">database.type</span><span> should always be </span><span class="c8">mysql</span><span>.</span></p><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.zhyyrwyzfk1u"></a><span>Executor Host and Port</span></h3><p class="c5"><span>The web client will need to know the executor server host and port. Use the following configuration settings in </span><span class="c8">azkaban.properties</span><span>. The executor.host is not necessary if the executor server is running locally to the web client.</span></p><p class="c9 c5"><span></span></p><a href="#" name="191c69dee6235ca9bcb78d0c5be00bfde76bb593"></a><a href="#" name="4"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c18"><p class="c5"><span class="c10">executor.port=12321</span></p><p class="c5"><span class="c10">executor.host=<url></span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.bho32ou5kytc"></a><span>Setting up the Web Client</span></h3><p class="c5"><span>Azkaban was built as a Jetty server. You are able to specify the ports and connection number that Azkaban will use. Please refer to Azkaban Web Server Settings section under Jetty settings.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>The color theme and name of the install can be customized as well, and is expecially useful in identifying multiple installed version of Azkaban.</span></p><h3 class="c5"><a name="h.hbjottygopcz"></a><span>Installing Viewer Plugins</span></h3><p class="c5"><span>Previous version of Azkaban contained an HDFS browser. This is now optional in newer Azkaban, and can be added as a plugin. It can be grabbed from github azkaban/azkaban-plugins.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>To install a viewer plugin, download and extract the plugin into the ./plugin/viewer directory.</span></p><p class="c5"><span>More instructions to come.</span></p><h3 class="c5"><a name="h.rqy8dl8k2ydr"></a><span>Running Web Server</span></h3><p class="c5"><span>The </span><span class="c8">bin</span><span> directory should contain a </span><span class="c8">azkaban-web-start.sh</span><span>. Use it to start up Azkaban Web Server. Use </span><span class="c8">azkaban-web-shutdown.sh</span><span> to shut azkaban down.</span></p><h2 class="c5"><a name="h.vfenajwcdi9l"></a><span>Azkaban W</span><span>eb Server Settings</span></h2><p class="c9 c5"><span></span></p><p class="c5"><span class="c15">General Properties</span></p><a href="#" name="ac0d6331714708cba02ad9ed5c39932a044ecae0"></a><a href="#" name="5"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr><td class="c7 c21"><p class="c1"><span class="c15">Property</span></p></td><td class="c7 c21"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c21"><p class="c1"><span class="c15">Default</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">azkaban.name</span></p></td><td class="c7"><p class="c1"><span class="c3">The name of the azkaban instance that will show up in the UI. Useful if you run more than one Azkaban instance.</span></p></td><td class="c7"><p class="c1"><span class="c3">Local</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">azkaban.label</span></p></td><td class="c7"><p class="c1"><span class="c3">A label to describe the Azkaban instance.</span></p></td><td class="c7"><p class="c1"><span class="c3">My Local Azkaban</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">azkaban.color</span></p></td><td class="c7"><p class="c1"><span class="c3">Hex value that allows you to set a style color for the Azkaban UI.</span></p></td><td class="c7"><p class="c1"><span class="c3">#FF3601 (red)</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">web.resource.dir</span></p></td><td class="c7"><p class="c1"><span class="c3">Sets the directory for the ui’s css and javascript files</span></p></td><td class="c7"><p class="c1"><span class="c3">src/web</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">default.timezone</span></p></td><td class="c7"><p class="c1"><span class="c3">The timezone that will be displayed by Azkaban.</span></p></td><td class="c7"><p class="c1"><span class="c3">America/Los_Angeles</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">user.manager.class</span></p></td><td class="c7"><p class="c1"><span class="c3">The user manager that is used to authenticate a user. The default is an XML user manager, but it can be overwritten to support other authethentication methods, such as JDNI.</span></p></td><td class="c7"><p class="c1"><span class="c3">azkaban.user.XmlUserManager</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">mail.sender</span></p></td><td class="c7"><p class="c1"><span class="c3">The email address that azkaban uses to send emails.</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">mail.host</span></p></td><td class="c7"><p class="c1"><span class="c3">The email server host machine</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">mail.user</span></p></td><td class="c7"><p class="c1"><span class="c3">The email server user name</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">mail.password</span></p></td><td class="c7"><p class="c1"><span class="c3">The email password user name</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">azkaban.should.proxy</span></p></td><td class="c7"><p class="c1"><span class="c3">Used by the HDFS browser. Set to true if using Hadoop 1.0+ with security turned on. Will soon be removed.</span></p></td><td class="c7"><p class="c1"><span class="c3">false</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">proxy.keytab.location</span></p></td><td class="c7"><p class="c1"><span class="c3">Used by the HDFS browser. Set to true if using Hadoop 1.0+ with security turned on. Will soon be removed.</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">proxy.user</span></p></td><td class="c7"><p class="c1"><span class="c3">The proxy user</span></p></td><td class="c7"><p class="c1 c9"><span class="c3"></span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">viewer.plugin.dir</span></p></td><td class="c7"><p class="c1"><span class="c3">Directory where viewer plugins will be installed</span></p></td><td class="c7"><p class="c1"><span class="c3">plugins/viewer</span></p></td></tr></tbody></table><p class="c5 c9"><span></span></p><p class="c5"><span class="c15">Jetty Properties</span></p><p class="c9 c5"><span></span></p><a href="#" name="73fd84b587922cb283ec53f2e3ce4dd180f648b4"></a><a href="#" name="6"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr><td class="c7 c13"><p class="c1"><span class="c15">Property</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Default</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.maxThreads</span></p></td><td class="c7"><p class="c1"><span class="c3">Max request threads</span></p></td><td class="c7"><p class="c1"><span class="c3">25</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.ssl.port</span></p></td><td class="c7"><p class="c1"><span class="c3">The ssl port</span></p></td><td class="c7"><p class="c1"><span class="c3">8443</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.keystore</span></p></td><td class="c7"><p class="c1"><span class="c3">The keystore file</span></p></td><td class="c7"><p class="c1"><span class="c3">keystore</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.password</span></p></td><td class="c7"><p class="c1"><span class="c3">The jetty password</span></p></td><td class="c7"><p class="c1"><span class="c3">password</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.keypassword</span></p></td><td class="c7"><p class="c1"><span class="c3">The keypassword</span></p></td><td class="c7"><p class="c1"><span class="c3">password</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.truststore</span></p></td><td class="c7"><p class="c1"><span class="c3">The trust store</span></p></td><td class="c7"><p class="c1"><span class="c3">keystore</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">jetty.trustpassword</span></p></td><td class="c7"><p class="c1"><span class="c3">The trust password</span></p></td><td class="c7"><p class="c1"><span class="c3">password</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><p class="c5"><span class="c15">MySQL Connection Properties</span></p><p class="c9 c5"><span></span></p><a href="#" name="7a6602c4e399d0b014db1d704d9e86bc50ccba44"></a><a href="#" name="7"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c7 c13"><p class="c1"><span class="c15">Property</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Default</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">database.type</span></p></td><td class="c7"><p class="c1"><span class="c3">The database type. Currently, the only database supported is mysql.</span></p></td><td class="c7"><p class="c1"><span class="c3">mysql</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.port</span></p></td><td class="c7"><p class="c1"><span class="c3">The port to the mysql db</span></p></td><td class="c7"><p class="c1"><span class="c3">3306</span></p></td></tr><tr class="c19"><td class="c7"><p class="c1"><span class="c3">mysql.host</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql host</span></p></td><td class="c7"><p class="c1"><span class="c3">localhost</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.database</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql database</span></p></td><td class="c7"><p class="c1"><span class="c3">azkaban2</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.user</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql user</span></p></td><td class="c7"><p class="c1"><span class="c3">azkaban</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.password</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql password</span></p></td><td class="c7"><p class="c1"><span class="c3">password</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.numconnections</span></p></td><td class="c7"><p class="c1"><span class="c3">The number of connections that Azkaban web client can open to the database </span></p></td><td class="c7"><p class="c1"><span class="c3">100</span></p></td></tr></tbody></table><p class="c9 c5"><span class="c15"></span></p><p class="c5"><span class="c15">Executor Server Properties</span></p><p class="c9 c5"><span class="c15"></span></p><a href="#" name="bdcd5f269529cc593edc385149bcf056e3c4e554"></a><a href="#" name="8"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr><td class="c7 c11"><p class="c1"><span class="c15">Properties</span></p></td><td class="c7 c11"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c11"><p class="c1"><span class="c15">Default</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">executor.port</span></p></td><td class="c7"><p class="c1"><span class="c3">The port for the azkaban executor server</span></p></td><td class="c7"><p class="c1"><span class="c3">12321</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">executor.host</span></p></td><td class="c7"><p class="c1"><span class="c3">The host for azkaban executor server</span></p></td><td class="c7"><p class="c1"><span class="c3">localhost</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><p class="c9 c5"><span></span></p><h2 class="c5"><a name="h.iukep39qys2g"></a><span>Setup the Executor Server</span></h2><h3 class="c5"><a name="h.eo9zrotnrlyo"></a><span>Download and Install</span></h3><ol class="c20" start="1"><li class="c0"><span>Download the </span><span class="c8">azkaban-executor-server</span><span> tar. Extract it into your azkaban executor install directory.</span></li><li class="c0"><span>Copy the jdbc jar into </span><span class="c8">./extlib</span><span> directory. Azkaban will automatically look to this directory for external (non-distributed) jars.</span></li></ol><p class="c5"><span>In the conf dir, there should be several files:</span></p><p class="c9 c5"><span></span></p><a href="#" name="4d5b1ed7d225a69ea785e7f16795265fe785708b"></a><a href="#" name="9"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c2 c13"><p class="c1"><span class="c15">File</span></p></td><td class="c2 c13"><p class="c1"><span class="c15">Description</span></p></td></tr><tr class="c4"><td class="c2"><p class="c1"><span class="c3">azkaban.properties</span></p></td><td class="c2"><p class="c1"><span class="c3">Used by Azkaban for runtime properties</span></p></td></tr><tr class="c4"><td class="c2"><p class="c1"><span class="c3">global.properties</span></p></td><td class="c2"><p class="c1"><span class="c3">Global static properties that are passed as shared properties to every workflow and job.</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><p class="c5"><span>Much like the Web Server, both the port and the database needs to be set up.</span></p><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.sdrhglkc6yj2"></a><span>Setting up the DB</span></h3><p class="c5"><span>Azkaban executor server will need to be set up to connect to the MySQL instance. You will need to configure the following properties in </span><span class="c8">azkaban.properties</span><span>.</span></p><p class="c9 c5"><span></span></p><a href="#" name="fb95f7517d06f3974a9be16406174eb530c8346b"></a><a href="#" name="10"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c18"><p class="c5"><span class="c10"> database.type=mysql<br> mysql.port=3306<br> mysql.host=localhost<br> mysql.database=azkaban2<br> mysql.user=azkaban<br> mysql.password=azkaban<br> mysql.numconnections=100</span></p></td></tr></tbody></table><p class="c5"><span><br>Currently MySQL is the only data store type supported in Azkaban. So </span><span class="c8">database.type</span><span> should always be </span><span class="c8">mysql</span><span>.</span></p><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.biw9yghh97xx"></a><span>Executor Host and Port</span></h3><p class="c5"><span>The executor port will need. Use the following configuration settings in </span><span class="c8">azkaban.properties.</span></p><p class="c9 c5"><span></span></p><a href="#" name="eb379d62e3101c642e09a417f3f38da1b07f5612"></a><a href="#" name="11"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c18"><p class="c5"><span class="c10">executor.maxThreads=50</span></p><p class="c5"><span class="c10">executor.port=12321</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><h2 class="c5"><a name="h.y6bkxaxvg7u6"></a><span>Azkaban Executor Server Settings</span></h2><p class="c5"><span class="c15">MySQL Connection Properties</span></p><p class="c9 c5"><span></span></p><a href="#" name="7a6602c4e399d0b014db1d704d9e86bc50ccba44"></a><a href="#" name="12"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c7 c13"><p class="c1"><span class="c15">Property</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c13"><p class="c1"><span class="c15">Default</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">database.type</span></p></td><td class="c7"><p class="c1"><span class="c3">The database type. Currently, the only database supported is mysql.</span></p></td><td class="c7"><p class="c1"><span class="c3">mysql</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.port</span></p></td><td class="c7"><p class="c1"><span class="c3">The port to the mysql db</span></p></td><td class="c7"><p class="c1"><span class="c3">3306</span></p></td></tr><tr class="c19"><td class="c7"><p class="c1"><span class="c3">mysql.host</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql host</span></p></td><td class="c7"><p class="c1"><span class="c3">localhost</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.database</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql database</span></p></td><td class="c7"><p class="c1"><span class="c3">azkaban2</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.user</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql user</span></p></td><td class="c7"><p class="c1"><span class="c3">azkaban</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.password</span></p></td><td class="c7"><p class="c1"><span class="c3">The mysql password</span></p></td><td class="c7"><p class="c1"><span class="c3">password</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">mysql.numconnections</span></p></td><td class="c7"><p class="c1"><span class="c3">The number of connections that Azkaban web client can open to the database </span></p></td><td class="c7"><p class="c1"><span class="c3">100</span></p></td></tr></tbody></table><p class="c9 c5"><span class="c15"></span></p><p class="c5"><span class="c15">Executor Server Properties</span></p><p class="c9 c5"><span class="c15"></span></p><a href="#" name="43739021fb6c97b63c7ad1875c3f0232c7b43223"></a><a href="#" name="13"></a><table cellpadding="0" cellspacing="0" class="c12"><tbody><tr class="c4"><td class="c7 c11"><p class="c1"><span class="c15">Properties</span></p></td><td class="c7 c11"><p class="c1"><span class="c15">Description</span></p></td><td class="c7 c11"><p class="c1"><span class="c15">Default</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">executor.port</span></p></td><td class="c7"><p class="c1"><span class="c3">The port for the azkaban executor server</span></p></td><td class="c7"><p class="c1"><span class="c3">12321</span></p></td></tr><tr class="c4"><td class="c7"><p class="c1"><span class="c3">executor.maxThreads</span></p></td><td class="c7"><p class="c1"><span class="c3">The maximum number flows that are accepted by the executor</span></p></td><td class="c7"><p class="c1"><span class="c3">50</span></p></td></tr><tr><td class="c7"><p class="c1"><span class="c3">azkaban.jobtype.plugin.dir</span></p></td><td class="c7"><p class="c1"><span class="c3">Dire</span></p></td><td class="c7"><p class="c1"><span class="c3">plugins/jobtypes</span></p></td></tr></tbody></table><p class="c9 c5"><span></span></p><h3 class="c5"><a name="h.ljlh0q6qwcs"></a><span>Installing JobType Plugins</span></h3><p class="c5"><span>Azkaban Executor can run command, javaprocess type jobs. Other jobs types can be added by dropping them into the jobtypes plugin directory. Common job types like ‘java’ type and ‘pig’ type can be downloaded from github azkaban/azkaban-plugins.</span></p><p class="c9 c5"><span class="c24 c15"></span></p><p class="c5"><span class="c24 c15">Upgrading Azkaban</span></p><p class="c5"><span>Having azkaban as a web server and a executor server in separate processes gives us the ability to roll the upgrade and not shut down long lived jobs.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>To do this, install the newer executor server and change the executor port. You’ll need to also update the executor server port for the web client. Restarting the web client should point to the new executor server.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>Any running jobs in the old executor should be auto detected by the web client. When the old executor finishes running its flow, it should be safe to shut the executor server down.</span></p><p class="c9 c5"><span></span></p><p class="c5"><span>Note that the scheduler runs in the web server. If you shut down the web server, you may skip any scheduled jobs during this time.</span></p><p class="c9 c5"><span></span></p></body></html>