azkaban-uncached
Changes
docs/AzkabanInstallInstructions.html 1(+1 -0)
Details
docs/AzkabanInstallInstructions.html 1(+1 -0)
diff --git a/docs/AzkabanInstallInstructions.html b/docs/AzkabanInstallInstructions.html
new file mode 100644
index 0000000..ae67614
--- /dev/null
+++ b/docs/AzkabanInstallInstructions.html
@@ -0,0 +1 @@
+<html><head><title>Azkaban Install Instructions</title><style type="text/css">ol{margin:0;padding:0}.c1{vertical-align:top;width:156pt;border-style:solid;border-color:#000000;border-width:1pt;padding:5pt 5pt 5pt 5pt}.c12{list-style-type:disc;margin:0;padding:0}.c14{max-width:468pt;background-color:#ffffff;padding:72pt 72pt 72pt 72pt}.c9{font-size:9pt;font-family:"Courier New"}.c11{padding-left:0pt;margin-left:36pt}.c20{color:inherit;text-decoration:inherit}.c19{color:#1155cc;text-decoration:underline}.c2{line-height:1.0;direction:ltr}.c16{background-color:#cccccc}.c13{height:18pt}.c17{background-color:#b7b7b7}.c15{font-size:18pt}.c10{font-weight:bold}.c8{height:0pt}.c7{height:11pt}.c6{background-color:#d9d9d9}.c21{font-size:24pt}.c0{font-style:italic}.c3{direction:ltr}.c18{font-size:14pt}.c5{border-collapse:collapse}.c4{font-size:10pt}.title{padding-top:24pt;line-height:1.15;text-align:left;color:#000000;font-size:36pt;font-family:"Arial";font-weight:bold;padding-bottom:6pt}.subtitle{padding-top:18pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:24pt;font-family:"Georgia";padding-bottom:4pt}li{color:#000000;font-size:11pt;font-family:"Arial"}p{color:#000000;font-size:11pt;margin:0;font-family:"Arial"}h1{padding-top:24pt;line-height:1.15;text-align:left;color:#000000;font-size:18pt;font-family:"Arial";font-weight:bold;padding-bottom:6pt}h2{padding-top:18pt;line-height:1.15;text-align:left;color:#000000;font-size:14pt;font-family:"Arial";font-weight:bold;padding-bottom:4pt}h3{padding-top:14pt;line-height:1.15;text-align:left;color:#666666;font-size:12pt;font-family:"Arial";font-weight:bold;padding-bottom:4pt}h4{padding-top:12pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:11pt;font-family:"Arial";padding-bottom:2pt}h5{padding-top:11pt;line-height:1.15;text-align:left;color:#666666;font-size:10pt;font-family:"Arial";font-weight:bold;padding-bottom:2pt}h6{padding-top:10pt;line-height:1.15;text-align:left;color:#666666;font-style:italic;font-size:10pt;font-family:"Arial";padding-bottom:2pt}</style></head><body class="c14"><p class="c3"><span class="c10 c21">Azkaban 2</span></p><p class="c7 c3"><span class="c10 c18"></span></p><p class="c3"><span class="c10 c18">Getting started</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Azkaban needs 3 different components to run. </span></p><ol class="c12" start="1"><li class="c11 c3"><span>There is the Web UI, which is where users will upload, schedule and submit jobs to run. </span></li><li class="c3 c11"><span>The Executor Server runs the jobs and may be remote.</span></li><li class="c11 c3"><span>A MySQL db is where the projects are uploaded, as well as the state of executing jobs are updated.</span></li></ol><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">1. Setup the DB.</span></p><p class="c3"><span>Azkaban uses MySQL to run. You will need to download the mysql jdbc connector jar since Azkaban doesn’t distribute it. Download it here: </span><span class="c19"><a class="c20" href="http://www.mysql.com/downloads/connector/j/">http://www.mysql.com/downloads/connector/j/</a></span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Once the MySQL DB is set up, you will need to create the tables that Azkaban will use. Extract the </span><span class="c0">azkaban-sql-script</span><span> archive. Use the</span><span class="c0"> *.sql </span><span>scripts to create the tables.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">2. Setup the Web UI</span></p><p class="c3"><span>Extract the </span><span class="c0">azkaban-web-server</span><span> archive to the install directory. Copy the mysql jdbc connector to the</span><span class="c0"> ./extlibs</span><span> directory.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Azkaban uses SSL, and so a keystore will need to be created. Follow the following instructions on how to configure Azkaban jetty for SSL. </span><span class="c19"><a class="c20" href="http://docs.codehaus.org/display/JETTY/How+to+configure+SSL">http://docs.codehaus.org/display/JETTY/How+to+configure+SSL</a></span></p><p class="c7 c3"><span></span></p><p class="c3"><span>In the </span><span class="c0">./conf </span><span>directory, there are several settings files. The </span><span class="c0">azkaban.properties</span><span> file is used by azkaban for its general settings. The </span><span class="c0">azkaban-users.xml </span><span>is used by the XmlUserManager for authentication, and the </span><span class="c0">global.properties</span><span> are the properties that are passed as shared properties to every workflow and job.</span></p><p class="c3 c7"><span></span></p><p class="c3"><span>By default, the method of authentication is the azkaban.user.XmlUserManager class, and uses the </span><span class="c0">azkaban-users.xml.</span><span> By implementing the azkaban.user.UserManager interface, you can override the authentication method.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>To use the XmlUserManager, just add a user entry to the xml file. Note that you’ll have to restart to server to pick up new users.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Example.</span></p><p class="c3"><span class="c9"><azkaban-users></span></p><p class="c3"><span class="c9"> <user username="azkaban" password="azkaban" roles="admin" groups="azkaban"/></span></p><p class="c3"><span class="c9"> <role name="admin" permissions="ADMIN" /></span></p><p class="c3"><span class="c9"></azkaban-users></span></p><p class="c7 c3"><span></span></p><p class="c3"><span>For hadoop, you will want to start Azkaban with the environtment variable HADOOP_HOME pointing to the hadoop cluster.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>The settings that will need to be set are the mysql settings to connect to the mysql db, and the azkaban executor server settings. When setting up the ExecutorServer, the ports must be the same.</span></p><p class="c7 c3"><span></span></p><p class="c7 c3"><span></span></p><p class="c7 c3"><span></span></p><p class="c3"><span>The following are the properties you may be able to set.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">General Properties</span></p><a href="#" name="c8f44b1e326d744c6c5f3f87b22097ef4d913cd6"></a><a href="#" name="0"></a><table cellpadding="0" cellspacing="0" class="c5"><tbody><tr><td class="c1 c6"><p class="c2"><span class="c10">Property</span></p></td><td class="c1 c6"><p class="c2"><span class="c10">Description</span></p></td><td class="c1 c6"><p class="c2"><span class="c10">Default</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">azkaban.name</span></p></td><td class="c1"><p class="c2"><span class="c4">The name of the azkaban instance that will show up in the UI. Useful if you run more than one Azkaban instance.</span></p></td><td class="c1"><p class="c2"><span class="c4">Local</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">azkaban.label</span></p></td><td class="c1"><p class="c2"><span class="c4">A label to describe the Azkaban instance.</span></p></td><td class="c1"><p class="c2"><span class="c4">My Local Azkaban</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">azkaban.color</span></p></td><td class="c1"><p class="c2"><span class="c4">Hex value that allows you to set a style color for the Azkaban UI.</span></p></td><td class="c1"><p class="c2"><span class="c4">#FF3601 (red)</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">web.resource.dir</span></p></td><td class="c1"><p class="c2"><span class="c4">Sets the directory for the ui’s css and javascript files</span></p></td><td class="c1"><p class="c2"><span class="c4">src/web</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">default.timezone</span></p></td><td class="c1"><p class="c2"><span class="c4">The timezone that will be displayed by Azkaban.</span></p></td><td class="c1"><p class="c2"><span class="c4">America/Los_Angeles</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">user.manager.class</span></p></td><td class="c1"><p class="c2"><span class="c4">The user manager that is used to authenticate a user. The default is an XML user manager, but it can be overwritten to support other authethentication methods, such as JDNI.</span></p></td><td class="c1"><p class="c2"><span class="c4">azkaban.user.XmlUserManager</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">mail.sender</span></p></td><td class="c1"><p class="c2"><span class="c4">The email address that azkaban uses to send emails.</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">mail.host</span></p></td><td class="c1"><p class="c2"><span class="c4">The email server host machine</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">mail.user</span></p></td><td class="c1"><p class="c2"><span class="c4">The email server user name</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">mail.password</span></p></td><td class="c1"><p class="c2"><span class="c4">The email password user name</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">azkaban.should.proxy</span></p></td><td class="c1"><p class="c2"><span class="c4">Used by the HDFS browser. Set to true if using Hadoop 1.0+ with security turned on. Will soon be removed.</span></p></td><td class="c1"><p class="c2"><span class="c4">false</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">proxy.keytab.location</span></p></td><td class="c1"><p class="c2"><span class="c4">Used by the HDFS browser. Set to true if using Hadoop 1.0+ with security turned on. Will soon be removed.</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">proxy.user</span></p></td><td class="c1"><p class="c2"><span class="c4">The proxy user</span></p></td><td class="c1"><p class="c2 c7"><span class="c4"></span></p></td></tr></tbody></table><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">Jetty Properties</span></p><a href="#" name="73fd84b587922cb283ec53f2e3ce4dd180f648b4"></a><a href="#" name="1"></a><table cellpadding="0" cellspacing="0" class="c5"><tbody><tr><td class="c1 c16"><p class="c2"><span class="c10">Property</span></p></td><td class="c1 c16"><p class="c2"><span class="c10">Description</span></p></td><td class="c1 c16"><p class="c2"><span class="c10">Default</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.maxThreads</span></p></td><td class="c1"><p class="c2"><span class="c4">Max request threads</span></p></td><td class="c1"><p class="c2"><span class="c4">25</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.ssl.port</span></p></td><td class="c1"><p class="c2"><span class="c4">The ssl port</span></p></td><td class="c1"><p class="c2"><span class="c4">8443</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.keystore</span></p></td><td class="c1"><p class="c2"><span class="c4">The keystore file</span></p></td><td class="c1"><p class="c2"><span class="c4">keystore</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.password</span></p></td><td class="c1"><p class="c2"><span class="c4">The jetty password</span></p></td><td class="c1"><p class="c2"><span class="c4">password</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.keypassword</span></p></td><td class="c1"><p class="c2"><span class="c4">The keypassword</span></p></td><td class="c1"><p class="c2"><span class="c4">password</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.truststore</span></p></td><td class="c1"><p class="c2"><span class="c4">The trust store</span></p></td><td class="c1"><p class="c2"><span class="c4">keystore</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">jetty.trustpassword</span></p></td><td class="c1"><p class="c2"><span class="c4">The trust password</span></p></td><td class="c1"><p class="c2"><span class="c4">password</span></p></td></tr></tbody></table><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">MySQL Connection Properties</span></p><p class="c7 c3"><span></span></p><a href="#" name="7a6602c4e399d0b014db1d704d9e86bc50ccba44"></a><a href="#" name="2"></a><table cellpadding="0" cellspacing="0" class="c5"><tbody><tr class="c8"><td class="c1 c16"><p class="c2"><span class="c10">Property</span></p></td><td class="c1 c16"><p class="c2"><span class="c10">Description</span></p></td><td class="c1 c16"><p class="c2"><span class="c10">Default</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">database.type</span></p></td><td class="c1"><p class="c2"><span class="c4">The database type. Currently, the only database supported is mysql.</span></p></td><td class="c1"><p class="c2"><span class="c4">mysql</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">mysql.port</span></p></td><td class="c1"><p class="c2"><span class="c4">The port to the mysql db</span></p></td><td class="c1"><p class="c2"><span class="c4">3306</span></p></td></tr><tr class="c13"><td class="c1"><p class="c2"><span class="c4">mysql.host</span></p></td><td class="c1"><p class="c2"><span class="c4">The mysql host</span></p></td><td class="c1"><p class="c2"><span class="c4">localhost</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">mysql.database</span></p></td><td class="c1"><p class="c2"><span class="c4">The mysql database</span></p></td><td class="c1"><p class="c2"><span class="c4">azkaban2</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">mysql.user</span></p></td><td class="c1"><p class="c2"><span class="c4">The mysql user</span></p></td><td class="c1"><p class="c2"><span class="c4">azkaban</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">mysql.password</span></p></td><td class="c1"><p class="c2"><span class="c4">The mysql password</span></p></td><td class="c1"><p class="c2"><span class="c4">password</span></p></td></tr><tr class="c8"><td class="c1"><p class="c2"><span class="c4">mysql.numconnections</span></p></td><td class="c1"><p class="c2"><span class="c4">The number of connections that Azkaban web client can open to the database </span></p></td><td class="c1"><p class="c2"><span class="c4">100</span></p></td></tr></tbody></table><p class="c7 c3"><span class="c10"></span></p><p class="c3"><span class="c10">Executor Server Properties</span></p><p class="c7 c3"><span class="c10"></span></p><a href="#" name="bdcd5f269529cc593edc385149bcf056e3c4e554"></a><a href="#" name="3"></a><table cellpadding="0" cellspacing="0" class="c5"><tbody><tr><td class="c1 c17"><p class="c2"><span class="c10">Properties</span></p></td><td class="c1 c17"><p class="c2"><span class="c10">Description</span></p></td><td class="c1 c17"><p class="c2"><span class="c10">Default</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">executor.port</span></p></td><td class="c1"><p class="c2"><span class="c4">The port for the azkaban executor server</span></p></td><td class="c1"><p class="c2"><span class="c4">12321</span></p></td></tr><tr><td class="c1"><p class="c2"><span class="c4">executor.host</span></p></td><td class="c1"><p class="c2"><span class="c4">The host for azkaban executor server</span></p></td><td class="c1"><p class="c2"><span class="c4">localhost</span></p></td></tr></tbody></table><p class="c7 c3"><span class="c10"></span></p><p class="c7 c3"><span class="c10"></span></p><p class="c7 c3"><span></span></p><p class="c3"><span>There are two pieces of Azkaban that must be installed: the web client and the executor server.</span></p><p class="c7 c3"><span class="c10"></span></p><p class="c3"><span class="c10">3. Setting up the Executor Server</span></p><p class="c3"><span>Download the azkaban executor tarball and extract it to the executor install directory.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Just like setting up the Web Server, you will need to copy the mysql jdbc connector jar to the </span><span class="c0">./extlib</span><span> directory. You will also need to change the conf/azkaban.properties to set the executor port and the mysql db settings information.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>You may need to also set the azkaban proxy if using hadoop 1.0 security.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span class="c10">4. Running everything </span></p><p class="c3"><span>Start both the web server and the executor server to run azkaban.</span></p><p class="c7 c3"><span class="c15 c10"></span></p><p class="c3"><span class="c10 c15">Upgrading Azkaban</span></p><p class="c3"><span>Having azkaban as a web server and a executor server in separate processes gives us the ability to roll the upgrade and not shut down long lived jobs.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>To do this, install the newer executor server and change the executor port. You’ll need to also update the executor server port for the web client. Restarting the web client should point to the new executor server.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Any running jobs in the old executor should be auto detected by the web client. When the old executor finishes running its flow, it should be safe to shut the executor server down.</span></p><p class="c7 c3"><span></span></p><p class="c3"><span>Note that the scheduler runs in the web server. If you shut down the web server, you may skip any scheduled jobs during this time.</span></p></body></html>
\ No newline at end of file