azkaban-aplcache

increase job name varchar size in Mysql table to allow long job …

5/31/2018 3:08:20 AM

name to survive (#1694)

Users reported that log missed when composed job name was too long. The reason is that Mysql doesn't allow to insert a row when name's length is bigger than 128. This PR increases the length to allow long job name to survive.

Since we don't know the Mysql prefix settings for other Azkaban users, we try to meet the requirement for defaults. In Default Mysql settings, the index key prefix length limit is 767 bytes for innoDB. We have to assume most users using innoDB.

Besides, users should gain difference regarding character seeting in Mysql. In latin1, each character is exactly one byte long. In utf8, a character can consist of more than one byte, usually 3 or 4. At our settings, we use latin1, so we can hold 512 character long name for a job.

Because varchar size enhancement is not a necessary upgrade, schema update file is not recommended here. If people run into the issues above, they should follow the below instructions:

maximize the Mysql varchar size limit introduced in this PR.
change the Mysql character setting to latin1, if your Azkaban only stores English or other latin characters.
For h2 database, there is no limit for the following entities, except the memory and storage capacity: maximum identifier length (table name, column name, and so on)
http://www.h2database.com/html/advanced.html#limits_limitations

Liang Tang

Commit: 94410db

Tree: 4162ad1

Parents: 7cd5123

Changes

azkaban-db/src/main/sql/create.execution_jobs.sql 2(+1 -1)

azkaban-db/src/main/sql/create.execution_logs.sql 8(+7 -1)

Details

azkaban-db/src/main/sql/create.execution_jobs.sql 2(+1 -1)

diff --git a/azkaban-db/src/main/sql/create.execution_jobs.sql b/azkaban-db/src/main/sql/create.execution_jobs.sql
index d973ae2..b712f52 100644
--- a/azkaban-db/src/main/sql/create.execution_jobs.sql
+++ b/azkaban-db/src/main/sql/create.execution_jobs.sql
@@ -3,7 +3,7 @@ CREATE TABLE execution_jobs (
   project_id    INT          NOT NULL,
   version       INT          NOT NULL,
   flow_id       VARCHAR(128) NOT NULL,
-  job_id        VARCHAR(128) NOT NULL,
+  job_id        VARCHAR(512) NOT NULL,
   attempt       INT,
   start_time    BIGINT,
   end_time      BIGINT,

azkaban-db/src/main/sql/create.execution_logs.sql 8(+7 -1)

diff --git a/azkaban-db/src/main/sql/create.execution_logs.sql b/azkaban-db/src/main/sql/create.execution_logs.sql
index 2d26e1b..9003820 100644
--- a/azkaban-db/src/main/sql/create.execution_logs.sql
+++ b/azkaban-db/src/main/sql/create.execution_logs.sql
@@ -1,6 +1,12 @@
+-- In table execution_logs, name is the combination of flow_id and job_id
+--
+-- prefix support and lengths of prefixes (where supported) are storage engine dependent.
+-- By default, the index key prefix length limit is 767 bytes for innoDB.
+-- from: https://dev.mysql.com/doc/refman/5.7/en/create-index.html
+
 CREATE TABLE execution_logs (
   exec_id     INT NOT NULL,
-  name        VARCHAR(128),
+  name        VARCHAR(640),
   attempt     INT,
   enc_type    TINYINT,
   start_byte  INT,