7/10/2018 1:19:35 AM
#1842
Issue:
azkaban.utils.ExecutorServiceUtilsTest > force_shutdown_after_timeout STANDARD_ERROR
log4j:WARN No appenders could be found for logger (azkaban.utils.ExecutorServiceUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
azkaban.utils.ExecutorServiceUtilsTest > force_shutdown_after_timeout FAILED
org.junit.ComparisonFailure: expected:<[tru]e> but was:<[fals]e>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at azkaban.utils.ExecutorServiceUtilsTest.force_shutdown_after_timeout(ExecutorServiceUtilsTest.java:60)
azkaban.utils.ExecutorServiceUtilsTest
It failed at
assertThat(this.isTestThreadInterrupted).isTrue();
It's possible that the task has not been scheduled yet or has not had time to process the interrupt request.
Fix:
Replaced the test with a mock based test strategy.
The test requires a newer version of JDK. JDK 1.8.05 failed.
1.8.133 worked.
See
https://stackoverflow.com/questions/37527038/mockito-object-is-not-an-instance-of-declaring-class/37528479#37528479
for more information.
|
7/9/2018 7:14:20 PM
3.49.0
Add a test to run a v2 flow file with the new DAG engine
This is a step towards building an integration test for the new DAG engine.
Refactored the DagBuilder API to make it easier to use.
Instead of using a DagBuilder class to link nodes, use the name of the
nodes directly.
This facility can also be used in the future to build the tools to run
flows locally for testing purposes.
* Fix a copy and paste error.
Should check non null of the dagProcessor parameter.
|
7/9/2018 7:07:52 PM
improvements needs to be done on project LRU cache feature:
1. The feature needs to be turned off by default.
2. Running disk calculation on our production clusters whose project dir size is 1.8T takes >= 10 mins.
3. Cleaning thread will be hanging likely due to https://stackoverflow.com/questions/13008526/runtime-getruntime-execcmd-hanging.
4. An alternative design is considered:
Create a file in each project directory and write the size of the project to the file when the project is created. The project files are not supposed to change after creation. Touch this file each time the project is used. This way, we can have a more efficient LRU algorithm based on last access time, not creation time.
Maintain the total size of the project cache in memory to avoid the overhead of re-calculating it. The size shouldn't change too often. This way we can afford to run the check more frequently.
Revert the change to get the release going forward for now and will make the improvements later.
|
7/9/2018 6:12:59 PM
dependency order. Otherwise it can happen that web server tries to launch an execution, but it fails as executor server hasn't started yet.
Also start ExecutorManager's threads (that periodically call executors to get updates) not sooner than when the web server is started. Those threads used to be started already as soon as Guice injection creates the ExecutorManager instance (which happens before executor server is started – even now that the order is fixed).
This is one step towards having everything run in "multi-executor" mode.
As you may know, currently there are two possible modes, controlled by azkaban.use.multiple.executors=true|false:
multi-executor mode, based on the executors DB table
single executor mode, based on fixed props executor.host & executor.port
That adds extra complexity and different branches of code are executed depending on the mode. However, everything can just as well be run as "multi-executor" (but just one executor if that's what you want, and naturally so in case of solo-server). It makes it also easier to reach a better test coverage when testing is always done in the same "mode" that's used in production configurations.
I also noticed that ExecutorManager loads executors from the DB at the time of Guice injection (before executor server is started). This may need to be changed as well to properly support multi-executor mode on a single server.
|
6/29/2018 6:54:09 PM
invalid address from users. (#1827)
|
|
|
|
|
6/22/2018 10:00:47 PM
endtime was showing current time when trigger instance is running.
|
6/21/2018 5:24:38 PM
computation of a flow
level++ returns the level variable BEFORE it is incremented. This means
the level of a node is never incremented and is always zero.
Hence, we need to increment the value by one before passing it to the
function.
Fixes #1793.
* followup: add tests for node level computation
* followup: refactor test according to review comments
|
6/20/2018 9:15:46 PM
test failed today on trunk
azkaban.jobExecutor.ProcessJobTest > testCancelAfterJobProcessCreation FAILED org.junit.ComparisonFailure: expected:<[fals]e> but was:<[tru]e> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at azkaban.jobExecutor.ProcessJobTest.testCancelAfterJobProcessCreation(ProcessJobTest.java:228)
Cause:
I suspect it is because of the travis CI machine being too busy to cancel
the job before it finishes.
Solution:
Increase the running time of the job from 1 second to 5 seconds.
This will not increase the test running time when the test is successful
since the job will be canceled as soon as possible.
However it will increase the test running time if the cancel logic
doesn't interrupt the job as expected.
Also removed the printout of the stacktrace from the job. This information
doesn't deliver much value and only makes the log noisier.
|
