Skip to content

NIFI-5318 Implement NiFi test harness - #3165

Closed
peter-gergely-horvath wants to merge 6 commits into
apache:masterfrom
peter-gergely-horvath:NIFI-5318
Closed

peter-gergely-horvath wants to merge 6 commits into
apache:masterfrom
peter-gergely-horvath:NIFI-5318

Conversation

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Hi @MikeThomsen I have opened this new pull request instead of #2872 according your suggestions. Here we have everything sorted out you mentioned earlier on. Now the build executes the test cases if you specify run-testharness-tests and you now have the option to get the flow file version updated to the version of NiFi you are using. :)

@MikeThomsen

Copy link
Copy Markdown
Contributor

Will try to get back into it soon. BTW, thanks for putting it all back onto a branch.

@@ -0,0 +1,28 @@

<?xml version="1.0" encoding="UTF-8"?>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this xml document looks like a real http response. We cannot use it as test material unless we can assert its license and it is ALv2 compatible. Its better to use entirely synthetic test data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it absolutely must be changed because the BBC is asserting its IP rights here:

<copyright><![CDATA[Copyright: (C) British Broadcasting Corporation, see http://news.bbc.co.uk/2/hi/help/rss/4498287.stm for terms and conditions of reuse.]]></copyright>

@MikeThomsen MikeThomsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peter-gergely-horvath the sample infringes on BBC IP, so it should be changed ASAP.

@@ -0,0 +1,28 @@

<?xml version="1.0" encoding="UTF-8"?>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it absolutely must be changed because the BBC is asserting its IP rights here:

<copyright><![CDATA[Copyright: (C) British Broadcasting Corporation, see http://news.bbc.co.uk/2/hi/help/rss/4498287.stm for terms and conditions of reuse.]]></copyright>

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Hi Guys, thank you for pointing this out. I've replaced it with some synthetic content.

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath Tried your latest change and still get these results:

[INFO] -------------------------------------------------------
[INFO] Running org.apache.nifi.testharness.samples.NiFiFlowTest
[INFO] Running org.apache.nifi.testharness.samples.NiFiMockFlowTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:46 min
[INFO] Finished at: 2018-12-12T11:20:20-05:00
[INFO] Final Memory: 36M/602M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on project nifi-testharness: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter5354893279908884469.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2018-12-12T11-18-44_222-jvmRun1 surefire3770690799997642032tmp surefire_03864170411723641611tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter8415059538712188018.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2018-12-12T11-18-44_222-jvmRun1 surefire7043843761330522886tmp surefire_12826311853280960897tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter5354893279908884469.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2018-12-12T11-18-44_222-jvmRun1 surefire3770690799997642032tmp surefire_03864170411723641611tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter8415059538712188018.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2018-12-12T11-18-44_222-jvmRun1 surefire7043843761330522886tmp surefire_12826311853280960897tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:496)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:443)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:295)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1124)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter8415059538712188018.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2018-12-12T11-18-44_222-jvmRun1 surefire7043843761330522886tmp surefire_12826311853280960897tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:686)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:535)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$700(ForkStarter.java:116)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:431)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:408)
[ERROR] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[ERROR] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[ERROR] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[ERROR] 	at java.lang.Thread.run(Thread.java:748)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

@MikeThomsen this issue has now been fixed. Please review / try it again: now it works properly.

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath confirmed that it worked. I'll need to spend some time looking at it again and trying to write my own tests as a POC to verify it works outside of the supplied tests. Thanks for keeping up with this.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Brilliant, let me know if you need any help / have questions. I have added tiny bit of fix there in the meanwhile -- should not affect anything.

@MikeThomsen

Copy link
Copy Markdown
Contributor

Any ideas? Got this after rerunning:

[INFO] 
[INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ nifi-testharness ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.nifi.testharness.samples.NiFiFlowTest
[INFO] Running org.apache.nifi.testharness.samples.NiFiMockFlowTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:02 min
[INFO] Finished at: 2019-01-16T07:08:46-05:00
[INFO] Final Memory: 36M/582M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on project nifi-testharness: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter1127929295710730022.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2019-01-16T07-06-53_087-jvmRun1 surefire2580769922376621977tmp surefire_01859622849316861122tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter103806013718313616.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2019-01-16T07-06-53_087-jvmRun1 surefire5630505416024306961tmp surefire_17051999035288672021tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter1127929295710730022.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2019-01-16T07-06-53_087-jvmRun1 surefire2580769922376621977tmp surefire_01859622849316861122tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter103806013718313616.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2019-01-16T07-06-53_087-jvmRun1 surefire5630505416024306961tmp surefire_17051999035288672021tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:496)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:443)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:295)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1124)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/michaelthomsen/workspace/nifi/nifi-testharness/nifi_testharness_nifi_home && /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -Xmx1G -Djava.net.preferIPv4Stack=true -jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire/surefirebooter103806013718313616.jar /Users/michaelthomsen/workspace/nifi/nifi-testharness/target/surefire 2019-01-16T07-06-53_087-jvmRun1 surefire5630505416024306961tmp surefire_17051999035288672021tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:686)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:535)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$700(ForkStarter.java:116)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:431)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:408)
[ERROR] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[ERROR] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[ERROR] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[ERROR] 	at java.lang.Thread.run(Thread.java:748)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

@peter-gergely-horvath

peter-gergely-horvath commented Jan 16, 2019 •

Copy link
Copy Markdown
Contributor Author

You should be able to fish out the failed test's output from the following files:

  • nifi-testharness/target/surefire-reports/org.apache.nifi.testharness.samples.NiFiMockFlowTest-output.txt
  • nifi-testharness/target/surefire-reports/org.apache.nifi.testharness.samples.NiFiMockFlowTest.txt

Maybe that helps. If not, please post their content to here.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

@joewitt, @MikeThomsen I am wondering if it would be possible to make some progress with this PR? (I understand you might be very busy)

Based on best of my knowledge, this feature (and code base) works properly:
I had successfully executed tests both in my IDE, via a local Maven command line build and in a Travis CI build.

I absolutely appreciate the dedication of ensuring that only quality code should become part of NiFi, but at the same time I think releasing something that might later be improved is part of the natural evolution:

  • This feature is a new testing utility, which cannot possibly break any existing (production) feature
  • All APIs introduced clearly state this might be evolving in the future (CAUTION: THIS IS AN EXPERIMENTAL API: EXPECT CHANGES! Efforts will be made to retain backwards API compatibility, but no guarantee is given.), so users can take that into consideration.

I had some previous projects where having a (even a primitive) NiFi test harness would have been of a huge relief.

Please let me know if I can help you in any way here.

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath been busy w/ the day job and building out our new graph database bundle, but yeah I'm trying to circle back to this. I'll start kicking the tires again soon.

@MikeThomsen

Copy link
Copy Markdown
Contributor

MongoTest_Int.xml.txt

NiFiMongoTest.java.txt

@peter-gergely-horvath I tried my own simple test with five steps that ended with PutMongo, and it crashed hard.

Here are the logs:

org.apache.nifi.testharness.samples.NiFiMongoTest-output.txt

}

private void installFlowFile(File fileToIncludeInGz) throws IOException {
Zip.gzipFile(fileToIncludeInGz, flowXmlGz);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markap14 @bbende Can one of you take a look here at how he's building up the test flow.xml.gz? I have a feeling he's making a bad assumption because the synchronizer code is blowing up in my log file in a place where it never does outside of this use case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NM, documentation problem! (Or lack thereof)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bbende @markap14 In case you didn't get an update, no need here.

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath I was able to get a test flow working. We'll need to work on some documentation and some cleanup soon. I think this is pretty close to being ready to merge as a MVP with the caveat that it is for advanced NiFi users that aren't afraid to sling a lot of Java code.

@asfgit asfgit closed this in 4323572 Mar 16, 2019
rwithers pushed a commit to rwithers/nifi that referenced this pull request May 3, 2019
NIFI-5318 Implement NiFi test harness: replaced original sample feed payload with synthetic content
NIFI-5318 Implement NiFi test harness: fixed test harness run crash issue; better reporting of paths
NIFI-5318 Implement NiFi test harness: added further states where NiFi version can be queried
NIFI-5318 Implement NiFi test harness: fixed incorrect class reference
NIFI-5318 Implement NiFi test harness: added type parameter bounding to setClassOfSingleProcessor to prevent configuring obviously incorrect classes
NIFI-5318 Updated project version.

This closes apache#3165

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants