Repository navigation
NIFI-5318 Implement NiFi test harness: initial commit of nifi-testharness - #2872
peter-gergely-horvath wants to merge 19 commits into
Conversation
lfrancke
left a comment
There was a problem hiding this comment.
This is fantastic to have, thank you!
I checked out the PR, built it and ran the two included tests. They ran successfully (after fixing my working directory).
I only have a few nit-picks
| * interface: | ||
| * <ul> | ||
| * <li> | ||
| * {@link Builder#setFlowXmlToInstallForTesting(File)} specifies the location of the NiFi binary |
There was a problem hiding this comment.
I think this should refer to setNiFiBinaryDistributionZip instead
| * the configuration to a to a temporary directory. | ||
| * <li> | ||
| * Symlinks all files from the temporary directory to the current working | ||
| * directory, causing the to hold a fully functional |
There was a problem hiding this comment.
missing word
causing the directory
| * <p> | ||
| * To perform a clean shutdown of the hosted NiFi instance, the caller is required to call | ||
| * {@link #stopAndCleanup()}, which will shut down NiFi and remove all temporary files, including | ||
| * the symlinks created int current working directory. |
| * directory at the same time. | ||
| * </li> | ||
| * <li> | ||
| * Currently, due to NiFi limitations, one {@TestNiFiInstance} can be started per JVM process. |
There was a problem hiding this comment.
wrong javadoc tag
missing "code"
|
|
||
| static File getBinaryDistributionZipFile(File binaryDistributionZipDir) { | ||
|
|
||
| File[] files = binaryDistributionZipDir.listFiles((dir, name) -> |
There was a problem hiding this comment.
This can return null which makes the next if statement fail with a NPE. I think the condition can be extended to
if (files == null || files.length == 0)
|
Hi @lfrancke I've fixed the issues you noticed. Can you please review the current state? |
|
Thanks! Unfortunately for some reason I can't see your new commit... (I should maybe also add that I'm not a committer, I wast just interested in this PR) |
|
@peter-gergely-horvath can you push that commit? |
|
@MikeThomsen The pull request contains the latest changes; after @lfrancke had made his comments, I corrected the issues mentioned, nuked my whole fork and force-pushed my new version. (I'm wondering what the preferred approach is for you, when there are some suggestions regarding a pull request: maybe it would be worth documenting this in NiFi dev manual?) |
|
@peter-gergely-horvath the consensus seems to be just keep pushing new commits and let us do a squashed commit for you because that helps GitHub maintain the context of feedback in a review. |
|
@MikeThomsen OK, in the future, I will follow that approach. For this pull request, please review as it is, since the current state contains everything in a squashed commit. |
MikeThomsen
left a comment
There was a problem hiding this comment.
Got a little ways in and noticed that your PR is coming from your master. I'll checkout your whole repo, but in the future please make sure to do a PR off a branch named after the Jira number.
| <jetty.version>9.4.3.v20170317</jetty.version> | ||
| </properties> | ||
|
|
||
| <dependencies> |
|
|
||
| public static void createLink(Path newLink, Path existingFile) { | ||
| try { | ||
| Files.createSymbolicLink(newLink, existingFile); |
There was a problem hiding this comment.
Have you tried this on Windows?
There was a problem hiding this comment.
No, I have not tested this on Windows. I would expect it to work properly, since Windows 10 does support symbolic links. (you just need the correct permission)
| public final class Constants { | ||
|
|
||
| static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest"); | ||
| static final File NIFI_ZIP_DIR = new File("../../nifi-assembly/target"); |
There was a problem hiding this comment.
Do these constants assume a particular location for the code that will instantiate the test harness? That NIFI_ZIP_DIR one looks particularly brittle.
There was a problem hiding this comment.
No, these are in the test cases directory: they are there as a sample (referred to in the documentation), and will not be packaged to the actual API artifact.
The paths there refer the NiFi bundle produced during the NiFi build, which seems to be a good default in case you are working on the test harness itself.
| to its configuration, including changing processor properties and replacing processor | ||
| classes with mocks. | ||
|
|
||
| In order to add the necessary classes to your project, |
There was a problem hiding this comment.
Just to clarify, you mean external projects that use NiFi. To use this with NiFi's code base, a developer would have to add something to the end of the build system that would run the tests after the assembly is built based on what I've seen reviewing this. Is that right or am I missing something?
Just want to make sure in case we have to answer questions from people come in and try to use this to test their contributions.
There was a problem hiding this comment.
Yes, absolutely. After reading the paragraph once again, I'm wondering if it would make sense to change us to you (referring to the user), but it is you call, really.
If you think it makes sense, can you do it while you merge/squash the commit?
MikeThomsen
left a comment
There was a problem hiding this comment.
I tried running your tests as a precursor to writing one of my own, and none of the tests for this module ran when I did mvn clean test or mvn clean install. Can you take a look at that?
|
Hi @MikeThomsen I intentionally have no test configuration in the project (at least for now): they are merely samples of what can be done, but they should not be executed as part of the core NiFi build. Please create a new Maven project quickstart project (that will have testing configuration enabled) and add the following (replacing once done, you can take the samples into your own project, where you can experiment with the test harness. I understand piggybacking on the tests directory is maybe not the perfect place to deliver samples, but given the circumstances I think it is acceptable and could maybe be improved in the future. |
|
@peter-gergely-horvath ok. I'm pretty sure that the surefire plugin can be disabled in the POM, but manually activated, so we'll need to look at that because those tests should be runnable if someone wants to modify the test harness and not roll their own test case. |
|
I think there should be a Maven module for the test harness which is disabled by default and can be activated with a flag like |
|
That makes sense: I'll look into that... once I have a tiny bit of time... |
|
Hi @MikeThomsen , hi @alopresto , OK, I've implemented my changes according your recommendations: we now have a dedicated profile for running tests, which is disabled by default. (If you decide to make the squashed commit, please use the original |
MikeThomsen
left a comment
There was a problem hiding this comment.
Did the following:
rm -rf ~/.m2/repositorymvn clean install -DskipTests=trueon NiFi from your branchmvn clean install -Prun-testharness-tests
Got this:
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 57.754 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest) Time elapsed: 57.419 s <<< FAILURE!
java.lang.AssertionError: Output directory not found: ./NiFiTest/NiFiReadTest
at org.apache.nifi.testharness.samples.NiFiMockFlowTest.testFlowCreatesFilesInCorrectLocation(NiFiMockFlowTest.java:104)
Once that is fixed, and I can build everything and run your tests, I'll write my own test and flow and we should be closing in on being done.
…d xpath is now private
…f NiFi flows against the NiFi installation
…t the fact these tests are not executed during the standard build of NiFi
|
Hi @MikeThomsen sorry for the delay, I have been just very busy. It turns out there have been some changes in the flow XML format, which I have fixed against the latest GA version: now all of the tests are executed properly. I have also introduced strict version checking: the NiFi version referenced in the flow file is checked against the actual version of the NiFi distribution used so that we can avoid confusion in the future. If a user tries to run the test harness with a flow created using a different version of NiFi than the one actually used, this will fail fast, preventing a non functioning installation leaving the user being confused why it does not work properly. |
Thanks. I've got a lot on my plate as well, but will try to find some time early this week to review. |
MikeThomsen
left a comment
There was a problem hiding this comment.
I ran through your changes, and noticed this:
[INFO] Running org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.698 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiFlowTest) Time elapsed: 25.405 s <<< ERROR!
java.lang.RuntimeException: Installation failed: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
at org.apache.nifi.testharness.samples.NiFiFlowTest.bootstrapNiFi(NiFiFlowTest.java:93)
Caused by: java.lang.RuntimeException: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
at org.apache.nifi.testharness.samples.NiFiFlowTest.bootstrapNiFi(NiFiFlowTest.java:93)
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiFlowTest) Time elapsed: 25.406 s <<< ERROR!
java.lang.IllegalStateException: Cannot transition from INSTALLATION_FAILED to STOPPED
at org.apache.nifi.testharness.samples.NiFiFlowTest.shutdownNiFi(NiFiFlowTest.java:123)
[INFO] Running org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 27.011 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest) Time elapsed: 26.812 s <<< ERROR!
java.lang.RuntimeException: Installation failed: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
at org.apache.nifi.testharness.samples.NiFiMockFlowTest.bootstrapNiFi(NiFiMockFlowTest.java:94)
Caused by: java.lang.RuntimeException: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
at org.apache.nifi.testharness.samples.NiFiMockFlowTest.bootstrapNiFi(NiFiMockFlowTest.java:94)
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest) Time elapsed: 26.812 s <<< ERROR!
java.lang.IllegalStateException: Cannot transition from INSTALLATION_FAILED to STOPPED
at org.apache.nifi.testharness.samples.NiFiMockFlowTest.shutdownNiFi(NiFiMockFlowTest.java:124)
Seems pretty brittle with respect to the NiFi version issue there. Any thoughts on what's going on here?
|
|
||
| // NOTE: you will have to have the NiFi distribution ZIP placed into this directory. | ||
| // Its version must be the same as the one referenced in the flow.xml, otherwise it will not work! | ||
| static final File NIFI_ZIP_DIR = new File(System.getProperty("user.home"), "Downloads"); |
There was a problem hiding this comment.
Since this is an internal unit test, it should be pointing to the build output of nifi-assembly.
There was a problem hiding this comment.
This will be relative to the nifi_testharness_nifi_home directory, from which the test cases are executed. Since the test case should clear up that directory automatically, I think this should be fine.
There was a problem hiding this comment.
That folder evaluated to $HOME/Downloads for my user account when I ran it. Are you expecting the user to sudo or something?
|
@MikeThomsen This is kind of tricky: NiFi flows refer the precise version of NiFi within the flow XML file, so you have something like this for the processor definitions: When you export a flow file to test, I would expect the you to use precisely the same NiFi version as the one used to export the flow file. I think there is a little bit of confusion here: please note that these tests do not test the test-harness. The test-harness itself is not tested (kind of a chicken-and-egg issue): these test cases are merely samples, which are referenced in the documentation: they demonstrate how an end-user could create test cases for his/her own flows. Please download the ZIP of NiFi 1.7.1 to the |
…pdated programmatically to match that of NiFi used
…ZIP is not there
… to the documentation
|
Closed an created a new pull request for the same issue from a clean state: #3165 |
All issues are fixed from the previous pull request comments.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.