Skip to content

NIFI-5318 Implement NiFi test harness: initial commit of nifi-testharness - #2872

Closed
peter-gergely-horvath wants to merge 19 commits into
apache:masterfrom
peter-gergely-horvath:master
Closed

peter-gergely-horvath wants to merge 19 commits into
apache:masterfrom
peter-gergely-horvath:master

Conversation

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor

All issues are fixed from the previous pull request comments.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@lfrancke lfrancke left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic to have, thank you!

I checked out the PR, built it and ran the two included tests. They ran successfully (after fixing my working directory).

I only have a few nit-picks

* interface:
* <ul>
* <li>
* {@link Builder#setFlowXmlToInstallForTesting(File)} specifies the location of the NiFi binary

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should refer to setNiFiBinaryDistributionZip instead

* the configuration to a to a temporary directory.
* <li>
* Symlinks all files from the temporary directory to the current working
* directory, causing the to hold a fully functional

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing word

causing the directory

* <p>
* To perform a clean shutdown of the hosted NiFi instance, the caller is required to call
* {@link #stopAndCleanup()}, which will shut down NiFi and remove all temporary files, including
* the symlinks created int current working directory.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

int -> in the?

* directory at the same time.
* </li>
* <li>
* Currently, due to NiFi limitations, one {@TestNiFiInstance} can be started per JVM process.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong javadoc tag
missing "code"


static File getBinaryDistributionZipFile(File binaryDistributionZipDir) {

File[] files = binaryDistributionZipDir.listFiles((dir, name) ->

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can return null which makes the next if statement fail with a NPE. I think the condition can be extended to

if (files == null || files.length == 0)

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Hi @lfrancke I've fixed the issues you noticed. Can you please review the current state?

@lfrancke

Copy link
Copy Markdown
Member

Thanks! Unfortunately for some reason I can't see your new commit...

(I should maybe also add that I'm not a committer, I wast just interested in this PR)

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath can you push that commit?

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

@MikeThomsen The pull request contains the latest changes; after @lfrancke had made his comments, I corrected the issues mentioned, nuked my whole fork and force-pushed my new version.

(I'm wondering what the preferred approach is for you, when there are some suggestions regarding a pull request: maybe it would be worth documenting this in NiFi dev manual?)

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath the consensus seems to be just keep pushing new commits and let us do a squashed commit for you because that helps GitHub maintain the context of feedback in a review.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

@MikeThomsen OK, in the future, I will follow that approach. For this pull request, please review as it is, since the current state contains everything in a squashed commit.

@MikeThomsen MikeThomsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got a little ways in and noticed that your PR is coming from your master. I'll checkout your whole repo, but in the future please make sure to do a PR off a branch named after the Jira number.

Comment thread nifi-testharness/pom.xml
<jetty.version>9.4.3.v20170317</jetty.version>
</properties>

<dependencies>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L&N looks good here.


public static void createLink(Path newLink, Path existingFile) {
try {
Files.createSymbolicLink(newLink, existingFile);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried this on Windows?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I have not tested this on Windows. I would expect it to work properly, since Windows 10 does support symbolic links. (you just need the correct permission)

public final class Constants {

static final File OUTPUT_DIR = new File("./NiFiTest/NiFiReadTest");
static final File NIFI_ZIP_DIR = new File("../../nifi-assembly/target");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these constants assume a particular location for the code that will instantiate the test harness? That NIFI_ZIP_DIR one looks particularly brittle.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, these are in the test cases directory: they are there as a sample (referred to in the documentation), and will not be packaged to the actual API artifact.
The paths there refer the NiFi bundle produced during the NiFi build, which seems to be a good default in case you are working on the test harness itself.

to its configuration, including changing processor properties and replacing processor
classes with mocks.

In order to add the necessary classes to your project,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, you mean external projects that use NiFi. To use this with NiFi's code base, a developer would have to add something to the end of the build system that would run the tests after the assembly is built based on what I've seen reviewing this. Is that right or am I missing something?

Just want to make sure in case we have to answer questions from people come in and try to use this to test their contributions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, absolutely. After reading the paragraph once again, I'm wondering if it would make sense to change us to you (referring to the user), but it is you call, really.
If you think it makes sense, can you do it while you merge/squash the commit?

@MikeThomsen MikeThomsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried running your tests as a precursor to writing one of my own, and none of the tests for this module ran when I did mvn clean test or mvn clean install. Can you take a look at that?

@peter-gergely-horvath

peter-gergely-horvath commented Sep 25, 2018 •

Copy link
Copy Markdown
Contributor Author

Hi @MikeThomsen I intentionally have no test configuration in the project (at least for now): they are merely samples of what can be done, but they should not be executed as part of the core NiFi build.
NiFi is a beast, starting and stopping it takes some time, I do not want to add that to the each NiFi build.

Please create a new Maven project quickstart project (that will have testing configuration enabled) and add the following (replacing ${nifi version} with the current one) to the dependencies:

<dependency>
	<groupId>org.apache.nifi</groupId>
	<artifactId>nifi-testharness</artifactId>
	<version>${nifi version}</version>
</dependency>

once done, you can take the samples into your own project, where you can experiment with the test harness.

I understand piggybacking on the tests directory is maybe not the perfect place to deliver samples, but given the circumstances I think it is acceptable and could maybe be improved in the future.

@MikeThomsen

Copy link
Copy Markdown
Contributor

@peter-gergely-horvath ok. I'm pretty sure that the surefire plugin can be disabled in the POM, but manually activated, so we'll need to look at that because those tests should be runnable if someone wants to modify the test harness and not roll their own test case.

@alopresto

Copy link
Copy Markdown
Contributor

I think there should be a Maven module for the test harness which is disabled by default and can be activated with a flag like mvn clean test -Ptest-harness.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

That makes sense: I'll look into that... once I have a tiny bit of time...

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Hi @MikeThomsen , hi @alopresto , OK, I've implemented my changes according your recommendations: we now have a dedicated profile for running tests, which is disabled by default.

(If you decide to make the squashed commit, please use the original peter-gergely-horvath as author)

@MikeThomsen MikeThomsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did the following:

  1. rm -rf ~/.m2/repository
  2. mvn clean install -DskipTests=true on NiFi from your branch
  3. mvn clean install -Prun-testharness-tests

Got this:

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 57.754 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest)  Time elapsed: 57.419 s  <<< FAILURE!
java.lang.AssertionError: Output directory not found: ./NiFiTest/NiFiReadTest
	at org.apache.nifi.testharness.samples.NiFiMockFlowTest.testFlowCreatesFilesInCorrectLocation(NiFiMockFlowTest.java:104)

Once that is fixed, and I can build everything and run your tests, I'll write my own test and flow and we should be closing in on being done.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Hi @MikeThomsen sorry for the delay, I have been just very busy. It turns out there have been some changes in the flow XML format, which I have fixed against the latest GA version: now all of the tests are executed properly.

I have also introduced strict version checking: the NiFi version referenced in the flow file is checked against the actual version of the NiFi distribution used so that we can avoid confusion in the future.

If a user tries to run the test harness with a flow created using a different version of NiFi than the one actually used, this will fail fast, preventing a non functioning installation leaving the user being confused why it does not work properly.

@MikeThomsen

Copy link
Copy Markdown
Contributor

Hi @MikeThomsen sorry for the delay, I have been just very busy.

Thanks. I've got a lot on my plate as well, but will try to find some time early this week to review.

@MikeThomsen MikeThomsen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran through your changes, and noticed this:

[INFO] Running org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.698 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiFlowTest)  Time elapsed: 25.405 s  <<< ERROR!
java.lang.RuntimeException: Installation failed: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
	at org.apache.nifi.testharness.samples.NiFiFlowTest.bootstrapNiFi(NiFiFlowTest.java:93)
Caused by: java.lang.RuntimeException: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
	at org.apache.nifi.testharness.samples.NiFiFlowTest.bootstrapNiFi(NiFiFlowTest.java:93)

[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiFlowTest)  Time elapsed: 25.406 s  <<< ERROR!
java.lang.IllegalStateException: Cannot transition from INSTALLATION_FAILED to STOPPED
	at org.apache.nifi.testharness.samples.NiFiFlowTest.shutdownNiFi(NiFiFlowTest.java:123)

[INFO] Running org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 27.011 s <<< FAILURE! - in org.apache.nifi.testharness.samples.NiFiMockFlowTest
[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest)  Time elapsed: 26.812 s  <<< ERROR!
java.lang.RuntimeException: Installation failed: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
	at org.apache.nifi.testharness.samples.NiFiMockFlowTest.bootstrapNiFi(NiFiMockFlowTest.java:94)
Caused by: java.lang.RuntimeException: The NiFi version referenced in the flow file ('1.7.1') does not match the version of NiFi being used ('1.8.0-SNAPSHOT')
	at org.apache.nifi.testharness.samples.NiFiMockFlowTest.bootstrapNiFi(NiFiMockFlowTest.java:94)

[ERROR] testFlowCreatesFilesInCorrectLocation(org.apache.nifi.testharness.samples.NiFiMockFlowTest)  Time elapsed: 26.812 s  <<< ERROR!
java.lang.IllegalStateException: Cannot transition from INSTALLATION_FAILED to STOPPED
	at org.apache.nifi.testharness.samples.NiFiMockFlowTest.shutdownNiFi(NiFiMockFlowTest.java:124)

Seems pretty brittle with respect to the NiFi version issue there. Any thoughts on what's going on here?


// NOTE: you will have to have the NiFi distribution ZIP placed into this directory.
// Its version must be the same as the one referenced in the flow.xml, otherwise it will not work!
static final File NIFI_ZIP_DIR = new File(System.getProperty("user.home"), "Downloads");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is an internal unit test, it should be pointing to the build output of nifi-assembly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be relative to the nifi_testharness_nifi_home directory, from which the test cases are executed. Since the test case should clear up that directory automatically, I think this should be fine.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That folder evaluated to $HOME/Downloads for my user account when I ran it. Are you expecting the user to sudo or something?

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

@MikeThomsen This is kind of tricky: NiFi flows refer the precise version of NiFi within the flow XML file, so you have something like this for the processor definitions:

      <class>org.apache.nifi.processors.standard.GetHTTP</class>
      <bundle>
        <group>org.apache.nifi</group>
        <artifact>nifi-standard-nar</artifact>
        <version>1.7.1</version>
      </bundle>

When you export a flow file to test, I would expect the you to use precisely the same NiFi version as the one used to export the flow file.

I think there is a little bit of confusion here: please note that these tests do not test the test-harness. The test-harness itself is not tested (kind of a chicken-and-egg issue): these test cases are merely samples, which are referenced in the documentation: they demonstrate how an end-user could create test cases for his/her own flows.

Please download the ZIP of NiFi 1.7.1 to the Downloads directory within your user home (referenced by org.apache.nifi.testharness.samples.Constants#NIFI_ZIP_DIR as
new File(System.getProperty("user.home"), "Downloads") ) and try running the test case again.

@peter-gergely-horvath

Copy link
Copy Markdown
Contributor Author

Closed an created a new pull request for the same issue from a clean state: #3165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants