Skip to content

RegTest: invert for test sections #17

Open

Description

@stormi

There are tests that I write before they can pass. They are TODOs, to remind me that there is something that I need to fix. It's not always something I can fix right away, because it may require fixing at the French library level first. Actually, there are also known issues in the French Inform 6 library and I would like to use RegTest for them too, as one of the maintainers.

A useful feature for this would be the ability to mark a test as xfail (stands for "expected to fail", I borrowed it from pytest). That is, such tests will pass if they fail, and fail if they pass. The test definition is ready for when the bug is fixed, and can serve as specifications for the fix yet to be done or objects/actions yet to be implemented in the game. The test output will mention that they are expectedly fail (with the XFAIL status code, for example), but they won't cause the whole tests execution to fail (won't cause a nonzero return code of RegTest).

Once the bug is fixed, or objects/actions implemented, the xfail test starts failing (pytest uses a specific code, for this: XPASS), so you can check that it is now expected to work and remove the xfail flag from it if it is to make it a regular test now protecting you against regressions.

One can likely do without this flag, by creating tests that match the error message or bad output, with the actual expected conditions from when it will work commented out, but it may be less elegant, and hardly works in complex tests where the failing conditions are early.

What I'm starting to do right now is to move all known to fail tests into separate test files, that I will have to check from time to time to extract now-working tests from them.

Activity

  1. stormi commented on Jul 14, 2024

    @stormi
    Author

    By the way, if you agree with the idea and the itch becomes strong enough for me, it's entirely possible I'd contribute this myself.

  2. erkyrath commented on Jul 15, 2024

    @erkyrath
    Owner

    You can write !test or {invert} test to get the "pass if it fails, fail if it passes" behavior.

    Or do you mean the entire test section, as in * test?

  3. stormi commented on Jul 15, 2024

    @stormi
    Author

    Looks like I didn't read the README carefully enough, so I missed {invert}. But indeed I meant the entire test section, because a failed action may cause what's after it to fail too (like an objet that is not in the inventory because the Take action failed).

    However, I believe I can start with {invert} and see whether it's enough for my needs.

    Thanks!

  4. changed the title [-]RegTest: `xfail` feature suggestion[/-] [+]RegTest: `invert` for test sections[/+] on Jul 15, 2024
  5. erkyrath commented on Jul 15, 2024

    @erkyrath
    Owner

    Ok, I'll keep this open as a suggestion for test sections. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions