Skip to content

Run tests as part of build/CI #1413

Closed
Closed

Description

@hoffie

There are test cases in src/testbench*.

  • Figure out how to run them.
  • Check if they can be run on an unmodified Jamulus code (I have seen various commented-out code related to "tests" within the code, not sure if this is related)
  • Check if running them as part of the build is useful and possible
  • Check if running them as part of CI (e.g. for PRs) is useful and possible

cc @pljones (as you suggested looking at the tests some time ago ;))

Activity

  1. atsampson commented on Apr 1, 2021

    @atsampson
    Contributor

    The code in testbench.h is a simple template-based fuzzer for the Jamulus protocol - that is, it generates valid messages with completely random field values. It's intended for testing the protocol parsers on the client and server side. To enable it, you uncomment the CTestBench like in main.cpp and recompile, then run Jamulus as usual - this'll cause it to run with an additonal thread that periodically generates a random message and feeds it into the parser.

    Because it's non-deterministic, and it needs to run for a very long time to have a chance of finding a problem, it's not really very useful as part of an automated test suite. It also needs updating for the current list of message types (e.g. it can't do PROTMESSID_CLM_RED_SERVER_LIST yet).

    Back in mid-2020 I experimented with using it to generate seeds for a coverage-directed fuzzer, afl-fuzz, which is much more efficient (but still non-deterministic). That work is in this branch, which also needs updating. The change to add the -Z option might be worth including in mainline Jamulus, since then the CI tests could at least feed a canned sequence of messages into the parsers...

  2. ann0see commented on Apr 22, 2022

    @ann0see
    Member

    @atsampson thanks for the comment. Could you please quickly also comment on #2428 maybe we can somehow use it with JSON-RPC?

    Nevertheless, closing as
    Duplicate: #1413

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions