You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parity compares exit codes, so two implementations wrong the same way are green #148
test_verdict_agrees_with_the_reference_harness runs both implementations over the corpus and asserts the overall verdicts match. That is one bit per run.
What it cannot see
A shared mistake. Both implementations pass, both are wrong, the test is green. There is no oracle in the comparison; each side is only checked against the other.
Compensating differences. One implementation misses a violation while the other reports a spurious one. Same exit code, opposite errors.
Which case disagreed. When it does fail, the assertion prints our failures and the tail of the reference's stdout, and a human diffs two differently-shaped reports by eye.
Shape
Compare per case rather than per run: each implementation emits a machine-readable report of case to outcome, and the suite diffs them against the expectations the corpus declares.
Two things fall out. The comparison stops being pairwise, so a third implementation costs nothing here - it runs the same suite and produces the same report shape, and no implementation needs to know about any other. And this package stops being both a participant and the judge, which is the part that does not survive a third port.
Depends on OO-LD/oold-schema#158, which is where the expectations have to exist before anything can be diffed against them. Until then the exit-code comparison is the honest maximum and should stay.
Activity