Repository navigation
Conformance tests: Add mechanism to allow errors on one of multiple lines #1680
Description
Sometimes type checkers differ on the exact line where an error is reported, e.g. issues with decorated functions may be reported on either the decorator line or the def line. For the automated conformance checking that I just added in #1678, we need some way to annotate such expected errors.
For example, this incorrect code produces a mypy error on line 3 and a pyright error on line 4:
from typing import final
@final
def f():
passMy proposed syntax is to add "# E or +1", meaning that an error should be reported either on the comment line or the line after. We would write the test case like this:
from typing import final
@final # E or +1: @final must be used on a method, not a global function
def f():
passOr alternatively:
from typing import final
@final
def f(): # E or -1: @final must be used on a method, not a global function
passThis syntax isn't the greatest but I haven't thought of anything better; suggestions welcome.
Activity
erictraut commentedon Apr 5, 2024
I don't think this will be sufficient. There are cases where errors can be separated by many lines. These are typically related to overloads, overrides, or redefinitions. We haven't hit many of these yet because these areas are still (mostly) unspecified or under-specified in the typing spec, but they're all areas that will need to be spec'ed eventually.
To handle cases like this in an automated way, we'd probably need to be able to tag multiple error sites such that they can be associated with each other. For example, we could use a numeric tag — something like
# E[4]. If this appears on two or more lines, it would mean "this error should be emitted on one of these lines". That would be more flexible than a+1or-1syntax.JelleZijlstra commentedon Apr 5, 2024
To be clear I was envisioning that any number would be accepted, e.g.
E or +3, but your solution is probably better because it doesn't require counting lines and marks every error location explicitly. I would modify it to say that the tag can be any string that is unique within the file, e.g.E[final1].Conformance tests: Add mechanism for marking that an error should app…
Conformance tests: Add mechanism for marking that an error should app…