Repository navigation
TypeGuard typechecking on filter is too strict #12682
Description
Activity
NB: the new behavior was introduced in python/typeshed#5661
This seems like a mypy bug, not a typeshed bug. It's picking the wrong overload.
Yeah, mypy isn't happy with this on 0.942 either, I think the bug is a type context related.
Any word on getting this fixed? Maybe the same issue - #12996
Has a workaround been identified in the meantime?
Reacted by Ben Kallus, Gaik Tamazian and Marcelo Drudi MirandaAlthough error message is bad (mypy sometimes randomly chooses "best" overload match to report error if none of overload matches), it is technically correct (assuming that best inferred type of the first argument in second and third call is
Callable[[Any], bool]). To illustrate, this crashes at runtime:from typing import Any, List, Optional path_parts: List[Optional[str]] = ["foo", None, "bar"] def always(x: Any) -> bool: return True "/".join(filter(always, path_parts))
So to support this we will need both:
- Special-case
boolconstructor to be a typeguard againstNone - Special-case
lambda x: isinstance(x, ...)to infer a type with TypeGuard
Both are possible but non-trivial (and still a bit hacky).
Reacted by Ilya Priven and Lucas- Special-case
I was personally getting this error whenever I was attempting to filter a
list[str]down to just the elements that contained a substring. However, my problem was that substring was from adict.get()that could resolve toNone, so this was actually a good catch from mypy.Narrowing the type with an assertion was enough to stop mypy from producing this error.
table_name = self.table_specifier.specifier.get("table") if not table_name: raise ... filtered = list(filter(lambda x: x in <>))
Another example snippet that triggers the issue.
Here mypy is happy
from typing_extensions import reveal_type non_none_strings = list(filter(lambda x: x is not None, [None, "value"])) reveal_type(non_none_strings)
the revealed type is
builtins.list[Union[builtins.str, None]]. Since I need to join the non-none strings, I specified the typefrom typing_extensions import reveal_type non_none_strings: list[str] = list(filter(lambda x: x is not None, [None, "value"])) reveal_type(non_none_strings) print(", ".join(non_none_strings))
which makes mypy complain with
asd.py:3: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[str | None], TypeGuard[str]]" [arg-type] asd.py:4: note: Revealed type is "builtins.list[builtins.str]"The workaround I had to use was to NOT pass the lambda at all
from typing_extensions import reveal_type non_none_strings: list[str] = list(filter(None, [None, "value"])) reveal_type(non_none_strings) print(", ".join(non_none_strings))
Reacted by ClaraIn my use-case I needed the first item of a list of dicts where a particular key equals a particular value. At first I used
nextandfilterlike this:item = next(filter(lambda v: v["my-key"] == "my-value", list_of_dicts), None) if item is None: ... else: ...
I got 3 mypy errors for the same line:
Argument 1 to "filter" has incompatible type "Callable[[Any], Any]"; expected "Callable[[Any], TypeGuard[Any]]" [arg-type] Argument 1 to "filter" has incompatible type "Callable[[Any], Any]"; expected "Callable[[Any], TypeGuard[dict[Any, Any] | None]]" [arg-type] Value of type "dict[Any, Any] | None" is not indexable [index]My workaround was to use
nextwith a generator expression:item = next((item for item in list_of_dicts if item["my-key"] == "my-value"), None) if item is None: ... else: ...
Reacted by andrewqa1, Bruno García Berrotarán, Reinier Maas and Peter- added a commit that references this issue
on Jun 11, 2024 I posted a message on the gitter chat asking if I should post this as an issue or not, because I have the intuition that it is linked to this issue. But if it is not, I can open an issue.
Typing is failing for a call tofilterin a given case and not in another case.To reproduce
from typing import Self, List, Optional class Foo: def __init__(self: Self, a: str, parent: Optional[Self] = None): if parent is not None: self.parent: Self = parent self.a = a class Bar: def __init__(self: Self): self.__field: List[Foo] = [] # mypy fails here def typing_failure(self: Self, other: Foo) -> Optional[Foo]: return next( filter( lambda node: node.parent.a == other.parent.a, self.__field, ), None, ) def typing_success(self: Self, other: Foo) -> Optional[Foo]: filtered = filter( lambda node: node.parent.a == other.parent.a, self.__field, ) return next( filtered, None, )
Expected Behavior
$> mypy test.py Success: no issues found in 1 source file
Actual Behavior
$> mypy test.py test.py:16: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool | Any]"; expected "Callable[[Foo], TypeGuard[Foo | None]]" [arg-type] test.py:16: error: Item "None" of "Foo | None" has no attribute "parent" [union-attr] test.py:16: error: Item "None" of "Foo | Any | None" has no attribute "a" [union-attr] Found 3 errors in 1 file (checked 1 source file)
Your Environment
- Mypy version used:
mypy 1.9.0 (compiled: yes) - Python version used: 3.12.3
- Mypy version used:
Bug Report
NB: This may be more properly an issue in Typeshed, or require changes in that project to fix, but our team encountered it via MyPy.
The type annotations on
builtins.filtercause correct and idiomatic usages to fail to typecheck:To Reproduce
Expected:
Actual result:
Your Environment
$> mypy --version mypy 0.950 (compiled: yes)$> python --version Python 3.10.4The following is what my editor (VSCode) pulls up when I investigate the type hints for
filter, though I doubt this is the same file that MyPy itself is consulting:Note that both offending lines conform to the third type signature, but fail the second.