Declare and use attributes of function objects

Python functions can have attributes, like

def some_function(x): ...
some_function.some_attribute = 42

The code above fails to typecheck, but there's some real code that uses those attributes in a pretty structured way. For example django tags some metadata about the function that allows formatting the function results when displaying on the admin (examples at https://docs.djangoproject.com/en/1.10/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_display). So it would be nice to be able to do something like:

class FunctionForAdmin(Callable[[], Any]):
    short_description: str
    boolean: bool = False
    admin_order_field: str
    empty_value_display: str

some_function: FunctionForAdmin
def some_function():
    return ...
some_function.short_description = "What a nice little function!"

And having it properly typechecked... is this possible now?

Oh wow, nice use case for PEP 526! I guess the equivalent pre-PEP-526 syntax would be

some_function = None  # type: FunctionForAdmin
def some_function():
    ...

(In a stub the initializer could be ... instead of None.)

Then all we have to do is change mypy so that this type-checks.

Type checking code like that could be tricky. Just def some_function() isn't really compatible with FunctionForAdmin using ordinary type checking rules, since it doesn't define the extra attributes and it also isn't an instance of the class. Rewriting it makes it clear:

...
def _some_function():
    return ...
some_function: FunctionForAdmin = _some_function   # not compatible
some_function.short_description = "What a nice little function!"
# even now boolean etc. are missing from the object

For this to make sense we could perhaps support some form of multi-step initialization and structural subtyping, but it's unclear how exactly. Also Callable is not valid as a base class. We could perhaps define __call__ instead, but that wouldn't be quite as pretty.

What about using a decorator instead that declares that a function conforms to a protocol (it doesn't define any attributes -- they need to come from somewhere else):

from typing import Protocol, apply_protocol

class AdminAttributes(Protocol):
    short_description: str
    boolean: bool = False
    ...

@apply_protocol(AdminAttributes)
def some_function():
    return ...
some_function.short_description = "..."

Here apply_protocol would generate an intersection type Intersection[AdminAttributes, <callable corresponding to the signature of the function>] for the function. This would have the benefits of working around the subtyping issue and preserving the exact signature of the function (via some magic).

@JukkaL

@JukkaL your notation looks ok for me (I wasn't proposing a particular notation, just the general idea in the best way I could write it).

Ignore some warnings when setting function attributes

any progress here? what's the latest way to work around this, other than just # type: ignore (my current method)

No, this hasn't reached criticality yet...

Maybe this is a situation where we can (re-)use @declared_type from #3291 to specify a more precise type for the function?

Perhaps, though how would you define the type of a function f(int) -> int that also has an attribute foo: int? You can't inherit from Callable[[int], int].

You can't inherit from Callable[[int], int].

It is possible at runtime (since practically it is just an alias to collections.abc.Callable), but mypy will complain about this. Maybe we can do exactly the same that we do with Tuple[...] -- subclassing Callable will create its copy but with a fallback to an actual instance?

Even if we allowed subclassing from Callable (like in the OP's initial comment), we'd still need some hack in mypy that allows a plain function to be assignable to a variable whose type is a subclass of Callable. See what Jukka wrote above.

we'd still need some hack in mypy that allows a plain function to be assignable to a variable whose type is a subclass of Callable. See what Jukka wrote above.

Exactly, this is why I propose to re-use @declared_type together with subclassing Callable:

class AdminAttributes(Callable[[int], int]):
    short_description: str
    boolean: bool = False
    ...

@declared_type(AdminAttributes)
def some_function(x: int) -> int:
    return ...
some_function.short_description = "..."

This way we will not need an additional decorator @apply_protocol, otherwise the semantics is the same as Jukka proposes.

But this doesn't follow from the rules for @declared_type -- it would require special-casing in mypy, because the "natural" type of some_function is not a subtype of AdminAttributes.

But this doesn't follow from the rules for @declared_type -- it would require special-casing in mypy, because the "natural" type of some_function is not a subtype of AdminAttributes.

Maybe I misunderstood @declared_type, but I thought it is always like this -- the type in @declared_type contains a more precise type for the function, so it should be the subtype of the type inferred by mypy, not vice versa, for example:

def magic_deco(func: Any) -> Any:
    # do some magic
    return func

@declared_type(Callable[[int], int])  # this is more precise than a "natural" type Any
@magic_deco
def fun(x: int) -> int:
    ...

Hm, I always thought it was the other way around. But what you say makes sense too.

I find the draft text for the PEP rather vague on this, and most unit tests, like your example above, skirt the issue by using Any, which is both a subtype and a supertype of everything.

If I look at the implementation it appears to be checking that the inferred type (i.e. before applying @declared_type) is a subtype of the declared type (i.e. the argument to @declared_type).

But this test seems to suggest that you're right.

Where did I go astray?

If I understand your description correctly, this should work:

P = ParamSpec("P")
R = TypeVar("R", covariant=True)

class FunctionWithMetadata(Protocol[P, R]):
    metadata: Metadata

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

def process(event_type: str) -> Callable[[Callable[P, R]], FunctionWithMetadata[P, R]]:
    def process_decorator(func: Callable[P, R]) -> FunctionWithMetadata[P, R]:
        func.metadata = getattr(func, "metadata", Metadata())
        func.metadata.events.append(event_type)
        return cast(FunctionWithMetadata[P, R], func)

    return process_decorator

Use `Protocol` instead of `Callable` in rate_limit_wrapper_test.

@ilevkivskyi

@ilevkivskyi:

(Raising priority to high since this is a popular feature request).

Also it looks like we didn't discuss another popular use case here: adding attributes from another class in a generic manner. For example ...

Such proxy class could for example also check that all attributes of a dataclass were read. That allows to persists type checks of read attributes and also ensure no attributes were kept not accessed. Would play handy when you try to read schema objects and apply those to models in webserver scenario

Support additinal attributes in callback protocols (#14084)

The idea suggested in #2087 (comment) unfortunately doesn't work with the latest mypy 1.0.0 (at the time of writing):

# asd.py
from typing import *

class Metadata:
    ...

P = ParamSpec("P")

R = TypeVar("R", covariant=True)

class FunctionWithMetadata(Protocol[P, R]):
    metadata: Metadata

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

def process(event_type: str) -> Callable[[Callable[P, R]], FunctionWithMetadata[P, R]]:
    def process_decorator(func: Callable[P, R]) -> FunctionWithMetadata[P, R]:
        func.metadata = getattr(func, "metadata", Metadata())
        func.metadata.events.append(event_type)
        return cast(FunctionWithMetadata[P, R], func)

    return process_decorator

$ mypy asd.py
asd.py:18: error: "Callable[P, R]" has no attribute "metadata"  [attr-defined]
asd.py:19: error: "Callable[P, R]" has no attribute "metadata"  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)

I was having the same issue (trying to add an action to Django admin) and Google kept bringing me here. I'm pretty new to typing/mypy, so I don't know if this is perfect, but in case its of use to anyone, I was able to solve my issue with the below workaround:
class AdminAttributes(Protocol):
    short_description: str


def admin_attr_decorator(func: Any) -> AdminAttributes:
    return func


@admin_attr_decorator
def action(modeladmin: admin.ModelAdmin, request: HttpRequest, queryset: QuerySet) -> None:
    do_stuff()


action.short_description = 'do stuff'
This worked for me, but If anyone did not know what a Protocol is (like me). It comes from this import from typing_extensions import Protocol https://mypy.readthedocs.io/en/stable/protocols.html

First, for mypy 1.0.0, to get this to work (including actually calling action()) I had to add a dummy __call__ to AdminAttributes, like this:

from typing import Any, Protocol

class AdminAttributes(Protocol):
    short_description: str
    def __call__(self):
        pass

def admin_attr_decorator(func: Any) -> AdminAttributes:
    return func

@admin_attr_decorator
def action() -> None:
    print(action.short_description)

action.short_description = 'doing stuff'
action()

However mypy also allows this, which is more concise, maybe more standard, and I think in most cases (maybe all?) achieves the same result:

class Action():
    short_description : str = "Default description"
    def __call__(self) -> None:
        print(self.short_description)
action: Action = Action()

action()
action.short_description = "doing stuff"
action()

Here's another way to achieve this that mypy does not allow:

class action():
    short_description: str = "Default description"
    def __new__(self) -> str: 
        print(self.short_description)
        return self.short_description

action.short_description = "doing stuff"
action()

mypy says:
test.py:5: error: Incompatible return type for "new" (returns "str", but must return a subtype of "action") [misc]

Maybe that's a mypy requirement, but it doesn't seem to be a python requirement:
discuss.python.org/t/metaclass-new-return-value/13376/6

I can even make an example based on this that uses no type annotations at all, and mypy rejects it even without --check-untyped-defs:

class foo():
    def print(self):  
        print("Hello World")
        return ("fooish")

class bar():
    ''' A callable class with a return value'''
    def __new__(self):  
        return foo()

bar().print()

This works fine in python 3.8 but mypy says:
test3.py:12: error: "bar" has no attribute "print" [attr-defined]

I guess I am surprised to see how much valid python mypy restricts, even for completely untyped code.

I'm new to mypy and may well miss the point, but it surprises me given that mypy is advertised as the reference implementation of typing PEPs, yet seems to be imposing non-reference restrictions on the language. Of course I get the argument above, that it may simply be really hard to do.

Here are the related bugs for __new__: #1020 #14426 #8330
It happens to be related because one may try to use both __new__ or function attributes to create a callable with an arbitrary return type and with "static" attributes that may or may not need to also be externally accessible, and in both cases find that mypy complains. It's also related because of the general issue discussed there regarding if the purpose of mypy should be to enforce good coding practices, in some cases without (I think) even any good way to ignore the enforcement. Should there even be any discussion of valid use cases in a reference implementation?

@alessio-b2c2

This is a long and very old issue. Note that you can basically accomplish what is needed using Protocols:

from typing import *

class FunctionForAdmin:
    short_description: str
    boolean: bool
    admin_order_field: str
    empty_value_display: str
    
    def __call__(self) -> Any: ...  # or whatever signature you want

# Option 1: Declare type of some_function as below, note you may have to # type: ignore[no-redef] or disable that error code
some_function: FunctionForAdmin

# Option 2: Declare a decorator like in https://github.com/python/mypy/issues/2087#issuecomment-462726600, then use it to decorate some_function

def some_function():
    return ...
    
reveal_type(some_function.short_description)

As others have already noted, in some cases you may be better served by using a normal class with __call__ instead of patching attributes onto functions.

If you want to discuss intersection types, python/typing#213 / https://github.com/CarliJoy/intersection_examples is a good place to do this.

If you have needs that are not met by the above, please open a new issue.

@alessio-b2c2 you need to type ignore in your inner function (or make the input type Any), and everything will work nicely. mypy's not going to understand the process of turning a function from Callable into FunctionWithMetadata

mypy: Improve typing for cmdutils/qute_args

Type-annotate serialization.py (pytorch#40862)

Navigation Menu

Repository navigation

Declare and use attributes of function objects #2087

58 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Declare and use attributes of function objects #2087

Description

Activity

gvanrossum commented on Sep 2, 2016

JukkaL commented on Sep 2, 2016

dmoisset commented on Sep 5, 2016

dgoldstein0 commented on Aug 22, 2017

gvanrossum commented on Aug 23, 2017

ilevkivskyi commented on Aug 23, 2017

gvanrossum commented on Aug 23, 2017

ilevkivskyi commented on Aug 23, 2017

gvanrossum commented on Aug 23, 2017

ilevkivskyi commented on Aug 23, 2017

gvanrossum commented on Aug 23, 2017

ilevkivskyi commented on Aug 23, 2017

gvanrossum commented on Aug 23, 2017

58 remaining items

erictraut commented on Jul 25, 2022

Murtagy commented on Oct 25, 2022

alessio-b2c2 commented on Feb 15, 2023

DougLeonard commented on Feb 16, 2023

DougLeonard commented on Feb 17, 2023

hauntsaninja commented on Aug 10, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions