[ty] Fix setattr call check precedence during attribute assignment #18347

thejchap · 2025-05-28T06:02:47Z

Summary

This PR changes the __setattr__ call check to be attempted first, prior to the "normal mechanism", as a defined __setattr__ should take precedence over setting an attribute on the instance dictionary directly.

if the return type of __setattr__ is Never, an invalid-assignment diagnostic is emitted

Once this is merged, a subsequent PR will synthesize a __setattr__ method with a Never return type for frozen dataclasses.

Ecosystem changes

it looks like support needs to be added for writing special attributes on Callables (at least I wasn't able to find tests that asserted these are supported). These diagnostics are getting emitted due to a CallDunderError::PossiblyUnbound.

This seems like an expected side-effect given the change in this PR - would it make sense to add a separate issue to track this?

Function objects also support getting and setting arbitrary attributes,

+ error[unresolved-attribute] yarl/_url.py:140:5: Can not assign object of `Literal["yarl"]` to attribute `__module__` on type `_T` with custom `__setattr__` method.
+ error[unresolved-attribute] optuna/_deprecated.py:185:17: Can not assign object of `Literal[""]` to attribute `__doc__` on type `CT` with custom `__setattr__` method.
+ error[unresolved-attribute] optuna/_deprecated.py:193:13: Can not assign object of `str` to attribute `__doc__` on type `CT` with custom `__setattr__` method.
+ error[unresolved-attribute] optuna/_experimental.py:129:17: Can not assign object of `Literal[""]` to attribute `__doc__` on type `CT` with custom `__setattr__` method.
+ error[unresolved-attribute] optuna/_experimental.py:133:13: Can not assign object of `str` to attribute `__doc__` on type `CT` with custom `__setattr__` method.

it's not obvious to me that the other ecosystem changes are related to this PR - would need to understand better how the comparison is happening

Test Plan

Existing tests + mypy_primer

github-actions · 2025-05-28T06:05:29Z

`mypy_primer` results

Changes were detected when running on open source projects

beartype (https://github.com/beartype/beartype)
- warning[possibly-unbound-attribute] beartype/_util/api/external/utilclick.py:108:5: Attribute `callback` on type `BeartypeableT` is possibly unbound
- Found 557 diagnostics
+ Found 556 diagnostics

comtypes (https://github.com/enthought/comtypes)
+ warning[unused-ignore-comment] comtypes/_post_coinit/misc.py:377:46: Unused blanket `type: ignore` directive
- Found 472 diagnostics
+ Found 473 diagnostics

werkzeug (https://github.com/pallets/werkzeug)
+ error[invalid-assignment] src/werkzeug/sansio/response.py:597:9: Object of type `def on_update(value: WWWAuthenticate) -> None` is not assignable to attribute `_on_update` on type `(Unknown & ~None) | WWWAuthenticate`
- Found 385 diagnostics
+ Found 386 diagnostics

scrapy (https://github.com/scrapy/scrapy)
- error[invalid-assignment] tests/test_extension_telnet.py:16:9: Implicit shadowing of function `_get_telnet_vars`
- Found 1100 diagnostics
+ Found 1099 diagnostics

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ error[unresolved-attribute] tests/test_config.py:70:5: Can not assign object of type `float` to attribute `chop_threshold` on type `Display` with custom `__setattr__` method.
- Found 2769 diagnostics
+ Found 2770 diagnostics

sphinx (https://github.com/sphinx-doc/sphinx)
- error[invalid-assignment] sphinx/builders/__init__.py:698:9: Object of type `None` is not assignable to attribute `record_dependencies` of type `DependencyList`
- Found 607 diagnostics
+ Found 606 diagnostics

mitmproxy (https://github.com/mitmproxy/mitmproxy)
- error[invalid-assignment] test/mitmproxy/addons/test_next_layer.py:374:17: Object of type `tuple[Literal["2001:db8::1"], Literal[443], Literal[0], Literal[0]] | tuple[Literal["192.0.2.1"], Literal[443]]` is not assignable to attribute `peername` of type `tuple[str, int] | None`
- error[invalid-assignment] test/mitmproxy/addons/test_tlsconfig.py:439:17: Object of type `None` is not assignable to attribute `alpn_offers` on type `Unknown | Server`
- Found 1804 diagnostics
+ Found 1802 diagnostics

streamlit (https://github.com/streamlit/streamlit)
- error[invalid-assignment] lib/tests/streamlit/runtime/scriptrunner/script_runner_test.py:233:9: Implicit shadowing of function `_run_script`
- Found 3301 diagnostics
+ Found 3300 diagnostics

dd-trace-py (https://github.com/DataDog/dd-trace-py)
- error[invalid-assignment] tests/tracer/test_pin.py:76:13: Invalid assignment to data descriptor attribute `service` on type `Pin` with custom `__set__` method
- Found 6845 diagnostics
+ Found 6844 diagnostics

No memory usage changes detected ✅

crates/ty_python_semantic/src/types/infer.rs

thejchap · 2025-05-31T01:59:53Z

@sharkdp this is ready for a look, left a few notes in the description

thejchap · 2025-06-01T05:52:46Z

re: this comment/example from #17974:

So it seems reasonable to me to check the setattr call for both of these assignments. And because we see Never as a return type, emit a diagnostic for both of these assignments (which seems like the user intent).

from typing import Never


class Frozen:
    existing: int = 1

    def __setattr__(self, name, value) -> Never:
        raise AttributeError("Attributes can not be modified")


instance = Frozen()
instance.non_existing = 2  # fails at runtime
instance.existing = 2  # also fails at runtime

i think this would be different if it were actually a frozen dataclass - in that case it should be:

from typing import Never
from dataclasses import dataclass


@dataclass(frozen=True)
class Frozen:
    existing: int = 1

instance = Frozen()
instance.non_existing = 2  # unresolved attribute
instance.existing = 2  # invalid assignment

or at least that's how mypy handles it

if that's the case, then this first implementation won't work - we'd get back an invalid-assignment for instance.non_existing = 2 instead of unresolved-attribute

seems like the logic would need to be something like:

user-defined/overridden __setattr__?
- frozen dataclass?
  - emit some diagnostic as this isn't allowed
- else
  - attempt __setattr__ call
else
- found as instance member?
  - attempt __setattr__ call
- else
  - unresolved-attribute

## Summary Came across this while debugging some ecosystem changes in #18347. I think the meta-type of a typevar-annotated variable should be equal to `type`, not `<class 'object'>`. ## Test Plan New Markdown tests.

sharkdp · 2025-06-03T14:21:10Z

Thank you for working on this!

I think that we need to understand the ecosystem changes more closely and write some tests here. While taking a look at the yarl changes in particular, I found a bug which I fixed in #18439. I rebased your branch on top of that (so that we can see the updated ecosystem results), and also added an initial set of tests. I also took the liberty to rewrite the metadata in your first commit, as it seemed completely off, potentially from a rebase/squash (previous state: 22620d5).

sharkdp · 2025-06-03T16:10:30Z

crates/ty_python_semantic/src/types/infer.rs

-                                    self.context.report_lint(&INVALID_ASSIGNMENT, target)
-                                {
-                                    builder.into_diagnostic(format_args!(
+                    Err(CallDunderError::PossiblyUnbound(_)) => true,


I think this might be causing the changes in beartype, which look surprising to me. Do we need to change this branch, now that we call __setattr__ unconditionally? We should probably add some tests with unions of classes that do/don't have a __setattr__ method.

i will add some tests with unions and look into this a little more - but also, did this comment/concern make sense at all? i just want to confirm that this is the right approach generally (doing the unconditional call)

doing the __setattr__ call on every attribute assignment would result in (i think) the wrong diagnostics in this example:

from typing import Never from dataclasses import dataclass @dataclass(frozen=True) class Frozen: existing: int = 1 instance = Frozen() instance.non_existing = 2 # would expect unresolved attribute, but unconditional call would result in invalid assignment instance.existing = 2 # invalid assignment as expected

Yes, I think it's good to bring this up, but I think we can handle this.

For an arbitrary class with a __setattr__ method, we don't know what the body of that custom __setattr__ method does. So I think it's fine to unconditionally call it, see if it returns Never (there could be multiple overloads and maybe it only returns Never conditionally — it would be nice to add a test for this), and show the invalid-assignment diagnostic if that's the case.

On the other hand, for synthesized __setattr__ methods of frozen dataclasses, we understand how the body looks like. It returns an AttributeError if the attribute doesn't exist, and it returns a FrozenInstanceError if it's an existing attribute. It seems reasonable to emit "unresolved attribute" for the first case, and invalid-assignment (potentially with a customized error message for frozen instances) in the second case.

thejchap · 2025-06-04T22:42:23Z

@sharkdp

I think that we need to understand the ecosystem changes more closely and write some tests here ... I rebased your branch on top of that

sounds good! thanks - i will investigate the remaining changes after your rebase

it seemed completely off, potentially from a rebase/squash

🤦‍♂️ thanks - was a little hasty there - appreciate that...

sharkdp · 2025-06-06T12:19:39Z

sounds good! thanks - i will investigate the remaining changes after your rebase

Please let me know if you need help with this.

…n-type

sharkdp

Sorry that this took so long. I brought this up to date, made a few modifications and added some new tests. I also reviewed the ecosystem changes and except for the beartype thing which I don't fully understand, I think the changes are good.

Thank you very much!

thejchap · 2025-07-08T16:07:07Z

@sharkdp thank you!

sharkdp · 2025-07-08T18:40:37Z

@sharkdp thank you!

Of course, thank you for your contribution. Are you interested in following this up with a modified version of https://github.com/thejchap/ruff/pull/2/files? If not, that's also completely fine.

thejchap · 2025-07-09T02:20:27Z

@sharkdp yep, I'll revisit that

## Summary Synthesize a `__setattr__` method with a return type of `Never` for frozen dataclasses. https://docs.python.org/3/library/dataclasses.html#frozen-instances https://docs.python.org/3/library/dataclasses.html#dataclasses.FrozenInstanceError ### Related astral-sh/ty#111 #17974 (comment) #18347 (comment) ## Test Plan New Markdown tests --------- Co-authored-by: David Peter <[email protected]>

thejchap commented May 28, 2025

View reviewed changes

crates/ty_python_semantic/src/types/infer.rs Show resolved Hide resolved

thejchap commented May 28, 2025

View reviewed changes

crates/ty_python_semantic/src/types/infer.rs Outdated Show resolved Hide resolved

MichaReiser added the ty Multi-file analysis & type inference label May 28, 2025

thejchap force-pushed the feature/setattr-return-type branch 4 times, most recently from 5172ba6 to 22620d5 Compare May 31, 2025 01:48

thejchap changed the title ~~[ty] Unconditional __setattr__ call check during attribute assignment~~ [ty] Fix __setattr__ call check precedence during attribute assignment May 31, 2025

thejchap marked this pull request as ready for review May 31, 2025 01:59

thejchap requested review from AlexWaygood, carljm and dcreager as code owners May 31, 2025 01:59

sharkdp mentioned this pull request Jun 3, 2025

[ty] Meta-type of type variables should be type[..] #18439

Merged

sharkdp force-pushed the feature/setattr-return-type branch from 22620d5 to 4265911 Compare June 3, 2025 14:17

sharkdp reviewed Jun 3, 2025

View reviewed changes

thejchap force-pushed the feature/setattr-return-type branch from 4265911 to ee5be7e Compare June 7, 2025 01:23

thejchap mentioned this pull request Jun 7, 2025

[ty] Add attribute assignment tests #18527

Closed

4 tasks

thejchap marked this pull request as draft June 11, 2025 23:30

thejchap and others added 2 commits June 11, 2025 20:25

Fix __setattr__ call check precedence during attribute assignment

159819b

Add initial set of tests

e504514

thejchap force-pushed the feature/setattr-return-type branch from ee5be7e to e504514 Compare June 12, 2025 00:28

AlexWaygood removed their request for review June 16, 2025 22:36

Merge remote-tracking branch 'origin/main' into feature/setattr-retur…

f5b5326

…n-type

sharkdp added the ecosystem-analyzer label Jul 8, 2025

sharkdp added 3 commits July 8, 2025 14:20

Fix behavior for possibly-bound setattr methods

a4c796e

Merge remote-tracking branch 'origin/main' into feature/setattr-retur…

5ec6c10

…n-type

Change diagnostic message

a644cf3

sharkdp marked this pull request as ready for review July 8, 2025 13:32

sharkdp approved these changes Jul 8, 2025

View reviewed changes

sharkdp merged commit 738692b into astral-sh:main Jul 8, 2025
37 checks passed

thejchap mentioned this pull request Jul 13, 2025

[ty] synthesize __setattr__ for frozen dataclasses #19307

Merged

[ty] Fix __setattr__ call check precedence during attribute assignment #18347

[ty] Fix __setattr__ call check precedence during attribute assignment #18347

Uh oh!

Conversation

thejchap commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Ecosystem changes

Test Plan

Uh oh!

github-actions bot commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

Uh oh!

Uh oh!

thejchap commented May 31, 2025

Uh oh!

thejchap commented Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharkdp commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sharkdp Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

thejchap Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

sharkdp Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

thejchap commented Jun 4, 2025

Uh oh!

sharkdp commented Jun 6, 2025

Uh oh!

sharkdp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thejchap commented Jul 8, 2025

Uh oh!

sharkdp commented Jul 8, 2025

Uh oh!

thejchap commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[ty] Fix setattr call check precedence during attribute assignment #18347

[ty] Fix setattr call check precedence during attribute assignment #18347

thejchap commented May 28, 2025 •

edited

Loading

github-actions bot commented May 28, 2025 •

edited

Loading

`mypy_primer` results

thejchap commented Jun 1, 2025 •

edited

Loading

sharkdp commented Jun 3, 2025 •

edited

Loading