pandas-stubs testing fails with 1.1.310 and 1.1.309, worked fine with 1.1.308

[Dr-Irv]

Create a python environment with python 3.10 and install from pypi

• pandas
• pandas-stubs
• numpy
• jinja2
• xarray
• pytest
• pyarrow
• matplotlib

Download the source directory of pandas-stubs 2.0.1.230501 here: https://github.com/pandas-dev/pandas-stubs/releases/tag/v2.0.1.230501 in either ZIP or tar.gz format

Extract the tests folder. Then do pyright tests . Using pyright basic type checking, pyright 1.1.308 reports no errors.
pyright 1.1.310 reports

c:\Code\Misc\pstubtst\tests\test_frame.py
  c:\Code\Misc\pstubtst\tests\test_frame.py:1449:32 - error: Operator "+" not supported for types "TimestampSeries" and "Timedelta" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_frame.py:2101:16 - error: Expression of type "DataFrame" cannot be assigned to return type "MyDataFrame[int]"
    "DataFrame" is incompatible with "MyDataFrame[int]" (reportGeneralTypeIssues)
c:\Code\Misc\pstubtst\tests\test_indexes.py
  c:\Code\Misc\pstubtst\tests\test_indexes.py:771:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_indexes.py:772:23 - error: "assert_type" mismatch: expected "Timestamp" but received "Scalar" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_indexes.py:773:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
c:\Code\Misc\pstubtst\tests\test_io.py
  c:\Code\Misc\pstubtst\tests\test_io.py:880:9 - warning: Expression value is unused (reportUnusedExpression)
c:\Code\Misc\pstubtst\tests\test_pandas.py
  c:\Code\Misc\pstubtst\tests\test_pandas.py:1906:40 - error: Cannot access member "month" for type "Index"
    Member "month" is unknown (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_pandas.py:1915:40 - error: Cannot access member "month" for type "Index"
    Member "month" is unknown (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_pandas.py:1924:77 - error: Cannot access member "month" for type "Index"
    Member "month" is unknown (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_pandas.py:1933:77 - error: Cannot access member "month" for type "Index"
    Member "month" is unknown (reportGeneralTypeIssues)
c:\Code\Misc\pstubtst\tests\test_scalars.py
  c:\Code\Misc\pstubtst\tests\test_scalars.py:549:23 - error: "assert_type" mismatch: expected "TimedeltaIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:551:21 - error: Operator "+" not supported for types "Timedelta" and "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:551:21 - error: "assert_type" mismatch: expected "TimedeltaSeries" but received "Unknown" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:583:23 - error: "assert_type" mismatch: expected "TimedeltaIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:585:21 - error: "assert_type" mismatch: expected "TimedeltaSeries" but received "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:616:23 - error: "assert_type" mismatch: expected "TimedeltaIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:642:23 - error: "assert_type" mismatch: expected "TimedeltaIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:644:21 - error: Operator "-" not supported for types "Series[Unknown]" and "Timedelta" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:644:21 - error: "assert_type" mismatch: expected "TimedeltaSeries" but received "Unknown" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1000:21 - error: "assert_type" mismatch: expected "np_ndarray_bool" but received "Literal[False]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1003:21 - error: "assert_type" mismatch: expected "np_ndarray_bool" but received "Literal[True]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1005:27 - error: Cannot access member "all" for type "bool"
    Member "all" is unknown (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1181:23 - error: "assert_type" mismatch: expected "TimedeltaSeries" but received "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1193:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1194:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1197:21 - error: Operator "+" not supported for types "Timestamp" and "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1197:21 - error: "assert_type" mismatch: expected "TimestampSeries" but received "Unknown" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1221:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1223:21 - error: Operator "-" not supported for types "Timestamp" and "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1223:21 - error: "assert_type" mismatch: expected "TimestampSeries" but received "Unknown" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1246:23 - error: "assert_type" mismatch: expected "TimestampSeries" but received "Series[Unknown]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1358:21 - error: "assert_type" mismatch: expected "Series[bool]" but received "Literal[False]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1361:21 - error: "assert_type" mismatch: expected "Series[bool]" but received "Literal[True]" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_scalars.py:1363:27 - error: Cannot access member "all" for type "bool"
    Member "all" is unknown (reportGeneralTypeIssues)
c:\Code\Misc\pstubtst\tests\test_series.py
  c:\Code\Misc\pstubtst\tests\test_series.py:1476:16 - error: Expression of type "Series[Unknown]" cannot be assigned to return type "MySeries[float]"
    "Series[Unknown]" is incompatible with "MySeries[float]" (reportGeneralTypeIssues)
c:\Code\Misc\pstubtst\tests\test_timefuncs.py
  c:\Code\Misc\pstubtst\tests\test_timefuncs.py:448:25 - error: Cannot access member "month" for type "Index"
    Member "month" is unknown (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_timefuncs.py:448:23 - error: "assert_type" mismatch: expected "_IntIndexType" but received "Unknown" (reportGeneralTypeIssues)

I don't understand why upgrading pyright picked up all these issues. Some may be correct, but others don't seem to me. I've only looked at a few of them. The following ones don't make sense:

  c:\Code\Misc\pstubtst\tests\test_frame.py:2101:16 - error: Expression of type "DataFrame" cannot be assigned to return type "MyDataFrame[int]"

Corresponding code to the above:

    T = TypeVar("T")

    class MyDataFrame(pd.DataFrame, Generic[T]):
        ...

    def func() -> MyDataFrame[int]:
        return MyDataFrame[int]({"foo": [1, 2, 3]})  # pyright is reporting error here

Above doesn't make sense because we are returning the MyDataFrame class, so why is pyright thinking that it is a DataFrame??

  c:\Code\Misc\pstubtst\tests\test_indexes.py:771:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_indexes.py:772:23 - error: "assert_type" mismatch: expected "Timestamp" but received "Scalar" (reportGeneralTypeIssues)
  c:\Code\Misc\pstubtst\tests\test_indexes.py:773:23 - error: "assert_type" mismatch: expected "DatetimeIndex" but received "Index" (reportGeneralTypeIssues)

    idt = pd.DatetimeIndex(["2022-08-14", "2022-08-20", "2022-08-24"])
    check(assert_type(idt, pd.DatetimeIndex), pd.DatetimeIndex, pd.Timestamp)  # pyright reporting error here
    check(assert_type(idt[0], pd.Timestamp), pd.Timestamp)  # pyright reporting error here
    check(assert_type(idt[[0, 2]], pd.DatetimeIndex), pd.DatetimeIndex, pd.Timestamp)   # pyright reporting error here

First one doesn't make sense because we are returning a DatetimeIndex class, so why does pyright think it is Index.
Second one and third one happen because of the error on the first one.

Those are the ones I had the patience to analyze, but your help in indicating what has changed in pyright that has created these new errors would be appreciated.

One other note - there are more errors with 1.1.309 (171 of them) than with 1.1.310 (36 of them) so that leaves me to believe that a bug was introduced in pyright.

[erictraut]

Thanks for the bug report. There were some major changes introduced in 1.1.309 and 11.310 related to constructor analysis and bidirectional type inference. These are intended to address some long-standing issues and shortcomings, but they required significant rework. It's quite likely that a regression was introduced or a latent bug was uncovered. I'll dig into it in more detail.

I think this underscores the utility of a broader set of regression tests that we can run on pandas-stubs and other source bases that rely on pyright for type checking. I've therefore re-opened #4811.

[erictraut]

It looks like almost all of the errors are related to the fact that several classes in the pandas-stubs declare a __new__ method that does not return Self. Instead, they return the type of the class in which they are declared. That means they won't work properly for subclasses.

This problem exists for DataFrame (defined in core/frame.pyi), Series (defined in core/series.pyi) and Index (defined in core/indexes/base.pyi). If I fix these __new__ method declarations to return Self (at least for the fallback override), it eliminates all but three of the type errors.

Pyright's previous handling of constructors let you get away with this sort of a bug. Pyright's new code properly reflects the declared return type for constructors. It also more closely reflects the runtime behavior of the type.__call__ implementation. So I consider this a bug in the stubs, not in pyright. (It will still be seen as a regression from the perspective of pandas and pyright users, which I regret. Ideally, I would have noticed this problem and worked with you to get it fixed in the stubs prior to releasing an updated version of pyright.)

After I fix the above three locations in the stubs, there are three errors remaining, and I'll probably need your help in determining whether those are intended or not. The first is on line 1449 of test_frame.py (in the test_types_regressions function). The other two are on lines 2604 and 2609 of the same file, in the test_align function.

[erictraut]

Looking more closely at the remaining three errors, I think all are explained by bugs in earlier versions of pyright. The latest version of pyright appears to produce the correct result. Let me know if you disagree with this assessment.

If my analysis is correct, then everything is working as intended in pyright, and there's no bug to fix here. I'm therefore marking this as "as designed". I'll keep it open until you've had a chance to review my findings.

[Dr-Irv]

Looking more closely at the remaining three errors, I think all are explained by bugs in earlier versions of pyright. The latest version of pyright appears to produce the correct result. Let me know if you disagree with this assessment.

Yes, I think pyright did better type inference in those 3 cases, so I have fixes that address those.

However, making the changes regarding using Self in the __new__() constructors creates major issues for our tests using mypy . We need the stubs to support both pyright and mypy.

I'm heading out on a 9 day vacation, so may not be able to really address this during that time. Please keep the issue open, or suggest a change without making the change to using Self .

[erictraut]

OK, thanks for the confirmation. I've spent a bit of time looking at the output of mypy, and I can't explain its behavior. For example, it evaluates the type of pd.Series([1.0, 2.2, 3.3]) as Series[str]!?!? I don't know where str is coming from in this case. This looks like a straight-up bug in mypy.

I don't think there's anything actionable for me to do in pyright at this time. It appears to be working correctly. Reverting my recent changes will break other code that is correctly annotated, so I don't think that's the right answer. Self is the correct return type for __new__ if you want these classes to work with subclasses.

My recommendation is for you to work with the mypy maintainers to better understand its behavior. They may be able to suggest a workaround and/or expedite a fix for their next release.

Have a good vacation!

[Dr-Irv]

I have a set of fixes that work for myoy and pyright except for one test on pyright. Issue is that things need to work with subclassing a Series without specifying its generic parameter. I will try to push that branch when I get connectivity in the next few days.

Need to be able to have __new__ return Series without specification of the type parameter with a particular set of overloads. I.e. something equivalent to Series[Unknown]

[Dr-Irv]

Pick up https://github.com/Dr-Irv/pandas-stubs/tree/pyright310 that fixes everything but one test.

mypy passes this version, and the only pyright test not passing is this one:

  c:\Code\pandas-stubs\tests\test_series.py:1610:16 - error: Expression of type "Series[Unknown]" cannot be assigned to return type "MySeries[float]"
    "Series[Unknown]" is incompatible with "MySeries[float]" (reportGeneralTypeIssues)

which comes from doing this:

    T = TypeVar("T")

    class MySeries(pd.Series, Generic[T]):
        ...

    def func() -> MySeries[float]:
        return MySeries[float]([1, 2, 3])   # pyright reports error here

This has to do with the last overload for __new__() in series.pyi, which returns Series as opposed to Self, because it needs to return a non-generic Series . If I change that to Self, then pyright is happy, but mypy is not in many places.

[erictraut]

Please work with the mypy folks on this. The correct return type here is Self, and mypy should be able to handle it correctly. As I mentioned above, this looks to be a straight-up bug in mypy, so hopefully it will be easy for them to diagnose and fix. Once they understand the root cause, they might be able to suggest a workaround.

[Dr-Irv]

I have reported multiple issues on code like this over the past 18 months so it is highly unlikely they will fix the bug in a reasonable amount of time.

From issues on pandas-stubs, we get more people using mypy than pyright so asking the mypy maintainers to fix something to support this because pyright made a change that is incompatible with myoy is not going to support the community at large.

I like pyright but we need to find some way of making stubs working with both type checkers without waiting for either type checker to fix a bug. We already jump through numerous hoops to make our stubs work with both myoy and pyright.

[erictraut]

I've spent a bunch of time investigating this issue further. I now understand (at a high level) why mypy is evaluating the type Series[str]. It appears to be a bug in mypy's handling of constrained type variables. If it doesn't solve a class-scoped constrained TypeVar, it simply assumes the first constraint type (which is str in the case of the S1 TypeVar used in the pd.Series class). It should use Any in this case.

Here's a simplified demonstration of the bug:

from typing import Generic, Self, TypeVar

S = TypeVar("S", int, str, bytes)

class Foo(Generic[S]):
    def __new__(cls) -> Self: ...

f = Foo()
reveal_type(f)  # Mypy reveals: Foo[int]; should be Foo[Any]

I seem to recall seeing a bug like this reported in the mypy issue tracker in the past, but I can't find it now. There are approximately 25 open bugs related to constrained type variables in mypy.

In my investigations, I made a number of observations in the pandas stubs that are worth mentioning:

1. The pd.Series.__new__ method has some pretty glaring bugs. Some of the overrides swap the index and dtype parameters. I presume this was done to accommodate the case where the dtype argument is specified by keyword. This can be handled properly by retaining the correct parameter order and adding a keyword-only separator (*). Let me know if you need more details. I definitely recommend addressing this bug because it will lead to incorrect type evaluations.
2. Some of the overloads for pd.Series.__new__ return types other than pd.Series (e.g. TimestampSeries, PeriodSeries). These other types appear nowhere within the pandas library. I presume they were added to the stubs as sort of a hack to expose certain methods that apply only for certain specializations of Series. There are better ways of handling this that more closely model the library's runtime behavior. One of the big problems with introducing these "made-up" subclasses is that Series cannot be properly subclassed. For example, if you create a subclass of Series (let's call it MySeries) but then pass arguments to the constructor that invoke an overload that returns a TimestampSeries, you will now have a TimestampSeries rather than a MySeries. If I were maintaining these stubs, I'd look at eliminating these made-up classes and folding the functionality into the main Series class. You can use a specialized type annotation for self to limit the behavior to a specific specialization of Series. For example, def __add__(self: Series[Timedelta], ....
3. The pd.Series class is making use of a constrained TypeVar. I'm not sure why this was done, but it's almost certainly not the right approach. A constrained TypeVar is rarely the right tool — and that's especially true when it comes to class-scoped type variables. There are many reasons why a constrained TypeVar is problematic, but to give you one example... A Series is able to contain heterogeneous types, but a constrained TypeVar doesn't permit something like Series[int | str]. I recommend that you switch from a constrained TypeVar to a regular TypeVar with a bound that is a union. Not only will this more correctly model the way Series works in pandas, it will also avoid the many bugs that mypy has with constrained TypeVars. I see that constrained TypeVars are used in a number of locations within the pandas stubs. I recommend looking at each of these uses and considering whether there's a better approach.

I'm sympathetic to the point you made about jumping through numerous hoops to make the pandas stubs work with both mypy and pyright, and I appreciate the contributions you and others have made here. However, I can't be beholden to mypy's bugs, limitations, and slow fix rates. Mypy and its contributors deserve a lot of credit for introducing static typing to the world of Python, but at this point I feel that mypy is holding things back more than moving things forward for the Python community. I don't see a good solution to this problem.

As for this specific issue, I think the changes you've made in your branch look good. The last overload for Series.__new__ is still "wrong" from a type perspective. It should be Self, but given mypy's buggy behavior, I don't think it's feasible for you to make that change. You might want to add a comment indicating that Self is the correct return type annotation but that Series is required to work around a bug in mypy.

You'll also need to suppress the pyright error reported in the test_pandera_generic test case. You can do this with a # pyright: ignore[reportGeneralTypeIssues] comment. Once again, I think it would be useful to add a comment indicating why this is needed.

You might want to consider making some of the other changes I mentioned above, but those can be done separately. Let me know if you have any follow-up questions about those suggestions.

I don't think there's anything left for me to do with this issue, so I'm going to close this bug report.