[SPARK-33457][PYTHON] Adjust mypy configuration

[zero323]

What changes were proposed in this pull request?

This pull request:

• Adds following flags to the main mypy configuration:
  • strict_optional
  • no_implicit_optional
  • disallow_untyped_defs

These flags are enabled only for public API and disabled for tests and internal modules.

Additionally, these PR fixes missing annotations.

Why are the changes needed?

Primary reason to propose this changes is to use standard configuration as used by typeshed project. This will allow us to be more strict, especially when interacting with JVM code. See for example #29122 (review)

Additionally, it will allow us to detect cases where annotations have unintentionally omitted.

Does this PR introduce any user-facing change?

Annotations only.

How was this patch tested?

dev/lint-python.

[zero323]

FYI @Fokko @HyukjinKwon

Still work in progress, any suggestions about desired flags (I considered show_error_codes, for clarity, and disallow_any_generics).

[HyukjinKwon]

My only concern is the maintenance cost for being strict. If it doesn't add non-trivial overhead, I am good to be more strict on the type hints.

[zero323]

My only concern is the maintenance cost for being strict. If it doesn't add non-trivial overhead, I am good to be more strict on the type hints.

For the first two options - we are already covered, as these were used in pyspark-stubs.

The third one required some fixes, but these are mostly things that should be there. In all other cases, we can either ignore the whole module (as with tests and such) or specific lines (for example I considered ignoring callJavaFunc as these annotations are hardly useful and functions are not exactly public).

So overall, I don't think there's much complexity added and using these, gives us more confidence that annotations are in a good shape.

[SparkQA]

Test build #131122 has finished for PR 30382 at commit 4de37b2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35725/

[SparkQA]

Test build #131124 has finished for PR 30382 at commit c42fe59.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35725/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35727/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35727/

[SparkQA]

Test build #131515 has finished for PR 30382 at commit 426c7bb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36119/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36119/

[HyukjinKwon]

@zero323, let me know when you think this is ready. Looks fine to me.

[Fokko]

Looks good!

[SparkQA]

Test build #131547 has finished for PR 30382 at commit 9182a98.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class PipelineReader(MLReader[Pipeline]):
• class PipelineModelReader(MLReader[PipelineModel]):

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36150/

[zero323]

@zero323, let me know when you think this is ready. Looks fine to me.

Thanks @HyukjinKwon. I went over all violations of disallow_any_generics and these are mostly intended (little or no additional value gained by addressing these). So I think it ready to review.

Looks good!

Thanks for reviewing @Fokko!

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36150/

[SparkQA]

Test build #131599 has finished for PR 30382 at commit 9182a98.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class PipelineReader(MLReader[Pipeline]):
• class PipelineModelReader(MLReader[PipelineModel]):

[HyukjinKwon]

retest this please

[SparkQA]

Test build #131614 has finished for PR 30382 at commit 9182a98.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class PipelineReader(MLReader[Pipeline]):
• class PipelineModelReader(MLReader[PipelineModel]):

[HyukjinKwon]

Merged to master.

[zero323]

Thanks @Fokko and @HyukjinKwon!