[SPARK-55266][INFRA] Add pre-commit hooks for format/lint

[gaogaotiantian]

What changes were proposed in this pull request?

Add pre-commit yaml file to allow users optionally enable pre-commit hook.

Why are the changes needed?

It's common that users forget to check lint/format and have to wait for 2 hours to realize it. This prevents that situation. Also python reformatter and lint is sub-second so it won't impact user too much.

This is also optional. User won't feel it without explicitly installing pre-commit

Does this PR introduce any user-facing change?

No

How was this patch tested?

Local pre-commit worked.

Was this patch authored or co-authored using generative AI tooling?

No

[github-actions]

JIRA Issue Information

=== Improvement SPARK-55266 ===
Summary: Add pre-commit hook file for python lint/format
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[zhengruifeng]

I think we should provide a requirements-lint.txt with all versions pinned, so that developers can run the python linter locally

[gaogaotiantian]

Developers can already run linters locally. This PR is to run it "automatically" before each commit. requirements.txt already has all the required tools.

[zhengruifeng]

I see, does dev/requirements exactly match the linter image now? The mypy check always fail in my local

get this after reinstall dev/requirements

starting mypy annotations test...
annotations failed mypy checks:
python/pyspark/ml/linalg/__init__.py:407: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/pandas/series.py:1085: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/pandas/series.py:1087: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/pandas/indexes/base.py:550: error: Unused "type: ignore[arg-type]" comment  [unused-ignore]
python/pyspark/mllib/linalg/__init__.py:460: error: Variable "numpy._typing.ArrayLike" is not valid as a type  [valid-type]
python/pyspark/mllib/linalg/__init__.py:460: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
Found 5 errors in 4 files (checked 1194 source files)

[zhengruifeng]

ok, we don't check mypy which is pretty slow (and flaky in my local).
Then LGTM

[Yicong-Huang]

I think it is slow because mypy is checking all the dependencies. can we let it not to do it?

[gaogaotiantian]

mypy is slow, especially without cache. It's pure python. astral and meta are both working on a rust-based type checker which are significantly faster than mypy, but neither of them is stable enough for production use.

[gaogaotiantian]

Yeah mypy is much slower and not as consistent across environments. We only do black + ruff for pre-commit.

Are there a lot of error reports for your mypy? I've updated some requirements while I'm fixing mypy, did you upgrade to the latest requirements.txt? There should be a few errors but not a lot. I'm working on a more stable mypy - including upgrading the version of mypy. Older versions have more inconsistent results on different library versions.

[gaogaotiantian]

The mypy check always fail in my local

I believe you are running lint on 3.12. The lint image is 3.11.

[Yicong-Huang]

Love this feature!

The mypy check always fail in my local

I believe you are running lint on 3.12. The lint image is 3.11.

BTW, updating lint image to 3.12 here in #54042.

[zhengruifeng]

The mypy check always fail in my local

I believe you are running lint on 3.12. The lint image is 3.11.

I have been using 3.13 for some time :)

[HyukjinKwon]

Merged to master.