Pydantic v2

[V02460]

Update Pydantic to v2.

Steps I took to create this change:

1. Remove import indirection
2. Run the Pydantic migration tool
3. Fix remaining issues
4. Enable strict mode and remove check_pydantic_models.py

Notable changes are

1. differing return type for threepid requests with an invalid email
2. changed logic for _metadata_url and _introspection_endpoint
3. changed definition of AnyEventId
4. Disabled strict mode for tuple and set types

Closes #15858
Closes #15979

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[anoadragon453]

Thank you very much for filing this - it saves the dev team a ton of effort ❤️

The sytest failures appear to be due to Sytest not following the spec - hooray for validation!

I've started on fixing them - by converting the tests to Complement and debugging them there, starting with: matrix-org/complement#811

Some minor comments below.

[anoadragon453]

Do we require 2.12?

Several Linux distributions do not carry that version, so will struggle to package Synapse: https://repology.org/project/python%3Apydantic/versions

In general we should aim to specify the lowest possible version in pyproject.toml that's compatible with the code. Then in poetry.lock we specify highest possible (so those distributing Synapse using pyenv's can get the latest fixes and improvements).

[V02460]

It’s dependent on the Python version. Pydantic 2.7.4 was the first version to give correct results. For Python 3.13 and Python 3.14 later versions are required, with Pydantic 2.12 being the first version to support Python 3.14.

We could discriminate based on Python version, something like pydantic>=2.7.4;python_version<'3.13', etc.

[V02460]

Pydantic 2.8 added support for Python 3.13. Because most of the distros in the table packaged at least Pydantic 2.8, I’d suggest the following:

pydantic = [
    { version = "~= 2.8", python = "< 3.14" },
    { version = "~= 2.12", python = ">= 3.14" },
]

[anoadragon453]

I see, I didn't realise >=2.7.4 was necessary. Indeed, I don't see any distros in that list with 2.7.*, so it would make sense to go to the next minor version up, 2.8*.

Your suggestion looks sound!

[V02460]

Done :)

[V02460]

Oh man, trial-olddeps is complaining. But I don’t get what’s going on just yet.

Edit: My first guess is a logic error in check_dependencies.py.

[anoadragon453]

You're correct. check_dependencies.py doesn't take the python attribute into account when determining which dependencies it needs to install. It then fails when attempting to install 2.12.x even though it's on Python 3.9.

I did some preliminary work on updating it in a separate branch. Let me bring that up into a PR.

[anoadragon453]

@V02460 I believe I've bypassed the previous old-deps errors and now we've moved on to something else (likely just changing a set to a list in the defined type?).

Heading to bed now, and can pick it up in the morning.

[anoadragon453]

#19110 has now merged, and I've updated this branch to latest develop. Assuming CI now passes, this should be ready to merge!

[V02460]

Looks like it. Nice work! 🎉