Vendor a TOML v1.0.0 compatible TOML parser

[hukkin]

What's the problem this feature will solve?

Pip currently only supports TOML v0.5.0. Reading a pyproject.toml containing TOML v1.0.0-rc.1+ syntax will cause a failure, even if the unsupported syntax is in a section that pip doesn't care about (e.g. [tool.*]).

Some projects in the Python ecosystem are hesitant to support TOML v1.0.0 because pip does not. Meanwhile other projects are using Poetry and its v1.0.0-rc.1 compatible tomlkit for building packages, thus potentially creating pip incompatible builds without noticing an issue.

Describe the solution you'd like

Vendor a TOML v1.0.0 compatible TOML parser. I suggest my very own Tomli which, in addition to being v1.0.0 compatible, fixes many bugs present in the toml library, parses 2x as fast, and is much smaller with only 50% the amount of lines of code.

Alternative Solutions

Stick to a TOML v0.5.0 compatible parser. Perhaps mention in the docs that projects where pyproject.toml includes TOML v1.0.0-rc.1+ syntax are not supported.

[hukkin]

On a completely separate but related note: shouldn't PEP 518 specify supported TOML spec version for pyproject.toml to avoid these sorts of issues? Otherwise the pyproject.toml shared by countless tools, each reading it with a parser of their own, is just asking for issues like this...

[uranusjr]

shouldn't PEP 518 specify supported TOML spec version for pyproject.toml

Good point, could you raise this on discuss.python.org? PEP amendment should happen there to reach the most related people.

[hukkin]

@uranusjr started a topic about that

[pradyunsg]

FWIW, I think we should bump up to TOML 1.0.0 here, and update our standards to that.

[hukkin]

@pradyunsg Would you be happy if I made a PR where I replace toml with Tomli?

The alternatives I can think of are:

• do the aforementioned, but with tomlkit (tomlkit is a fairly large lib though and does many things not needed here)
• create and maintain a v1.0.0 compatible toml fork and vendor that
• wait for github user uiri to bring toml up-to-date with v1.0.0 and vendor the updated version

[pfmoore]

I haven't done any analysis of the various TOML libraries, but I'm personally OK with switching to Tomli. The design goals of small, fast and read-only seem very compatible with what we want.

[pradyunsg]

Would you be happy if I made a PR where I replace toml with Tomli?

Yup yup, and @pfmoore said what I wanted to say.

wait for github user uiri to bring toml up-to-date with v1.0.0

This, I've been working toward.

[hukkin]

@pradyunsg your last sentence confused me a bit, haha. So to be 100% certain before I put in the work for a PR:

This, I've been working toward.

Are you still working toward this and want to have it in pip? Or shall I make a PR with tomli (note, NOT toml 😄 )?

[uranusjr]

Pradyun meant he was working toward having toml release a TOML 1.0 compatible version, see uiri/toml#300.

And yes, please do make a PR for the switch to tomli.