Lock the source tree when running setuptools, to protect concurrent builds

[oconnor663]

Fixes #13703.

If I repeat the repro steps from my comment on that issue with this PR, the errors are gone:

$ cat >setup.py <<EOF
import setuptools
setuptools.setup(name="scratch", version="4")
EOF
$ for i in `seq 10` ; do
    ~/uv/target/fast-build/uv build --wheel &
done
[...lots of build output, no errors...]

There's also no performance impact:

$ hyperfine "/tmp/before build --wheel" "/tmp/after build --wheel"
Benchmark 1: /tmp/before build --wheel
  Time (mean ± σ):     255.8 ms ±   3.0 ms    [User: 225.7 ms, System: 41.5 ms]
  Range (min … max):   251.4 ms … 260.5 ms    11 runs

Benchmark 2: /tmp/after build --wheel
  Time (mean ± σ):     255.3 ms ±   2.6 ms    [User: 219.7 ms, System: 47.3 ms]
  Range (min … max):   252.8 ms … 262.0 ms    11 runs

Summary
  /tmp/after build --wheel ran
    1.00 ± 0.02 times faster than /tmp/before build --wheel

[charliermarsh]

I don't think locking the output directory is actually sufficient because IIRC setuptools uses a build directory in the source directory? So, like, concurrent builds to different output directories would still fail?

[oconnor663]

Ah true, I still get failures with something like this:

$ for i in `seq 10` ; do
    ~/uv/target/fast-build/uv build --wheel --out-dir /tmp/mybuild$i &
done

[oconnor663]

I've switched to locking (the hash of) the source dir. The following now succeeds with no errors, and the builds run in series:

$ cat >setup.py <<EOF
import setuptools
setuptools.setup(name="scratch", version="4")
EOF
$ for i in `seq 10` ; do
    ~/uv/target/fast-build/uv build --wheel --out-dir `mktemp -d` &
done

I'm considering refactoring this a bit to stop putting so many file locks directly in /tmp. Going to put another PR on top of this one, which we can take or not.

[oconnor663]

@zanieb suggested that special-casing setuptools might make more sense than taking a lock under /tmp. I'm going to look into that. (In that case we could still choose to take #14225, but it would no longer be related to this RP.)

[oconnor663]

Just put up a new iteration. I've kept the same locking strategy (a cache_digest-named file under temp_dir()), but I've made it specific to setuptools, and we take the lock before we invoke any setuptools commands, including get_requires_for.... I noticed that even that read-only-sounding command creates *.egg-info, and even though I haven't seen that lead to failures when multiple processes do it at once, it seems safer to protect it. @konstin do you have time to take a look?

[oconnor663]

These are drive-by cleanup.

[konstin]

This makes sense to me

[charliermarsh]

I'm not a huge fan of special-casing setuptools but I don't feel strongly enough to advocate for a change.

[charliermarsh]

(Actually just ignore my comment, it's alright.)

[oconnor663]

(rebasing to pick up test fixes from main)