Stronger locking for parallel operations

[konstin]

We need to ensure that there a no race conditions when two uv subcommands in parallel. This includes:

• Parallel filesystem modifications in interpreters/venvs
• TOCTOU differences between determining packages and performing changes
• Installing as well as parallel editing of pyproject.toml, including TOCTOU errors with

We already enforce these guarantees with locks for most subcommands. (Follow-up to #12751.)

The required locking generally applies to operations of the same kind, such as uv sync, but it can also be cross-operation, e.g. uv pip list could fail reading the installed packages if uv sync removes some mid-operation.

An exception is the cache, where we use symlinks for atomic operations to allow safe parallel modification. If we need multiple locks, e.g. pyproject.toml and the interpreter, we should define and document an order for the locks and use it globally to prevent deadlocks.

Important

• Ensure that uv run locks when syncing the envirnoment.
  uv run calls the same logic as uv sync internally, so we can use the same locking pattern
  Prevent concurrent updates of the environment in uv run #14153
• Ensure that uv version --bump locks when syncing the environment.
• Lock build directories for setuptools (concurrent wheel builds fail oddly #13703)
  Lock the source tree when running setuptools, to protect concurrent builds #14174
• Ensure that all other subcommands already use the correct locking.
• uv cache operations don't clash with other uv operations Make uv cache clean parallel process safe #15888

Less Important

• Determine if there are reliable testing strategies, when uv testing currently uses subprocesses.
  Integrating shuttle is not feasible with several uv processes, but maybe a script that locally runs all modifying uv subcommands in a loop in parallel and verifies that we en in a sound state?
• uv add, uv remove and uv version --bump should lock pyproject.toml before computing edits.
  The use case here is that these operations may be started by an IDE/LSP while another operation is e.g. still in its sync phase and/or may roll back in the end.
  We can likely test this by having a build we control in the sync phase.
• uv add and uv remove should not lock a global interpreter path when they're not actually going to sync anything (e.g. inline script metadata without an existing cached or locked venv)

Nice to have

• Are there portable filesystem read-write locks to allow e.g. blocking uv pip list from reading while uv sync is editing?
• Does uv init/uv venv and venv creation in general need to lock?

[charliermarsh]

To clarify, we already lock where it's known to matter and it's considered safe to run uv concurrently, barring oversights like #13869. This issue is about improving those guarantees, and making it easier and more rigorous for us to enforce.

[charliermarsh]

I don't know if we should lock for read operations like uv pip list. Locking at the operation level is pretty coarse and I'd prefer to avoid it where we can.

[petamas]

@konstin

An exception is the cache, where we use symlinks for atomic operations to allow safe parallel modification.

@charliermarsh

To clarify, we already lock where it's known to matter and it's considered safe to run uv concurrently, barring oversights like #13869.

I'm not sure how relevant it is for this issue, but I want to point out that concurrent cache writes are still not safe on Windows using uv 0.7.11: #11002

[leviska]

uv run calls the same logic as uv sync internally, so we can use the same locking pattern

I just want to clarify, the language of this phrase can be misleading in a way that run and sync "use the same locking pattern" but "different locks (or something)".
From my user perspective, they must be the same, because for me uv run is the same as uv sync && uv just_run, so any combination of sync and run should not race with each other. I.e., uv sync | uv run should block one of the commands (for the syncing part)

[konstin]

The issue is written for guiding the internal development process, as a user you won't observe any of this and uv sync and uv run will work the same and lock the same, even if their implementations have slightly different code paths due to internal complexities that the subcommands are abstracting over.

The circumstances you need to hit one of those case where something fails are already rare, as we're already very defensive in our filesystem operations, for example using temp dirs and symlinks for atomic operations and we already have locks in many places (which is also why it took so long for #12751 to surface compare to many way obscurer bugs).

[b-phi]

I think we're also getting bit by this. Similar to this issue, we use uv to synchronize Python environments across large ray clusters. In our case, we mount a shared uv cache volume to all workers and run something like uv run --all-packages --link-mode symlink to create the environments and start our jobs. We experienced errors like

Failed to install: ... Caused by: failed to remove file ... No such file or directory (os error 2)

And other times more serious things like missing modules after the environment has been setup

ModuleNotFoundError: No module named 'botocore.session'

[konstin]

Does this happen with --link-mode copy too, or only with --link-mode symlink?