Skip to content

Add relocatable installs to support concurrency-safe cached environments#5509

Merged
charliermarsh merged 2 commits intomainfrom
charlie/safe
Jul 29, 2024
Merged

Add relocatable installs to support concurrency-safe cached environments#5509
charliermarsh merged 2 commits intomainfrom
charlie/safe

Conversation

@charliermarsh
Copy link
Copy Markdown
Member

@charliermarsh charliermarsh commented Jul 27, 2024

Summary

The idea here is similar to what we do for wheels: we create the CachedEnvironment in the archive-v0 bucket, then symlink it to its content-addressed location. This ensures that we can always recreate these environments without concern for whether anyone else is accessing them.

Part of the challenge here is that we want the virtual environments to be relocatable, because we're now building them in one location but persisting them in another. This requires that we write relative (rather than absolute) paths to scripts and entrypoints. The main risk with relocatable virtual environments is that the scripts and entrypoints themselves are not relocatable, because they use a relative shebang. But that's fine for cached environments, which are never intended to leave the cache.

Closes #5503.

@charliermarsh charliermarsh added bug Something isn't working enhancement New feature or improvement to existing functionality preview Experimental behavior labels Jul 27, 2024
@charliermarsh charliermarsh force-pushed the charlie/safe branch 2 times, most recently from 6bcb8e1 to 67c2105 Compare July 27, 2024 17:26
@charliermarsh charliermarsh requested review from konstin and zanieb July 27, 2024 17:51
@charliermarsh charliermarsh enabled auto-merge (squash) July 29, 2024 00:18
@charliermarsh charliermarsh disabled auto-merge July 29, 2024 00:32
@charliermarsh charliermarsh merged commit 9af0ae2 into main Jul 29, 2024
@charliermarsh charliermarsh deleted the charlie/safe branch July 29, 2024 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or improvement to existing functionality preview Experimental behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make CachedEnvironment robust to concurrent modifications

1 participant