-
Notifications
You must be signed in to change notification settings - Fork 84
build: added environment.lock for 'stable' builds #75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
247d052
Update production.lock
jandom 93b4dee
feat: use .lock files instead of environment.yml which may drift
jandom 3e4c78b
use a dedicated conda env and auto-activate it
jandom 13b0707
move to "deny-all, allow-list" in dockerignore
jandom 39bdb82
test: with the environment.lock we can take a fresher version of mini…
jandom f303e56
support building both from .lock and .yaml
jandom a8f6f5d
change default to 'yaml'
jandom 6e790ce
review comments: make the platform explicit
jandom File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| # Full performance multi-stage build with complete CUDA toolchain | ||
| ARG CUDA_BASE_IMAGE_TAG=12.2.2-cudnn8-devel-ubuntu22.04 | ||
| ARG CUDA_BASE_IMAGE_TAG=12.1.1-cudnn8-devel-ubuntu22.04 | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is unrelated and snuck in via #70, turning back to the old version we used |
||
| FROM nvidia/cuda:${CUDA_BASE_IMAGE_TAG} AS builder | ||
|
|
||
| # Install complete build dependencies including CUDA compiler tools | ||
|
|
@@ -23,9 +23,10 @@ RUN wget -P /tmp \ | |
| ENV PATH=/opt/conda/bin:$PATH | ||
| ENV CONDA_PREFIX=/opt/conda | ||
|
|
||
| # Copy and install dependencies with aggressive cleanup | ||
| COPY environments/production.yml /opt/openfold3/environment.yml | ||
| RUN mamba env update -n base --file /opt/openfold3/environment.yml \ | ||
| # Copy and install dependencies from lock file for reproducible builds | ||
| # To regenerate the lock file, see docker/DOCKER.md | ||
| COPY environments/production.lock /opt/openfold3/production.lock | ||
| RUN mamba install -n base --file /opt/openfold3/production.lock \ | ||
| && mamba clean --all --yes \ | ||
| && conda clean --all --yes | ||
|
|
||
|
|
@@ -52,7 +53,7 @@ ENV TORCH_CUDA_ARCH_LIST="8.0;8.6;9.0" | |
| # python3 -c "import deepspeed; print('DeepSpeed ops loaded successfully')" | ||
|
|
||
| # Devel stage - use devel image for full CUDA support | ||
| ARG CUDA_BASE_IMAGE_TAG=12.2.2-cudnn8-devel-ubuntu22.04 | ||
| ARG CUDA_BASE_IMAGE_TAG=12.1.1-cudnn8-devel-ubuntu22.04 | ||
| FROM nvidia/cuda:${CUDA_BASE_IMAGE_TAG} AS devel | ||
|
|
||
| # Install devel dependencies | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Dockerfile for generating conda environment lock files | ||
| # This produces a fully-pinned lock file for reproducible builds | ||
| # | ||
| # Usage: | ||
| # docker build -f docker/Dockerfile.update-reqs -t openfold3-update-reqs . | ||
| # docker run --rm openfold3-update-reqs > environments/production.lock | ||
|
|
||
| FROM mambaorg/micromamba:1.5.10 | ||
|
|
||
| USER root | ||
|
|
||
| # Install conda-lock | ||
| RUN micromamba install -y -n base -c conda-forge conda-lock \ | ||
| && micromamba clean --all --yes | ||
|
|
||
| USER $MAMBA_USER | ||
|
|
||
| COPY --chown=$MAMBA_USER:$MAMBA_USER environments/production.yml /tmp/environment.yml | ||
|
|
||
| # Generate explicit lock file for linux-64 | ||
| # The explicit format is directly consumable by mamba/conda | ||
| RUN micromamba run -n base conda-lock lock \ | ||
| --mamba \ | ||
| --platform linux-64 \ | ||
| --file /tmp/environment.yml \ | ||
| --kind explicit \ | ||
| --filename-template '/tmp/production-{platform}.lock' | ||
|
|
||
| # Output the lock file to stdout when container runs | ||
| CMD ["cat", "/tmp/production-linux-64.lock"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add documentation about where the current
production.lockis generated? Specifically, what kind of instance / system, and any other variables that are relevant to environment resolution.For my understanding: Do we expect the
production.lockto change if the system is a GPU / CPU? Or should it be the same because we specify the same docker base image with CUDA?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that's a good point – currently the updating and generation is kind of synonymous (it doesn't say that). And it absolutely is platform specific, as you mentioned in your comments later. Yeah, it's specific to the platform (linux64, arm64, etc). The GPU/CPU point is conditional on what's in the environment.yaml – if that pulls a CPU-version of torch, that's what will be installed in the env.