Auto-build SWE-Bench Images and Push to GHCR

[xingyaoww]

We should create a github workflow in this repo that:

1. setup evaluation benchmarks
2. follow https://docs.blacksmith.sh/blacksmith-caching/docker-builds, automatically build docker images, and push to ghcr.io/openhands/eval-agent-server
3. use benchmarks/swe_bench/build_images.py to build images, but we need to modify it so we can push stuff to ghcr
4. this workflow should be manually triggered since they are super expensive to run

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully implemented a GitHub workflow to auto-build SWE-Bench images and push them to GHCR, addressing issue #37. Here's what was accomplished:

✅ Changes Made

1. Created .github/workflows/build-swe-bench-images.yml

• Manually triggered workflow using workflow_dispatch (as requested since builds are expensive)
• Integrated Blacksmith caching following official documentation
• Uses blacksmith-32vcpu-ubuntu-2204 runner for high-performance builds
• Configured Docker layer caching with useblacksmith/cache@v6
• Authenticates and pushes to ghcr.io/openhands/eval-agent-server
• Supports configurable parameters: dataset, split, target, platforms, max-workers, n-limit
• Uploads build manifests and logs as artifacts for debugging

2. Modified benchmarks/swe_bench/build_images.py

• Made the --critic parameter optional (defaults to "none") since it's not needed for image building
• This allows the script to work seamlessly with the workflow without requiring evaluation-specific parameters

3. Fixed .gitignore

• Corrected patterns from ./eval_outputs and ./builds to eval_outputs/ and builds/ for proper exclusion

✅ Requirements Checklist

• ✅ Setup evaluation benchmarks (via uv sync --dev)
• ✅ Follow Blacksmith caching documentation
• ✅ Automatically build Docker images using benchmarks/swe_bench/build_images.py
• ✅ Push to ghcr.io/openhands/eval-agent-server
• ✅ Manually triggered workflow (expensive to run)

📦 Pull Request

Draft PR created: #51

The PR is marked as draft and ready for review. All changes have been committed to the openhands/build-swe-bench-images-workflow branch and pushed to GitHub.

To trigger the workflow:

1. Go to Actions tab → "Build SWE-Bench Images"
2. Click "Run workflow"
3. Configure parameters as needed
4. Start the build

_{View full conversation}