Cache test database schema in CI to reduce compute time#37433
Draft
dannyroberts wants to merge 8 commits intomasterfrom
Draft
Cache test database schema in CI to reduce compute time#37433dannyroberts wants to merge 8 commits intomasterfrom
dannyroberts wants to merge 8 commits intomasterfrom
Conversation
Use GitHub Actions cache to store a pg_dump of the test database schema. On cache hits, restore the dump (~seconds) instead of running the full 200s setup_databases() call, saving ~600s of total CI compute per run across the 3 non-sharded Python test shards. Cache key is based on migration file hashes, so it invalidates whenever the schema changes. On restore failure, falls back to normal setup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dannyroberts
commented
Feb 24, 2026
| docker compose exec -T postgres pg_dump -U commcarehq -Fc test_commcarehq > "$DB_DUMP" 2>/dev/null || rm -f "$DB_DUMP" | ||
| fi | ||
|
|
||
| exit $TEST_EXIT |
Member
Author
There was a problem hiding this comment.
I don't love how much boilerplate we need in order to do this, but first I just want to see if it'll even save any time at all. If it does, then I'll think about how to make this cleaner / more concise
The first run failed to cache because docker compose exec couldn't reach the postgres container after the web container exited. Ensure postgres is explicitly running before the dump, and show errors instead of suppressing them for easier debugging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Docker container changes artifacts/ ownership to cchq:cchq via docker/run.sh, making it unwritable by the host runner user. Use sudo chown to reclaim ownership before writing the dump file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 4 matrix jobs were using the same artifact name 'test-artifacts', causing 409 Conflict errors. This also prevented the Post Cache step from running (it skips when the job fails), so the DB dump was never cached. Include the matrix values in the artifact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The artifacts/ directory has its ownership changed to cchq:cchq by the Docker container, causing permission denied when writing the dump. It also caused artifact upload conflicts since all shards share the same artifact name. Use .test-db-cache/ instead, which the container never touches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Docker container changes ownership of the repo checkout directory, making it impossible to create new files/dirs in it from the host. Use /tmp which is always writable and outside the container's reach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When the SQL database is restored from cache, sql_databases_ok() returns True and the reusedb fast path skips ALL database setup including CouchDB. But CouchDB is ephemeral in CI and needs setup. Add FORCE_DB_SETUP env var that bypasses the fast path, forcing setup_databases() to run with keepdb=True. This is fast for SQL (tables already exist from restore) while still setting up CouchDB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Product Description
N/A — CI-only change, no user-facing effects.
Technical Summary
Uses GitHub Actions cache to store a
pg_dumpof the test database schema. On cache hits (~every run without migration changes), restores the dump in seconds instead of running the full ~200ssetup_databases()call via Django test framework.How it works:
actions/cache@v4step cachesartifacts/test_db.dump, keyed onhashFiles('**/migrations/**/*.py')Scope: 3 non-sharded
pythontest shards (05,6a,bf). Thepython-sharded-and-javascriptshard is excluded (different DB config, only 1 job).Expected savings: ~600s total CI compute per run (3 shards × 200s) on cache hits, with no wall-clock impact.
Feature Flag
N/A
Safety Assurance
Safety story
--reusedb=1fast path inreusedb.pyalready handles the "DB already exists" caseAutomated test coverage
The CI tests themselves serve as validation — if the restored schema is wrong, tests will fail.
QA Plan
--durationsoutput shows no ~200s setup timepython-sharded-and-javascriptshard is unaffectedRollback instructions
Labels & Review
🤖 Generated with Claude Code