[OSDEV-1514] Database. Upgrade the PostgreSQL version to 13#479
Conversation
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
had a problem deploying
to
Quality Environment
GitHub Actions
Failure
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
React App | Jest test suite - Code coverage reportTotal: 28.31%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
Dedupe Hub App | Unittest test suite - Code coverage reportTotal: 56.14%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
Contricleaner App | Unittest test suite - Code coverage reportTotal: 98.91%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
Countries App | Unittest test suite - Code coverage reportTotal: 100%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
…jor version upgrade and change the reference to TF RDS module to the WIP one
…jor version upgrade
…ule and increase the Postgres version 13 back
…repo to the latest one.
📝 WalkthroughWalkthroughThis pull request encompasses a comprehensive upgrade of the PostgreSQL database from version 12 to version 13 across multiple environments and components. The changes include updating Dockerfiles, Terraform configuration files, migration scripts, and deployment workflows. The upgrade involves modifying base images, extension versions, and introducing new parameters to control version upgrades and immediate application of changes. Changes
Sequence DiagramsequenceDiagram
participant Deployment as Deployment Workflow
participant Terraform as Terraform Config
participant Database as PostgreSQL Database
participant Extensions as Database Extensions
Deployment->>Terraform: Update RDS configuration
Terraform->>Database: Upgrade PostgreSQL version
Database->>Extensions: Upgrade PostGIS
Database->>Extensions: Upgrade pg_trgm
Database->>Database: Refresh statistics
Possibly related PRs
Suggested reviewers
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms (13)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
src/django/api/migrations/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py (2)
10-14: Consider simplifying the function name for readability.The function name
refresh_pg_statistic_and_perform_upgrading_pg_extensionsis quite long and may impact readability. Consider shortening it to something likeupgrade_pg_extensionsorrefresh_stats_and_upgrade_extensionsfor clarity.
18-43: Use triple-double quotes for the docstring to follow PEP 257 conventions.PEP 257 recommends using triple-double quotes (
""") for docstrings. This helps maintain consistency across the codebase.doc/release/RELEASE-NOTES.md (2)
51-54: Consider enhancing the release instructions.While the release instructions reference the Confluence article well, consider adding:
- A brief summary of the most critical pre and post-upgrade steps directly in the release notes
- Key warning signs to watch for during the upgrade
🧰 Tools
🪛 Markdownlint (0.37.0)
52-52: Expected: 2; Actual: 4
Unordered list indentation(MD007, ul-indent)
53-53: Expected: 2; Actual: 4
Unordered list indentation(MD007, ul-indent)
54-54: Expected: 2; Actual: 4
Unordered list indentation(MD007, ul-indent)
Line range hint
15-54: Fix markdown formatting issues.There are some minor markdown formatting inconsistencies:
- Remove trailing punctuation from headings
- Fix unordered list indentation (should be 2 spaces instead of 4)
🧰 Tools
🪛 Markdownlint (0.37.0)
15-15: Punctuation: ':'
Trailing punctuation in heading(MD026, no-trailing-punctuation)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (21)
.github/workflows/deploy_to_aws.yml(1 hunks)deployment/environments/terraform-development.tfvars(1 hunks)deployment/environments/terraform-preprod.tfvars(1 hunks)deployment/environments/terraform-production.tfvars(1 hunks)deployment/environments/terraform-staging.tfvars(1 hunks)deployment/environments/terraform-test.tfvars(1 hunks)deployment/terraform/anonymize_db_job.tf(1 hunks)deployment/terraform/anonymized_database_dump_scheduled_task/docker/Dockerfile(1 hunks)deployment/terraform/database.tf(1 hunks)deployment/terraform/database_anonymizer_scheduled_task/docker/database_anonymizer.py(0 hunks)deployment/terraform/variables.tf(2 hunks)doc/release/RELEASE-NOTES-TEMPLATE.md(1 hunks)doc/release/RELEASE-NOTES.md(16 hunks)src/anon-tools/Dockerfile.dump(2 hunks)src/anon-tools/Dockerfile.restore(1 hunks)src/anon-tools/anon.sql(0 hunks)src/anon-tools/do_dump.sh(1 hunks)src/anon-tools/initdb.sql(0 hunks)src/database/Dockerfile.local(2 hunks)src/django/api/migrations/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py(1 hunks)src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql(1 hunks)
💤 Files with no reviewable changes (3)
- deployment/terraform/database_anonymizer_scheduled_task/docker/database_anonymizer.py
- src/anon-tools/anon.sql
- src/anon-tools/initdb.sql
✅ Files skipped from review due to trivial changes (4)
- deployment/terraform/anonymize_db_job.tf
- deployment/terraform/anonymized_database_dump_scheduled_task/docker/Dockerfile
- .github/workflows/deploy_to_aws.yml
- src/anon-tools/do_dump.sh
🧰 Additional context used
🪛 Markdownlint (0.37.0)
doc/release/RELEASE-NOTES-TEMPLATE.md
12-12: Punctuation: ':'
Trailing punctuation in heading
(MD026, no-trailing-punctuation)
doc/release/RELEASE-NOTES.md
15-15: Punctuation: ':'
Trailing punctuation in heading
(MD026, no-trailing-punctuation)
52-52: Expected: 2; Actual: 4
Unordered list indentation
(MD007, ul-indent)
53-53: Expected: 2; Actual: 4
Unordered list indentation
(MD007, ul-indent)
54-54: Expected: 2; Actual: 4
Unordered list indentation
(MD007, ul-indent)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: run-integration-test-code-quality
- GitHub Check: run-django-code-quality
- GitHub Check: get-base-branch-django-cov
🔇 Additional comments (20)
src/django/api/migrations/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py (1)
1-51: LGTM!The migration script correctly executes the required SQL commands to refresh
pg_statisticand upgrade PostgreSQL extensions. It follows Django's migration patterns and properly sets dependencies.deployment/environments/terraform-preprod.tfvars (1)
17-18: Confirming the PostgreSQL version upgrade settings.The
rds_engine_versionandrds_parameter_group_familyhave been correctly updated to reflect the upgrade to PostgreSQL 13.deployment/environments/terraform-staging.tfvars (2)
22-23: Verify the impact ofapply_immediatelyand major version upgrade settings.Setting
rds_apply_immediately = truewill apply changes immediately upon deployment, which may cause downtime. Ensure that this is acceptable for the staging environment and align with your deployment strategy.
15-16: Confirming the PostgreSQL version upgrade settings.The
rds_engine_versionandrds_parameter_group_familyhave been correctly updated to reflect the upgrade to PostgreSQL 13.deployment/environments/terraform-development.tfvars (1)
17-18: LGTM! RDS configuration for PostgreSQL 13 upgrade looks correct.The configuration properly sets up PostgreSQL 13 upgrade with appropriate parameters:
- Engine version and parameter group family are aligned
- Major version upgrade is enabled
- Changes will be applied immediately
Also applies to: 24-25
deployment/environments/terraform-production.tfvars (2)
16-17: LGTM! RDS configuration for PostgreSQL 13 upgrade looks correct.The configuration properly sets up PostgreSQL 13 upgrade with appropriate parameters for production.
Also applies to: 23-24
21-21: Verify if Multi-AZ should be enabled for production.The
rds_multi_az = falsesetting might pose a risk to high availability in production. Consider enabling Multi-AZ for better fault tolerance.deployment/environments/terraform-test.tfvars (1)
17-18: LGTM! RDS configuration for PostgreSQL 13 upgrade looks correct.The configuration properly sets up PostgreSQL 13 upgrade with appropriate parameters for the test environment.
Also applies to: 24-25
src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql (2)
1-2: LGTM! Database statistics refresh is appropriate after version upgrade.The
ANALYZE VERBOSEcommand will help optimize query performance by updating statistics after the major version upgrade.
3-4: Verify PostgreSQL 13 extension version compatibility.Let's verify that the specified extension versions are compatible with PostgreSQL 13 in AWS RDS:
src/anon-tools/Dockerfile.restore (1)
1-1: Base image update looks good!The upgrade to
postgis/postgis:13-3.4-alpinealigns with the PostgreSQL 13 upgrade objective.Let's verify the image availability and check for any security advisories:
✅ Verification successful
✅ Verified: postgis/postgis:13-3.4-alpine is valid and secure
- Image is actively maintained (last update: Oct 14, 2024)
- No vulnerabilities reported
- Consistent with PostgreSQL 13 usage across 5 other Dockerfiles in the codebase
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify the postgis image exists and check for vulnerabilities docker manifest inspect postgis/postgis:13-3.4-alpine 2>/dev/null || echo "Image not found" # Check for CVEs in PostgreSQL 13 gh api graphql -f query=' { securityVulnerabilities(first: 5, ecosystem: DOCKER, package: "postgres") { nodes { advisory { summary severity publishedAt } vulnerableVersionRange firstPatchedVersion { identifier } } } }'Length of output: 871
Script:
#!/bin/bash # Check image availability on Docker Hub curl -s "https://hub.docker.com/v2/repositories/postgis/postgis/tags/?name=13-3.4-alpine" | jq . # Check for PostgreSQL security advisories with correct ecosystem gh api graphql -f query=' { securityVulnerabilities(first: 5, ecosystem: CONTAINER, package: "postgres") { nodes { advisory { summary severity publishedAt } vulnerableVersionRange firstPatchedVersion { identifier } } } }' # Search for any other PostgreSQL 13 related images in the codebase rg -l "postgres.*13|postgis.*13" --type dockerfileLength of output: 2164
Script:
#!/bin/bash # Search for other PostgreSQL 13 related images with correct parameters rg -l "postgres.*13|postgis.*13" -g "Dockerfile*" # Alternative approach to check security advisories curl -s "https://hub.docker.com/v2/repositories/postgis/postgis/tags/?name=13-3.4-alpine" | \ jq -r '.results[].images[] | select(.architecture=="amd64") | .digest' | \ xargs -I {} curl -s "https://hub.docker.com/v2/repositories/postgis/postgis/tags/?name=13-3.4-alpine/vulnerabilities" || \ echo "No vulnerability data available"Length of output: 587
src/anon-tools/Dockerfile.dump (2)
1-1: Base image update looks good!The upgrade to
postgis/postgis:13-3.4-alpinealigns with the PostgreSQL 13 upgrade objective.
36-37: Verify the impact of removed initdb.sql and ENTRYPOINT changes.The switch from ENTRYPOINT to CMD and removal of initdb.sql might affect database initialization. Please ensure:
- Required PostgreSQL extensions are initialized elsewhere
- The container startup behavior remains correct
Let's check for any extension initialization code in other files:
✅ Verification successful
Changes to container initialization are verified and safe ✅
The switch from ENTRYPOINT to CMD and removal of initdb.sql are appropriate because:
- PostgreSQL and PostGIS initialization is handled by the base image (postgis/postgis:13-3.4-alpine)
- Database startup is properly managed in do_dump.sh using docker-entrypoint.sh
- No custom extensions were required in initdb.sql
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Search for PostgreSQL extension creation commands rg -l 'CREATE EXTENSION' --type sql # Search for other initialization scripts fd -e sql -e sh -p 'init|setup'Length of output: 71
Script:
#!/bin/bash # Find do_dump.sh and examine its contents fd -t f "do_dump.sh" --exec cat {} # Look for Docker-related files in the anon-tools directory fd -p "Dockerfile|docker-compose" src/anon-tools/ --exec cat {} # Search for PostgreSQL related configuration rg -l "postgresql|postgres|pg_" src/anon-tools/Length of output: 5439
doc/release/RELEASE-NOTES-TEMPLATE.md (1)
9-11: Documentation improvements look good!The addition of the "Database changes" section and the correction of "Schema changes" improve the template's clarity and completeness.
Also applies to: 15-16
src/database/Dockerfile.local (2)
1-8: Excellent documentation of version constraints!The comments clearly explain the rationale behind using only the major version in the Docker image while targeting PostgreSQL 13.15 in AWS RDS.
34-42: PostGIS upgrade looks good, but verify compatibility.The upgrade to PostGIS 3.4.2 aligns with the PostgreSQL 13 upgrade.
Let's verify PostGIS compatibility:
✅ Verification successful
PostGIS 3.4.2 compatibility verified successfully
The upgrade to PostGIS 3.4.2 is explicitly verified to be compatible with PostgreSQL 13.15 in AWS RDS, as documented in the project. The version choice is deliberate and consistent across all environments.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check PostGIS compatibility matrix curl -s https://postgis.net/documentation/release-notes/ | grep -A 5 "PostGIS 3.4.2" # Verify if any other files reference PostGIS version rg "postgis-3\.4\.[0-9]" --type=DockerfileLength of output: 166
Script:
#!/bin/bash # Find all Dockerfiles and search for PostGIS/PostgreSQL versions fd Dockerfile --exec grep -i "postgis\|postgres" {} \; # Look for version information in documentation and config files fd -e md -e yaml -e yml -e conf -e env --exec grep -i "postgis\|postgres" {} \; # Search for database setup or migration files fd -e sql --exec grep -i "postgis\|postgres" {} \;Length of output: 8798
deployment/terraform/database.tf (1)
79-80: Verify compatibility with the new module version.The upgrade to version 3.1.0 of the terraform-aws-postgresql-rds module needs verification.
Let's check for breaking changes:
✅ Verification successful
Module version 3.1.0 is compatible with the changes
The module update specifically introduces the
allow_major_version_upgradeandapply_immediatelyvariables that are being utilized in this PR, along with PostgreSQL 13 support. No breaking changes are present.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check module release notes or changelog curl -s -H "Accept: application/vnd.github.v3+json" \ "https://api.github.com/repos/opensupplyhub/terraform-aws-postgresql-rds/releases/tags/3.1.0"Length of output: 2616
deployment/terraform/variables.tf (2)
73-77: PostgreSQL version upgrade changes look good!The PostgreSQL version has been correctly updated from 12 to 13 in both the engine version and parameter group family variables.
117-127: Well-structured control variables for RDS upgrade!The new variables provide good control over the major version upgrade process:
rds_allow_major_version_upgradeto control major version upgradesrds_apply_immediatelyto control when changes are applieddoc/release/RELEASE-NOTES.md (1)
13-14: Documentation for PostgreSQL upgrade is comprehensive!The release notes effectively document:
- The version upgrade from PostgreSQL 12 to 13
- The scope of environments affected
- The extension upgrades (postgis and pg_trgm)
It's me. Forget change account |
src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql
Outdated
Show resolved
Hide resolved
|
4 participants
[OSDEV-1514]
apply immediatelyflag to perform the PostgreSQL major version upgrade in AWS.src/anon-tools/do_dump.shfile and in the name of the folderdatabase_anonymizer_sheduled_task. Removed the unusedsrc/anon-tools/anon.sqlfile and the redundantsrc/anon-tools/initdb.sqlfile. Removed commented-out code in thesrc/anon-tools/Dockerfile.dumpanddeployment/terraform/database_anonymizer_scheduled_task/docker/database_anonymizer.pyfiles.rds_allow_major_version_upgradeandrds_apply_immediatelyTerraform variables to enable or disable major version upgrades and theapply immediatelyflag, depending on the environment.