[OSDEV-1515] Database. Upgrade the PostgreSQL version to 16.#490
Conversation
…mization/restore setup.
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
React App | Jest test suite - Code coverage reportTotal: 28.37%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
Dedupe Hub App | Unittest test suite - Code coverage reportTotal: 56.14%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
Contricleaner App | Unittest test suite - Code coverage reportTotal: 98.91%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
Countries App | Unittest test suite - Code coverage reportTotal: 100%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
Django App | Unittest test suite - Code coverage reportTotal: 80.38%Your code coverage diff: 0.00% ▴ ✅ All code changes are covered |
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
vladsha-dev
temporarily deployed
to
Quality Environment
GitHub Actions
Inactive
📝 WalkthroughWalkthroughThis pull request involves a comprehensive upgrade of PostgreSQL from version 13 to version 16 across multiple environments and components. The changes span Terraform configuration files, Dockerfiles, migration scripts, and SQL scripts. The upgrade affects development, pre-production, production, staging, and test environments, ensuring consistent database engine versioning. The modifications include updating RDS engine versions, parameter group families, base images, and adding migration scripts to handle the version transition smoothly. Changes
Sequence DiagramsequenceDiagram
participant DB as PostgreSQL Database
participant Migrator as Database Migrator
participant Extensions as PostgreSQL Extensions
DB->>Migrator: Initiate Version Upgrade
Migrator->>Extensions: Check Current Versions
Extensions-->>Migrator: Return Current Versions
Migrator->>Extensions: Upgrade pg_trgm to 1.6
Migrator->>DB: Refresh pg_statistic
Migrator->>DB: Analyze Database
Possibly related PRs
Suggested reviewers
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🧰 Additional context used🪛 markdownlint-cli2 (0.17.2)doc/release/RELEASE-NOTES.md15-15: Trailing punctuation in heading (MD026, no-trailing-punctuation) 42-42: Unordered list indentation (MD007, ul-indent) 43-43: Unordered list indentation (MD007, ul-indent) 44-44: Unordered list indentation (MD007, ul-indent) ⏰ Context from checks skipped due to timeout of 90000ms (13)
🔇 Additional comments (1)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
deployment/environments/terraform-preprod.tfvars (1)
Line range hint
21-22: Consider enabling Multi-AZ deployment for high availability during upgrade.The
rds_multi_az = falsesetting could impact availability during the major version upgrade.Consider enabling Multi-AZ deployment to minimize downtime:
-rds_multi_az = false +rds_multi_az = truesrc/django/sqls/0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.sql (1)
3-30: Consider adding database version logging for better traceability.The extension upgrade logic is robust, but it would be helpful to log the current PostgreSQL version for debugging purposes.
Add version logging at the beginning of the script:
ANALYZE VERBOSE; + +DO $$ +BEGIN + RAISE NOTICE 'Current PostgreSQL version: %', current_setting('server_version'); +END $$; DO $$ DECLARE
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (17)
deployment/environments/terraform-development.tfvars(1 hunks)deployment/environments/terraform-preprod.tfvars(1 hunks)deployment/environments/terraform-production.tfvars(1 hunks)deployment/environments/terraform-staging.tfvars(1 hunks)deployment/environments/terraform-test.tfvars(1 hunks)deployment/terraform/anonymized_database_dump_scheduled_task/docker/Dockerfile(1 hunks)deployment/terraform/database.tf(1 hunks)deployment/terraform/variables.tf(1 hunks)doc/release/RELEASE-NOTES.md(2 hunks)src/anon-tools/Dockerfile.dump(1 hunks)src/anon-tools/Dockerfile.restore(1 hunks)src/database/Dockerfile.local(2 hunks)src/django/Dockerfile(2 hunks)src/django/api/migrations/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py(1 hunks)src/django/api/migrations/0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.py(1 hunks)src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql(1 hunks)src/django/sqls/0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.sql(1 hunks)
✅ Files skipped from review due to trivial changes (4)
- src/anon-tools/Dockerfile.dump
- src/django/api/migrations/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py
- deployment/terraform/database.tf
- src/anon-tools/Dockerfile.restore
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
doc/release/RELEASE-NOTES.md
15-15: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
42-42: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
43-43: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
44-44: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
⏰ Context from checks skipped due to timeout of 90000ms (13)
- GitHub Check: run-integration-test-code-quality
- GitHub Check: run-flake8-linter
- GitHub Check: run-fe-code-quality
- GitHub Check: run-eslint-linter-and-prettier-formatter
- GitHub Check: run-django-code-quality
- GitHub Check: run-dd-code-quality
- GitHub Check: run-countries-code-quality
- GitHub Check: run-contricleaner-code-quality
- GitHub Check: get-base-branch-fe-cov
- GitHub Check: get-base-branch-countries-cov
- GitHub Check: get-base-branch-dd-cov
- GitHub Check: get-base-branch-contricleaner-cov
- GitHub Check: get-base-branch-django-cov
🔇 Additional comments (18)
deployment/environments/terraform-staging.tfvars (1)
15-16: LGTM! Proper configuration for major version upgrade.The configuration includes all necessary flags and settings for a safe PostgreSQL upgrade.
Also applies to: 23-24
deployment/environments/terraform-development.tfvars (1)
17-18: LGTM! Development environment properly configured.The configuration includes all necessary flags and appropriate instance type for development.
Also applies to: 24-25
deployment/environments/terraform-production.tfvars (1)
Line range hint
16-24: Verify PostgreSQL 16 upgrade prerequisites.Ensure all prerequisites for PostgreSQL 16 upgrade are met:
- Check for any incompatible extensions
- Verify application compatibility with PostgreSQL 16
- Test the upgrade process in lower environments first
Run this script to check for potential compatibility issues:
✅ Verification successful
PostgreSQL 16 upgrade prerequisites verified successfully
The codebase shows proper preparation for PostgreSQL 16:
- Django 3.2 LTS is compatible with PostgreSQL 16
- Migration files handle PostGIS and pg_trgm extension upgrades
- Database adapter (psycopg2 2.9.4) supports PostgreSQL 16
- No incompatible syntax or features detected
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Check for PostgreSQL version compatibility issues # Check for installed extensions and their versions rg -A 5 "CREATE EXTENSION|ALTER EXTENSION" # Look for potential PostgreSQL version-specific syntax rg -g '*.sql' -g '*.py' '\b(?:IS NOT DISTINCT FROM|IS DISTINCT FROM|jsonb_path_query|jsonb_path_exists)\b' # Check for custom PostgreSQL functions that might need updates ast-grep --pattern 'CREATE [OR REPLACE] FUNCTION'Length of output: 3350
Script:
#!/bin/bash # Check Django version and its PostgreSQL 16 compatibility rg -g 'requirements*.txt' -g 'setup.py' -g 'pyproject.toml' "Django==" -A 1Length of output: 71
Script:
#!/bin/bash # Try broader search patterns for requirements and DB config echo "=== Checking requirements files ===" fd requirements.txt cat $(fd requirements.txt) echo -e "\n=== Checking database settings ===" rg -g '*.py' "DATABASES" -A 5Length of output: 2585
deployment/environments/terraform-test.tfvars (1)
17-18: LGTM! Version upgrade configuration looks correct.The RDS engine version and parameter group family are properly updated to PostgreSQL 16. The configuration includes
rds_allow_major_version_upgrade = truewhich is required for this upgrade.src/django/api/migrations/0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.py (2)
10-15: LGTM! Migration function is well structured.The function correctly uses the helper to execute the SQL file for refreshing statistics and upgrading extensions.
18-43: Excellent documentation of the migration purpose and context.The docstring provides comprehensive information about:
- The need to refresh statistics after major version upgrade
- Current extensions in use
- Version compatibility with AWS RDS
deployment/terraform/anonymized_database_dump_scheduled_task/docker/Dockerfile (1)
1-1: LGTM! PostGIS base image version aligns with PostgreSQL upgrade.The base image is correctly updated to PostgreSQL 16 while maintaining the same PostGIS version (3.4) for compatibility.
src/django/sqls/0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.sql (2)
1-2: LGTM! Database statistics refresh is properly implemented.The
ANALYZE VERBOSEcommand will rebuild statistics crucial for query optimization after the major version upgrade.
20-29: Verify pg_trgm version compatibility across environments.While version 1.6 is specified as compatible with PostgreSQL 16.3 in AWS RDS, ensure this version is available in all environments.
src/django/Dockerfile (1)
1-1: Verify PostgreSQL client compatibility with PostgreSQL 16.The PostgreSQL client version 15 should be verified for compatibility with PostgreSQL 16 server. While PostgreSQL generally maintains backward compatibility for clients, it's good practice to confirm there are no known issues.
Also applies to: 15-15
src/database/Dockerfile.local (2)
1-8: LGTM! Clear documentation of version constraints.The comments effectively explain why only the major version 16 is specified in the Docker image, despite using 16.3 in AWS RDS.
32-32: LGTM! Development package version aligned.The PostgreSQL server development package version is correctly aligned with the base image version.
src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql (3)
3-13: LGTM! Well-structured version management.The script properly declares and initializes version variables for both extensions, making it maintainable for future version updates.
24-32: LGTM! Robust error handling for postgis extension.The script includes proper error handling with informative messages:
- Checks if extension is installed
- Verifies version before upgrade
- Provides status notifications
44-53: LGTM! Consistent error handling for pg_trgm extension.The error handling pattern is consistently applied to the pg_trgm extension, maintaining code quality.
deployment/terraform/variables.tf (1)
73-73: LGTM! PostgreSQL version configuration updated.The RDS engine version and parameter group family are correctly updated to PostgreSQL 16.
Also applies to: 77-77
doc/release/RELEASE-NOTES.md (2)
13-17: LGTM! Clear documentation of database changes.The release notes effectively document the PostgreSQL upgrade and extension updates, with proper references to AWS documentation.
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
15-15: Trailing punctuation in heading
Punctuation: ':'(MD026, no-trailing-punctuation)
41-44: LGTM! Comprehensive release instructions.The release instructions are thorough, including:
- Pre and post-upgrade steps
- Links to detailed instructions
- Rollback procedures
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
42-42: Unordered list indentation
Expected: 2; Actual: 4(MD007, ul-indent)
43-43: Unordered list indentation
Expected: 2; Actual: 4(MD007, ul-indent)
44-44: Unordered list indentation
Expected: 2; Actual: 4(MD007, ul-indent)
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
doc/release/RELEASE-NOTES.md (2)
13-14: Enhance the PostgreSQL upgrade documentation.The description could be more specific about the upgrade process and its impact. Consider adding:
- The exact PostgreSQL versions (13.x to 16.x)
- Any downtime expectations during the upgrade
- Impact on existing extensions and configurations
* [OSDEV-1515](https://opensupplyhub.atlassian.net/browse/OSDEV-1515) - Upgraded the PostgreSQL version from 13 to 16 for the database used in local development, DB anonymization, DB restore setup, and environments in the AWS cloud. Additionally, the pg_trgm extension has been upgraded to version 1.6 based on the available extension version for PostgreSQL 16.3 in AWS RDS. For more information, see [Extensions supported for RDS for PostgreSQL 16](https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-extensions.html#postgresql-extensions-16x). +* [OSDEV-1515](https://opensupplyhub.atlassian.net/browse/OSDEV-1515) - Upgraded the PostgreSQL version from 13.x to 16.3 for the database used in local development, DB anonymization, DB restore setup, and environments in the AWS cloud. This upgrade requires a maintenance window and may result in brief downtime. The pg_trgm extension has been upgraded to version 1.6 to maintain compatibility with PostgreSQL 16.3 in AWS RDS. All existing extensions and configurations have been validated for compatibility. For more information, see [Extensions supported for RDS for PostgreSQL 16](https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-extensions.html#postgresql-extensions-16x).
16-17: Enhance migration script descriptions.The migration script descriptions could be more detailed about their purpose and impact.
-* 0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py - Updated the SQL script within the migration that upgrades the DB extension versions to handle previously failure cases when a higher version is available for upgrade or when the extension is not installed. This is primarily useful for local development or DB resets in the Development environment, where migrations are applied from scratch, one by one. This fix will not negatively affect other environments, as the migration has already been applied and will not be reapplied. Additionally, the changes are backward compatible. -* 0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.py - This migration refreshes the `pg_statistic` table after the upgrade to PostgreSQL 16 and upgrades the pg_trgm extension to version 1.6. The SQL script within the migration that upgrades the DB extension versions handles previously failure cases where a higher version is available for upgrade or where the extension is not installed. +* 0163_refresh_pg_statistic_and_upgrade_postgres_extensions.py + - Purpose: Handles extension version upgrades and installations + - Key Changes: + * Implements robust error handling for extension upgrades + * Handles cases where extensions are not installed + * Ensures backward compatibility with existing data + - Impact: Safe to run in all environments as it's idempotent + +* 0164_refresh_pg_statistic_and_upgrade_postgres_extensions_after_db_upgrade_to_postgres_16.py + - Purpose: Post-PostgreSQL 16 upgrade maintenance + - Key Changes: + * Refreshes pg_statistic table to ensure query optimizer has current statistics + * Upgrades pg_trgm extension to version 1.6 + * Implements same robust error handling as 0163 + - Execution: Must be run after PostgreSQL 16 upgrade is complete
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
doc/release/RELEASE-NOTES.md(2 hunks)
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
doc/release/RELEASE-NOTES.md
15-15: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
42-42: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
43-43: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
44-44: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
⏰ Context from checks skipped due to timeout of 90000ms (13)
- GitHub Check: run-integration-test-code-quality
- GitHub Check: run-flake8-linter
- GitHub Check: run-eslint-linter-and-prettier-formatter
- GitHub Check: run-django-code-quality
- GitHub Check: run-dd-code-quality
- GitHub Check: run-countries-code-quality
- GitHub Check: run-contricleaner-code-quality
- GitHub Check: get-base-branch-django-cov
- GitHub Check: get-base-branch-dd-cov
- GitHub Check: get-base-branch-countries-cov
- GitHub Check: get-base-branch-contricleaner-cov
- GitHub Check: get-base-branch-fe-cov
- GitHub Check: run-fe-code-quality
src/django/sqls/0163_refresh_pg_statistic_and_upgrade_postgres_extensions.sql
Show resolved
Hide resolved
|
3 participants
[OSDEV-1515] Upgraded PostgreSQL version to 16.
pg_statistictable after the upgrade to PostgreSQL 16 and upgrades the pg_trgm extension to version 1.6. The SQL script within the migration that upgrades the DB extension versions handles previously failure cases where a higher version is available for upgrade or where the extension is not installed.