[OSEDV-814] Upgrade Django and Python

[VadimKovalenkoSNF]

Fix OSDEV-814

1. Upgrade Python: 3.8 → 3.11
2. Upgrade Django: 3.2.17 →5.2.10
3. Upgrade Python and Django packages to conform compatibility

Note: DedupeHub was not affected as it is a separate service.

[barecheck]

React App | Jest test suite - Code coverage report

Total: 40.32%

Your code coverage diff: 0.00% ▴

✅ All code changes are covered

[coderabbitai]

📝 Walkthrough

Walkthrough

Bumps Python and many dependencies; migrates rest_auth → dj_rest_auth and django-ckeditor → django-ckeditor-5 with compatibility shims and migrations; converts Event index declaration to models.Index; standardizes timezone handling to Django default; updates Dockerfile and release notes.

Changes

Cohort / File(s)	Summary
Python & Docker src/django/Dockerfile	Base image bumped to python:3.11-slim-bookworm; removed aiokafka sed/fork install steps.
Dependency manifest src/django/requirements.txt	Large dependency upgrades (Django → 5.2.10, dj-rest-auth, django-ckeditor-5, many others); added kafka-python==2.0.2.
REST auth migration src/django/api/serializers/user/user_password_reset*.py, src/django/api/views/auth/*, src/django/oar/settings.py, src/django/oar/urls.py	Replaced rest_auth imports/includes with dj_rest_auth equivalents and renamed REST_AUTH settings mapping; updated URL includes and routing wiring.
CKEditor migration & compatibility shim src/django/api/admin.py, src/django/api/models/partner_field.py, src/django/ckeditor/*, src/django/api/migrations/0196_switch_partner_field_source_by_editor.py	Switched editor usage to django-ckeditor-5; added src/django/ckeditor shim modules (__init__, fields.py, widgets.py) for legacy imports; migration alters PartnerField.source_by.
Database index change src/django/api/models/event.py, src/django/api/migrations/0197_add_event_index.py	Replaced index_together with explicit models.Index on ['content_type','object_id']; migration drops legacy indexes and creates api_event_content_object_idx.
Timezone handling updates src/django/api/close_list.py, src/django/api/limitation/date/...monthly_date_limitation.py, .../yearly_date_limitation.py, src/django/api/middleware.py, src/django/api/reports.py, src/django/api/serializers/facility/utils.py	Replaced explicit UTC usage with timezone.get_default_timezone(); normalize naive datetimes where needed; added get_start_date() and get_api_block_until() to MonthlyDateLimitation.
Admin & model tweaks src/django/api/admin.py, src/django/api/models/partner_field.py	Admin widget switched to CKEditor5Widget(config_name='default'); model field RichTextField → CKEditor5Field (args adjusted).
Minor logic & docs src/django/api/processing.py, src/django/api/throttles.py, doc/release/RELEASE-NOTES.md	Broadened a type check to isinstance; minor docstring wording change; added Release 2.19.0 notes.
URLs & routing src/django/oar/urls.py	Added ckeditor5 internal URL; switched rest_auth includes to dj_rest_auth; bound ApiBlockViewSet.as_view({'get':'list'}) and renamed internal route.
Migrations src/django/api/migrations/0196_switch_partner_field_source_by_editor.py, src/django/api/migrations/0197_add_event_index.py	Migration to alter PartnerField.source_by to CKEditor5Field; migration to drop legacy Event indexes and add api_event_content_object_idx.

Sequence Diagram(s)

(Skipped)

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• [OSDEV-1785] Link to the forked aiokafka@0.8.0 #534 — Directly related Dockerfile changes and aiokafka install handling.
• [OSDEV-2083] Update python and OpenSearch client versions #704 — Overlapping base-image and dependency/requirements updates.
• [OSDEV-2185] [Climate TRACE] Display Source By Text #792 — Overlapping PartnerField.source_by edits and related migrations/admin changes.

Suggested reviewers

• protsack-stephan
• vlad-shapik

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.75% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: upgrading Django and Python versions as part of OSDEV-814.
Description check	✅ Passed	The description is directly related to the changeset, detailing the specific version upgrades and package compatibility updates.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch OSDEV-814-upgrade-django-python

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/django/api/throttles.py (1)

5-9: Upgrade djangorestframework to 3.16.x or later for Django 5.1.3 compatibility.

The project pins djangorestframework==3.15.2 in src/django/requirements.txt, but Django REST Framework 3.15.x does not have official support for Django 5.1. The package is configured to use Django 5.1.3 with Python 3.11, which requires DRF 3.16.x or later. Update the dependency to a compatible version.

📜 Review details

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5bc888f and 9172af1.

📒 Files selected for processing (14)

• src/django/Dockerfile
• src/django/api/apps.py
• src/django/api/migrations/0190_add_event_index.py
• src/django/api/models/event.py
• src/django/api/processing.py
• src/django/api/serializers/facility/utils.py
• src/django/api/serializers/user/user_password_reset_confirm_serializer.py
• src/django/api/serializers/user/user_password_reset_serializer.py
• src/django/api/throttles.py
• src/django/api/views/auth/login_to_oar_client.py
• src/django/api/views/auth/logout_to_oar_client.py
• src/django/oar/settings.py
• src/django/oar/urls.py
• src/django/requirements.txt

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-08-14T08:13:20.055Z

Learnt from: VadimKovalenkoSNF
Repo: opensupplyhub/open-supply-hub PR: 717
File: src/django/requirements.txt:48-52
Timestamp: 2025-08-14T08:13:20.055Z
Learning: In the Open Supply Hub project, heavy ML/data processing dependencies (splink, geopy, numpy, pandas) are kept in the main requirements.txt file rather than being split into separate worker requirements because the record linkage feature is controlled by a runtime-toggleable Django waffle switch (`enable_record_linkage`) that can be enabled/disabled through Django admin. Separate images would overcomplicate the CI/CD process and could cause runtime failures if the feature is enabled but dependencies are missing.

Applied to files:

• src/django/requirements.txt

🧬 Code graph analysis (1)

src/django/api/processing.py (1)

src/django/api/models/facility/facility_list_item.py (1)

• FacilityListItem (10-195)

🪛 Ruff (0.14.10)

src/django/api/apps.py

13-13: Consider (*tuple(names), 'index_together') instead of concatenation

Replace with (*tuple(names), 'index_together')

(RUF005)

src/django/api/models/event.py

10-15: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

src/django/api/migrations/0190_add_event_index.py

5-7: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

---

9-21: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

src/django/oar/settings.py

27-27: Consider (*options.DEFAULT_NAMES, 'index_together') instead of concatenation

Replace with (*options.DEFAULT_NAMES, 'index_together')

(RUF005)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)

• GitHub Check: run-dd-code-quality
• GitHub Check: run-integration-test-code-quality
• GitHub Check: run-eslint-linter-and-prettier-formatter
• GitHub Check: run-flake8-linter
• GitHub Check: get-base-branch-dd-cov
• GitHub Check: run-django-code-quality
• GitHub Check: get-base-branch-django-cov
• GitHub Check: run-contricleaner-code-quality
• GitHub Check: get-base-branch-fe-cov
• GitHub Check: run-countries-code-quality
• GitHub Check: run-fe-code-quality
• GitHub Check: get-base-branch-countries-cov
• GitHub Check: get-base-branch-contricleaner-cov

🔇 Additional comments (14)

src/django/api/processing.py (1)

81-82: Good refactoring to use isinstance() for type checking.

This change follows Python best practices. The isinstance() check is more Pythonic and flexible than strict type equality, as it properly handles inheritance hierarchies. The codebase contains no subclasses of FacilityListItem, so this is a safe modernization that improves code maintainability.

src/django/api/throttles.py (1)

14-14: Docstring improvement: removes redundancy and clarifies wording.

The updated docstring is clearer, removing the redundant "custom" and simplifying the phrasing. The change improves code clarity without affecting functionality.

src/django/api/views/auth/login_to_oar_client.py (1)

4-4: Import migration looks correct.

The migration from rest_auth.views.LoginView to dj_rest_auth.views.LoginView is appropriate. Since the custom post() method handles authentication independently using Django's authenticate() and login(), this change should be compatible with dj-rest-auth 7.x.

src/django/api/models/event.py (1)

10-15: Good migration from index_together to explicit indexes.

This correctly updates from the deprecated index_together syntax to explicit models.Index(), which is the recommended approach for Django 4.2+ and required for Django 5.x where index_together is fully removed.

src/django/api/serializers/facility/utils.py (1)

3-3: Timezone change is functionally equivalent.

Using datetime.timezone.utc instead of django.utils.timezone.utc works identically for the astimezone() call. Both represent UTC. This is a valid change, though django.utils.timezone remains fully supported in Django 5.x if consistency with other Django code is preferred.

Also applies to: 267-267

src/django/Dockerfile (1)

1-1: Python 3.11 base image upgrade looks good.

The upgrade to python:3.11-slim-bookworm aligns with Django 5.1.3 requirements. The removal of the aiokafka workaround simplifies the build process.

src/django/api/views/auth/logout_to_oar_client.py (1)

3-3: Import migration looks correct.

The migration from rest_auth.views.LogoutView to dj_rest_auth.views.LogoutView is appropriate. The custom post() method uses Django's logout() directly, so it's independent of the parent class's implementation details.

src/django/api/serializers/user/user_password_reset_confirm_serializer.py (1)

1-1: Verify self.user and self.set_password_form attribute availability in dj-rest-auth 7.x.

The import migration is correct, but dj-rest-auth 7.x's PasswordResetConfirmSerializer may have different internal attributes. The custom save() method relies on self.user and self.set_password_form being populated by the parent class's validation. Verify these attributes exist after calling the parent's validation methods.

dj-rest-auth PasswordResetConfirmSerializer user set_password_form attributes

Also applies to: 5-10

src/django/api/serializers/user/user_password_reset_serializer.py (1)

1-1: Verify custom save() method compatibility with dj-rest-auth 7.x.

The import migration to dj-rest-auth is correct, but dj-rest-auth 7.x PasswordResetSerializer.save() requires a specific parameter signature. Ensure the custom save() method at lines 25-47 implements the correct signature and passes compatible parameters to reset_form.save().

src/django/api/migrations/0190_add_event_index.py (1)

1-22: LGTM! Migration correctly transitions to explicit index declaration.

The migration properly uses AlterModelOptions to add an explicit index, which is the modern approach for Django 5.x. This replaces the deprecated index_together Meta option.

Note: The static analysis warnings (RUF012) about ClassVar annotations are false positives—Django's migration framework requires dependencies and operations as standard class attributes.

src/django/oar/settings.py (2)

15-15: Approve compatibility workaround for legacy migrations.

The import and compatibility code correctly addresses Django 5.x's removal of index_together from DEFAULT_NAMES. This allows existing migrations that reference the deprecated index_together Meta option to continue working during the upgrade transition.

Note: The static analysis hint (RUF005) suggesting tuple unpacking is valid but represents a minor style improvement that can be optionally addressed.

Optional style improvement

-    options.DEFAULT_NAMES = options.DEFAULT_NAMES + ('index_together',)
+    options.DEFAULT_NAMES = (*options.DEFAULT_NAMES, 'index_together')

Also applies to: 25-27

---

131-131: LGTM! Migration from django-rest-auth to dj-rest-auth is necessary.

The package swap from rest_auth to dj_rest_auth is required because django-rest-auth is no longer maintained. The dj-rest-auth fork provides Django 5.x compatibility and active maintenance.

Also applies to: 135-135

src/django/oar/urls.py (1)

148-149: LGTM! URL patterns updated for dj-rest-auth migration.

The URL includes are correctly updated from rest_auth to dj_rest_auth, aligning with the changes in settings.py and the overall migration to the maintained authentication package.

src/django/requirements.txt (1)

2-2: Verify Django 5.1.3 compatibility and security advisories for major upgrades.

This PR performs major framework upgrades including Django 3.2.17 → 5.1.3 (skipping 4.x) and implicitly Python 3.8 → 3.11. While the commit history shows an intermediate Django 4.2.6 step, the final jump to 5.1.3 introduces significant changes.

Key concerns:

1. Django 5.1.3 verification: Confirm this version exists and check for security advisories
2. Breaking changes: Django 5.x has numerous breaking changes that must be addressed
3. Python 3.11 compatibility: Verify all dependencies support Python 3.11
4. Security advisories: Check for known vulnerabilities in pinned versions

Note: The PR description appears to have swapped version numbers—it states "Python: 3.2.17 → 5.1.3; Django: 3.8 → 3.11" when it should be "Django: 3.2.17 → 5.1.3; Python: 3.8 → 3.11".

What is the latest stable version of Django 5.1 and are there any security advisories for Django 5.1.3?

What are the major breaking changes in Django 5.0 and Django 5.1 compared to Django 3.2?

Does djangorestframework 3.15.2 support Django 5.1 and Python 3.11?

Also applies to: 4-4, 6-8, 10-10, 13-13, 15-16, 18-19, 21-34, 36-39, 41-44, 48-50

[barecheck]

Dedupe Hub App | Unittest test suite - Code coverage report

Total: 55.73%

Your code coverage diff: 0.00% ▴

✅ All code changes are covered

[barecheck]

Contricleaner App | Unittest test suite - Code coverage report

Total: 98.75%

Your code coverage diff: 0.00% ▴

✅ All code changes are covered

[barecheck]

Countries App | Unittest test suite - Code coverage report

Total: 100%

Your code coverage diff: 0.00% ▴

✅ All code changes are covered

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/django/oar/settings.py (1)

409-409: STATICFILES_STORAGE is ignored in Django 5.2; local dev loses SPA behavior.

STATICFILES_STORAGE was removed in Django 5.1—this line has no effect. The STORAGES dict at lines 610-617 is only defined when not DEBUG or (AWS_S3_ENDPOINT_URL and not TESTING). In local DEBUG mode without AWS_S3_ENDPOINT_URL, STORAGES is never set, so Django defaults to StaticFilesStorage instead of SPAStaticFilesStorage, breaking SPA routing (index.html at root, fallback behavior).

Define STORAGES unconditionally with the SPA staticfiles backend, then conditionally override the "default" backend for S3:

🔧 Proposed fix

-STATICFILES_STORAGE = 'spa.storage.SPAStaticFilesStorage'
+# Django 5.2+ requires STORAGES dict; STATICFILES_STORAGE is removed.
+STORAGES = {
+    "default": {
+        "BACKEND": "django.core.files.storage.FileSystemStorage",
+    },
+    "staticfiles": {
+        "BACKEND": "spa.storage.SPAStaticFilesStorage",
+    },
+}

Then update the conditional block to only override the "default" backend:

 if not DEBUG or (AWS_S3_ENDPOINT_URL and not TESTING):
-    STORAGES = {
-        "default": {
-            "BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
-        },
-        "staticfiles": {
-            "BACKEND": "spa.storage.SPAStaticFilesStorage",
-        },
-    }
+    STORAGES["default"] = {
+        "BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
+    }

Also applies to: 609-617

🧹 Nitpick comments (1)

src/django/oar/settings.py (1)

376-376: Remove deprecated USE_L10N setting.

USE_L10N was deprecated in Django 4.0 and removed in Django 5.0. In Django 5.2, localization is always enabled, making this setting a no-op. Consider removing it to clean up dead code.

♻️ Proposed fix

 USE_I18N = True
 
-USE_L10N = True
-
 USE_TZ = True

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/django/oar/settings.py`:
- Line 409: Remove the deprecated STATICFILES_STORAGE assignment and replace it
by unconditionally defining the STORAGES dict in settings.py; keep conditional
logic only for the "default" backend (e.g., choose between the SPA backend
'oar.storage.SPAStaticFilesStorage' and whatever other default you had) while
always providing a "staticfiles" or equivalent key in STORAGES so Django has a
staticfiles backend fallback; update references to STATICFILES_STORAGE
accordingly and ensure STORAGES is declared at module scope (replace the
STATICFILES_STORAGE = 'oar.storage.SPAStaticFilesStorage' line and ensure the
existing conditional that previously set STORAGES at line ~610 is consolidated
into a single unconditional STORAGES definition that uses a conditional
expression for the "default" entry.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/django/oar/settings.py (1)

17-30: Remove the index_together compatibility shim—it is ineffective and relies on unstable internals.

In Django 5.2, index_together was removed from runtime Meta support (Django 5.1+). The shim does not help with old migrations: Django's migration framework handles backward compatibility through dedicated operations (AlterIndexTogether, RenameIndex), not via runtime Meta validation. Monkey-patching DEFAULT_NAMES (an internal, unstable API with no public guarantees) only prevents a TypeError if code tries to define index_together in Meta, but does nothing to actually support the removed feature. Delete lines 26–29.

🤖 Fix all issues with AI agents

In `@src/django/oar/settings.py`:
- Around line 37-55: Update CKEDITOR_5_CONFIGS by removing invalid plugin names
from the "removePlugins" list so only supported identifiers are used; locate the
CKEDITOR_5_CONFIGS dict and the "removePlugins" key and replace the entries that
are not part of django-ckeditor-5 0.2.19 (e.g., remove CKBox, EasyImage,
ImageUpload) leaving only valid plugin names such as CKFinderUploadAdapter,
Image, ImageCaption, ImageStyle, ImageToolbar, MediaEmbed, Table, and
TableToolbar so the build will actually disable the intended features.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/django/oar/settings.py`:
- Around line 609-617: STORAGES is currently defined conditionally and uses
django.contrib.staticfiles.storage.ManifestStaticFilesStorage for "staticfiles",
which breaks SPA behavior; change the configuration so STORAGES is defined
unconditionally and set the "staticfiles" backend to
spa.storage.SPAStaticFilesStorage (replace ManifestStaticFilesStorage with
spa.storage.SPAStaticFilesStorage) and keep the "default" S3 backend as-is
(S3Boto3Storage) so that SPA routing and asset serving work in DEBUG and
production regardless of AWS_S3_ENDPOINT_URL or TESTING.

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/django/oar/settings.py (1)

409-409: Critical: STATICFILES_STORAGE is ignored in Django 5.1+ — remove and migrate to STORAGES.

STATICFILES_STORAGE was removed in Django 5.1. This line is now a no-op and the SPA staticfiles backend (spa.storage.SPAStaticFilesStorage) is not being used. This breaks SPA functionality:

• index.html not served at root
• Client-side routing fallback not working
• Static assets not served at both /static/... and /... paths

Remove this line and update the STORAGES configuration (lines 609-623) to use spa.storage.SPAStaticFilesStorage for the staticfiles backend.

Suggested fix

Remove line 409 entirely, then update STORAGES (see next comment).

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/django/oar/settings.py`:
- Around line 607-621: The DEBUG branch of the STORAGES dict is missing the
"default" backend which Django 5.2 no longer provides implicitly; update the
DEBUG branch STORAGES to include a "default" key using a filesystem storage
backend (e.g., django.core.files.storage.FileSystemStorage) so default_storage
and FileField work in development, keeping the existing "staticfiles" entry
intact.

[protsack-stephan]

Great work, one small comment.

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/django/oar/settings.py (1)

376-376: Remove deprecated USE_L10N setting.

USE_L10N was deprecated in Django 4.0 and removed in Django 5.0. The project uses Django 5.2.10, where this setting no longer has any effect and may cause configuration warnings. Remove this line entirely, as localization is always enabled in Django 5.0+.

🧹 Nitpick comments (1)

src/django/oar/settings.py (1)

26-29: Acceptable workaround, but consider migration squashing long-term.

This monkey-patch allows legacy migrations using index_together to load without editing historical migration files. While this works, it relies on Django internals that could change. Consider using squashmigrations to consolidate old migrations and convert index_together to indexes in the squashed migration, removing the need for this hack in future releases.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@src/django/oar/settings.py`:
- Around line 153-165: The migration file 0185_add_source_by_to_partner_field.py
still imports the legacy ckeditor package (relying on the compatibility shim in
the ckeditor package __init__), so update the migration to stop depending on the
shim: replace the legacy import from "ckeditor" with the correct import from the
new package (django_ckeditor_5) or the exact class/field module provided by
django_ckeditor_5 used in that migration, ensuring the migration references the
new package names directly (or, if impossible, add a one-line compatibility
alias inside the migration to map the old name to the new one) and then remove
reliance on the ckeditor compatibility shim.
- Around line 37-55: Update the CKEDITOR_5_CONFIGS mapping in settings.py to
remove the invalid plugin identifier "CKFinderUploadAdapter" from the
"removePlugins" list in the "default" config of CKEDITOR_5_CONFIGS; replace it
with the valid identifier "UploadAdapter" if you intend to disable upload
adapters, or simply delete that entry entirely so only valid plugin names
("Image", "ImageCaption", "ImageStyle", "ImageToolbar", "MediaEmbed", "Table",
"TableToolbar") remain, ensuring CKEditor 5 initializes correctly.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[vlad-shapik]

LGTM
Thank you!

[protsack-stephan]

Great work!

[VadimKovalenkoSNF]

@coderabbitai resolve

[coderabbitai]

✅ Actions performed

Comments resolved.