Skip to content
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
eee6d63
Create PR
vladsha-dev Nov 6, 2024
31013ba
Create POST endpoint
vladsha-dev Nov 6, 2024
63d1066
Set up the code structure to handle the contribution moderation event
vladsha-dev Nov 18, 2024
512b79d
remove print
vladsha-dev Nov 18, 2024
c8bed49
Implemented source(API or SLC) validation.
vladsha-dev Nov 19, 2024
81afe75
Implemented triggering of ContriCleaner within the processor to colle…
vladsha-dev Nov 21, 2024
53160f5
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Nov 21, 2024
4afe200
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Nov 22, 2024
621e710
Implement geocoding for data if coordinates information is missing.
vladsha-dev Nov 22, 2024
1734ba5
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Nov 22, 2024
32b88ce
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Nov 25, 2024
abd6a08
Implemented the final saving of the moderation event with a PENDING s…
vladsha-dev Nov 26, 2024
804e1a8
Fix MR conflict
vladsha-dev Nov 27, 2024
fe6127c
Add type hints
vladsha-dev Nov 28, 2024
2e85972
Fix MR conflict
vladsha-dev Nov 28, 2024
3a78e99
Fix and update ContriCleaner and Countries tests
vladsha-dev Nov 29, 2024
3050a24
Revert setting of the OAR_CLIENT_KEY value in the compose file
vladsha-dev Nov 29, 2024
5562d85
Create a test for throttling and permission checks.
vladsha-dev Dec 2, 2024
6a3bda8
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Dec 2, 2024
6e89a04
Create tests
vladsha-dev Dec 2, 2024
26e8f3f
Fix tests
vladsha-dev Dec 2, 2024
2863bce
Add print for debugging
vladsha-dev Dec 2, 2024
0eae448
Fix test
vladsha-dev Dec 3, 2024
de0bdad
Create tests
vladsha-dev Dec 3, 2024
cb7e536
Write tests
vladsha-dev Dec 4, 2024
2739c14
Fix MRs
vladsha-dev Dec 4, 2024
c1d4732
Fix MR conflict
vladsha-dev Dec 4, 2024
c147bdc
Update release notes and fix flake8 errors
vladsha-dev Dec 4, 2024
1d7d44b
Update release notes
vladsha-dev Dec 4, 2024
f26e9a1
Remove redundant space
vladsha-dev Dec 4, 2024
2cda5bc
Merge branch 'main' into OSDEV-1158-api-create-post-endpoint-for-v1-p…
vladsha-dev Dec 5, 2024
8d99859
Handle comments
vladsha-dev Dec 5, 2024
1cc6465
Add dot in the end of the comment
vladsha-dev Dec 5, 2024
9ee130d
Handle comments
vladsha-dev Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions doc/release/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,38 @@ This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html

### Database changes
#### Migrations:
* *Describe migrations here.*
* 0162_update_moderationevent_table_fields.py - This migration updates the ModerationEvent table and its constraints.

#### Scheme changes
* *Describe scheme changes here.*
* [OSDEV-1158](https://opensupplyhub.atlassian.net/browse/OSDEV-1158) - The following updates to the ModerationEvent table have been made:
1. Set `uuid` as the primary key.
2. Make `geocode_result` field optional. It can be blank if lat and lng
have been provided by user.
3. Remove redundant `blank=False` and `null=False` constraints, as these are
the default values for model fields in Django and do not need to be
explicitly set.
4. Make `contributor` field non-nullable, as the field should not be left
empty. It is required to have information about the contributor.
5. Allow `claim` field to be blank. This change reflects the fact that
a moderation event may not always be related to a claim, so the field can
be left empty.

### Code/API changes
* [OSDEV-1453](https://opensupplyhub.atlassian.net/browse/OSDEV-1453) - The `detail` keyword instead of `message` has been applied in error response objects for V1 endpoints.
* [OSDEV-1346](https://opensupplyhub.atlassian.net/browse/OSDEV-1346) - Disabled null values from the response of the OpenSearch. Disabled possible null `os_id`, `claim_id` and `source` from `PATCH api/v1/moderation-events/{moderation_id}` response.
* [OSDEV-1410](https://opensupplyhub.atlassian.net/browse/OSDEV-1410) - Introduced new POST `/api/moderation-events/{moderation_id}/production-locations` endpoint
* [OSDEV-1346](https://opensupplyhub.atlassian.net/browse/OSDEV-1346) - Disabled null values from the response of the OpenSearch. Disabled possible null `os_id`, `claim_id` and `source` from `PATCH /api/v1/moderation-events/{moderation_id}/` response.
* [OSDEV-1410](https://opensupplyhub.atlassian.net/browse/OSDEV-1410) - Introduced a new POST `/api/v1/moderation-events/{moderation_id}/production-locations/` endpoint
* [OSDEV-1449](https://opensupplyhub.atlassian.net/browse/OSDEV-1449) - **Breaking changes** to the following endpoints:
- GET `v1/moderation-events`
- GET `v1/production-locations`
- GET `v1/moderation-events/`
- GET `v1/production-locations/`

**Changes include:**
- Refactored `sort_by` parameter to improve sorting functionality.
- Split `search_after` parameter into `search_after_value` and `search_after_id` for better pagination control.

* [OSDEV-1158](https://opensupplyhub.atlassian.net/browse/OSDEV-1158) - The following features and improvements have been made:
1. Introduced a new POST `/api/v1/production-locations/` endpoint based on the API v1 specification. This endpoint allows the creation of a new moderation event for the production location creation with the given details.
2. Removed redundant redefinition of paths via the `as_view` method for all the v1 API endpoints since they are already defined via `DefaultRouter`.

### Architecture/Environment changes
* [OSDEV-1170](https://opensupplyhub.atlassian.net/browse/OSDEV-1170) - Added the ability to automatically create a dump from the latest shared snapshot of the anonymized database from Production environment for use in the Test and Pre-Prod environments.

Expand All @@ -40,6 +55,7 @@ This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html

### Release instructions:
* Ensure that the following commands are included in the `post_deployment` command:
* `migrate`
* `reindex_database`


Expand Down
41 changes: 39 additions & 2 deletions src/django/api/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,12 +192,49 @@ class NumberOfWorkersRanges:
}]


class ErrorMessages:
GEOCODED_NO_RESULTS = "The address you submitted can not be geocoded."
class APIErrorMessages:
GEOCODED_NO_RESULTS = 'The address you submitted can not be geocoded.'
MAINTENANCE_MODE = ('Open Supply Hub is undergoing maintenance and '
'not accepting new data at the moment. Please '
'try again in a few minutes.')


class FacilitiesDownloadSettings:
DEFAULT_LIMIT = 10000


# API v1
class APIV1CommonErrorMessages:
COMMON_REQ_BODY_ERROR = 'The request body is invalid.'
COMMON_INTERNAL_ERROR = (
'An unexpected error occurred while processing the request.'
)
COMMON_REQ_QUERY_ERROR = 'The request query is invalid.'
MAINTENANCE_MODE = (
'Open Supply Hub is undergoing maintenance and not accepting new data '
'at the moment. Please try again in a few minutes.'
)


class APIV1LocationContributionErrorMessages:
GEOCODED_NO_RESULTS = (
'A valid address could not be found for the provided country and '
'address. This may be due to incorrect, incomplete, or ambiguous '
'information. Please verify and try again.'
)

@staticmethod
def invalid_data_type_error(data_type: str) -> str:
return ('Invalid data. Expected a dictionary (object), '
f'but got {data_type}.')


# If the error isn’t field-specific, the non_field_errors key will be used
# for issues spanning multiple fields or related to the overall data
# object.
NON_FIELD_ERRORS_KEY = 'non_field_errors'


class APIV1LocationContributionKeys:
LNG = 'lng'
LAT = 'lat'
4 changes: 2 additions & 2 deletions src/django/api/facility_actions/processing_facility_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import traceback
from typing import Any, Dict, List, Union

from api.constants import ErrorMessages, ProcessingAction
from api.constants import APIErrorMessages, ProcessingAction
from api.extended_fields import create_extendedfields_for_single_item
from api.facility_actions.processing_facility import ProcessingFacility
from api.geocoding import geocode_address
Expand Down Expand Up @@ -246,7 +246,7 @@ def __handle_geocode_result(
else:
item.status = FacilityListItem.GEOCODED_NO_RESULTS
result['status'] = item.status
result['message'] = ErrorMessages.GEOCODED_NO_RESULTS
result['message'] = APIErrorMessages.GEOCODED_NO_RESULTS

item.processing_results.append(
{
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Generated by Django 3.2.17 on 2024-11-26 16:27

import django.contrib.postgres.indexes
from django.db import migrations, models
import django.db.models.deletion
import uuid


class Migration(migrations.Migration):
'''
This migration performs the following updates to the ModerationEvent table:
1. Set uuid as the primary key.
2. Make geocode_result field optional. It can be blank if lat and lng
have been provided by user.
3. Remove redundant blank=False and null=False constraints, as these are
the default values for model fields in Django and do not need to be
explicitly set.
4. Make contributor field non-nullable, as the field should not be left
empty. It is required to have information about the contributor.
5. Allow claim field to be blank. This change reflects the fact that
a moderation event may not always be related to a claim, so the field can
be left empty.
'''

dependencies = [
('api', '0161_create_disable_list_uploading'),
]

operations = [
migrations.RemoveField(
model_name='moderationevent',
name='id',
),
migrations.AlterField(
model_name='moderationevent',
name='claim',
field=models.OneToOneField(blank=True, help_text='Linked claim id for this production location.', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='moderation_event_claim', to='api.facilityclaim'),
),
migrations.AlterField(
model_name='moderationevent',
name='contributor',
field=models.ForeignKey(help_text='Linked contributor responsible for this moderation event.', on_delete=django.db.models.deletion.PROTECT, related_name='moderation_event_contributor', to='api.contributor'),
),
migrations.AlterField(
model_name='moderationevent',
name='geocode_result',
field=models.JSONField(blank=True, default=dict, help_text='Result of the geocode operation.'),
),
migrations.AlterField(
model_name='moderationevent',
name='uuid',
field=models.UUIDField(db_index=True, default=uuid.uuid4, editable=False, help_text='Unique identifier to make moderation event table more reusable across the app.', primary_key=True, serialize=False, unique=True),
),
]
12 changes: 3 additions & 9 deletions src/django/api/models/moderation_event.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ class Source(models.TextChoices):
SLC = 'SLC', 'SLC'

uuid = models.UUIDField(
primary_key=True,
default=uuid.uuid4,
editable=False,
unique=True,
Expand Down Expand Up @@ -54,7 +55,6 @@ class Source(models.TextChoices):
contributor = models.ForeignKey(
Contributor,
on_delete=models.PROTECT,
null=True,
related_name='moderation_event_contributor',
help_text='Linked contributor responsible for this moderation event.'
)
Expand All @@ -72,20 +72,18 @@ class Source(models.TextChoices):
FacilityClaim,
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name='moderation_event_claim',
help_text='Linked claim id for this production location.'
)

request_type = models.CharField(
max_length=6,
null=False,
choices=RequestType.choices,
help_text='Type of moderation record.'
)

raw_data = models.JSONField(
null=False,
blank=False,
default=dict,
help_text=(
'Key-value pairs of the non-parsed row and '
Expand All @@ -94,8 +92,6 @@ class Source(models.TextChoices):
)

cleaned_data = models.JSONField(
null=False,
blank=False,
default=dict,
help_text=(
'Key-value pairs of the parsed row and '
Expand All @@ -104,8 +100,7 @@ class Source(models.TextChoices):
)

geocode_result = models.JSONField(
null=False,
blank=False,
blank=True,
default=dict,
help_text=(
'Result of the geocode operation.'
Expand All @@ -116,7 +111,6 @@ class Source(models.TextChoices):
max_length=8,
choices=Status.choices,
default=Status.PENDING,
null=False,
help_text='Moderation status of the production location.'
)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from dataclasses import dataclass, field
from typing import Dict

from rest_framework import status

from api.models.moderation_event import ModerationEvent


@dataclass
class CreateModerationEventDTO:
contributor_id: int
raw_data: Dict
request_type: str
cleaned_data: Dict = field(default_factory=dict)
source: str = ''
geocode_result: Dict = field(default_factory=dict)
errors: Dict = field(default_factory=dict)
status_code: int = status.HTTP_202_ACCEPTED
moderation_event: ModerationEvent = None
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from abc import ABC, abstractmethod

from api.moderation_event_actions.creation.dtos.create_moderation_event_dto \
import CreateModerationEventDTO


class EventCreationStrategy(ABC):
@abstractmethod
def serialize(
self,
event_dto: CreateModerationEventDTO) -> CreateModerationEventDTO:
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from api.moderation_event_actions.creation.event_creation_strategy \
import EventCreationStrategy
from api.moderation_event_actions.creation.location_contribution \
.processors.contribution_processor import ContributionProcessor
from api.moderation_event_actions.creation.location_contribution \
.processors.source_processor import SourceProcessor
from api.moderation_event_actions.creation.location_contribution \
.processors.production_location_data_processor \
import ProductionLocationDataProcessor
from api.moderation_event_actions.creation.location_contribution \
.processors.geocoding_processor import GeocodingProcessor
from api.moderation_event_actions.creation.dtos.create_moderation_event_dto \
import CreateModerationEventDTO


class LocationContribution(EventCreationStrategy):
'''
The class-based algorithm that validates, extracts, and geocodes the
retrieved location data from the user before saving it as
a moderation event.
'''

def serialize(
self,
event_dto: CreateModerationEventDTO) -> CreateModerationEventDTO:
entry_location_data_processor = self.__setup_location_data_processors()

processed_location_data = entry_location_data_processor.process(
event_dto
)

return processed_location_data

@staticmethod
def __setup_location_data_processors() -> ContributionProcessor:
location_data_processors = (
SourceProcessor(),
ProductionLocationDataProcessor(),
GeocodingProcessor()
)
for index in range(len(location_data_processors) - 1):
location_data_processors[index].set_next(
location_data_processors[index + 1]
)

entry_location_data_processor = location_data_processors[0]

return entry_location_data_processor
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
from __future__ import annotations
from abc import ABC, abstractmethod
import logging

from rest_framework import status

from api.moderation_event_actions.creation.dtos.create_moderation_event_dto \
import CreateModerationEventDTO
from api.constants import APIV1CommonErrorMessages

# Initialize logger.
log = logging.getLogger(__name__)


class ContributionProcessor(ABC):
'''
The class that defines the common interface for location contribution
processors. Essentially, it is used to implement the Chain of
Responsibility pattern, allowing the graceful stopping of moderation event
data processing in case an error occurs, or allowing the entire chain of
processors to run through in case of a successful pass.
'''

_next: ContributionProcessor = None

def set_next(self, next: ContributionProcessor) -> None:
self._next = next

@abstractmethod
def process(
self,
event_dto: CreateModerationEventDTO) -> CreateModerationEventDTO:
if self._next:
return self._next.process(event_dto)

log.error(
('[API V1 Location Upload] Internal Moderation Event Creation '
'Error: The non-existing next handler for processing the '
'moderation event creation related to location contribution was '
'called in the last handler of the chain.')
)
event_dto.errors = {
'detail': APIV1CommonErrorMessages.COMMON_INTERNAL_ERROR
}
event_dto.status_code = status.HTTP_500_INTERNAL_SERVER_ERROR

return event_dto
Loading