Unable to configure retry behavior for status codes 400-499

[Mantisus]

In BasicCrawler._should_retry_request, we always return False for status codes 400-499.

    def _should_retry_request(self, context: BasicCrawlingContext, error: Exception) -> bool:
        if context.request.no_retry:
            return False

        # Do not retry on client errors.
        if isinstance(error, HttpStatusCodeError) and is_status_code_client_error(error.status_code):
            return False

The is_status_code_client_error function enforces this behavior:

def is_status_code_client_error(value: int) -> bool:
    """Returns `True` for 4xx status codes, `False` otherwise."""
    return 400 <= value <= 499  # noqa: PLR2004

This makes it impossible to configure retry behavior for these codes.

[vdusek]

Related to #708

[monk3yd]

Having this issue with my spider, errors 403 and 406 from the target website are impossible to retry. Is there any way to bypass this issue for now? I instantiate a RequestQueue within the failed request handler but it doesn't seem to requeue the request, even if I change its unique_key

   @crawler.failed_request_handler
    async def failed_request_handler(context: Union[ParselCrawlingContext, BasicCrawlingContext], error: Exception) -> None:
        context.log.error(f"failed_request_handler is processing {str(error)} for url {context.request.url} ...")

        if isinstance(error, HttpStatusCodeError):
            if context.session:
                context.log.error("retiring session ...")
                context.session.retire()

            timestamp = datetime.now().strftime('%Y%m%d%H%M%S')
            rq = await RequestQueue.open(name="rq")
            req = Request.from_url(url=context.request.url, label=context.request.label, unique_key=f"{context.request.unique_key}_retry_{timestamp}")
            await rq.add_request(req, forefront=True)

            context.log.error("requeue...")

            failed = {
                "id": context.request.id,
                "url": context.request.url,
                "error": str(error),
                "request": context.request.model_dump()
            }
            dataset = await Dataset.open(name="errors")
            await dataset.push_data(failed)

[Mantisus]

Hey @monk3yd, regarding your failed_request_handler. I have a few questions about the RequestQueue:

1. Are you using the same named queue RequestQueue.open(name="rq") in your crawler, or do you only open it inside the handler?
2. Note that if you don't explicitly pass a named queue to the crawler, it uses the default queue (you can open it using just RequestQueue.open()).

[monk3yd]

Hey @Mantisus, thank you for your response! I finally patched the issue by doing this

def is_status_code_client_error(value: int) -> bool:
    """Returns `True` for 4xx status codes, `False` otherwise."""
    if value == 403 or value == 406:
        value = 500
    return 400 <= value <= 499  # noqa: PLR2004

and something like #708

if context.session and context.session.is_blocked_status_code(
    status_code=status_code,
    additional_blocked_status_codes=[403, 406],
) and status_code not in self._http_client._ignore_http_error_status_codes :

This will enable retry for this status codes, though its a pretty hacky fix hehe

[Mantisus]

Hey @monk3yd.

Yes, this approach would work too.

Hopefully, when we're done with #812, you won't have to do something like this. 😅