Issues with payload in POST requests

[francomanca93]

Issues with payload in POST requests

Description

When using the crawlee.http_crawler._http_crawler module, I'm encountering a HttpStatusCodeError with a 404 status code. This error occurs during the _make_http_request operation and causes the crawler to reach the maximum number of retries.

The issue with the 404 error in the crawler seems to be related to the way i'm handling the request payload.

Steps to Reproduce

1. Initialize a new Crawlee project and set up an HttpCrawler with either the CurlImpersonateHttpClient or the HttpxHttpClient:

import asyncio
import json
from crawlee import Request
from crawlee.http_crawler import HttpCrawler, HttpCrawlingContext
from crawlee.http_clients.curl_impersonate import CurlImpersonateHttpClient
from crawlee.http_clients import HttpxHttpClient

async def main() -> None:
    #http_client = CurlImpersonateHttpClient(
    #    persist_cookies_per_session=True,
    #)
    # Or use HttpxHttpClient:
    http_client = HttpxHttpClient(
        persist_cookies_per_session=True,
    )

    crawler = HttpCrawler(
        http_client=http_client,
        max_requests_per_crawl=20,
    )

    url = "https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088"
    payload = {
        'ShowAllTickets': True,
        'HideDuplicateTicketsV2': False,
        'Quantity': 2,
        'IsInitialQuantityChange': False,
        'PageSize': 20,
        'CurrentPage': 2,
        'SortBy': 'NEWPRICE',
        'SortDirection': 0,
        'Sections': '',
        'Rows': '',
        'Seats': '',
        'SeatTypes': '',
        'TicketClasses': '',
        'ListingNotes': '',
        'PriceRange': '0,100',
        'InstantDelivery': False,
        'EstimatedFees': True,
        'BetterValueTickets': True,
        'PriceOption': '',
        'HasFlexiblePricing': False,
        'ExcludeSoldListings': False,
        'RemoveObstructedView': False,
        'NewListingsOnly': False,
        'PriceDropListingsOnly': False,
        'SelectBestListing': False,
        'ConciergeTickets': False,
        'Method': 'IndexSh'
    }
    payload_bytes = json.dumps(payload).encode()

    print(f"0. Start: {payload_bytes}")

    initial_req = Request.from_url(
        url=url,
        method="POST",
        payload=payload_bytes,
        use_extended_unique_key=True,
    )

    @crawler.router.default_handler
    async def request_handler(context: HttpCrawlingContext) -> None:
        # Handle the response here
        pass

    # Run the crawler
    await crawler.run([initial_req])

if __name__ == '__main__':
    asyncio.run(main())
    

2. The crawler encounters a 404 error during the crawl operation and raises the following traceback:

   [crawlee.http_crawler._http_crawler] ERROR Request failed and reached maximum retries
         Traceback (most recent call last):
           File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_context_pipeline.py", line 62, in __call__
             result = await middleware_instance.__anext__()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_crawler/_http_crawler.py", line 101, in _make_http_request
             result = await self._http_client.crawl(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/curl_impersonate.py", line 152, in crawl
             self._raise_for_error_status_code(
           File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/_base.py", line 153, in _raise_for_error_status_code
             raise HttpStatusCodeError('Error status code returned', status_code)
       crawlee.errors.HttpStatusCodeError: Error status code returned (status code: 404).
   

But if I pass the request as this:

    initial_req = Request.from_url(
        url=url,
        method="POST",
        payload=payload,  # payload as dictionary
        use_extended_unique_key=True,
    )

The error is the following:

Traceback (most recent call last):
  File "/Development/scraper/src/test_clients.py", line 175, in <module>
    asyncio.run(main())
  File "/Library/Application Support/uv/python/cpython-3.12.7-macos-x86_64-none/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Application Support/uv/python/cpython-3.12.7-macos-x86_64-none/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Application Support/uv/python/cpython-3.12.7-macos-x86_64-none/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Development/scraper/src/test_clients.py", line 159, in main
    initial_req = Request.from_url(
                  ^^^^^^^^^^^^^^^^^
  File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/_request.py", line 321, in from_url
    unique_key = unique_key or compute_unique_key(
                               ^^^^^^^^^^^^^^^^^^^
  File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/_utils/requests.py", line 126, in compute_unique_key
    payload_hash = _get_payload_hash(payload)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/_utils/requests.py", line 151, in _get_payload_hash
    return compute_short_hash(payload_in_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/_utils/crypto.py", line 17, in compute_short_hash
    hash_object = sha256(data)
                  ^^^^^^^^^^^^
TypeError: object supporting the buffer API required

Environment

• Python version: 3.12
• Crawlee version: 0.4.0

[janbuchar]

Hello @francomanca93 and thank you for your interest in Crawlee! I ran your snippet and I believe that it behaves correctly (the first one, I mean). I re-checked this with plain curl in a console and the website does in fact return the 404 HTTP status code if I send an equivalent request:

curl -v -X POST https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 -d @payload.json
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host www.viagogo.es:443 was resolved.
* IPv6: (none)
* IPv4: 23.212.110.33, 23.212.110.35
*   Trying 23.212.110.33:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=CH; L=Genve; O=Viagogo AG; CN=*.viagogo.com
*  start date: Mar 26 00:00:00 2024 GMT
*  expire date: Apr 26 23:59:59 2025 GMT
*  subjectAltName: host "www.viagogo.es" matched cert's "*.viagogo.es"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to www.viagogo.es (23.212.110.33) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.viagogo.es]
* [HTTP/2] [1] [:path: /Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088]
* [HTTP/2] [1] [user-agent: curl/8.11.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 624]
* [HTTP/2] [1] [content-type: application/x-www-form-urlencoded]
> POST /Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 HTTP/2
> Host: www.viagogo.es
> User-Agent: curl/8.11.0
> Accept: */*
> Content-Length: 624
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 624 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 404
< content-type: text/html; charset=utf-8
< content-length: 68804

So, this doesn't seem to be an issue of Crawlee, just irregular behavior of the website you're trying to scrape. Try to experiment with HTTP request headers to see if you can get it to return what you actually need.

[francomanca93]

You have to use curl-impersonate to get the data, here an example:

from rich import print
import httpx
from curl_cffi import requests
import json

url = "https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088"
payload = {...THE ABOVE PAYLOAD...}

# payload_bytes = json.dumps(payload).encode("utf-8")
# I cannot get the data passing the data in bytes format, but I can if I will pass in a dict format 

curl_response = requests.post(url, data=payload)
curl_response = curl_response.json()

print(f"Total items: {len(curl_response['items'])}")
print(f"totalCount: {curl_response['totalCount']}")
print(f"currentPage: {curl_response['currentPage']}")
print(f"pageSize: {curl_response['pageSize']}")
print(f"itemsRemaining: {curl_response['itemsRemaining']}")

My output:

Related with the issue I have activate logging as follow:

    crawler = HttpCrawler(
        http_client=http_client,
        max_requests_per_crawl=20,
        configuration=Configuration(verbose_log=True),
    )

I have also modified the method format from CrawleeLogFormatter class line 127:

        logger_name_string = f'{_LOG_NAME_COLOR}[{record.name} - {record.lineno}]{Style.RESET_ALL} '

And I have added some logger.debug() to check deeply and I found this:

Here the complete log:

�[crawlee._utils.recurring_task - 23] DEBUG Calling RecurringTask.__init__(func=_log, delay=0:01:00)...
[crawlee._request - 319] DEBUG -----Request payload: b'{"ShowAllTickets": true, "HideDuplicateTicketsV2": false, "Quantity": 2, "IsInitialQuantityChange": false, "PageSize": 20, "CurrentPage": 2, "SortBy": "NEWPRICE", "SortDirection": 0, "Sections": "", "Rows": "", "Seats": "", "SeatTypes": "", "TicketClasses": "", "ListingNotes": "", "PriceRange": "0,100", "InstantDelivery": false, "EstimatedFees": true, "BetterValueTickets": true, "PriceOption": "", "HasFlexiblePricing": false, "ExcludeSoldListings": false, "RemoveObstructedView": false, "NewListingsOnly": false, "PriceDropListingsOnly": false, "SelectBestListing": false, "ConciergeTickets": false, "Method": "IndexSh"}'
[crawlee.storages._request_queue - 243] DEBUG ---DELETE--- Showing batch data [Request(url='https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088', unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', method='POST', headers=HttpHeaders(root={}), payload=b'{"ShowAllTickets": true, "HideDuplicateTicketsV2": false, "Quantity": 2, "IsInitialQuantityChange": false, "PageSize": 20, "CurrentPage": 2, "SortBy": "NEWPRICE", "SortDirection": 0, "Sections": "", "Rows": "", "Seats": "", "SeatTypes": "", "TicketClasses": "", "ListingNotes": "", "PriceRange": "0,100", "InstantDelivery": false, "EstimatedFees": true, "BetterValueTickets": true, "PriceOption": "", "HasFlexiblePricing": false, "ExcludeSoldListings": false, "RemoveObstructedView": false, "NewListingsOnly": false, "PriceDropListingsOnly": false, "SelectBestListing": false, "ConciergeTickets": false, "Method": "IndexSh"}', user_data=UserData(crawlee_data=None, label=None), retry_count=0, no_retry=False, loaded_url=None, handled_at=None, id='ykSDp6ltgy8G3xw', json_=None, order_no=None)]...
[crawlee.storages._request_queue - 246] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='ykSDp6ltgy8G3xw', unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee.storages._request_queue - 247] DEBUG ---DELETE--- Response: {'processed_requests': [{'id': 'ykSDp6ltgy8G3xw', 'unique_key': 'POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', 'was_already_present': False, 'was_already_handled': False}], 'unprocessed_requests': []}
[crawlee.memory_storage_client._memory_storage_client - 138] DEBUG Storage was already purged on start.
[crawlee._autoscaling.autoscaled_pool - 122] DEBUG Starting the pool


[crawlee.http_crawler._http_crawler - 217] INFO  Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [0]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 0.0      │
│ requests_total                │ 0        │
│ crawler_runtime               │ 0.024688 │
└───────────────────────────────┴──────────┘
[crawlee._autoscaling.autoscaled_pool - 213] INFO  current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[crawlee._utils.system - 65] DEBUG Calling get_cpu_info()...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.statistics._statistics - 239] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue - 502] DEBUG Queue head still returned requests that need to be processed (or that are locked by other clients)
[crawlee._autoscaling.autoscaled_pool - 240] DEBUG Scheduling a new task
[crawlee.storages._request_queue - 480] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool - 240] DEBUG Scheduling a new task
[crawlee.storages._request_queue - 480] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool - 236] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee.storages._request_queue - 283] DEBUG ---DELETE--- Fetching request with ID: ykSDp6ltgy8G3xw
[crawlee.storages._request_queue - 285] DEBUG ---DELETE--- Fetched request: url='https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088' unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088' method='POST' headers=HttpHeaders(root={}) payload=b'b\'{"ShowAllTickets": true, "HideDuplicateTicketsV2": false, "Quantity": 2, "IsInitialQuantityChange": false, "PageSize": 20, "CurrentPage": 2, "SortBy": "NEWPRICE", "SortDirection": 0, "Sections": "", "Rows": "", "Seats": "", "SeatTypes": "", "TicketClasses": "", "ListingNotes": "", "PriceRange": "0,100", "InstantDelivery": false, "EstimatedFees": true, "BetterValueTickets": true, "PriceOption": "", "HasFlexiblePricing": false, "ExcludeSoldListings": false, "RemoveObstructedView": false, "NewListingsOnly": false, "PriceDropListingsOnly": false, "SelectBestListing": false, "ConciergeTickets": false, "Method": "IndexSh"}\'' user_data=UserData(crawlee_data=None, label=None) retry_count=0 no_retry=False loaded_url=None handled_at=None id='ykSDp6ltgy8G3xw' json_=None order_no=Decimal('1731074723267.712116')
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 291] DEBUG Worker task finished
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system - 75] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system - 65] DEBUG Calling get_cpu_info()...
[crawlee._utils.system - 75] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system - 65] DEBUG Calling get_cpu_info()...
[crawlee._utils.system - 75] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.http_crawler._http_crawler - 1012] DEBUG An exception occurred during the initialization of crawling context
      Traceback (most recent call last):
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_context_pipeline.py", line 62, in __call__
          result = await middleware_instance.__anext__()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_crawler/_http_crawler.py", line 94, in _make_http_request
          result = await self._http_client.crawl(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/curl_impersonate.py", line 145, in crawl
          self._raise_for_error_status_code(
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/_base.py", line 153, in _raise_for_error_status_code
          raise HttpStatusCodeError('Error status code returned', status_code)
      crawlee.errors.HttpStatusCodeError: Error status code returned (status code: 404).

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_basic_crawler.py", line 929, in __run_task_function
          await wait_for(
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/_utils/wait.py", line 37, in wait_for
          return await asyncio.wait_for(operation(), timeout.total_seconds())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Library/Application Support/uv/python/cpython-3.12.7-macos-x86_64-none/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
          return await fut
                 ^^^^^^^^^
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_basic_crawler.py", line 1027, in __run_request_handler
          await self._context_pipeline(context, self.router)
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_context_pipeline.py", line 70, in __call__
          raise ContextPipelineInitializationError(e, crawling_context) from e
      crawlee.errors.ContextPipelineInitializationError
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.http_crawler._http_crawler - 787] ERROR Request failed and reached maximum retries
      Traceback (most recent call last):
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/basic_crawler/_context_pipeline.py", line 62, in __call__
          result = await middleware_instance.__anext__()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_crawler/_http_crawler.py", line 94, in _make_http_request
          result = await self._http_client.crawl(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/curl_impersonate.py", line 145, in crawl
          self._raise_for_error_status_code(
        File "/Development/scraper/.venv/lib/python3.12/site-packages/crawlee/http_clients/_base.py", line 153, in _raise_for_error_status_code
          raise HttpStatusCodeError('Error status code returned', status_code)
      crawlee.errors.HttpStatusCodeError: Error status code returned (status code: 404).
[crawlee._autoscaling.autoscaled_pool - 291] DEBUG Worker task finished
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._autoscaling.autoscaled_pool - 254] DEBUG `is_finished_function` reports that we are finished
[crawlee._autoscaling.autoscaled_pool - 263] DEBUG Terminating - no running tasks to wait for
[crawlee._autoscaling.autoscaled_pool - 149] INFO  Waiting for remaining tasks to finish
[crawlee._autoscaling.autoscaled_pool - 159] DEBUG Pool cleanup finished
[crawlee.statistics._statistics - 239] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.http_crawler._http_crawler - 452] INFO  Error analysis: total_errors=1 unique_errors=1
[crawlee.http_crawler._http_crawler - 467] INFO  Final request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 1        │
│ retry_histogram               │ [1]      │
│ request_avg_failed_duration   │ 2.690305 │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 21       │
│ request_total_duration        │ 2.690305 │
│ requests_total                │ 1        │
│ crawler_runtime               │ 2.760437 │
└───────────────────────────────┴──────────┘

[francomanca93]

...following the previous comment...

I have even tried the following for test and it works:

1. I have modified the file _types.py line 55:

HttpPayload: TypeAlias = bytes | dict

2. I have modified the file _utils.requests.py:

# ----- before -----
def _get_payload_hash(payload: HttpPayload | None) -> str:
    payload_in_bytes = b'' if payload is None else payload
    return compute_short_hash(payload_in_bytes)
    
# ------- after ------ (only for tests)
def _get_payload_hash(payload: HttpPayload | None) -> str:
    if isinstance(payload, dict):
        import json
        payload = json.dumps(payload).encode('utf-8')
    payload_in_bytes = b'' if payload is None else payload
    return compute_short_hash(payload_in_bytes)

The script:

import asyncio
import json
from crawlee import Request
from crawlee.http_crawler import HttpCrawler, HttpCrawlingContext
from crawlee.http_clients.curl_impersonate import CurlImpersonateHttpClient
from crawlee.http_clients import HttpxHttpClient
from crawlee.configuration import Configuration

async def main() -> None:
    http_client = CurlImpersonateHttpClient(
        persist_cookies_per_session=True,
    )
    # Or use HttpxHttpClient:
    #http_client = HttpxHttpClient(
    #    persist_cookies_per_session=True,
    #    configuration=Configuration(verbose_log=True),
    #)

    crawler = HttpCrawler(
        http_client=http_client,
        max_requests_per_crawl=20,
        configuration=Configuration(verbose_log=True),
    )

    url = "https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088"
    payload = {... THE ABOVE PAYLOAD ..}
    payload_bytes = json.dumps(payload).encode() # <--- it doesn't work

    initial_req = Request.from_url(
        url=url,
        method="POST",
        payload=payload,  # payload in dict format
        use_extended_unique_key=True,
    )

    @crawler.router.default_handler
    async def request_handler(context: HttpCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        response = context.http_response.read().decode('utf-8')
        json_response = json.loads(response)
        context.log.info(f"Total items: {len(json_response['items'])}")
        context.log.info(f"totalCount: {json_response['totalCount']}")
        context.log.info(f"currentPage: {json_response['currentPage']}")
        context.log.info(f"pageSize: {json_response['pageSize']}")
        context.log.info(f"itemsRemaining: {json_response['itemsRemaining']}")

    # Run the crawler
    await crawler.run([initial_req])

if __name__ == '__main__':
    asyncio.run(main())

My output:

Here the complete log:

❯ python src/test_clients.py                                                                                                                                                  ─╯
[crawlee._utils.recurring_task - 23] DEBUG Calling RecurringTask.__init__(func=_log, delay=0:01:00)...
[crawlee._request - 319] DEBUG -----Request payload: {'ShowAllTickets': True, 'HideDuplicateTicketsV2': False, 'Quantity': 2, 'IsInitialQuantityChange': False, 'PageSize': 20, 'CurrentPage': 2, 'SortBy': 'NEWPRICE', 'SortDirection': 0, 'Sections': '', 'Rows': '', 'Seats': '', 'SeatTypes': '', 'TicketClasses': '', 'ListingNotes': '', 'PriceRange': '0,100', 'InstantDelivery': False, 'EstimatedFees': True, 'BetterValueTickets': True, 'PriceOption': '', 'HasFlexiblePricing': False, 'ExcludeSoldListings': False, 'RemoveObstructedView': False, 'NewListingsOnly': False, 'PriceDropListingsOnly': False, 'SelectBestListing': False, 'ConciergeTickets': False, 'Method': 'IndexSh'}


[crawlee.storages._request_queue - 243] DEBUG ---DELETE--- Showing batch data [Request(url='https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088', unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', method='POST', headers=HttpHeaders(root={}), payload={'ShowAllTickets': True, 'HideDuplicateTicketsV2': False, 'Quantity': 2, 'IsInitialQuantityChange': False, 'PageSize': 20, 'CurrentPage': 2, 'SortBy': 'NEWPRICE', 'SortDirection': 0, 'Sections': '', 'Rows': '', 'Seats': '', 'SeatTypes': '', 'TicketClasses': '', 'ListingNotes': '', 'PriceRange': '0,100', 'InstantDelivery': False, 'EstimatedFees': True, 'BetterValueTickets': True, 'PriceOption': '', 'HasFlexiblePricing': False, 'ExcludeSoldListings': False, 'RemoveObstructedView': False, 'NewListingsOnly': False, 'PriceDropListingsOnly': False, 'SelectBestListing': False, 'ConciergeTickets': False, 'Method': 'IndexSh'}, user_data=UserData(crawlee_data=None, label=None), retry_count=0, no_retry=False, loaded_url=None, handled_at=None, id='ykSDp6ltgy8G3xw', json_=None, order_no=None)]...
[crawlee.storages._request_queue - 246] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='ykSDp6ltgy8G3xw', unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee.storages._request_queue - 247] DEBUG ---DELETE--- Response: {'processed_requests': [{'id': 'ykSDp6ltgy8G3xw', 'unique_key': 'POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', 'was_already_present': False, 'was_already_handled': False}], 'unprocessed_requests': []}
[crawlee.memory_storage_client._memory_storage_client - 138] DEBUG Storage was already purged on start.
[crawlee._autoscaling.autoscaled_pool - 122] DEBUG Starting the pool
[crawlee.http_crawler._http_crawler - 217] INFO  Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [0]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 0.0      │
│ requests_total                │ 0        │
│ crawler_runtime               │ 0.024998 │
└───────────────────────────────┴──────────┘
[crawlee._utils.system - 65] DEBUG Calling get_cpu_info()...
[crawlee._autoscaling.autoscaled_pool - 213] INFO  current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.statistics._statistics - 239] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue - 502] DEBUG Queue head still returned requests that need to be processed (or that are locked by other clients)
[crawlee._autoscaling.autoscaled_pool - 240] DEBUG Scheduling a new task
[crawlee.storages._request_queue - 480] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool - 240] DEBUG Scheduling a new task
[crawlee.storages._request_queue - 480] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool - 236] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee.storages._request_queue - 283] DEBUG ---DELETE--- Fetching request with ID: ykSDp6ltgy8G3xw
[crawlee.storages._request_queue - 285] DEBUG ---DELETE--- Fetched request: url='https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088' unique_key='POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088' method='POST' headers=HttpHeaders(root={}) payload={'ShowAllTickets': True, 'HideDuplicateTicketsV2': False, 'Quantity': 2, 'IsInitialQuantityChange': False, 'PageSize': 20, 'CurrentPage': 2, 'SortBy': 'NEWPRICE', 'SortDirection': 0, 'Sections': '', 'Rows': '', 'Seats': '', 'SeatTypes': '', 'TicketClasses': '', 'ListingNotes': '', 'PriceRange': '0,100', 'InstantDelivery': False, 'EstimatedFees': True, 'BetterValueTickets': True, 'PriceOption': '', 'HasFlexiblePricing': False, 'ExcludeSoldListings': False, 'RemoveObstructedView': False, 'NewListingsOnly': False, 'PriceDropListingsOnly': False, 'SelectBestListing': False, 'ConciergeTickets': False, 'Method': 'IndexSh'} user_data=UserData(crawlee_data=None, label=None) retry_count=0 no_retry=False loaded_url=None handled_at=None id='ykSDp6ltgy8G3xw' json_=None order_no=Decimal('1731076460936.870098')
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 291] DEBUG Worker task finished
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system - 75] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee.storages._request_queue - 490] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["ykSDp6ltgy8G3xw"]})
[crawlee.storages._request_queue - 553] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "ykSDp6ltgy8G3xw", "unique_key": "POST|e3b0c442|6793db94|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool - 238] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system - 65] DEBUG Calling get_cpu_info()...
[crawlee._utils.system - 75] DEBUG Calling get_memory_info()...
[crawlee.http_crawler._http_crawler - 169] INFO  Processing https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 ...
[crawlee.http_crawler._http_crawler - 172] INFO  Total items: 20
[crawlee.http_crawler._http_crawler - 173] INFO  totalCount: 73
[crawlee.http_crawler._http_crawler - 174] INFO  currentPage: 2
[crawlee.http_crawler._http_crawler - 175] INFO  pageSize: 20
[crawlee.http_crawler._http_crawler - 176] INFO  itemsRemaining: 33
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 125] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager - 127] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager - 135] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee._autoscaling.autoscaled_pool - 291] DEBUG Worker task finished
[crawlee._autoscaling.autoscaled_pool - 254] DEBUG `is_finished_function` reports that we are finished
[crawlee._autoscaling.autoscaled_pool - 263] DEBUG Terminating - no running tasks to wait for
[crawlee._autoscaling.autoscaled_pool - 149] INFO  Waiting for remaining tasks to finish
[crawlee._autoscaling.autoscaled_pool - 159] DEBUG Pool cleanup finished
[crawlee.statistics._statistics - 239] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.http_crawler._http_crawler - 467] INFO  Final request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 1        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [1]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ 1.332043 │
│ requests_finished_per_minute  │ 43       │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 1.332043 │
│ requests_total                │ 1        │
│ crawler_runtime               │ 1.407498 │
└───────────────────────────────┴──────────┘

[vdusek]

Hi @francomanca93, this issue should be fixed now. We're releasing a new patch version, 0.4.2, next week. If you'd like to check it out sooner, you can try the latest beta version - https://pypi.org/project/crawlee/#history.

[francomanca93]

Hi @vdusek. I have tested the new version and I don't have any problem if I make one request without the argument use_extended_unique_key in my initial_req but If I use the use_extended_unique_key=True I cannot continue with other request to the same page but with different payload. You can see this below. Steps I have followed:

1. I have installed the beta version: uv pip install crawlee==0.4.2b3
2. I want to extract the information through other pages I get from initial request, so I added in my request_handler await context.add_requests([req]). Complete code:

Here the complete python script :

import asyncio
import json
from urllib.parse import urlencode
from crawlee import Request
from crawlee.http_crawler import HttpCrawler, HttpCrawlingContext
from crawlee.http_clients.curl_impersonate import CurlImpersonateHttpClient
from crawlee.http_clients import HttpxHttpClient
from crawlee.configuration import Configuration

async def main() -> None:
    http_client = CurlImpersonateHttpClient(
        persist_cookies_per_session=True,
    )
    # Or use HttpxHttpClient:
    #http_client = HttpxHttpClient(
    #    persist_cookies_per_session=True,
    #    configuration=Configuration(verbose_log=True),
    #)

    crawler = HttpCrawler(
        http_client=http_client,
        max_requests_per_crawl=20,
        configuration=Configuration(verbose_log=True),
    )

    url = "https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088"
    # I have added the header as you've added in the PR. 
    headers={'content-type': 'application/x-www-form-urlencoded'}
    payload = {
        'ShowAllTickets': True,
        'HideDuplicateTicketsV2': False,
        'Quantity': 2,
        'IsInitialQuantityChange': False,
        'PageSize': 20,
        'CurrentPage': 1,
        'SortBy': 'NEWPRICE',
        'SortDirection': 0,
        'Sections': '',
        'Rows': '',
        'Seats': '',
        'SeatTypes': '',
        'TicketClasses': '',
        'ListingNotes': '',
        'PriceRange': '0,100',
        'InstantDelivery': False,
        'EstimatedFees': True,
        'BetterValueTickets': True,
        'PriceOption': '',
        'HasFlexiblePricing': False,
        'ExcludeSoldListings': False,
        'RemoveObstructedView': False,
        'NewListingsOnly': False,
        'PriceDropListingsOnly': False,
        'SelectBestListing': False,
        'ConciergeTickets': False,
        'Method': 'IndexSh'
    }

    # I have encoded the payload using urlencode
    payload_bytes = urlencode(payload)

    initial_req = Request.from_url(
        url=url,
        method="POST",
        headers=headers,
        payload=payload_bytes,
        # I'm using the extended unique key
        use_extended_unique_key=True,
    )

    @crawler.router.default_handler
    async def request_handler(context: HttpCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        response = context.http_response.read().decode('utf-8')
        json_response = json.loads(response)
        context.log.info(f"Total items: {len(json_response['items'])}")
        context.log.info(f"totalCount: {json_response['totalCount']}")
        context.log.info(f"currentPage: {json_response['currentPage']}")
        context.log.info(f"pageSize: {json_response['pageSize']}")
        context.log.info(f"itemsRemaining: {json_response['itemsRemaining']}")

        # I have updated the payload here while creating the new request
        # to go through the next page and can get all the items I need.
        if json_response['itemsRemaining'] > 0:
            payload['CurrentPage'] += 1
            req = Request.from_url(
                url=url,
                method="POST",
                headers=headers,
                payload=urlencode(payload),
                # I'm using the extended unique key here as well
                use_extended_unique_key=True,
            )
            await context.add_requests([req])

    # Run the crawler
    await crawler.run([initial_req])

if __name__ == '__main__':
    asyncio.run(main())

With the above 👆 code I'm having this issue 👇:

I have fixed it in this way in the file crawlee._utils.crypto:

def compute_short_hash(data: bytes, *, length: int = 8) -> str:
    """Computes a hexadecimal SHA-256 hash of the provided data and returns a substring (prefix) of it.

    Args:
        data: The binary data to be hashed.
        length: The length of the hash to be returned.

    Returns:
        A substring (prefix) of the hexadecimal hash of the data.
    """
    logger.debug(f"Data: {data}")
    logger.debug(f"data Type: {type(data)}")
    data = str(data).encode('utf-8')
    logger.debug(f"Data bytes: {data}")
    hash_object = sha256(data)
    return hash_object.hexdigest()[:length]

After the above fix I have added, this is the complete log:

Here the complete log :

❯ python src/test_clients.py                                                                                                                                                  ─╯
[crawlee._utils.recurring_task] DEBUG Calling RecurringTask.__init__(func=_log, delay=0:01:00)...
[crawlee._utils.crypto] DEBUG Data: ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=1&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh
[crawlee._utils.crypto] DEBUG data Type: <class 'str'>
[crawlee._utils.crypto] DEBUG Data bytes: b'ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=1&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh'
[crawlee._utils.crypto] DEBUG Data: b'content-type:application/x-www-form-urlencoded'
[crawlee._utils.crypto] DEBUG data Type: <class 'bytes'>
[crawlee._utils.crypto] DEBUG Data bytes: b"b'content-type:application/x-www-form-urlencoded'"
[crawlee.storages._request_queue] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='QTDBO9YnbNFBG4s', unique_key='POST|6382465e|3fd3ad83|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee.memory_storage_client._memory_storage_client] DEBUG Storage was already purged on start.
[crawlee._autoscaling.autoscaled_pool] DEBUG Starting the pool
[crawlee.http_crawler._http_crawler] INFO  Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [0]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 0.0      │
│ requests_total                │ 0        │
│ crawler_runtime               │ 0.025215 │
└───────────────────────────────┴──────────┘
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[crawlee._utils.system] DEBUG Calling get_cpu_info()...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.statistics._statistics] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue] DEBUG Queue head still returned requests that need to be processed (or that are locked by other clients)
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "QTDBO9YnbNFBG4s", "unique_key": "POST|6382465e|3fd3ad83|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["QTDBO9YnbNFBG4s"]})
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "QTDBO9YnbNFBG4s", "unique_key": "POST|6382465e|3fd3ad83|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["QTDBO9YnbNFBG4s"]})
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "QTDBO9YnbNFBG4s", "unique_key": "POST|6382465e|3fd3ad83|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee.http_crawler._http_crawler] INFO  Processing https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 ...
[crawlee.http_crawler._http_crawler] INFO  Total items: 22
[crawlee.http_crawler._http_crawler] INFO  totalCount: 76
[crawlee.http_crawler._http_crawler] INFO  currentPage: 1
[crawlee.http_crawler._http_crawler] INFO  pageSize: 20
[crawlee.http_crawler._http_crawler] INFO  itemsRemaining: 56
[crawlee._utils.crypto] DEBUG Data: ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=2&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh
[crawlee._utils.crypto] DEBUG data Type: <class 'str'>
[crawlee._utils.crypto] DEBUG Data bytes: b'ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=2&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh'
[crawlee._utils.crypto] DEBUG Data: b'content-type:application/x-www-form-urlencoded'
[crawlee._utils.crypto] DEBUG data Type: <class 'bytes'>
[crawlee._utils.crypto] DEBUG Data bytes: b"b'content-type:application/x-www-form-urlencoded'"
[crawlee.storages._request_queue] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='NnSc70HhiswcrCR', unique_key='POST|6382465e|33002a2a|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG Queue head still returned requests that need to be processed (or that are locked by other clients)
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "NnSc70HhiswcrCR", "unique_key": "POST|6382465e|33002a2a|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["NnSc70HhiswcrCR"]})
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "NnSc70HhiswcrCR", "unique_key": "POST|6382465e|33002a2a|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system] DEBUG Calling get_cpu_info()...
[crawlee._utils.system] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.http_crawler._http_crawler] INFO  Processing https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 ...
[crawlee.http_crawler._http_crawler] INFO  Total items: 21
[crawlee.http_crawler._http_crawler] INFO  totalCount: 76
[crawlee.http_crawler._http_crawler] INFO  currentPage: 2
[crawlee.http_crawler._http_crawler] INFO  pageSize: 20
[crawlee.http_crawler._http_crawler] INFO  itemsRemaining: 36
[crawlee._utils.crypto] DEBUG Data: ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=3&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh
[crawlee._utils.crypto] DEBUG data Type: <class 'str'>
[crawlee._utils.crypto] DEBUG Data bytes: b'ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=3&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh'
[crawlee._utils.crypto] DEBUG Data: b'content-type:application/x-www-form-urlencoded'
[crawlee._utils.crypto] DEBUG data Type: <class 'bytes'>
[crawlee._utils.crypto] DEBUG Data bytes: b"b'content-type:application/x-www-form-urlencoded'"
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["NnSc70HhiswcrCR"]})
[crawlee.storages._request_queue] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='YCtBXrRIetjQpN5', unique_key='POST|6382465e|e6d2f7de|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "NnSc70HhiswcrCR", "unique_key": "POST|6382465e|33002a2a|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["YCtBXrRIetjQpN5"]})
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "YCtBXrRIetjQpN5", "unique_key": "POST|6382465e|e6d2f7de|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee.http_crawler._http_crawler] INFO  Processing https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 ...
[crawlee.http_crawler._http_crawler] INFO  Total items: 20
[crawlee.http_crawler._http_crawler] INFO  totalCount: 76
[crawlee.http_crawler._http_crawler] INFO  currentPage: 3
[crawlee.http_crawler._http_crawler] INFO  pageSize: 20
[crawlee.http_crawler._http_crawler] INFO  itemsRemaining: 16
[crawlee._utils.crypto] DEBUG Data: ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=4&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh
[crawlee._utils.crypto] DEBUG data Type: <class 'str'>
[crawlee._utils.crypto] DEBUG Data bytes: b'ShowAllTickets=True&HideDuplicateTicketsV2=False&Quantity=2&IsInitialQuantityChange=False&PageSize=20&CurrentPage=4&SortBy=NEWPRICE&SortDirection=0&Sections=&Rows=&Seats=&SeatTypes=&TicketClasses=&ListingNotes=&PriceRange=0%2C100&InstantDelivery=False&EstimatedFees=True&BetterValueTickets=True&PriceOption=&HasFlexiblePricing=False&ExcludeSoldListings=False&RemoveObstructedView=False&NewListingsOnly=False&PriceDropListingsOnly=False&SelectBestListing=False&ConciergeTickets=False&Method=IndexSh'
[crawlee._utils.crypto] DEBUG Data: b'content-type:application/x-www-form-urlencoded'
[crawlee._utils.crypto] DEBUG data Type: <class 'bytes'>
[crawlee._utils.crypto] DEBUG Data bytes: b"b'content-type:application/x-www-form-urlencoded'"
[crawlee.storages._request_queue] DEBUG Added 1 requests to the queue, response: processed_requests=[ProcessedRequest(id='yBqnykRpfz7TQSc', unique_key='POST|6382465e|e2033248|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088', was_already_present=False, was_already_handled=False)] unprocessed_requests=[]
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG Queue head still returned requests that need to be processed (or that are locked by other clients)
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Scheduling a new task
[crawlee.storages._request_queue] DEBUG There are still ids in the queue head that are pending processing ({"queue_head_ids_pending": 1})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new tasks - already running at desired concurrency
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "yBqnykRpfz7TQSc", "unique_key": "POST|6382465e|e2033248|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee.storages._request_queue] DEBUG There are still requests in progress (or zombie) ({"in_progress": ["yBqnykRpfz7TQSc"]})
[crawlee.storages._request_queue] DEBUG Skipping request from queue head, already in progress or recently handled ({"id": "yBqnykRpfz7TQSc", "unique_key": "POST|6382465e|e2033248|https://www.viagogo.es/entradas-deportes/futbol/real-madrid-c-f-entradas/e-153769088", "in_progress": true, "recently_handled": false})
[crawlee._autoscaling.autoscaled_pool] DEBUG Not scheduling new task - no task is ready
[crawlee._utils.system] DEBUG Calling get_cpu_info()...
[crawlee._utils.system] DEBUG Calling get_memory_info()...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Awaiting listener task...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Listener task completed.
[crawlee.events._event_manager] DEBUG LocalEventManager.on.listener_wrapper(): Removing listener task from the set...
[crawlee.http_crawler._http_crawler] INFO  Processing https://www.viagogo.es/Entradas-Deportes/Futbol/Real-Madrid-C-F-Entradas/E-153769088 ...
[crawlee.http_crawler._http_crawler] INFO  Total items: 16
[crawlee.http_crawler._http_crawler] INFO  totalCount: 76
[crawlee.http_crawler._http_crawler] INFO  currentPage: 4
[crawlee.http_crawler._http_crawler] INFO  pageSize: 20
[crawlee.http_crawler._http_crawler] INFO  itemsRemaining: 0
[crawlee._autoscaling.autoscaled_pool] DEBUG Worker task finished
[crawlee._autoscaling.autoscaled_pool] DEBUG `is_finished_function` reports that we are finished
[crawlee._autoscaling.autoscaled_pool] DEBUG Terminating - no running tasks to wait for
[crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
[crawlee._autoscaling.autoscaled_pool] DEBUG Pool cleanup finished
[crawlee.statistics._statistics] DEBUG Persisting state of the Statistics (event_data=is_migrating=False).
[crawlee.http_crawler._http_crawler] INFO  Final request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 4        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [4]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ 0.611377 │
│ requests_finished_per_minute  │ 93       │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 2.445509 │
│ requests_total                │ 4        │
│ crawler_runtime               │ 2.593437 │
└───────────────────────────────┴──────────┘

[Mantisus]

Hey @francomanca93, payload should be bytes

payload_bytes = urlencode(payload).encode()

# ----- and-----

req = Request.from_url(
    url=url,
    method="POST",
    headers=headers,
    payload=urlencode(payload).encode(),
    # I'm using the extended unique key here as well
    use_extended_unique_key=True,
)

will fix it for you.

[francomanca93]

Thank you @Mantisus, that fixed my issue.