Simplify usage of cookies

[barjin]

While it's currently possible to work with Cookies (by manually updating the HTTP headers of the request and parsing for the set-cookie headers in the response) in both impit language bindings, it's far from ergonomic.

httpx (in Python) exposes their own wrapper over CookieJar, which might be a good starting point.

The fetch API in JS doesn't seem to have a dedicated cookies option, but adding one (something like fetch('https://example.org', {cookies: {...}})) might actually result in a net positive (despite diverting from the original API).

[barjin]

One of the bigger design questions is - who should own the cookie jar?

Most (if not all) of the impit methods are fully stateless (besides the cache storing the h3-enabled servers). This matches the Crawlee design, where the cookies are stored in the Session class (and therefore rotating the Session switches up the cookies too).

This is different from what httpx does by default:

# courtesy of @Mantisus 
import httpx

client = httpx.Client()

response = client.get("https://httpbin.org/cookies/set?a=1")
response = client.get("https://httpbin.org/cookies/set?b=2")
response = client.get("https://httpbin.org/cookies")

print(response.json()) # {'cookies': {'a': '1', 'b': '2'}}

Even without specifying any cookies= arguments, the client stores the cookies and attaches them to the subsequent requests.

[Mantisus]

I would say that if we want to follow the httpx API. This is a pretty important part of it 🙂.

But maybe @B4nan, @vdusek, @janbuchar will have other thoughts.

[B4nan]

Technically, that's behavior, not API, right? I would stay away from that personally, I like the statelessness we have now, but maybe others will have different opinions.

If we would do this, it should work the same in the JS version, and it would need to be configurable so we can disable that in crawlee.

[barjin]

that's behavior, not API, right

True, but IMO the end goal of impit is for the users to be able to do something like

- from httpx import Client
+ from impit import Client

or

import { fetch } from 'impit'

and fix all their blocking issues in either language. I'm not too sure how fetch handles cookies (a quick Google search says it likely doesn't).

+1 for opt-out, I also like the stateless behavior better, but 🤷‍♂ whether that should be the default for Python impit.

[garysassano]

I requested a similar feature for Bun's Cookie API.

One of the bigger design questions is - who should own the cookie jar?

I don't know about Rust, but for Node there's a very well-maintained library by Salesforce: tough-cookie