[FANBOX] Occasional 403 Forbidden for posts (both public and restricted) using proper/fresh cookies #8976
Description
(Before you spend any time on this, this is not a critical issue since it typically solves itself. Thanks for your hard work and patience!)
Possibly normal behavior but gallery-dl tends to run into 403s (CF? stops before switching to nginx) even for public posts. Here's a --print-traffic redacted dump when this happens and another to compare with a successful query (added 'comments' to highlight what was on the redacted lines)
OS: W10 x64
Cookie origin: Browser (FF ESR latest)
Python and libs: Python 3.13.11, requests 2.32.5, urllib3 2.6.3
gallery-dl versions tested:
fd6bc39
17e1d25
>gallery-dl https://official.fanbox.cc/posts/* --print-traffic
403 occurrence:
send: b'GET *'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Date: *
header: Content-Type: text/html; charset=UTF-8
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: X-Frame-Options: SAMEORIGIN
header: Referrer-Policy: same-origin
header: Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
header: Expires: *
header: Set-Cookie: * //CF
header: Vary: Accept-Encoding
header: Server: cloudflare
header: CF-RAY: *
header: Content-Encoding: br
header: alt-svc: h3=":443"; ma=86400
[fanbox][warning] Skipping post * (HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.info?postId=*')
Successful response (both can happen in the same CLI call if 2+ posts are passed as arguments or an URL list)
send: b'GET *'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: *
header: Content-Type: application/json; charset=utf-8
header: Content-Length: *
header: Connection: keep-alive
header: CF-RAY: *
header: content-security-policy: frame-ancestors 'self' https://*.fanbox.cc
header: vary: X-UserId,Accept-Encoding
header: access-control-allow-origin: https://www.fanbox.cc
header: access-control-allow-headers: Content-Type, X-CSRF-Token
header: access-control-allow-credentials: true
header: expires: *
header: Cache-Control: no-store, no-cache, must-revalidate
header: pragma: no-cache
header: Set-Cookie: * //FBSSID
header: Set-Cookie: * //PPA
header: Set-Cookie: * //PPN
header: Set-Cookie: * //CF
header: x-userid: *
header: x-content-type-options: nosniff
header: Content-Encoding: gzip
header: x-frame-options: SAMEORIGIN
header: cf-cache-status: DYNAMIC
header: Server: cloudflare
header: alt-svc: h3=":443"; ma=86400
send: b'GET *'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: *
header: Content-Type: image/jpeg
header: Content-Length: *
header: Connection: keep-alive
header: X-Content-Type-Options: nosniff
header: Expires: *
header: Cache-Control: max-age=86400
header: Age: *
header: Via: http/1.1 f003 (second), http/1.1 f055 (second)
.\gallery-dl\fanbox\official\*_000.jpg
Maybe high traffic causing CF-ch resolution to fail since this typically works as intended even within the same query and also works fine a few hours later (gallery-dl can't do much about the former since that would be a server issue; if the latter and maybe some odd 429 case - didn't try more than two small posts and hardly ever downloading much to avoid hard flags - that might as well just be the Fanbox API returning a 403 when it shouldn't). Not really thinking of tainted/blacklisted IPs (posts would likely all fail instantly at the CF-ch resolution level) but not ruling that out either, CF has been blacklisting a lot of rays from rude/excessive use (mostly bots but also human users who forget to go easy on the servers).
conf is set to be as polite as possible to again avoid any flags.
Thanks in advance if you have any thoughts about this occasional issue - and apologies for the very long post!