niyaniya.moe not working after site revamp

[h36asdy45]

https://niyaniya.moe has had a site revamp and no longer works, below is the logs after a basic download.

C:\Windows\system32>"C:\Users\vboxuser\Documents\gallery-dl.exe" https://niyaniya.moe/g/12487/a8529b1b286d --verbose
[gallery-dl][debug] Version 1.28.5 - Executable (stable/windows)
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.3
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://niyaniya.moe/g/12487/a8529b1b286d'
[koharu][debug] Using KoharuGalleryExtractor for 'https://niyaniya.moe/g/12487/a8529b1b286d'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.schale.network:443
[urllib3.connectionpool][debug] https://api.schale.network:443 "GET /books/detail/12487/a8529b1b286d HTTP/11" 200 1751
[koharu][error] An unexpected error occurred: KeyError - 'data'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[koharu][debug]
Traceback (most recent call last):
File "gallery_dl\job.pyc", line 152, in run
File "gallery_dl\extractor\common.pyc", line 713, in items
File "gallery_dl\extractor\koharu.pyc", line 143, in images
KeyError: 'data'

[WorldTeacher]

Adding on to this:
[gallery-dl][debug] Version 1.28.5 - Executable (stable/linux)
[gallery-dl][debug] Python 3.10.12 - Linux-6.13.2-arch1-1-x86_64-with-glibc2.41
[gallery-dl][debug] requests 2.31.0 - urllib3 2.3.0
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://niyaniya.moe/?cat=4'
[koharu][debug] Using KoharuSearchExtractor for 'https://niyaniya.moe/?cat=4'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.schale.network:443
[urllib3.connectionpool][debug] https://api.schale.network:443 "GET /books?cat=4&page=1 HTTP/1.1" 200 5162
[koharu][error] An unexpected error occurred: KeyError - 'public_key'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[koharu][debug]
Traceback (most recent call last):
File "gallery_dl/job.py", line 152, in run
File "gallery_dl/extractor/koharu.py", line 53, in _pagination
KeyError: 'public_key'

[lIIIIllIllllllIl]

I can confirm this bug is still on the latest release.

[viatana35]

The issue is not related to the site revamp. Instead, it appears to be due to changes in the API. After reviewing the source code, it seems that the API no longer provides all the required data needed for the download libraries to function properly.

[mikf]

It's a bit more complicated then that, I'm afraid.

Basic gallery information can be retrieved with the currently used

GET
https://api.schale.network/books/detail/14216/6c67076fdd45

but to get actual download links / information, it needs a crt value which can only be gotten by doing some form of Cloudflare challange in the background.

POST
https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/b/flow/ov1/1756740490:1740298494:jN8hC35aQ84qsHFx3DVwriGusWbWBaFrW2Vke6ddoWE/9165d4262f2de505/VctHFZ87yyAsCpvZ9.QMJfW1WVfNxisUZpV.WvnQ5ds-1740299162-1.1.1.1-Ksk2OCetizxHJxpjOTsPHsBUxbmqAGKj85gpfSaB2YwZfxFunJPqk3OsyFbgz_Bt

# gets `crt` value
POST
https://auth.schale.network/clearance

# gets `data` aka download information
POST
https://api.schale.network/books/detail/14216/6c67076fdd45?crt=32286069-9823-4a83-ab73-2aa265c82b11

[a84r7a3rga76fg]

Will it work if we import cookies?

[mikf]

Fixed in 1b9e1ff, but downloading galleries now requires a crt value as well as your browser's user-agent thanks to Cloudflare "protection".

[a84r7a3rga76fg]

How to get the crt value?

edit: Nvm, I ctrl+F5'd the configuration page and see how now. I was banned after downloading two small galleries. I evaded the ban and tried to get the last 20 gallery URLs, I was banned before any gallery URL had been retrieved.