Some Internet Archive crawlers are challenged

[benselme]

Describe the bug

We have noticed that some of our pages in the Wayback machine show the Anubis challenge page, like this one:

https://web.archive.org/web/20260718175718/https://www.erudit.org/en/journals/ae/2012-v88-n4-ae01274/1023796ar/

But not all: https://web.archive.org/web/20260728190436/https://www.erudit.org/fr/revues/ae/2020-v96-n4-ae06831/1087010ar/

The problem might be that the IP addresses list in /data/crawlers/internet-archive.yaml is not up to date. At least this site: https://sourcecidr.com/internet-archive/ shows additional addresses for Internet Archive's crawlers.

Steps to reproduce

Go to this address: https://web.archive.org/web/20260718175718/https://www.erudit.org/en/journals/ae/2012-v88-n4-ae01274/1023796ar/

Expected behavior

It should show an archived version of this page: https://www.erudit.org/en/journals/ae/2012-v88-n4-ae01274/1023796ar/
And not an archived version of the Anubis challenge page

Your operating system and its version.

Kubuntu 24.04

Your browser and its version.

Firefox 153

Additional context

No response

[rimrul]

At least this site: https://sourcecidr.com/internet-archive/ shows additional addresses for Internet Archive's crawlers.

That site seems to be mixing up AS7941 (Internet archive, archive.org) and AS399784 (Internet archive Canada, internetarchivecanada.org) two seemingly unrelated entities.

https://bgp.he.net/AS7941#_prefixes lists only 207.241.224.0/20, 208.70.24.0/21 and 2620:0:9c0::/48