[Twitter] Scraping search 401's and even SUSPENDS your account!

[kosshi-net]

I've been hitting this issue constantly past few days. I am trying to download old content inaccessible via /media (the tweet limit issue) by using the search (@from:account since:date until:date &f=media), but it's impossible. You get the 15 minute ratelimit every 10 posts, and get logged out of all your accounts (causing the 401) if you let it run long enough.

But even worse, if you keep logging in and trying, you may get slapped with "Violating our rules against evading suspension. ". I've never ever been suspended from twitter, so this is an automatic false positive (which I got reversed by appeal thankfully).
Now I've created alt accounts to keep trying without risking my main, and they all get suspended pretty quickly for the same reason. And I've tried from different IP addresses and changed the user-agent of gallery-dl, but it still keeps suspending me. (Note that if I just normally download /media etc, the accounts wont get suspended)

The extremely aggressive ratelimit, singing out and suspensions, all of this combined makes me think whatever gallery-dl is doing while scraping search results, is tripping something at twitter.

If you just use the offical website to do the search, copy the URLs of results and gallery-dl those, everything goes smoothly. Wont hit any ratelimits.

Using gallery-dl 1.26.9

[a84r7a3rga76fg]

Are you still getting banned? I got banned on several accounts between April 29 and April 30. I'm also not getting rate limited anymore.

These are my cooldowns

			"sleep-request": 2,
			"sleep": 2,

[kosshi-net]

Are you still getting banned? I got banned on several accounts between April 29 and April 30. I'm also not getting rate limited anymore.

These are my cooldowns

			"sleep-request": 2,
			"sleep": 2,

If I scrape without sleeps, I still get aggressively ratelimited just like before. Didn't test if it still leads to 401 or suspensions.
With --sleep 2 --sleep-request 2 I get ratelimited after 50-100 files, not the 10-20 like before, and I've yet to be logged out once. This seems to be a working solution, though I haven't tested this for long enough to be sure. But thanks!

If I don't get suspended or 401'd with these sleeps in the next few days, I will close this issue.

[Twi-Hard]

This is what I've been doing to get no rate limiting at all which I assume lowers the chance they'll have a problem with me.

"sleep": 5
"sleep-request": 30

I tried "sleep": 4 and "sleep-request": 20 but towards the end of the day it rate limited me until the next day.

[mikf]

Maybe it would help if gallery-dl waited for Twitter's rate limits to reset before it hits a 429 error, i.e. before x-rate-limit-remaining reaches 0?

Also, API requests sent by a browser include a x-client-transaction-id header, which gallery-dl currently does not include. Mostly because I have no idea how to generate its values and reading Twitter's minimized JS code to find out how is a pain.

[Twi-Hard]

I assume that could ensure we go the maximum speed we can go? That would be really nice (even if it doesn't do that part)

[Butterfly-Dragon]

could we set the wait time? like:

"ratelimit": "wait:3600",

waits 1 hour between retries or something? Thanks.

[Scripter17]

Also, API requests sent by a browser include a x-client-transaction-id header, which gallery-dl currently does not include. Mostly because I have no idea how to generate its values and reading Twitter's minimized JS code to find out how is a pain.

I don't suppose you already looked through Obfio's work on the matter? If it's outdated I'd be willing to have a go at it.

https://github.com/obfio/twitter-tid-deobf

https://github.com/obfio/twitter-tid-generator

[mikf]

@Butterfly-Dragon 162d426

[mikf]

I've finally managed to actually sit down and read through all three blog posts, only to arrive at

It's completely useless.

The header doesn't make Twitter "trust" your request any more or less. I guess it's just there to gather data right now,

On one hand, it would have been really nice having a potential solution to all the logout/account suspended/etc problems. On the other, it's kind of nice knowing that I don't have to put in any more work in this regard for the time being.

[jojo2833432]

So what would be the solution? I had about 3 throwaway accounts given permanent suspensions. I tried the --sleep 3-10 and the --sleep-request method. But still was suspended.

[NCLnclNCL]

Also, API requests sent by a browser include a x-client-transaction-id header, which gallery-dl currently does not include. Mostly because I have no idea how to generate its values and reading Twitter's minimized JS code to find out how is a pain.

I don't suppose you already looked through Obfio's work on the matter? If it's outdated I'd be willing to have a go at it.

https://github.com/obfio/twitter-tid-deobf

https://github.com/obfio/twitter-tid-generator

yes, is it