[Twitter] Timeline extraction by search may stop early due to empty response

[fireattack]

This issue seems to stem from Twitter's own GraphQL API, but I believe there is still room for improvement in how gallery-dl handles it.

When downloading a user's timeline via search, the API sometimes returns an empty response even though the user still has older tweets available.

Unfortunately, it's difficult to establish a reliable, reproducible test case. Yesterday, I could reproduce it consistently with one account using the following command and configuration:

gallery-dl https://x.com/ilife_official --verbose --cookies /my/cookies.txt -d D:\nodoka_tweet -o cursor=2_1835703201129201872/DAADDAABCgABGXm6VeBagNAKAAIYRwESNZpwLQAIAAIAAAACCAADAAAAAAgABAAAABYKAAUbQScxjcAnEAoABhtBJzGNvH2QAAA

{
    "extractor": {
        "base-directory": "./",
        "url-metadata": "gdl_url",
        "path-metadata": "gdl_path",
        "twitter": {
            "directory": [""],
            "filename": "{author['name']}-{tweet_id}-{date:%Y%m%d_%H%M%S}UTC-{num}.{extension}",
            "retweets": "original",
            "text-tweets": false,
            "pinned": true,
            "quoted": true,
            "unavailable": true
        },
...

[gallery-dl][debug] Version 1.30.6-dev - Git HEAD: bfee2a37
[gallery-dl][debug] Python 3.11.1 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.32.3 - urllib3 2.3.0
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/ilife_official'
[twitter][debug] Using TwitterUserExtractor for 'https://x.com/ilife_official'
[twitter][debug] Using TwitterTimelineExtractor for 'https://x.com/ilife_official/timeline'
[twitter][debug] cookies: Loading cookies from 'D:\temp\twittercookies.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/ck5KkZ8t5cOmoLssopN99Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22ilife_official%22%2C%22withGrokTranslatedBio%22%3Afalse%7D&features=%7B%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Atrue%2C%22subscriptions_feature_can_gift_premium%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22subscriptions_verification_info_is_identity_verified_enabled%22%3Atrue%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%7D&fieldToggles=%7B%22withAuxiliaryUserLabels%22%3Atrue%7D HTTP/1.1" 200 1610
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/4fpceYZ6-YQCx_JSl_Cn_A/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3AiLiFE_official+max_id%3A1835703201129201872+include%3Aretweets+include%3Anativeretweets+filter%3Alinks%22%2C%22count%22%3A100%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Latest%22%2C%22withGrokTranslatedBio%22%3Afalse%2C%22cursor%22%3A%22DAADDAABCgABGXm6VeBagNAKAAIYRwESNZpwLQAIAAIAAAACCAADAAAAAAgABAAAABYKAAUbQScxjcAnEAoABhtBJzGNvH2QAAA%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 None
D:\nodoka_tweet\iLiFE_official-1749013132310454752-20240121_101640UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1748948933219590159-20240121_060134UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1748674580393398274-20240120_115123UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1748674580393398274-20240120_115123UTC-2.jpg
D:\nodoka_tweet\iLiFE_official-1748649368624992584-20240120_101112UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1748576484988445021-20240120_052135UTC-1.jpg
[twitter][debug] Skipping quote of 1748342377876529644 (deleted)
D:\nodoka_tweet\iLiFE_official-1748299560043323739-20240119_110111UTC-1.mp4
D:\nodoka_tweet\iLiFE_official-1747965642882646264-20240118_125419UTC-1.jpg
[twitter][debug] Skipping quote of 1747944475966750750 (deleted)
D:\nodoka_tweet\iLiFE_official-1747631265799385147-20240117_144537UTC-1.mp4
D:\nodoka_tweet\iLiFE_official-1747614539657699622-20240117_133910UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1747614539657699622-20240117_133910UTC-2.jpg
D:\nodoka_tweet\iLiFE_official-1747559548066697568-20240117_100039UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1747215795078431074-20240116_111441UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746869994368155671-20240115_122036UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746842506510361037-20240115_103122UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746827080904720458-20240115_093005UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746525049727893670-20240114_132955UTC-1.mp4
D:\nodoka_tweet\iLiFE_official-1746487782363783482-20240114_110150UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746479681988440449-20240114_102938UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1746125112389304483-20240113_110042UTC-1.jpg
[twitter][debug] Cursor: 2_1835703201129201872/DAADDAABCgABGXm6VeBagNAKAAIYO3uJ0VqAowAIAAIAAAACCAADAAAAAAgABAAAABcKAAUbQScxjcAnEAoABhtBJzGNvFaAAAA
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/4fpceYZ6-YQCx_JSl_Cn_A/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3AiLiFE_official+max_id%3A1835703201129201872+include%3Aretweets+include%3Anativeretweets+filter%3Alinks%22%2C%22count%22%3A100%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Latest%22%2C%22withGrokTranslatedBio%22%3Afalse%2C%22cursor%22%3A%22DAADDAABCgABGXm6VeBagNAKAAIYO3uJ0VqAowAIAAIAAAACCAADAAAAAAgABAAAABcKAAUbQScxjcAnEAoABhtBJzGNvFaAAAA%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 None
D:\nodoka_tweet\iLiFE_official-1746109956712182213-20240113_100029UTC-1.jpg
[twitter][debug] Skipping quote of 1745982872362877425 (deleted)
D:\nodoka_tweet\iLiFE_official-1745056175119643103-20240110_121308UTC-1.mp4
D:\nodoka_tweet\iLiFE_official-1744962381854494740-20240110_060026UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1744962381854494740-20240110_060026UTC-2.jpg
D:\nodoka_tweet\iLiFE_official-1744661514861310193-20240109_100454UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1744661514861310193-20240109_100454UTC-2.jpg
D:\nodoka_tweet\iLiFE_official-1744308807461908748-20240108_104322UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1744308807461908748-20240108_104322UTC-2.jpg
[twitter][debug] Skipping quote of 1744268781625692357 (deleted)
D:\nodoka_tweet\iLiFE_official-1743976703410757987-20240107_124342UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743958679039393804-20240107_113204UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743943431439937563-20240107_103129UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743922041651429676-20240107_090629UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743905556887380365-20240107_080059UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743890776319508506-20240107_070215UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743873632705863964-20240107_055408UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743847310822641758-20240107_040932UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743831170809565367-20240107_030524UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743618584188752154-20240106_130039UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1743610453345898628-20240106_122821UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1742848392563441997-20240104_100012UTC-1.jpg
D:\nodoka_tweet\iLiFE_official-1742848392563441997-20240104_100012UTC-2.jpg
[twitter][debug] Cursor: 2_1835703201129201872/DAADDAABCgABGXm6VeBagNAKAAIYL9dhCBrBTQAIAAIAAAACCAADAAAAAAgABAAAABgKAAUbQScxjcAnEAoABhtBJzGNvC9wAAA
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/4fpceYZ6-YQCx_JSl_Cn_A/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3AiLiFE_official+max_id%3A1835703201129201872+include%3Aretweets+include%3Anativeretweets+filter%3Alinks%22%2C%22count%22%3A100%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Latest%22%2C%22withGrokTranslatedBio%22%3Afalse%2C%22cursor%22%3A%22DAADDAABCgABGXm6VeBagNAKAAIYL9dhCBrBTQAIAAIAAAACCAADAAAAAAgABAAAABgKAAUbQScxjcAnEAoABhtBJzGNvC9wAAA%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 320

As you can see, the process stops at a tweet from 20240104_100012UTC, even though the account still has tweets going back to 2020. The reason is that the API response contains zero tweets, which triggers the early return here:

gallery-dl/gallery_dl/extractor/twitter.py

Line 2071 in bfee2a3

return extr._update_cursor(None)

Suggested improvement: when this situation occurs (an empty response even though older tweets exist), gallery-dl could attempt the search again using an updated max_id, specifically the lowest tweet ID retrieved in the previous batch. This would prevent the false positive "end of timeline" condition and allow the extraction to continue until the true end of the user’s timeline.

[mikf]

Searches no longer stop immediately when receiving no Tweets: f94eedb
and I've tried implementing your max_id suggestion: d182749

I tested both cursor-based and max_id-based pagination on one whole account and got the exact same 990 results in each case.

[fireattack]

Searches no longer stop immediately when receiving no Tweets: f94eedb and I've tried implementing your max_id suggestion: d182749

I tested both cursor-based and max_id-based pagination on one whole account and got the exact same 990 results in each case.

Thanks! For the first change, I assume it will just try it a few more times? In my case (which unfortunately cannot reproduce anymore) it won't help because that cursor consistently return false positive of end; but it's good to have it I guess.

But the second change definitely would be helpful!

[mikf]

For the first change, I assume it will just try it a few more times?

No, it will continue with the next internal cursor value, even if the current batch of results is empty, i.e. it won't immediately trigger the early return you pointed out.

[fireattack]

Thanks, I see.

I don't know if the cursor in these empty response(s) would be actually updated though (I forgot to save some before), but we will see.

[mikf]

If it is not updated and stays the same as before, the other return condition will trigger:

gallery-dl/gallery_dl/extractor/twitter.py

Lines 2091 to 2092 in d182749

	if not cursor or cursor == variables.get("cursor"):
	return extr._update_cursor(None)

[fireattack]

Thanks for the continuous refining. I still haven't encountered the false-positive again, but at least when the "true" end of the TL was reached the cursor does change even if its empty, FWIW.

[nisehime]

Oh man, I always thought gallery-dl was doing those attempts to get more from the search before stopping? Wasn't somebody talking about it in the past, because it's kinda an old problem. Now I'm paranoided that I missed something

Edit
Yeah, it was here #544 and I was explaining it to hitomi-downloader dev back then lol. I guess the min_position thing is not here anymore?

[nisehime]

The search issue is even worse than I thought. First of all, using filter:media or filter:links + max_id: for certain user gives me no results. Not that surprising since it's also an old issue with twitter search, which is why I used to suggest making a query without filter. However, last time I did so I was getting rate limited quickly since Elon changes.

I wanted to try it out right now just to confirm that's still the case, but then I got something else.
https://x.com/search?q=from%3Abambi2000line max_id%3A1739556385141256653&src=typed_query&f=live

This search works for me in the browser (I get results), but doesn't work in gallery-dl (no results). What gives? Endpoint differences?

Also, it would be nice to have a log message when there's no results in the search at all when using timeline extractor (ie media + search).

[mikf]

This search works for me in the browser (I get results), but doesn't work in gallery-dl (no results). What gives? Endpoint differences?

gallery-dl/gallery_dl/extractor/twitter.py

Line 1457 in 88bfc09

"count": 100,

gallery-dl gets results when this value is lowered to a more "reasonable" amount, like 20 (the same as a browser uses).

[nisehime]

"count": 100,

From what I can tell this doesn't even work anyway, you still get 20 tweets or something in response, so safe to change it to 20 I guess.