[twitter] download stops halfway with no error

[nisehime]

https://x.com/haimurakiyotaka

Executed:

gallery-dl -A 1 -o output.skip=false -o extractor.modules=twitter -o extractor.twitter.quoted="false" -o retweets="false" -o output.logfile.format="[{asctime}][{name}][{levelname}] {message} [url={extractor.url},id={keywords.tweet_id}]" -o output.logfile.path="./logs/twitter_log.txt" https://x.com/id:1223536376307568640/

The last downloaded id was 1622185556515110915 after that I just got Use '-o cursor=1/DAABCgABG2Cd1Gn__-4KAAIWuqZK2pqAAQgAAwAAAAIAAA' to continue downloading from the current position with no errors, it just stopped. Trying again with cursor still gives just info log message with nothing else

F:\gallery-dl>gallery-dl -A 1 -o output.skip=false -o extractor.modules=twitter -o extractor.twitter.quoted="false" -o retweets="false" -o output.logfile.format="[{asctime}][{name}][{levelname}] {message} [url={extractor.url},id={keywords.tweet_id}]" -o output.logfile.path="./logs/twitter_log.txt" -o "cursor=1/DAABCgABG2Cd1Gn__-4KAAIWuqZK2pqAAQgAAwAAAAIAAA" -v https://x.com/id:1223536376307568640/
[gallery-dl][debug] Version 1.30.8 - Executable (stable/windows)
[gallery-dl][debug] Python 3.13.7 - Windows-8.1-6.3.9600-SP0
[gallery-dl][debug] requests 2.32.5 - urllib3 2.5.0
[gallery-dl][debug] Configuration Files ['F:\\gallery-dl\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/id:1223536376307568640/'
[twitter][debug] Using TwitterUserExtractor for 'https://x.com/id:1223536376307568640/'
[twitter][debug] Using TwitterTimelineExtractor for 'https://x.com/id:1223536376307568640/timeline'
[twitter][debug] cookies: Loading cookies from './cookies/twitter_cookies.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/8r5oa_2vD0WkhIAOkY4TTA/UserByRestId?variables=%7B%22userId%22%3A%221223536376307568640%22%7D&features=%7B%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Atrue%2C%22subscriptions_feature_can_gift_premium%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D HTTP/1.1" 200 1136←[0m
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/jCRhbOzdgOHp6u9H4g2tEg/UserMedia?variables=%7B%22userId%22%3A%221223536376307568640%22%2C%22count%22%3A100%2C%22includePromotedContent%22%3Afalse%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22cursor%22%3A%22DAABCgABG2Cd1Gn__-4KAAIWuqZK2pqAAQgAAwAAAAIAAA%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atru
e%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_wi
th_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D HTTP/1.1" 200 17182
[twitter][debug] Using download archive './archive/twitter_db.sqlite3'
[twitter][info] Use '-o cursor=1/DAABCgABG2Cd1Gn__-4KAAIWuqZK2pqAAQgAAwAAAAIAAA' to continue downloading from the current position

config

{

    "extractor": {

        "base-directory": "./galleries/",

        "twitter": {
            "cookies": "./cookies/twitter_cookies.txt",
            "directory": ["twitter", "{author[name]} ({author[id]})"],
            "filename": "[{date:%y-%m-%d}] {tweet_id}_p{num}.{extension}",
            "archive": "./archive/twitter_db.sqlite3",
            "actions": { "debug:Skipping \\d+ \\(deleted\\)" : "level = info" },
            "twitpic": true,
            "retweets": true,
            "quoted": true,
            "unavailable": true,
            "tweet": {
                  "directory": ["twitter_tweet", "{author[name]} ({author[id]})"],
                  "archive": null
            },
            "likes": { "archive": null }
        },

    },

    "output": {
        "unsupportedfile": {
            "path": "./logs/unsupported.txt",
            "mode": "a"
        },

        "errorfile": {
            "path": "./logs/errorfile.txt",
            "mode": "a"
        },

        "logfile": {
            "mode": "a",
            "path": "./logs/generic_log.txt",
            "format": "[{asctime}][{name}][{levelname}] {message} [url={extractor.url}]"
        },

        "private": true,

        "ansi": false
    },

    "cache": {
        "file": "./cache.sqlite3"
    },

    "downloader": {
        "mtime": false
    }
}

[nisehime]

Oh yeah, im idiot. Forgot about -A 1 when trying again lol. Now it works, but still not clear why it stopped first time. There's no errors in log file, juts info about cursor

[nisehime]

The issue is happening with other users too. For example @bunta_kikumaru. At some point of downloading it just hits error

[twitter][error] Unable to retrieve Tweets from this timeline←[0m
[twitter][info] Use '-o cursor=2_1904449371594510727/DAADDAABCgABGm32nP6bUYcKAAIZUk0taBoQ5wAIAAIAAAACCAADAAAAAAgABAAAACoKAAUbl5VDnoAnEAoABhuXlUOeeXBQAAA' to continue downloading from the current position

When I restart with cursor, it continues just fine until it hits the same error again, having me to manually restart with cursor every time.

Here's what --write-pages gave me before it hit the error.

[twitter][info] Writing 'https://x.com/i/api/graphql/4fpceYZ6-YQCx_JSl_Cn_A/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3Abunta_kikumaru+max_id%3A1904449371594510727+filter%3Alinks%22%2C%22count%22%3A20%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Latest%22%2C%22withGrokTranslatedBio%22%3Afalse%2C%22cursor%22%3A%22DAADDAABCgABGm32nP6bUYcKAAIZUk0taBoQ5wAIAAIAAAACCAADAAAAAAgABAAAACoKAAUbl5VDnoAnEAoABhuXlUOeeXBQAAA%22%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22
communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3A
true%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D' response to 'F:\gallery-dl\25_https_x.com_i_api_graphql_4fpceYZ6-YQCx_JSl_Cn_A_SearchTimeline_variables_%7B%22rawQuery%22%3A%22from%3Abunta_kikumaru+max_id%3A1904449371594510727+filter%3Alinks%22%2C%22count%22%3A20%2C%22querySource%22%3A%22typed_query%22%2C%22product%.txt'

{
	"errors": [
		{
			"message": "Dependency: Unspecified",
			"locations": [
				{
					"line": 4,
					"column": 7
				}
			],
			"path": [
				"search_by_raw_query",
				"search_timeline",
				"timeline"
			],
			"extensions": {
				"name": "DependencyError",
				"source": "Server",
				"retry_after": 0,
				"code": 0,
				"kind": "Operational",
				"tracing": {
					"trace_id": "08d5b25b2b647cca"
				}
			},
			"code": 0,
			"kind": "Operational",
			"name": "DependencyError",
			"source": "Server",
			"retry_after": 0,
			"tracing": {
				"trace_id": "08d5b25b2b647cca"
			}
		}
	],
	"data": {
		"search_by_raw_query": {
			"search_timeline": {}
		}
	}
}

[nisehime]

@mikf can you fix please... literally can't download anyone. Just make it to retry automatically from the cursor anytime this error happens (but still show a warning), shouldn't be a lot of work.

[mikf]

The last downloaded id was 1622185556515110915

I got to 1224041034516070400 before it hit [twitter][debug] No Tweet results (1/3) before stopping.

The first media Tweet that I can find by searching in a browser appears to be 1224041034516070400 as well, so ... Works on my machine ™️ (I know, I know, ...)

---

"message": "Dependency: Unspecified", is most likely an internal Twitter error, I'd think, so these should always be retried.

[nisehime]

"message": "Dependency: Unspecified", is most likely an internal Twitter error, I'd think, so these should always be retried.

The problem is that im having this problem frequently. That's why im asking to make an auto retry.

[mikf]

Currently working on it

[nisehime]

Keep in mind

However, you would need to make it to ignore Abort option because sometimes starting from cursor would start download from before the last downloaded file, leading to a skip and therefore potential false abort trigger.

[mikf]

What I am working on should never be able to trigger abort since it is a retry on Dependency: Unspecified errors, which happens before any Tweets of that batch are processed.

[nisehime]

There were moments when I restarted from a cursor (inlcuding original post of this issue) and the cursor was set before the last downloaded tweet, meaning it had to skip some of them.

[mikf]

First iteration done: acf281a

This basically retries API requests when they would result in a Unable to retrieve Tweets from this timeline error and include a server related error message from Twitter's side, like "Dependency: Unspecified"

No restarting/retrying with cursor value yet.

[nisehime]

[twitter][warning] API errors (1/5):
- 'Dependency: Unspecified'
[twitter][warning] API errors (2/5):
- 'Dependency: Unspecified'
[twitter][warning] API errors (3/5):
- 'Dependency: Unspecified'
[twitter][warning] API errors (4/5):
- 'Dependency: Unspecified'
[twitter][warning] API errors (5/5):
- 'Dependency: Unspecified'
[twitter][info] Use '-o cursor=2_1904449371594510727/DAADDAABCgABGm32nP6bUYcKAAIYHgkG4pqxwAAIAAIAAAACCAADAAAAAAgABAAAAGEKAAUbl5VDnoAnEAoABhuXlUOecQvgAAA' to continue downloading from the current position
[twitter][error] An unexpected error occurred: AttributeError - 'LoggerAdapter' object has no attribute 'traceback'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
Traceback (most recent call last):
  File "gallery_dl\extractor\twitter.py", line 1914, in _pagination_tweets
KeyError: 'timeline'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gallery_dl\job.py", line 159, in run
  File "gallery_dl\extractor\twitter.py", line 85, in items
  File "gallery_dl\extractor\twitter.py", line 804, in tweets
  File "gallery_dl\extractor\twitter.py", line 1979, in _pagination_tweets
gallery_dl.exception.AbortExtraction: Unable to retrieve Tweets from this timeline

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gallery_dl\job.py", line 160, in run
  File "gallery_dl\job.py", line 228, in dispatch
  File "gallery_dl\job.py", line 459, in handle_queue
  File "gallery_dl\job.py", line 167, in run
AttributeError: 'LoggerAdapter' object has no attribute 'traceback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "__main__.py", line 20, in <module>
  File "gallery_dl\__init__.py", line 346, in main
  File "gallery_dl\job.py", line 190, in run
AttributeError: 'LoggerAdapter' object has no attribute 'traceback'
[PYI-5408:ERROR] Failed to execute script '__main__' due to unhandled exception!

[nisehime]

For some reason the cursor it gave is worse than I had before. Using this cursor starts with tweets from 2025-03, while this #8317 (comment) cursor that I initially used starts from 2024-08, not to mention the last downloaded tweet was 2023-12. Not the current position at all.

After trying again it now starts from 2022-12, wtf

Siiiiiiighhh whatever, the cursor seems accurate. For some reasons it now skip the WHOLE YEAR of tweets when using search. This site drives me insane.