Retries on yt-dlp fails

[gphg]

Tested only on x.com, but I expect it may happen on other services that using yt-dlp as video downloader.

Outside the gallery-dl, this can be solved by pass -R 100 (retries at higher number). However, on gallery-dl once error occur, it skips without retries.

This error likely occur on network problems, eg short blackout or disconnection. At the moment, to get the skipped gallery be downloaded, you have to rerun the same command and url.

[mikf]

gallery-dl already forwards its retries setting to yt-dlp.

gallery-dl/gallery_dl/downloader/ytdl.py

Lines 25 to 27 in 8d672fb

	retries = self.config("retries", extractor._retries)
	self.ytdl_opts = {
	"retries": retries+1 if retries >= 0 else float("inf"),

yt-dlp's extractor_retries uses a default of 3, but that apparently doesn't apply.

    The following options are used by the extractors:
    extractor_retries: Number of times to retry for known errors (default: 3)

[gphg]

I managed to get it downloads the leftovers failed download, however, it downloads the same video again.

1853214868037362148_1.mp4.json
1853215011688305122_1.mp4
1853215011688305122_1.mp4.json
1853215104680006050_1.1.mp4
1853215104680006050_1.2.mp4
1853215104680006050_1.mp4.json
1853215104680006050_2.1.mp4
1853215104680006050_2.2.mp4
1853215104680006050_2.mp4.json
1853215210066022674_1.mp4
1853215210066022674_1.mp4.json
1853215598676701355_1.1.mp4
1853215598676701355_1.2.mp4
1853215598676701355_1.3.mp4
1853215598676701355_1.mp4.json
1853215598676701355_2.1.mp4
1853215598676701355_2.2.mp4
1853215598676701355_2.3.mp4
1853215598676701355_2.mp4.json
1853215598676701355_3.1.mp4
1853215598676701355_3.2.mp4
1853215598676701355_3.3.mp4
1853215598676701355_3.mp4.json
1853215761201828175_1.mp4
1853215761201828175_1.mp4.json
1853215836225327593_1.mp4
1853215836225327593_1.mp4.json
1853970243229495472_1.jpg
1853970243229495472_1.jpg.json

I think my configuration is broken.

Edit:
I haven't checked it and I can't confirm these are the same. I'll report later.

Edit 2:
sha256sum

$ sha256sum 1853215598676701355_*
19e05637063e1c712391ff0149c6fde90b8a4939b7ff87e8597d0f1823b28a78  1853215598676701355_1.1.mp4
a833bc15bf0f488f701c97f012b4320463f21eaebcb5be21d1d6e4cd01c77a0e  1853215598676701355_1.2.mp4
5c9409d97af8f7c4f5a615fbe8291fcffb71259fabd473fcdbec0345a09629a4  1853215598676701355_1.3.mp4
99e86e2b158b84631595ab8622e4c0aea328faab96bb1a5363e5af09dc7fb22e  1853215598676701355_1.mp4.json
19e05637063e1c712391ff0149c6fde90b8a4939b7ff87e8597d0f1823b28a78  1853215598676701355_2.1.mp4
a833bc15bf0f488f701c97f012b4320463f21eaebcb5be21d1d6e4cd01c77a0e  1853215598676701355_2.2.mp4
5c9409d97af8f7c4f5a615fbe8291fcffb71259fabd473fcdbec0345a09629a4  1853215598676701355_2.3.mp4
a0b07d0dc8802fe1f72c0c9803ca5f7316f23c0241ff8f2c859b7cc7fbdf19af  1853215598676701355_2.mp4.json
19e05637063e1c712391ff0149c6fde90b8a4939b7ff87e8597d0f1823b28a78  1853215598676701355_3.1.mp4
a833bc15bf0f488f701c97f012b4320463f21eaebcb5be21d1d6e4cd01c77a0e  1853215598676701355_3.2.mp4
5c9409d97af8f7c4f5a615fbe8291fcffb71259fabd473fcdbec0345a09629a4  1853215598676701355_3.3.mp4
a01cc5210d40a90275784e3ef302a72ab769b2c17add2e68d5eddfa3ff92c2e3  1853215598676701355_3.mp4.json

Edit 3:
I posted my configuration file here:
#8253 (reply in thread)

Edit 4:
This is my yt-dlp configuration file:

--retries 100
--embed-metadata
--output "%(extractor)s %(display_id|id)s %(title).200s.%(ext)s"

[gphg]

@mikf can we reconsider this as bug?

[gphg]

gallery-dl --retries 100 --filter "extension in ['mp4', 'mkv', 'm4v', 'avi', 'mov', 'ts']" https://x.com/Yazanios/media
[cookies][info] Extracted 89 cookies from Firefox
[twitter][info] Initializing client transaction keys
./gallery-dl/twitter/Yazanios/1988517723232461296_1.mp4
./gallery-dl/twitter/Yazanios/1985067203943661949_1.mp4
./gallery-dl/twitter/Yazanios/1984533886345588905_1.mp4
[downloader.ytdl][error] ERROR: [twitter] 1983210484707807600: Unable to download JSON metadata: HTTPSConnectionPool(host='x.com', port=443): Read timed out. (read timeout=20.0) (caused by TransportError("HTTPSConnectionPool(host='x.com', port=443): Read timed out. (read timeout=20.0)"))
[download][error] Failed to download 1983210484707807600_1.mp4
./gallery-dl/twitter/Yazanios/1982152639555993776_1.mp4
./gallery-dl/twitter/Yazanios/1977005521896116350_1.mp4
./gallery-dl/twitter/Yazanios/1976867337791062264_1.mp4
./gallery-dl/twitter/Yazanios/1976689252080701639_1.mp4

Partial content of ~/.gallery-dl.conf

{
  "postprocessors": [
        {
            "name": "metadata",
            "mode": "json",
            "directory": "./"
        },
        {
            "name": "mtime"
        }
  ],
  "downloader": {
    "ytdl": {
      "cmdline-args": [
        "--retries",
        "100",
        "--embed-metadata",
        "--cookies-from-browser",
        "firefox"
      ]
    }
  },
  "extractor": {
    "path-restrict": "windows",
    "twitter": {
      "text-tweets": true,
      "retweets": true,
      "cookies": ["firefox"],
      "sleep": [14.9, 29.2],
      "sleep-request": [13.8, 33.6],
      "videos": "ytdl",
      "retries": 99
    }
  }
}

Partial content of ~/.config/yt-dlp/config

--retries 100
--embed-metadata
--output "%(extractor)s %(display_id|id)s %(title).200s.%(ext)s"
--sleep-requests 1

[mikf]

I've refactored the ytdl downloader a bit and added some basic retry functionality: 402f536
It currently retries on every error though and doesn't distinguish between (non) retry-able errors yet.

[gphg]

I haven't encounter any error after upgraded it. This issue has been solved. Thanks for the updates and great software!