Questions, Feedback, Suggestions #6

[mikf]

Continuation of the central place for any sort of question or suggestion not deserving their own separate issue.

Links to previous instances: #6582 #5262 #146 #74 #11
Discord: https://discord.gg/rSzQwRvGnE

[taskhawk]

How can I get the download URLs (like what you get with --get-urls) of files that fail to download during a normal run? I tried writing to a file with a metadata postprocessor for the error event but doesn't seem to work as I expected.

    "write-failed-url": {
      "name": "metadata",
      "event": "error",
      "directory": ".failed",
      "extension": "txt"
    },

Edit: Wanted to write just the download URL to the file but I tried writing all metadata first to test if the event worked.

[Ramelstag]

is there currently a way to have it so gallery-dl ignores the file extension when checking if an item already exists in the output folder, without making it not add file extensions to the output file names? for example, lets say that im keeping and updating a local stash of a website, and then i update it, (which is convenient since gallery-dl by default doesn't re-download existing ones and using the range option i can have it not check the entire thing, instead, only the few dozen newest items or so) and then a new gif is added, but since gifs are pretty bad, i decide to re-encode it into a video and have it nicely encoded, but then after updating the stash again, it would re-download the original gif file. a workaround it to create an empty file with the same name but the previous file extension, but that would get messy if the format im trying to replace is common on the website. would it be feasible to add this functionality if it's not currently present?

[hrxn]

^ Have you already tried to use the archive option? If I'm not mistaken here, this should do what you're trying to achieve here, I'd guess. In short, everything that gets downloaded is stored in an archive file, by its unique identifier (when using default settings) - that way you can ignore/avoid any messy details like file extensions etc.

[Ramelstag]

yes i did hear about the archive function, but i don't think it would work for my usecase, and i don't know how easy or how difficult it would be to put all of the stuff i already have on one of the archives, and i really don't want to re-download everything because i put a lot of work into re-enconding/optimizing basically all items

[taskhawk]

It still sounds like using an archive will help you with what you asked, gallery-dl will check the archive instead of checking if the file with .gif extension still exists, which is what triggers the re-download when you already replaced it with the video file.

To populate your archive with what you already downloaded, I would say first download the newest files the way you're currently doing it so you're up to date, and then add the archive to your command or config from now on, and use the --simulate option once so it writes to the archive without downloading files, use without --range so it process all available files from the URL you're using:

gallery-dl --download-archive path/archive.sqlite3 --simulate "$URL"

After that, look into --abort instead of --range if you just want the newest files, seems a better fit.

[mikf]

@taskhawk

files that fail to download

Enable metadata-url and let it write just that value.

Config

{
    "extractor": {
        "metadata-url": "_file_url",
        "postprocessors": "write-failed-url"
    },

    "postprocessor": {
        "write-failed-url": {
          "name": "metadata",
          "event": "error",
          "directory": ".failed",
          "extension": "txt",
          "format": "{_file_url}"
        }
    }
}

[mikf]

What do you all think of giving Discord a(nother) try?

I created a gallery-dl Discord server (Invite Link) way back in 2023, but more or less abandoned it immediately due to reasons (unrelated to Discord)
Another way to ask questions and communicate in general might be useful.

[taskhawk]

Personally, I barely use it, usually when it's my only option left to get help or need to get in contact with some staff.

My main issues with it are mostly privacy concerns and also communities that use it as a poor replacement for something that would be better off like a site, a wiki, or a forum.

Other than that it's a great chat tool, lots of people already use it so it's probably a good way for them to get quick support and stay on the loop of announcements. But, it's another thing you need to manage.

[hrxn]

I think it's a good idea. I even have more than one discord account, actually, but haven't found a way to better juggle these, yet..

[williamdotio]

I think the default directory format for the koofr extractor should be

["{category}","{title} ({id})"]

Instead of

Directory format (default):
["{category}","{date:%Y-%m-%d} {title}"]

[oncoremarco]

When I try and download some multi-image sets in Deviantart, it will only download the first file and give me an "Empty File" warning and "Failed to download," but I can see the files on the page. I've tried downloading normally and with "-o public-0." It seems these multi-image sets are just a nightmare to deal with in DA because I've also been going back and looking at downloaded folders and about a quarter of my downloads of these sets are half-grey failed downloads too.