Questions, Feedback and Suggestions #2

[mikf]

Continuation of the old issue as a central place for any sort of question or suggestion not deserving their own separate issue.

#11 had gotten too big, took several seconds to load, and was closed as a result.
There is also https://gitter.im/gallery-dl/main if that seems more appropriate.

[hrxn]

A bit of feedback:
I've been downloading some stuff from Tumblr now, and everything seems to be working like a charm so far. Kudos to you.
Including --write-unsupported FILE, detecting and saving embedded links from Vine, Instagram and links to other external sites. Safe to assume it'll work just the same with embeds from YouTube and Vimeo, which I thought to be pretty common on Tumblr, but it's entirely possible that the processed blogs so far actually don't had a single one. I try that later on my own test blog, just to be sure.

This also led me to another idea/suggestion:
How about a --write-log FILE feature?
I'm aware that there is something like tee, and even something similar for PowerShell, but I think the purpose for the logging feature (at least at default setting) would not be to replicate the full output printed to the console, no, only any relevant errors or warnings, i.e. like 404, 403 and also if there occurs some timeout or something, which can be caused easily by connectivity issues, not by gallery-dl itself.

Not sure, just an idea...

[mikf]

Sounded like a useful feature, so I tried to put something together: 97f4f15. It should more or less behave like one would expect, but there are at least two things that might be better handled otherwise:

Edit: Never mind the points below, I thought about it and decided to change its behavior. It is no longer persistent and stores exactly the same log messages as shown on screen (c9a9664).

• Log files are currently persistent across gallery-dl invocations and can grow indefinitely. Other options would be deleting its old content each time or even log rotations (Python has built-in support for that as well).
• Debug log messages will never be written to a log file, even when using '--verbose'. It would be easier to copy the --verbose output from a text file instead of a console window, but writing all of the debug output to an otherwise concise log file didn't seem like a good idea.

[hrxn]

One small addition to logging behaviour:

Not entirely sure without a test case at hand now, but how's the current output for images on Tumblr relying on the fallback mechanism? I don't know if I remember that correctly, but it appeared that trying to download some specific image (e.g. some old images, old URL scheme etc.) with s3.amazonaws which then resulted in an error (403) prints the corresponding messages to the terminal, but the next successful download had the same Post ID in the name, so that would be the fallback URL used here.
I hope you know what I mean, if not I'll try some random blogs again and wait for that error to appear and copy the messages here.
Because the question is, should the error be printed to the output/logged to file in this case here? Because even if the "raw" URL does not work and we run into an actual 403, the fallback URL/mechanism still works and downloads that image instead successfully, so it's a bit debatable if that would really qualify as a real "error" 😄

Edit:

Had the opportunity to think about this again, and I'm not sure if it's actually worth to bother. Sure, it may be less than ideal (Think of your average end-user©® and the reaction: "OMG It says there's some error"), but this can be solved by simply, uh, explaining the stuff. And if this really warrants to deal with errors that are actually not so much of an error, by having to handle different error types, error classes, error codes and whatever, just for the sake of what, actually, consistency (?), I'm not really sure anymore.

[mikf]

I think it is actually worth to bother. With the way things were, it was impossible to tell if all files have been downloaded successfully by just looking at a log file, and error messages from an image with fallback URL were kind of misleading as well, since the image download in question did succeed in the end.

I added two more logging messages to hopefully remedy this (db7f04d):

1. Failed to download <filename> when an image could not be downloaded, even when using fallback URLs.
2. Trying fallback URL #<number> to indicate that the last error message is not fatal.

Maybe it would be better to categorize all HTTP errors as warnings and only show the Failed to download … message as definite error?

[hrxn]

Maybe it would be better to categorize all HTTP errors as warnings and only show the Failed to download … message as definite error?

Yeah, sounds good.
👍

[rachmadaniHaryono]

@mikf,

• which extractor class from common module should i use to make CustomExtractor? is there any requirement for each extractor class?
• can you explain the extractor class' attribute? (e.g. from BooruExtractor have 'basecategory', 'filename_fmt', 'api_url', 'per_page', 'page_start', 'page_limit', etc)
• is there any rule on how metadata attribute on extractor class should be built, or is it up to each extractor?

[mikf]

1. Generally you should use the basic Extractor class, but, as always, it depends. There are some general extractor sub-classes (BooruExtractor, FoolslideExtractor, FoolfuukaExtractor, ...) and it might also be helpful to just copy an existing extractor module and adjust it to your needs.
   As for requirements: set the category, subcategory, filename_fmt, directory_fmt and pattern class attributes to some reasonable values (see, for example, slideshare.py).

2. category and subcategory are essentially an extractor's name and are used for config-lookup.
   directory_fmt and filename_fmt are default values for the directory and filename options.
   pattern is a list of regex-strings. An extractor is used if one of them matches the given URL. The resulting match-object is the second parameter to an extractor's __init__() method.
   basecategoryhas to do with shared config values, just ignore it.

   The other attributes you listed are BooruExtractor-specific:

   • api_url: URL to send API requests to
   • per_page: number of post-entries per page
   • page_start: the first page (0 or 1 depending on site)
   • page_limit: largest valid page number

3. You kind of asked the same thing before.
   It is up to each extractor, but similar ones should use the same key-names. For image-metadata, you should always provide the filename extension as extension or at least set it to None.

[rachmadaniHaryono]

is there any rule on how metadata attribute on extractor class should be built, or is it up to each extractor?

You kind of asked the same thing before.
It is up to each extractor, but similar ones should use the same key-names. For image-metadata, you should always provide the filename extension as extension or at least set it to None.

actually i got the wrong impression from chan.py. i thought that to use the match object later on items method it should be stored on class' attribute metadata. but after i look at slideshare.py, any name can be used (e.g. user and presentation)

---

here is what i can come up with https://gist.github.com/rachmadaniHaryono/e7d40fcc5b9cd6ecc1f9151c4f0f5d84

full code https://github.com/rachmadaniHaryono/RedditImageGrab/blob/master/redditdownload/api.py

this module will not download a file, but it will only extract from url

[rachmadaniHaryono]

@mikf can you give example for 6a07e38 ?

[mikf]

from my_project import module_with_extractors
class SomeExtractor(Extractor):
    ...

from gallery_dl import extractor
extractor.add(SomeExtractor)
extractor.add_module(module_with_extractors)

You should use these functions instead of manually manipulating extractor._cache and relying on implementation details.

[ChiChi32]

I'm doing something wrong? And i try option in config file, don't working.

[hrxn]

Which version of gallery-dl is that? Can you run gallery-dl -v please?

[Bfgeshka]

Can we have percent-encoding conversions for saved files? I.e. replacing %20 in filename with whitespace, %22 with ", etc.