Questions, Feedback and Suggestions

[hrxn]

A central place for these things might be a good idea.

This thread could serve as a starting point, results will eventually be collected in the project wiki, if appropriate and useful.

Edited 2017-04-15
For conciseness

Edited 2017-05-04
Removed nonsensical checklist thing

---

[mikf]

This is actually a really good idea, especially since I'm very hesitant/lazy about documenting things or writing text in general.

edit: The more I think about it, the less satisfied I am with the previous explanation, so here is version 2.

• Extractors produce a collection of key-value pairs. To find out what these look like, use the --list-keywords option:

$ gallery-dl --list-keywords http://www.pixiv.net/member_illust.php?id=11
Keywords for directory names:
artist-id:   11
artist-name: pixiv事務局
artist-nick: pixiv
category:    pixiv
subcategory: user

Keywords for filenames:
age_limit:       all-age
artist-id:       11
artist-name:     pixiv事務局
artist-nick:     pixiv
book_style:      right_to_left
...
category:        pixiv
content_type:    None
created_time:    2017-03-31 13:50:53
extension:       jpg
favorite_id:     0
height:          865
id:              62178245
...

• These key-value pairs are used to generate directory- and filenames by plugging them into format strings. For directories this is a list of format strings to work around the different path segment separators in Windows and UNIX systems (backslash \ or slash /).

• Each extractor has a default format string for directory- and filenames. For pixiv this is

    directory_fmt = ["{category}", "{artist-id}-{artist-nick}"]
    filename_fmt = "{category}_{artist-id}_{id}{num}.{extension}"

• The default values can be overwritten in your configuration file by setting the appropriate directory and filename values.

{
    "extractor":
    {
        "pixiv":
        {
            "directory": ["my pixiv images", "{artist-id}"],
            "filename": "{id}.{extension}"
        }
    }
}

• The category of each extractor is a keyword supplied in every key-value pair collection. It can therefore be used in every format string and has been chosen to be the first segment of every default format string for directory names, but that can, as explained above, be changed.

(edit end)

If something still doesn't make sense, just tell me and I will try to explain this a bit better.

[hrxn]

Very good to know, thank you.

Checked some profiles with --list-keywords, very useful, and returns exactly what expected. Everything according to plan, at least on the extraction side :)

I realized what caused the slight confusion (for me): The default format string set by the extractor gets overwritten by the output format defined in gallery-dl.conf, got that, all working as expected so far.

What put me a bit off was this:
https://github.com/mikf/gallery-dl/blob/master/gallery-dl.conf#L17-L30

Because pixiv seems to be a bit of a special case here.
Defining two different formats for directory, because pixiv makes use of two different "sub-extractors" ( for lack of a better word): "user": {..}, and "bookmark": {..}

I think these are called objects in JSON parlance..

Now, if I want to use my own directory and filename values in gallery-dl.conf,
along the lines of your given example:

{
    "extractor":
    {
        "pixiv":
        {
            "directory": ["my pixiv images", "{artist-id}"],
            "filename": "{id}.{extension}"
        }
    }
}

I put these two definitions into the "pixiv" object, that is, one level above the "user" and "bookmark' objects, right? This way, both definitions from each object get overwritten with the customized output format. A bit non-obvious, but this might just be me. And as long as it's working, nothing to complain here ;-)

[mikf]

Because pixiv seems to be a bit of a special case here.

What you have discovered here is true for all extractors, especially those with more than one extractor per module, and not just pixiv. In general the configuration value located the "deepest" inside the dictionary- or object-tree is used. If non is found, the config system falls back to the default value.

An example:

{
    "extractor":
    {
        "pixiv":
        {
            "user": { "filename": "A" },
            "filename": "B"
        },
        "deviantart":
        {
            "image": { "filename": "C"}
        },
        "filename": "D"
    }
}

With a configuration file like the one above, the following is going to happen:

• The pixiv.user extractor will use "A"
• All other pixiv extractors will use "B"
• The deviantart.image extractor will use "C"
• All other extractors, including the other deviantart extractors, will use "D"
• None will use their default value

I put these two definitions into the "pixiv" object, that is, one level above the "user" and "bookmark' objects, right? This way, both definitions from each object get overwritten with the customized output format.

Yes, if you have those two definitions at this place, then all pixiv extractors (there are 4 in total) will use these instead of their default format strings.

If you want to dig even deeper, take a look at the inner loop of the config.interpolate function. For example for the pixiv.user extractor this function gets called like so:

    directory = config.interpolate(["extractor", "pixiv", "user", "directory"], default)

This function first searches the top-most level for a value with key "directory" and stores this value if it finds it. It then descends into the "extractor" object and, again, searches this level for a value with key "directory". The same goes on with "pixiv" and "user" until it finally reaches the end.
If at any point something goes wrong and an exception gets thrown, which happens if for example the "pixiv" object doesn't exist, then the value stored up to this point gets returned.

[hrxn]

Okay, got it. Also, found all 4 pixiv extractors ;-)

Very nice, and very flexible. Ultimately, every possible variant can be customized. Excellent.

Just threw some pixiv URLs at the program, can confirm everything works indeed as described! (Including these multiple images per entry/"work", I did a manual recount ;-)

On to the next one..

Okay, this probably is a newbie question, but it looks like exhentai isn't a real site? There is e-hentai, seems like they are related (sister sites?). And you apparently need an e-hentai account first (and some dark magic, probably) before you can use exhentai. I will read a bit into this first.

Pretty sure that is the first time I've ever encountered something like this.

But this theory has a little flaw: If these two sites are indeed related, I'd assume that they don't differ much on the technical side, if at all. But trying some gallery links from e-hentai got me this:

C:\Users\Hrxn>gallery-dl "https://e-hentai.org/g/1047429/525823ef87/"
[gallery-dl][error] No suitable extractor found for 'https://e-hentai.org/g/1047429/525823ef87/'

C:\Users\Hrxn>gallery-dl "https://e-hentai.org/g/1047407/f00ba6d6cf/"
[gallery-dl][error] No suitable extractor found for 'https://e-hentai.org/g/1047407/f00ba6d6cf/'

C:\Users\Hrxn>gallery-dl "https://e-hentai.org/g/1047272/a003dfb22b/"
[gallery-dl][error] No suitable extractor found for 'https://e-hentai.org/g/1047272/a003dfb22b/'

C:\Users\Hrxn>gallery-dl "https://e-hentai.org/g/1047010/d8b62a3c87/"
[gallery-dl][error] No suitable extractor found for 'https://e-hentai.org/g/1047010/d8b62a3c87/'

C:\Users\Hrxn>gallery-dl "https://e-hentai.org/g/1047424/0218b04f9c/"
[gallery-dl][error] No suitable extractor found for 'https://e-hentai.org/g/1047424/0218b04f9c/'

C:\Users\Hrxn>gallery-dl --version
0.8.1-dev

C:\Users\Hrxn>

Or is there another specific reason for this?

[mikf]

exhentai is basically the "dark" version e-hentai with all the non-advertiser-friendly stuff enabled.

You should be able to access this site by doing this:

• create an account on e-hentai.org
• wait a week or so, use the regular site a bit, maybe play a bit of hentaiverse
• clear all your exhentai.org, e-hentai.org, forums.e-hentai.org cookies
• re-login on e-hentai
• visit exhentai

In the past the domain of the regular site was g.e-hentai.org and I haven't updated the extractor to also accept e-hentai.org.
You can just change the URLs a bit and replace the - with an x or put a g. in front. It all falls back to the same code that relies on having access to exhentai.

[hrxn]

Okay, made an account, will frequent the site a bit and see how it works out then..

Can't test it before, because b603b59 changed the expression pattern, and that part works, but it's still the exhentai extractor and therefore requires credentials for authentication. Which is not really an issue, don't get me wrong.

I will test some other sites in the meantime, and will update my initial post accordingly.

[mikf]

I don't know if visiting the regular site and so on is even necessary, that is just what I did when I created an account for unit testing and couldn't access exhentai immediately.

Speaking of which: I didn't want to make my unit testing accounts any more public than necessary (for, i hope, obvious reasons), but I should probably just share them with you. Take a look at this.

[hrxn]

I don't know if visiting the regular site and so on is even necessary, that is just what I did when I created an account for unit testing and couldn't access exhentai immediately.

I'm not sure, but other random sources on the Internet indicate that this is actually the case.

Speaking of which: I didn't want to make my unit testing accounts any more public than necessary (for, i hope, obvious reasons), but I should probably just share them with you. [...]

Yes, obviously. That is nice, but it won't be necessary, I've already made an account and started using it a bit. Besides, creating and using different accounts for different sites and services doesn't really bother me at all. If there is some longer gap between my responses, it's only because I'm busy with something else ;-)
I use Keepass for handling this stuff, which is a really great program, as you probably know. It's so good, they should invent a new word for it (great cross-platform alternative: KeepassXC)

---

Another thing, which I think belongs here, because it's not an issue or bug, but maybe a possible suggestion:

There is another feature on DeviantArt I wasn't aware of before: The Journal.

I noticed it while using gallery-dl with this profile: http://inkveil-matter.deviantart.com/

The site states: 190 deviations.
gallery-dl download: 155 files.

Luckily, there is a statistics page which explains this:
http://inkveil-matter.deviantart.com/stats/gallery/

InkVeil-Matter has 93,840 pageviews in total; their 35 journals and the 155 deviations in their gallery were viewed 733,738 times.

35 Journal entries, so 190 in total.

Shamelessly copied from the DeviantArt Wikipedia page:

Journals are like personal blogs for the member pages, and the choice of topic is up to each member; some use it to talk about their personal or art-related lives, others use it to spread awareness or marshal support for a cause.

Not sure if that is useful at all. I clicked around a bit, and saw nothing I would consider as missing.
Embeds from their own gallery, or from any other, and some links to some drawing feature of DeviantArt I also didn't know of before: Muro, can be seen when visiting sta.sh for example, which also belongs to them as it seems.

I don't know, not sure If I even really understand this feature yet.

Anyway, forgive me my wall of text here, I just wanted to let you know, just in case this is news to you as well ;-)

[mikf]

I use Keepass for handling this stuff, which is a really great program, as you probably know. It's so good, they should invent a new word for it (great cross-platform alternative: KeepassXC)

Thank you for the suggestion but I'm going to stay with my trusty GPG-encrypted plain text file :)

Another thing, which I think belongs here, because it's not an issue or bug, but maybe a possible suggestion

Even if this platform here is called an issue tracker, feel free (and even encouraged) to create new "issues" if you want to suggest or request a feature or support for a new site.

There is another feature on DeviantArt I wasn't aware of before: The Journal.

This seems to be just a collection of blog posts, which might contain references to other deviantart- or sta.sh images. There shouldn't be any images missing: 190 deviations consisting of 155 real deviations and 35 journal entries seems about right to me.
I could add an extractor to fetch those references and download all the images of a journal entry if you want me to.

Anyway, forgive me my wall of text here, I just wanted to let you know, just in case this is news to you as well ;-)

No worries, I don't mind walls of text and actually wasn't aware of the journal or muro, so thanks for telling me.

Sorry asking about it, this tool have a feature to able remembering what image is already downloaded, without checking local directory? iirc on package is already have sqlite dll, right?

Thank you.

[mikf]

No, I am sorry, but such a feature does currently not exist.
gallery-dl only skips downloads if a file with the same name already exists, but there is at this time no other way of "remembering" if an image has been downloaded before.
SQLite, as you have noted, is already being used, but that is only to cache login sessions and the like across separate gallery-dl invocations.

Feel free to open a separate issue If you want a feature like this being implemented, but please explain in greater detail what you actually want to do and/or need this feature for.

[hrxn]

Just saw the new commit adding options for skipping files.

A change from fc9223c#diff-283aceda91c5f7f10981253611f9f950

    def _exists_abort(self):
        if self.has_extension and os.path.exists(self.realpath):
            raise exception.StopExtraction()
        return False

Current extractor run, in this context, means just the 'active' URL, right?
Because I'm not sure yet what the expected behaviour would be if gallery-dl is used like this:
gallery-dl --input-file FILE

Maybe a case for an additional option. Or rather not, I'm still not sure about it, need to make up my mind first probably.