aryion: Better deal with stacked directories

[kattjevfel]

Currently when you download something in a folder that's in a folder (and possibly yet another folder!) you get a path like this:
./gallery-dl/aryion/jameshoward/Patreon Stuff - Alpha - Chapter 1/jameshoward-590664-Alpha_RP_01_04_COVER copy.jpg.

Ideally, looking at the webpage and actual folder structure (jameshoward > Patreon Stuff > Alpha - Chapter 1 > Alpha - Chapter 1 - COVER), it should be ./gallery-dl/aryion/jameshoward/Patreon Stuff/Alpha - Chapter 1/jameshoward-590664-Alpha_RP_01_04_COVER copy.jpg

I looked at the directory_fmt but I only made a mess trying to do something about it myself :(

[mikf]

The default directory_fmt uses {path:J - } to combine all elements in path with - as separator. It would also be possible to access those elements individually and create nested directories that way, but this is rather awkward since gallery-dl only supports a fixed number of entries in directory_fmt:

"directory": ["{path[0]:?//}", "{path[1]:?//}", "{path[2]:?//}", "{path[3]:?//}", "{path[4]:?//}"]

This fetches each individual element from path (path[0]) and ignores errors in case it doesn't exist (:?//). Empty segments in directory get ignored. And of course this only works up to a depth of 5.

[kattjevfel]

Well I suppose 5 is better than 1, so I've now set it it like this:

        "aryion": {
            "filename": "{filename}.{extension}",
            "directory": ["{category}", "{user!l}", "{path[0]:?//}", "{path[1]:?//}", "{path[2]:?//}", "{path[3]:?//}", "{path[4]:?//}"],
            "username": "x",
            "password": "x"
        },

Which indeed produces the desired effect, even if a quite ugly workaround :p
/mnt/jupiter/Temp/gallery-dl/aryion/jameshoward/Patreon Stuff/Alpha - Chapter 1/jameshoward-590664-Alpha_RP_01_04_COVER copy.jpg

Is the limit of 5 something that will be ever changed, or should I just close this issue as solved?

[mikf]

The maximum depth is only limited by how many {path[N]:?//} entries there are. You can use 10, 20, or how every many you think are necessary, but it can only deal with a static/fixed number of entries.

Having some way of producing dynamic-length directory structures would be nice, but I don't have a good idea of how to accomplish that while leaving everything else intact.

[rautamiekka]

Having some way of producing dynamic-length directory structures would be nice, but I don't have a good idea of how to accomplish that while leaving everything else intact.

Well, do you have even a tiny start of an idea ? Maybe others can help with it.

[mikf]

The first step would be not applying ´clean_segment()` to each whole directory name, but only to the results of format strings.

gallery-dl/gallery_dl/util.py

Line 859 in 2e80610

append(self.clean_segment(segment))

Next there'd need to be a formatting option to "expand" a list of strings into subdirectories (basically "/".join(list)) while only applying clean_segment() to the intermediate names and not the entire result, since that would wipe all /s.

And while we're at it, we might as well have the directory option as be single string instead of a fixed-size list with one element for each subdirectory.

[bluerthanever]

Hey. I ran into similar situation lately, when I found out this issue.

How about instead of returning a single result in StringFormatter.format_map, we modify the formatters, and make it return a list or an iterator,
and then in path.PathFormat.build_directory or path.PathFormat.build_directory_conditional, we process each segment returned from fmt, and then append it to segments?

like the following:
I have not yet fully understand how __init__ and format_map of formatters work yet. so can't give an example now.
I guess we need to add a new format/conversion? Like list:I/, I for iterate.

in path.PathFormat.build_directory or path.PathFormat.build_directory_conditional

            for fmt in formatters:
                for segment in fmt(kwdict):
                    segment = segment.strip()
                    if strip:
                        segment = segment.rstrip(strip)
                    if segment:
                        append(self.clean_segment(segment))

[bluerthanever]

Hmm.... dug a little deeper and it seems to be more complicated than I had thought. need to change a lot of places

[bluerthanever]

Hmm... I went for an easier approach and succeeded.

Here's what I did:

In `formatter':

def build_format_func(format_spec):
    if format_spec:
        fmt = format_spec[0]
        if fmt == "?":
            return _parse_optional(format_spec)
        if fmt == "L":
            return _parse_maxlen(format_spec)
        if fmt == "J":
            return _parse_join(format_spec)
        if fmt == "R":
            return _parse_replace(format_spec)
        if fmt == "D":
            return _parse_datetime(format_spec)
        if fmt == "i":
            return _parse_iterable(format_spec)
        return _default_format(format_spec)
    return format

......


def _parse_iterable(format_spec):
    _, format_spec = format_spec.split("/", 1)
    fmt = build_format_func(format_spec)

    def iteration(obj):
        try:
            for i in obj:
                yield fmt(i)
        except TypeError as e:
            yield fmt(obj)
    return iteration

in path:

from inspect import isgenerator
......

    def build_directory(self, kwdict):
        """Apply 'kwdict' to directory format strings"""
        segments = []
        append = segments.append
        strip = self.strip

        try:
            for fmt in self.directory_formatters:
                segs = fmt(kwdict)
                if not isinstance(segs, list) and not isgenerator(segs):
                    segs = [segs]
                for segment in segs:
                    segment = segment.strip()
                    if strip:
                        # remove trailing dots and spaces (#647)
                        segment = segment.rstrip(strip)
                    if segment:
                        append(self.clean_segment(segment))
            return segments
        except Exception as exc:
            raise exception.DirectoryFormatError(exc)

    def build_directory_conditional(self, kwdict):
        segments = []
        append = segments.append
        strip = self.strip

        try:
            for condition, formatters in self.directory_conditions:
                if condition(kwdict):
                    break
            else:
                formatters = self.directory_formatters
            for fmt in formatters:
                segs = fmt(kwdict)
                if not isinstance(segs, list) and not isgenerator(segs):
                    segs = [segs]
                for segment in segs:
                    segment = segment.strip()
                    if strip:
                        # remove trailing dots and spaces (#647)
                        segment = segment.rstrip(strip)
                    if segment:
                        append(self.clean_segment(segment))
            return segments
        except Exception as exc:
            raise exception.DirectoryFormatError(exc)

kind of not so pretty... hopefully this would be of use.