Save downloaded media in memory via Python API

[thcrt]

Using gallery-dl programmatically as a Python module, I can of course download to a file like normal, as shown in #642:

>>> import gallery_dl as dl
>>> job = dl.job.DownloadJob('https://www.reddit.com/r/aww/comments/1igojpv/this_loafingblepping_cat_in_istanbul/')
>>> job.run()
./gallery-dl/reddit/aww/1igojpv This loafing_blepping cat in Istanbul.jpg
0

I can then see the metadata:

>>> meta = list(job.extractor.items())[0][1]
>>> print(meta["title"])
This loafing/blepping cat in Istanbul

But the media itself is still saved to a file on disk, in this case ./gallery-dl/reddit/aww/1igojpv This loafing_blepping cat in Istanbul.jpg.

Is there a way to instead save the media to a BytesIO object or similar?

My goal is to then save the raw data to a database, and this would mean I could avoid the extra steps of writing and then reading and then deleting the saved file on disc.

Cheers!

[thcrt]

Also, if this isn't actually feasible, is there a way that I can get the downloaded file name(s)?

e.g.

[
    "./gallery-dl/reddit/aww/1igyy6e 01 I think my dog was a model in a past life.jpg",
    "./gallery-dl/reddit/aww/1igyy6e 02 I think my dog was a model in a past life.jpg",
    "./gallery-dl/reddit/aww/1igyy6e 03 I think my dog was a model in a past life.jpg",
    "./gallery-dl/reddit/aww/1igyy6e 04 I think my dog was a model in a past life.jpg"
]

[mikf]

You should be able to do this by overwriting PathFormat.open() and returning a BytesIO object instead of a handle to a real file. You also might want to call DownloadJob.initialize() manually to have access to pathfmt immediately.

[thcrt]

Unfortunately, subclassing PathFormat to override .open() can't work, from what I can tell.

HttpDownloader.download_impl() can certainly take a BytesIO instance instead of FileIO, but it's structured in such a way that the handle to the IO object is closed after writing. The method then (on success) returns True, rather than returning the IO object, essentially throwing away the reference. There's also the issue that running DownloadJob.run() is very likely to produce multiple files, so a single BytesIO instance wouldn't really be enough.

At least, this is the impression I have from briefly looking through the code. Of course, I'm not very familiar with how it's structured, so I could be missing something.

---

To be clear- I'm not criticising your design decisions, nor am I suggesting that you should undertake a massive rework of your project to fit it to an architecture that better suits my use case! Both would be completely unreasonable, and I don't want to give that impression. If there's something I'm missing that would make what I'm trying to do a lot easier, I'd love to know about it; otherwise, I'm still able to use the software as it was intended. :^)

[mikf]

from gallery_dl import job, path
import io
import sqlite3


class Bytes(io.BytesIO):

    def close(self):
        pass

    def realclose(self):
        super().close()


class CustomPath(path.PathFormat):

    def open(self, mode=None):
        self.data = data = Bytes()
        return data

    def finalize(self):
        cursor.execute(
            "INSERT INTO files (name, data) VALUES (?, ?)",
            (self.filename, self.data.getvalue(),),
        )

        self.data.realclose()


path = "/tmp/files.sqlite"
db = sqlite3.connect(path)
cursor = db.cursor()
cursor.execute(
    "CREATE TABLE IF NOT EXISTS files "
    "(name TEXT PRIMARY KEY, data BLOB) "
    "WITHOUT ROWID"
)


url = "https://imagetwist.com/f1i2s4vhvbrq/test.png"
job = job.DownloadJob(url)

job.initialize()
job.pathfmt = CustomPath(job.extractor)
job.run()


db.commit()
db.close()

[thcrt]

Well, now I look like an idiot. That works, of course. Thank you!

Only one issue I've seen: downloading an Imgur gallery doesn't work with this. Providing it with an Imgur gallery link (example) causes the image to be downloaded as normal to ./gallery-dl/imgur/<gallery_name>/<file_name>. The overridden methods on CustomPath are seemingly never called. This occurs whether the gallery contains only 1 image or multiple images. I can link directly to a single image (example) and it downloads in-memory and gets saved to the database. All other supported sites that I've tested don't have this problem.

[thcrt]

Something similar happens when passing a link with a ytdl: prefix. However, because yt-dlp supports being used as a Python library directly (and supports some video sites that are out of scope for gallery-dl anyway), I'll probably just add a manual integration, so I don't really need help with that specifically.

[mikf]

downloading an Imgur gallery doesn't work with this

Ah yes, I far too often forget about child Extractors and Jobs, which don't use the same "modifications" as applied to their parent - job.pathfmt = CustomPath(job.extractor) in this case. I guess overwriting the global PathFormat class would be an option.

path.PathFormat = CustomPath

Code

from gallery_dl import job, path
import io
import sqlite3


class Bytes(io.BytesIO):

    def close(self):
        pass

    def realclose(self):
        super().close()


class CustomPath(path.PathFormat):

    def open(self, mode=None):
        self.data = data = Bytes()
        return data

    def finalize(self):
        cursor.execute(
            "INSERT INTO files (name, data) VALUES (?, ?)",
            (self.filename, self.data.getvalue(),),
        )
        self.data.realclose()

path.PathFormat = CustomPath


path = "/tmp/files.sqlite"
db = sqlite3.connect(path)
cursor = db.cursor()
cursor.execute(
    "CREATE TABLE IF NOT EXISTS files "
    "(name TEXT PRIMARY KEY, data BLOB) "
    "WITHOUT ROWID"
)


url = "https://imgur.com/gallery/im-going-to-stick-to-wiseposting-from-now-on-guys-Q80vmS1"
status = job.DownloadJob(url).run()


db.commit()
db.close()

[thcrt]

That makes sense! Thanks for being so helpful and patient with me, haha. I hope it's not a chore.

I actually didn't end up taking that approach, because my implementation of a PathFormat subclass doesn't write to the database directly, rather, it has an attribute files: dict[str, bytes] that is updated in finalize() with self.files[self.filename] = self._data.getvalue(). job.pathfmt.files is then read by the calling code.

Instead, I subclassed DownloadJob to override handle_queue(), adding the child job it creates to a list as a property of the parent, so it can be accessed by the caller. To do this, I had to essentially copy the implementation of the original, because otherwise there is no way I can see to externally access a Job's children. I also added getters for metadata and files that recursively get data from the job and its descendants.

In practice, this means I have a module that looks like this:

gallery_dl_interface.py

from io import BytesIO
from typing import Any, Self

import gallery_dl.extractor  # type: ignore
from gallery_dl.extractor.common import Extractor  # type: ignore
from gallery_dl.job import DownloadJob, Job  # type: ignore
from gallery_dl.path import PathFormat  # type: ignore
from gallery_dl.text import nameext_from_url  # type: ignore

type KWDict = dict[str, Any]


class PersistentBytes(BytesIO):
    def close(self) -> None:
        pass

    def real_close(self) -> None:
        return super().close()


# mypy: allow_subclassing_any
class InMemoryFormat(PathFormat):
    files: dict[str, bytes]
    kwdict: KWDict

    def __init__(self, extractor: Extractor):
        super().__init__(extractor)
        self.files = {}

    def open(self, mode: str) -> PersistentBytes:
        self._data = data = PersistentBytes()
        return data

    def finalize(self) -> None:
        self.files[self.filename] = self._data.getvalue()
        self._data.real_close()


class InMemoryDownloadJob(DownloadJob):
    extractor: Extractor
    _skipcnt: int
    children: list[Self]

    def __init__(self, extractor: str | Extractor, parent: Job | None = None):
        super().__init__(extractor, parent)
        self.children = []
        self.initialize()
        self.pathfmt = InMemoryFormat(self.extractor)

    @property
    def metadata(self) -> KWDict:
        metadata = self.pathfmt.kwdict
        for child in self.children:
            metadata = metadata | child.metadata
        return metadata

    @property
    def files(self) -> dict[str, bytes]:
        files = self.pathfmt.files
        for child in self.children:
            files = files | child.files
        return files

    def handle_queue(self, url: str, kwdict: KWDict) -> None:
        if url in self.visited:
            return
        self.visited.add(url)

        extractor: Extractor | None
        cls: Extractor | None
        if cls := kwdict.get("_extractor"):
            extractor = cls.from_url(url)
        else:
            extractor = gallery_dl.extractor.find(url)
            if extractor:
                if self._extractor_filter is None:
                    self._extractor_filter: Any = self._build_extractor_filter()
                if not self._extractor_filter(extractor):
                    extractor = None

        if not extractor:
            self._write_unsupported(url)
            return

        child_job = self.__class__(extractor, self)

        parent_metadata = self.extractor.config2("parent-metadata", "metadata-parent")
        if parent_metadata:
            if isinstance(parent_metadata, str):
                data = self.kwdict.copy()
                if kwdict:
                    data.update(kwdict)
                child_job.kwdict[parent_metadata] = data
            else:
                if self.kwdict:
                    child_job.kwdict.update(self.kwdict)
                if kwdict:
                    child_job.kwdict.update(kwdict)

        while True:
            try:
                if self.extractor.config("parent-skip"):
                    child_job._skipcnt = self._skipcnt

                status = child_job.run()

                if self.extractor.config("parent-skip"):
                    self._skipcnt = child_job._skipcnt

                if status:
                    self.status |= status
                    if status & 95 and "_fallback" in kwdict and self.fallback:
                        fallback = kwdict["_fallback"] = iter(kwdict["_fallback"])
                        try:
                            url = next(fallback)
                        except StopIteration:
                            pass
                        else:
                            nameext_from_url(url, kwdict)
                            self.handle_url(url, kwdict)

                break

            except gallery_dl.exception.RestartExtraction:
                pass

        self.children.append(child_job)

I can then call it from code that looks like this:

from dataclasses import dataclass, field
from pprint import pp

import gallery_dl_interface as gdi

URL = "https://imgur.com/gallery/im-going-to-stick-to-wiseposting-from-now-on-guys-Q80vmS1"


# Imagine this class is part of an ORM and gets sent to a database somewhere
@dataclass
class Media:
    filename: str
    data: bytes

    def __repr__(self) -> str:
        return f"Media called {self.filename} with {len(self.data)} bytes"


# Imagine this class is part of an ORM and gets sent to a database somewhere
@dataclass
class Post:
    source: str
    title: str
    author: str
    text: str
    media: list[Media] = field(default_factory=list)


job = gdi.InMemoryDownloadJob(URL)
job.run()

post = Post(
    source=job.metadata["category"],
    title=job.metadata["album"]["title"],
    author=job.metadata["album"]["account"]["username"],
    text=job.metadata["description"],
    media=[Media(filename, data) for filename, data in job.files.items()],
)

pp(post)

This implementation seems to work pretty well for me so far. If you like, I'd be very happy to submit a PR that adds this as part of gallery-dl itself.

Thanks again for your help :)

[thcrt]

One quirk I noticed was how, if I had downloaded something with gallery-dl manually on the command line, the instance of gallery-dl being run programmatically would see that the file had already been downloaded and hence wouldn't actually write any data to the IO buffer.

This took me an embarrassingly long time to debug, although it's obvious in hindsight. I was staring at my screen for so long trying to figure out "why is this specific post not downloading!?!?" when it was just seen by the code as already done.

Just pointing this out in case anyone else in the future is getting a crisis from the same thing :^)

[mikf]

If you like, I'd be very happy to submit a PR that adds this as part of gallery-dl itself.

Thanks, but seeing how this would involve "modern" Python code, I have to decline.

[thcrt]

LOL, fair enough ;)

I'll probably separate it into its own package, as a kind of wrapper. Although I don't expect anyone will actually ever use it.

Thanks again for all your help - I really appreciate it :)

[thcrt]

I've written some code that essentially calls the Extractors directly: https://github.com/thcrt/gallery-dl-bytes

It doesn't fully support all the configuration of gallery-dl, but it seems pretty solid for my own use case. It's also fully typed (pyright strict mode) and for that purpose even includes some type stubs for gallery-dl itself. Of course, it's licensed also under GPL-2.0.

Just thought I'd let you know, and also mention its existence here in case anyone else finds it useful in the future.