Document how to use as library

[lyz-code]

Hi, I intend to use gallery-dl as a library for a program to periodically fetch the selected sources.

I already do it with youtube-dl as it's documented in their docs.

Is there a simple way to do the following with gallery-dl?

ydl_opts = {
    'format': 'bestaudio/best',
    'postprocessors': [{
        'key': 'FFmpegExtractAudio',
        'preferredcodec': 'mp3',
        'preferredquality': '192',
    }],
    'logger': MyLogger(),
    'progress_hooks': [my_hook],
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
    ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc'])

I've seen in this issue that you can process one url:

>>> from gallery_dl import job
>>> j = job.DataJob("https://imgur.com/0gybAXR")
>>> j.run()
[ ... ]

But it doesn't downloads the file, nor it works with gallery links such as https://www.deviantart.com/{{ user }}
Thank you

[mikf]

Use a DownloadJob instance to actually download stuff.
A DataJob object will only collect the data returned by its Extractor and not do much else with it.

Setting config options should be done via the functions in config.py, like config.set(), or by directly manipulating the _config dict in there. You can load config files with config.load().

For example:

from gallery_dl import config, job

config.load()  # load default config files
config.set(("extractor",), "base-directory", "/tmp/")
config.set(("extractor", "imgur"), "filename", "{id}{title:?_//}.{extension}")

for url in urls:
    job.DownloadJob(url).run()

[lyz-code]

Thank you @mikf, it helped a lot.

For others reading this issue, to know which options you need to set use the two config examples (1 and 2) with the options description. Here are some options I've set:

config.set(('extractor',), "archive", '~/.gallery-dl/archive.sql')
config.set(('extractor',), "base-directory", '~/downloads')
config.set(('extractor', 'deviantart'), "image-range", '1-10')
config.set(('extractor', 'deviantart'), "flat", False)
config.set(('extractor', 'deviantart'), "metadata", True)
config.set(
    ('extractor',),
    'postprocessors',
    [
        {
            "name": "metadata",
            "mode": "json",
        }
    ]
)

I'm still unable to configure the output, what am I doing wrong?

config.set(('output',), 'mode', 'terminal')
config.set(
    ('output',),
    'log',
    {
        "level": "info",
        "format": {
            "debug": "\u001b[0;37m{name}: {message}\u001b[0m",
            "info": "\u001b[1;37m{name}: {message}\u001b[0m",
            "warning": "\u001b[1;33m{name}: {message}\u001b[0m",
            "error": "\u001b[1;31m{name}: {message}\u001b[0m"
        }
    },
)
config.set(
    ('output',),
    'logfile',
    {
        "path": "log.txt",
        "mode": "w",
        "level": "debug"
    },
)
config.set(
    ('output',),
    "unsupportedfile",
    {
        "path": "unsupported.txt",
        "mode": "a",
        "format": "{asctime} {message}",
        "format-date": "%Y-%m-%d-%H-%M-%S"
    },
)

It produces the following config._config

 'output': {'log': {'format': {'debug': '\x1b[0;37m{name}: {message}\x1b[0m',
                               'error': '\x1b[1;31m{name}: {message}\x1b[0m',
                               'info': '\x1b[1;37m{name}: {message}\x1b[0m',
                               'warning': '\x1b[1;33m{name}: {message}\x1b[0m'},
                    'level': 'info'},
            'logfile': {'level': 'debug', 'mode': 'w', 'path': 'log.txt'},
            'mode': 'auto',
            'unsupportedfile': {'format': '{asctime} {message}',
                                'format-date': '%Y-%m-%d-%H-%M-%S',
                                'mode': 'a',
                                'path': 'unsupported.txt'}}}

Which is similar to the config example, but neither unsupported.txt, nor log.txt are being created.

Thanks

[mikf]

All logging output is done via Python's logging module.

You can use that to configure and attach your own handlers to the root logger,
or you call initialize_logging(), configure_logging(), and setup_logging_handler() from output.py after setting your output options.

Take a look at main() and search for output. to see how this is "normally" done.

For example

import logging
from gallery_dl import output

# initialze logging and setup logging handler to stderr
output.initialize_logging(logging.INFO)
# apply config options to stderr handler and create file handler
output.configure_logging(logging.INFO)
# create unsupported-file handler
output.setup_logging_handler("unsupportedfile", fmt="{message}")

[rpdelaney]

@mikf would you accept a PR documenting how to do this?

[mikf]

@rpdelaney Sure. I'd be happy about any sort of contribution, especially documentation. Let me know if you need anything or if I should explain how certain things (are supposed to) work.

[opsoyo]

All logging output is done via Python's logging module.

You can use that to configure and attach your own handlers to the root logger,
or you call initialize_logging(), configure_logging(), and setup_logging_handler() from output.py after setting your output options.

Take a look at main() and search for output. to see how this is "normally" done.

For example

import logging
from gallery_dl import output

# initialze logging and setup logging handler to stderr
output.initialize_logging(logging.INFO)
# apply config options to stderr handler and create file handler
output.configure_logging(logging.INFO)
# create unsupported-file handler
output.setup_logging_handler("unsupportedfile", fmt="{message}")

😅 I tried understanding this without success. For a split async moment, I simply use StringIO for stdout and stderr to capture and match with RegEx. Thankfully this small Discord bot won't mind the hacky method.

[rachmadaniHaryono]

config example

{ "output": {
    "log": { "level": "debug" },
    "#": "write logging messages to a separate file",
    "logfile": { "path": "/home/user/log.log", "mode": "a", "level": "debug" },
    "#": "write unrecognized URLs to a separate file",
    "unsupportedfile": { "path": "/home/user/unsupported.log", "mode": "a" }
}}

import logging
from gallery_dl import config, output
from gallery_dl.exception import NoExtractorError
from gallery_dl.extractor.common import get_soup
from gallery_dl.job import DataJob
#  load config before setting up logging
config.load()
# initialze logging and setup logging handler to stderr
output.initialize_logging(logging.DEBUG)
# apply config options to stderr handler and create file handler
output.configure_logging(logging.DEBUG)
# create unsupported-file handler
output.setup_logging_handler("unsupportedfile", fmt="{message}")
url = 'https://www.reddit.com/r/Hololive/comments/rcqpgr/'
job = DataJob(url)
job.run()
# process `job.data`

if you want to supress job.run

import os
with open(os.devnull, "w") as f:
    job.file = f
    job.run()

[53845714nF]

Does anyone of you know how to output all urls, like with the -g flag, but as a Python list and not on the stdout?
I know I need the job.UrlJob class. I also tried to monky patch some functions (run, dispatchand handle_url) but it didn't work.

Any ideas?

[rachmadaniHaryono]

have you tried last code?

after job.run you can get the output from job.data

[mikf]

@53845714nF just copy the job.UrlJob code, remove anything you don't need, and store any URLs in a list. You should end up with something like

class UrlJob(Job):

    def __init__(self, url, parent=None):
        Job.__init__(self, url, parent)
        self.urls = []

    def handle_url(self, url, _):
        self.urls.append(url)

Accessing URLs afterwards is then just

>>> j = UrlJob("imgur.com/asdqwe")
>>> j.run()
0
>>> j.urls
['https://i.imgur.com/asdqw.jpg']