Setting custom log handler when using gallery_dl as import / Custom Log Handler Option

[ZizzyDizzyMC]

I've been trying unsuccessfully to add a log handler to gallery_dl when used as an import.

import logging
import gallery_dl

class MyLogger(logging.Handler):
    
    #def handle(*args):
    def emit(*args, **kwags):
        print('--- args ---')
        for item in args:
            print(item)
        print('---')
        
# --- main ---

logger = logging.getLogger()
logger.setLevel(logging.INFO) 
my_logger = MyLogger(logging.INFO)
logger.addHandler(my_logger)
gallery_dl.job.DownloadJob("imgur.com/8372ne").run()

The resulting output will be:

--- args ---
<MyLogger (info)>
<LogRecord: imgur, 40, /home/ubuntu/bot-venv/lib/python3.10/site-packages/gallery_dl/job.py, 105, "%s: %s">
---
[imgur][error] HttpError: '404 Not Found' for 'https://api.imgur.com/post/v1/media/8372n'
4

Unfortunately I've tried grabbing ahold of getLogger('gallery-dl') also without success. This is related to #642 however this is also a request for an addition of using custom loggers similar to yt_dlp. I'm sure I can add it, I just cannot find it. Without being able to capture status results and messages in real time I am unable to act on items that return statuses I need to act on, such as 404's or other status codes in addition to file failures etc.

I appear to be getting the LogRecord as definied in the Logger class in output.py - however the msg contents of this log record are always blank with "%s: $s"

I am not wanting to write to a file, as seems to be the way for the entirety of output.py.

I appreciate your time, having to make a post was my last resort as I've given up.

[mikf]

I think your main problem comes from not properly implementing emit().

I've taken your code, reordered and renamed things a bit, copy-pasted parts of StreamHandler.emit() (part of Python's standard library), and it works just fine.

import logging
import gallery_dl


class MyLogger(logging.Handler):

    def emit(self, record):
        msg = self.format(record)
        print(f"MyLogger: {msg}")


my_handler = MyLogger(logging.INFO)

root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(my_handler)

gallery_dl.job.DownloadJob("imgur.com/8372ne").run()

$ python log.py 
MyLogger: HttpError: '404 Not Found' for 'https://api.imgur.com/post/v1/media/8372n'

---

Without being able to capture status results and messages in real time I am unable to act on items that return statuses I need to act on, such as 404's or other status codes in addition to file failures etc.

Derive a custom class from DownloadJob() (or plain Job()) and run that. This should give you all the control you need, or at the very least more than reacting to logging messages.

this is also a request for an addition of using custom loggers similar to yt_dlp

Override Job.get_logger() and let it return whatever Logger-like object you want.

I've tried grabbing ahold of getLogger('gallery-dl') also without success

This logger only gets defined and returned by gallery_dl.output.initialize_logging(), which does not automatically get called when importing gallery_dl as a module.

[mikf]

You might also want to take a look at https://docs.python.org/3/library/logging.html to learn about Python's logging infrastructure, or at least https://docs.python.org/3/library/logging.html#logging.LogRecord to get an overview over all attributes of the record argument of emit().

[ZizzyDizzyMC]

I wanted to take a moment and thank you, now that I've had a chance to rest. I had been stressing about that for a few days.

It works great, and I see that I was doing this wrongly, and was unable to come up with the correct method on my own.

I've come across a new issue that I'll make a different issue for but I'm going to close this one (and I'll contribute back to the documentation discussion / issue with this knowledge as I'm sure it'll be helpful in the future.)