Random "database is locked" errors in metadata processor

[cenodis]

Running gallery-dl with a metadata processor against a large set of accounts in quick succession (such as with -i) will sometimes cause the postprocessor to fail when trying to open its archive database. It always occurs at the start of a new account and causes the process to block for multiple seconds.

Example error:

[postprocessor.metadata][warning] Failed to open metadata archive at '<REDACTED>/meta-archive/{instance}.sqlite3' (OperationalError: database is locked)

Config excerpt:

"postprocessors": [
{
    "name": "metadata",
    "event": "post",
    "mode": "json",
    "filename": "{id}.json",
    "skip": false,
    "archive": "<REDACTED>/meta-archive/{instance}.sqlite3",
    "archive-pragma": ["locking_mode=EXCLUSIVE", "journal_mode=WAL", "synchronous=NORMAL"],
    "archive-prefix": "{instance}",
    "archive-format": "{id}",
    ...
},
...
}]

[cenodis]

Disclaimer: I am not very familiar with gallery-dls archive mechanism. It's possible I overlooked some things here.

Taking a cursory look through the source code it seems to me that the postprocessor archive is never closed?

The DownloadArchive class (archive.py) includes this line to support closing:

self.close = con.close

Looking at the DownloadJob class (job.py) it does properly close the archive after completion:

if self.archive:
    if not self.status:
        self.archive.finalize()
    self.archive.close()

In contrast, the common postprocessor (called by the metadata processor via self._init_archive) does connect to the archive:

self.archive = archive.connect(
    archive_path,
    archive_prefix,
    archive_format,
    archive_table,
    "file",
    options.get("archive-pragma"),
    job.pathfmt.kwdict,
    "_archive_" + self.name,
)

But neither the common nor metadata postprocessor seem to call self.archive.close() at any point. I also cannot find anything that would suggest the close call to happen from somewhere else.
To me this seems like a resource being leaked, resulting in the stale connection potentially holding onto its lock after the job completes, which in turn blocks the database access for the next job.

I don't know for sure if this is the cause for the locked databases. But to my knowledge sqlite connections should always be closed after use anyway.

[cenodis]

I have tested the commit with a list of ~60 accounts and it seems to have resolved the problem. Looks like the resource leak really was the cause.