[fandom/wikimedia] KeyError: 'metadata' caused by non-existent files

[ClosedPort22]

I have occasionally run into this error when scraping several Fandom wikis lately:

[fandom][error] An unexpected error occurred: KeyError - 'metadata'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[fandom][debug] 
Traceback (most recent call last):
  File "/home/[redacted]/programming/gallery-dl/gallery_dl/job.py", line 153, in run
    for msg in extractor:
               ^^^^^^^^^
  File "/home/[redacted]/programming/gallery-dl/gallery_dl/extractor/wikimedia.py", line 107, in items
    self.prepare_image(image)
    ~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/[redacted]/programming/gallery-dl/gallery_dl/extractor/wikimedia.py", line 85, in prepare_image
    for m in image["metadata"] or ()}
             ~~~~~^^^^^^^^^^^^
KeyError: 'metadata'

Upon closer inspection, it appears that this is an expected behavior, though it is only documented in the changelog of MediaWiki version 1.34:

In the response to queries that use prop=imageinfo, entries for non-existing files (indicated by the filemissing field) now omit the following fields, since they are meaningless in this context: timestamp, userhidden, user, userid, anon, size, width, height, pagecount, duration, commenthidden, parsedcomment, comment, thumburl, thumbwidth, thumbheight, thumbmime, thumberror, url, sha1, metadata, extmetadata, commonmetadata, mime, mediadtype, bitdepth. Clients that process these fields should first check if filemissing is set. Fields that are supported even if the file is missing include: canonicaltitle, archivename (deleted files only), descriptionurl, descriptionshorturl.

So far I have only seen this happen with image-revisions greater than 1. Here's a quick fix that I've been using personally, since I noticed that entries with filemissing set seem to always be returned as the last element of imageinfo:

diff --git a/gallery_dl/extractor/wikimedia.py b/gallery_dl/extractor/wikimedia.py
index 2e8136f1..a103a06b 100644
--- a/gallery_dl/extractor/wikimedia.py
+++ b/gallery_dl/extractor/wikimedia.py
@@ -104,6 +104,12 @@ class WikimediaExtractor(BaseExtractor):
             yield Message.Directory, info
 
             for info["num"], image in enumerate(images, 1):
+                # https://www.mediawiki.org/wiki/Release_notes/1.34
+                if "filemissing" in image:
+                    self.log.warning(
+                        "File %s (or its revision) is missing",
+                        image["canonicaltitle"].partition(":")[2])
+                    continue
                 self.prepare_image(image)
                 image.update(info)
                 yield Message.Url, image["url"], image

This would break the continuity of the sequence number if invalid entries appear in the middle of imageinfo, and I'm not sure if that would be acceptable.

[mikf]

This would break the continuity of the sequence number if invalid entries appear in the middle of imageinfo, and I'm not sure if that would be acceptable.

We could do the num increment manually instead of relying on enumerate()

[ClosedPort22]

This would break the continuity of the sequence number if invalid entries appear in the middle of imageinfo, and I'm not sure if that would be acceptable.

We could do the num increment manually instead of relying on enumerate()

Got it. I wasn't sure whether num was intended to behave like this so I thought I'd ask first. While trying to fix this I searched through the code and found extractors with the same issue as my current fix (leaving gaps in num):

gallery-dl/gallery_dl/extractor/scrolller.py

Lines 58 to 67 in c269339

	for num, media in enumerate(album, 1):
	sources = media.get("mediaSources")
	if not sources:
	self.log.warning("%s/%s: Missing media file",
	post.get("id"), num)
	continue
	src = max(sources, key=self._sort_key)
	src["num"] = num
	files.append(src)
	return files

gallery-dl/gallery_dl/extractor/instagram.py

Lines 230 to 237 in c269339

	for num, item in enumerate(items, 1):
	try:
	image = item["image_versions2"]["candidates"][0]
	except Exception:
	self.log.warning("Missing media in post %s",
	data["post_shortcode"])
	continue

The Reactor and Weibo extractors, however, don't use enumerate and are not affected:

gallery-dl/gallery_dl/extractor/reactor.py

Lines 99 to 121 in c269339

	num = 0
	date = text.parse_datetime(data["datePublished"])
	user = data["author"]["name"]
	description = text.unescape(data["description"])
	title, _, tags = text.unescape(data["headline"]).partition(" / ")
	post_id = text.parse_int(
	data["mainEntityOfPage"]["@id"].rpartition("/")[2])
	if not tags:
	title, tags = tags, title
	tags = tags.split(" :: ")
	tags.sort()
	for image in images:
	url = text.extr(image, ' src="', '"')
	if not url:
	continue
	if url.startswith("//"):
	url = "http:" + url
	width = text.extr(image, ' width="', '"')
	height = text.extr(image, ' height="', '"')
	image_id = url.rpartition("-")[2].partition(".")[0]
	num += 1

gallery-dl/gallery_dl/extractor/weibo.py

Lines 106 to 121 in c269339

	num = 0
	for file in files:
	url = file["url"]
	if not url:
	continue
	if url.startswith("http:"):
	url = f"https:{url[5:]}"
	if "filename" not in file:
	text.nameext_from_url(url, file)
	if file["extension"] == "json":
	file["extension"] = "mp4"
	if file["extension"] == "m3u8":
	url = f"ytdl:{url}"
	file["_ytdl_manifest"] = "hls"
	file["extension"] = "mp4"
	num += 1

(There could be more of both types of extractors)