[kemono] AttributeError: 'str' object has no attribute 'copy'

[XCanG]

>gallery-dl --verbose https://kemono.cr/patreon/user/17152737/post/126135488
[gallery-dl][debug] Version 1.31.4
[gallery-dl][debug] Python 3.14.2 - Windows-10-10.0.19043-SP0
[gallery-dl][debug] requests 2.32.5 - urllib3 2.6.2
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://kemono.cr/patreon/user/17152737/post/126135488'
[kemono][debug] Using KemonoPostExtractor for 'https://kemono.cr/patreon/user/17152737/post/126135488'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): kemono.cr:443
[urllib3.connectionpool][debug] https://kemono.cr:443 "GET /api/v1/patreon/user/17152737/post/126135488 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://kemono.cr:443 "GET /api/v1/patreon/user/17152737/post/126135488/revisions HTTP/1.1" 200 733
[kemono][error] An unexpected error occurred: AttributeError - 'str' object has no attribute 'copy'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemono][debug]
Traceback (most recent call last):
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 84, in items
    for post in posts:
                ^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 251, in <genexpr>
    self._revisions_post(post) for post in posts)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 266, in _revisions_post
    rev["revision_hash"] = self._revision_hash(rev)
                           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 309, in _revision_hash
    rev["file"] = rev["file"].copy()
                  ^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'copy'

In conf it is setup to "revisions": "unique"

[XCanG]

If I add print to this moment I would see those revisions (last one crashes):

{'id': '126135488', 'user': '17152737', 'service': 'patreon', 'title': 'Weekly content!', 'content': '<p>...</p>', 'embed': {}, 'shared_file': False, 'published': '2025-04-07T15:59:57', 'edited': '2025-08-23T13:18:36', 'file': {'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}, 'attachments': [], 'poll': None, 'captions': None, 'tags': None}

{'id': '126135488', 'user': '17152737', 'service': 'patreon', 'title': 'Weekly content!', 'content': '<p>...</p>', 'embed': {}, 'shared_file': False, 'published': '2025-04-07T15:59:57', 'edited': '2025-04-07T15:59:57', 'file': {'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}, 'attachments': [{'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}], 'poll': None, 'tags': None, 'captions': None}

{'id': '126135488', 'user': '17152737', 'service': 'patreon', 'title': 'Weekly content!', 'content': '<p>...</p>', 'embed': {}, 'shared_file': False, 'published': '2025-04-07T15:59:57', 'edited': '2025-04-07T15:59:57', 'file': {'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}, 'attachments': [{'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}], 'poll': None, 'tags': None, 'captions': None}

{'id': '126135488', 'user': '17152737', 'service': 'patreon', 'title': 'Weekly content!', 'content': '<p>...</p>', 'embed': {}, 'shared_file': False, 'published': '2025-04-07T15:59:57', 'edited': '2025-04-07T15:59:57', 'file': {'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}, 'attachments': [{'name': '448451741.png', 'path': '/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png'}], 'poll': None, 'tags': None, 'captions': None}

{'id': '126135488', 'user': '17152737', 'service': 'patreon', 'title': 'Weekly content!', 'content': '<p>...</p>', 'embed': '{}', 'shared_file': False, 'published': '2025-04-07T15:59:57', 'edited': '2025-04-07T15:59:57', 'file': '{"name": "448451741.png", "path": "/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png"}', 'attachments': ['{"name": "448451741.png", "path": "/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png"}'], 'poll': None, 'tags': None, 'captions': None}

In this case all of the values, not just file is stringified JSON. Attachments as well.

[XCanG]

So the fix could be something like:

rev["file"] = file.copy() if isinstance((file := rev["file"]), dict) else json.loads(file)
rev["file"].pop("name", None)
rev["attachments"] = [a.copy() if isinstance(a, dict) else json.loads(a) for a in rev["attachments"]]
for a in rev["attachments"]:
    a.pop("name", None)

However if it fixed here, in another place error arrives:

[kemono][debug]
Traceback (most recent call last):
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 130, in items
    for file in itertools.chain.from_iterable(
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
            g(post) for g in generators):
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 131, in <genexpr>
    g(post) for g in generators):
    ~^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 225, in _attachments
    attachment["type"] = "attachment"
    ~~~~~~~~~~^^^^^^^^
TypeError: 'str' object does not support item assignment

[XCanG]

This could be fixed by something like this:

def _attachments(self, post):
    for attachment in post["attachments"]:
        if isinstance(attachment, str):
            attachment = json.loads(attachment)
        attachment["type"] = "attachment"
    return post["attachments"]

But this will propagate further:

[kemono][error] An unexpected error occurred: TypeError - string indices must be integers, not 'str'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemono][debug]
Traceback (most recent call last):
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 132, in items
    url = file["path"]
          ~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'

[XCanG]

I was tried this fix:

for file in itertools.chain.from_iterable(
        g(post) for g in generators):
    if isinstance(file, str):
        file = json.loads(file)
    url = file["path"]

However now it create some sort of different issue:

[kemono][error] An unexpected error occurred: KeyError - 'type'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemono][debug]
Traceback (most recent call last):
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 141, in items
    if file["type"] not in duplicates and hash in hashes:
       ~~~~^^^^^^^^
KeyError: 'type'

So I think I missed something and it better to do a different way.

Looking at the file, it was equal to: {"name": "448451741.png", "path": "/b3/4d/b34d2ad89a59efa3746643c657310043bbada32751138aab7d23523fd1a5b765.png"}

So for now I use this approach:

if file.get("type") not in duplicates and hash in hashes:

to skip the key in this buggy scenario.

---

This will show that in other place another issue happened:

[kemono][error] An unexpected error occurred: TypeError - 'str' object does not support item assignment. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemono][debug]
Traceback (most recent call last):
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 158, in run
    msg = self.dispatch(extractor)
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\job.py", line 209, in dispatch
    for msg, url, kwdict in messages:
                            ^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 130, in items
    for file in itertools.chain.from_iterable(
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
            g(post) for g in generators):
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 131, in <genexpr>
    g(post) for g in generators):
    ~^^^^^^
  File "C:\Python\Python314\Lib\site-packages\gallery_dl\extractor\kemono.py", line 223, in _file
    file["type"] = "file"
    ~~~~^^^^^^^^
TypeError: 'str' object does not support item assignment

This is fixed with:

def _file(self, post):
    file = post["file"]
    if isinstance(file, str):
        file = json.loads(file)
    if not file or "path" not in file:
        return ()
    file["type"] = "file"
    return (file,)

After all those changes everything else worked. But I think there should be a better solution to parse those strings somewhere in 1 place, rather than patching in all those places.

[mikf]

file and attachments are almost always a (list of) dict, except in this one specific case...

[XCanG]

Well, I can understand that.

I was quite wondered how you could manage such a big project without relying more on types and tools like ruff. Sure that it add some overhead and take more time to fill with data, but to me, an outsider, looking at code with almost all of the variables being Any is hard to understand what is going on and where stuff are comes from. Like during my debugging when I find that "type" is inner parameter I couldn't find the way it is preserved (I find where it is assigned, but it was not clear how it keep further down the line).

And some variables even thow an error in IDE, but that is because it doesn't know about it existance, but it somehow somewhere assigned.

[mikf]

diff --git a/gallery_dl/extractor/kemono.py b/gallery_dl/extractor/kemono.py
index 2b44a24d..fc8f23ca 100644
--- a/gallery_dl/extractor/kemono.py
+++ b/gallery_dl/extractor/kemono.py
@@ -303,2 +303,7 @@ class KemonoExtractor(Extractor):
     def _revision_hash(self, revision):
+        if isinstance(revision["file"], str):
+            revision["file"] = util.json_loads(revision["file"])
+            revision["attachments"] = [
+                util.json_loads(a) for a in revision["attachments"]]
+
         rev = revision.copy()

Then again, this should probably be checked at a more "central" place...