Update to 6.9.1 breaks merging due to stream length checks

[rkrell]

I get now
pypdf.errors.LimitReachedError: Declared stream length of 77439792 exceeds maximum allowed length
in my merges which worked regularly until 6.9.0.

It is probably not a good idea to use static stream lengths here, but at least to have this value (MAX_DECLARED_STREAM_LENGTH) configurable.

Environment

pypy3.11-7.3.20, Linux 64-bit, Debian 11

$ python -m platform
Linux-5.10.0-39-amd64-x86_64-with-glibc2.31

$ python -c "import pypdf;print(pypdf._debug_versions)"
pypdf==6.9.1, crypt_provider=('cryptography', '46.0.5'), PIL=12.1.1

Traceback

This is the complete traceback I see:

Traceback (most recent call last):
  File "/home/user/project/printhouse_jobs/merge.py", line 1305, in merge
    results = [task.result() for task in completed]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/printhouse_jobs/merge.py", line 1305, in <listcomp>
    results = [task.result() for task in completed]
               ^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/pypy3.11-7.3.20/lib/pypy3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/user/.pyenv/versions/pypy3.11-7.3.20/lib/pypy3.11/asyncio/tasks.py", line 279, in __step
    result = coro.throw(exc)
             ^^^^^^^^^^^^^^^
  File "/home/user/project/printhouse_jobs/merge.py", line 341, in _do_merge
    writer = await pdf_filter.merge()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/printhouse_jobs/pagelayout/default.py", line 33, in merge
    await add_pypdf_page(merger, self.base_dir, file)
  File "/home/user/project/printhouse_jobs/utils/canvas.py", line 109, in add_pypdf_page
    await asyncify(writer.append, limiter=CapacityLimiter(1))(r)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/asyncer/_main.py", line 375, in wrapper
    return await run_sync(
           ^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/asyncer/_compat.py", line 25, in run_sync
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/anyio/to_thread.py", line 63, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/anyio/_backends/_asyncio.py", line 2502, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/pypy3.11-7.3.20/lib/pypy3.11/asyncio/futures.py", line 287, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^
  File "/home/user/.pyenv/versions/pypy3.11-7.3.20/lib/pypy3.11/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/home/user/.pyenv/versions/pypy3.11-7.3.20/lib/pypy3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/_writer.py", line 2621, in append
    self.merge(
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/_writer.py", line 2702, in merge
    srcpages[pg.indirect_reference.idnum] = self.add_page(
                                            ^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/_writer.py", line 611, in add_page
    return self._add_page(page, len(self.flattened_pages), excluded_keys)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/_writer.py", line 514, in _add_page
    "PageObject", page_org.clone(self, False, excluded_keys).get_object()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 308, in clone
    d__._clone(self, pdf_dest, force_duplicate, ignore_fields, visited)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 419, in _clone
    v.clone(pdf_dest, force_duplicate, ignore_fields)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 308, in clone
    d__._clone(self, pdf_dest, force_duplicate, ignore_fields, visited)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 419, in _clone
    v.clone(pdf_dest, force_duplicate, ignore_fields)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 308, in clone
    d__._clone(self, pdf_dest, force_duplicate, ignore_fields, visited)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 419, in _clone
    v.clone(pdf_dest, force_duplicate, ignore_fields)
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_base.py", line 363, in clone
    obj = self.get_object()
          ^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_base.py", line 382, in get_object
    return self.pdf.get_object(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/_reader.py", line 506, in get_object
    retval = read_object(self.stream, self)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 1490, in read_object
    return DictionaryObject.read_from_stream(stream, pdf, forced_encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.pyenv/versions/project-pypy3.11-7.3.20/lib/pypy3.11/site-packages/pypdf/generic/_data_structures.py", line 649, in read_from_stream
    raise LimitReachedError(f"Declared stream length of {length} exceeds maximum allowed length.")
pypdf.errors.LimitReachedError: Declared stream length of 77439792 exceeds maximum allowed length.

[stefan6419846]

which worked regularly until 6.9.0.

The relevant change happened in version 6.8.0 to mitigate a memory issue with invalid length values. Version 6.9.1 should not have corresponding changes as far as I am aware.

It is probably not a good idea to use static stream lengths here, but at least to have this value (MAX_DECLARED_STREAM_LENGTH) configurable.

This already is configurable, see the docs: https://pypdf.readthedocs.io/en/latest/user/security.html#filters (There are some undocumented limit values which are configurable less easily, but you are not hitting them here.)

At the moment, this requires monkey-patching:

import pypdf.filters

pypdf.filters.MAX_DECLARED_STREAM_LENGTH = 100_000_000

...

[rkrell]

which worked regularly until 6.9.0.

The relevant change happened in version 6.8.0 to mitigate a memory issue with invalid length values. Version 6.9.1 should not have corresponding changes as far as I am aware.

OK, 19th of March I updated from 6.8.0 to 6.9.1, but apparently didn't hit this issue yet.
So we met a bigger one now. We usually process quite "big" files with embedded high quality images.

It is probably not a good idea to use static stream lengths here, but at least to have this value (MAX_DECLARED_STREAM_LENGTH) configurable.

This already is configurable, see the docs: https://pypdf.readthedocs.io/en/latest/user/security.html#filters (There are some undocumented limit values which are configurable less easily, but you are not hitting them here.)

At the moment, this requires monkey-patching:

import pypdf.filters

pypdf.filters.MAX_DECLARED_STREAM_LENGTH = 100_000_000

...

Aha, great, thanks for the hint.

[stefan6419846]

We usually process quiet "big" files with embedded high quality images.

The default limits usually are a trade-off - while they work for most of the smaller workflows and avoid excessive memory usage, they might not work for more complex use cases. I personally overwrite some of these values as well to allow proper processing.