Linear scanning in CompileUnit.iter_DIEs()

[sevaa]

CompileUnit.iter_DIEs() is unnecessarily convoluted in terms of parse logic. Its purpose it to return the DIEs as they appear in the info section, but instead it performs recursive tree traversal via CompileUnit.iter_DIE_children(), which contains the more complicated logic of going over immediate children of a DIE. On a siblingless DIE tree (which is rare in practice but allowed by the format), this will contain extra work of parsing the whole subtree only to throw away the children of children.

Physically, DIEs are stored linearly. Logically, they are a tree. iter_DIEs() is expected to return a flat collection, but it does so by building a tree (out of a flat collection) and then flattening it. This PR changes iter_DIEs() to parse sequentially instead.

As a matter of performance, it shaves 5-7% from the linear scan time using iter_DIEs(). Tested on one rather artificial example; the speedup on real life DWARF might vary. Does not fix #623, but somewhat helps them.

As a downside, it duplicates the caching and the tree building logic already present in CompileUnit.iter_DIE_children().

---

On a side note, one reason I got started with this, I have an extensive collection of crash reports with a very similar condition that I've only recently tracked down to a root cause - the DW_AT_sibling on levels below the first had a bogus value. All affected binaries are PowerPC ELF files, so my working theory is that this is - or was - a bug in a PPC compiler. No way to reproduce, but I've been chasing this for a while. #580 was about that too.

Point is, the current version of iter_DIEs() relies on DW_AT_sibling because it's built on top of tree traversal. The one in this PR does not, so it's not susceptible to the issue (the immediate children traversal function still is). Far it be from me to introduce special cases for malformed DWARF, but I get my wins where I can :)

---

On a side side note, you can shave good 30% off scan time if you skip setting the parent/terminator, caching, and the possibility of supplemental DWARF - true firehose. That would look like this:

    def iter_DIEs_really_fast(self):
        stm = self.dwarfinfo.debug_info_sec.stream
        pos = self.cu_die_offset
        end_pos = self.cu_offset + self.size
        while pos < end_pos:
            die = DIE(self, stm, pos)
            yield die
            pos += die.size

I'm of half the mind to throw this in just for the benefit of @chall1123 with #623 :)

[eliben]

It'd also be good to add the extra context from the issue to the PR description / commit info, and link to the issue

[eliben]

Not sure what's happening here. Aren't we supposed to yield from _iter_die_subtree?

[sevaa]

You mean instead of
supp_die.cu._iter_DIE_subtree(supp_die)
it should have been
yield supp_die.cu._iter_DIE_subtree(supp_die)
? That was my first stab too, but apparently when you do that (at least in Python 3.13), a generator object is yielded to the caller, instead of its contents.

[eliben]

yield from?

Otherwise, I don't see what this call is even doing. _iter_DIE_subtree yields stuff, it's not usually called for side effects.

If you remove this call, do tests still pass?

[sevaa]

yield from is right. And the autotest didn't catch that. It does now.

[sevaa]

It'd also be good to add the extra context from the issue to the PR description / commit info, and link to the issue

What other context and what issue? The ticket about the crashes caused by bogus DW_AT_sibling values is in my own private bug tracker, not on Github. I was going to look around if there was a known compiler ticket about this, but I haven't yet.

[eliben]

It'd also be good to add the extra context from the issue to the PR description / commit info, and link to the issue

What other context and what issue? The ticket about the crashes caused by bogus DW_AT_sibling values is in my own private bug tracker, not on Github. I was going to look around if there was a known compiler ticket about this, but I haven't yet.

I mean the context from this comment: #623 (comment)