readNativeFrames Performance Optimizations

[suyashkumar]

This is an issue to track various investigations and changes to optimize readNativeFrames implementation, which I noticed in pprof profiles seemed to take up a majority of the CPU time in parsing DICOMs with Native pixel data.

More comprehensive tests and benchmarks were added in #160 to aid with this.

[suyashkumar]

#157 led to some significant performance improvements by reducing slice allocations.

[suyashkumar]

Also, on this subject, @kaxap has observed that eliminating calls to binary.Read may reduce slice allocations and improve performance a bit, so that's something we should try. In particular, doing something like the following to read uint8, uint16, uint32s, etc:

buf := make([]byte, bitsAllocated/8)
for pixel := 0; pixel < pixelsPerFrame*samplesPerPixel; pixel++ {
  _ := io.ReadFull(d, buf)
  val := int(binary.LittleEndian.Uint16(buf))
}

The main advantage here is reusing the buffer slice (since bitsAllocated should be constant), instead of binary.Read creating one within.

[kaxap]

it is also possible to lazily load frames from file which will result in smaller memory footprint (some dicom files can be gigabytes in size)

[suyashkumar]

This is somewhat along the lines of #162--which is allowing interested parties to not read the pixel data, or perhaps, to pause reading before actually parsing the PixelData (which can be done with the Parser API, but not at the frame by frame level, but at the tag by tag level). We can also read in the bytes and do nothing with unpacking them until a user wants the image.Image, which is almost what we do today. I'm leaning towards something similar to current behavior + an option to not read PixelData (or any blocklist of tags potentially). We also already have a "streaming" option where frames can be consumed by a consumer as they are parsed, instead of having to wait for the full set of frames (or the whole dicom) to be parsed.

What do you think? Also, would love to learn more about your use cases if you have any particular ones. Cheers!

[kaxap]

Most of the cases all I want to do is to get some tags and a single frame rendered (for a thumbnail or a preview). Lazy mode (when we save offsets in file for each frame instead of reading all of them into RAM) looks very appealing in this case.

[bbeckrest]

Regarding memory usage, we've recently had our server (32GB RAM) run out of memory while attempting to parse a 1GB zipfile containing dicoms.

Using pprof, I've benchmarked parsing a 55MB zipfile containing 222 dicom images (each image 526KB, total ~117MB unzipped) and noticed readNativeFrames() ballooning memory usage to ~1.74GB.

Is this high memory usage (15x) expected?

[suyashkumar]

Interesting, thank you for reporting! I don't think this is expected. Out of curiosity, what version of the library are you using? Does it include some of the recent optimizations included in this issue?

I'll take a closer look tonight and dive deep if needed this weekend.