Local/S3 zipped Zarr is not slower than Zarr

[olli4]

I implemented an async fsspec file system for reading from a zip file in another fsspec filesystem (S3, local). Using it, reading from a zipped Zarr is not slower than reading from a Zarr.

[csaybar]

Hi @olli4,

Thanks for commenting on the code. As @lnicola pointed out here #5 , several of my assumptions regarding Zarr+ZIP being slower were incorrect.

JTLUK, I've been experimenting with GDAL's support for Zarr+ZIP (via VFS) and comparing it to Cloud Optimized GeoTIFFs (COGs), using the same compression algorithm, chunking (512x512), and INTERLEAVE=TILE. COGs still retrieve spectral profiles faster, but I suspect this is due to GDAL's current implementation (GDAL_HTTP_MERGE_CONSECUTIVE_RANGES) rather than an inherent format thing. Once GDAL supports ZEP 2, I would expect the same performance.

I'm not sure which creation options you're using for COG, but I'm fairly certain the results you're seeing for SAFE are not accurate if you consider the discussion here: https://discourse.pangeo.io/t/cloud-optimized-access-to-sentinel-2-jpeg2000/5072

[olli4]

We are benchmarking reading satellite image time series which favor multidimensional formats like Zarr, I think mainly because a single Zarr read will retrieve data for multiple time points, not so much because of any improvement in compression efficiency. The ESA satellite image dissemination zipped Zarr examples I've seen have been for single images to which our time series read benchmarking does not directly apply.

For COG creation, for each image, we stored all channels of the same resolution in the same file and used rio-cogeo with default options.

Also with zipped Zarr it would be interesting to try combining reads of files that are stored consecutively inside a zip file. It is not super easy because Zarr-Python's FsspecStore does single-file _cat_file() awaits rather than reading multiple files in a single call. Would need to go deeper.

Thanks for that link on SAFE. It explains why we have seen many short HTTP range requests scanning through a JPEG2000 file when trying to read just a small patch (see picture). Our SAFE benchmarking has been for reading the current SAFE products that have no Tile Length Markers (TLMs) to jump straight to the data of interest. Great ideas in that thread (to build an external TLM index for primary SAFEs or to add TLMs to SAFEs), but not something we are likely to benchmark ourselves.

[csaybar]

By default, rio-cogeo uses the PIXEL interleave configuration. Switching to TILE can improve performance for partial reads. Although it will make files bigger. IDK if this might be relevant to your experiments.

https://gdal.org/en/stable/drivers/raster/cog.html#general-creation-options

It’s also worth noting that GDAL and Zarr-Python implement different caching. I usually get a huge boost when I configure GDAL for reading COGs.

https://developmentseed.org/titiler/advanced/performance_tuning/

[mkitti]

The main issue with zip files is that they contain a single flat central directory. For a limited number of chunks this should not be a problem. However, if there a single large array with many chunks among other arrays, the lookup time for chunks may become problematic depending on the zip implementation. Some zip implementations parse the entire central directory into a hash map or table, in which case there might be a high initial upfront cost but subsequent lookups may be relatively fast, O(1), in terms of time complexity. Other implementations may maintain the central directory of a list and iterate through it for every query, O(N) time complexity.

ZEP 2 shards at least isolate the index to each array. I would encourage the use of ZEP 2 shards even within a ZIP file, especially as the number of chunks is high.

In summary, the accessibility of zipped Zarr needs to be evaluated in terms of the number of chunks and how the central directory is parsed.

[csaybar]

My post was about Zarr in an EO context.
https://discourse.eopf.copernicus.eu/