Support for Unicode string arrays

[ksugar]

Hi @bogovicj , it would be great to support Unicode string arrays.
Here is the minimal example:

import zarr
import numpy as np

za = zarr.open("string_test.zarr", mode="w", zarr_format=2)
za["string"] = np.array(["string"])
za.info
# Name        :
# Type        : Group
# Zarr format : 2
# Read-only   : True
# Store type  : LocalStore
za["string"]
# <Array file://test.zarr/string shape=(1,) dtype=<U6>
za["string"].info
# Type               : Array
# Zarr format        : 2
# Data type          : FixedLengthUTF32(length=6, endianness='little')
# Fill value         :
# Shape              : (1,)
# Chunk shape        : (1,)
# Order              : C
# Read-only          : True
# Store type         : LocalStore
# Filters            : ()
# Compressors        : (Zstd(level=0),)
# No. bytes          : 24

Thanks in advance.

[mkitti]

Will fixed length strings be sufficient?

[ksugar]

The DataType like >U6 needs to be supported?

[ksugar]

At the moment, we can assume the length is not so long (e.g. 256 chars maximum).

[mkitti]

Encoding the strings as UTF-8 might be more portable and efficient.

UTF-32 support might require an additional dependency.

https://bugs.openjdk.org/browse/JDK-8310251#:~:text=+%20*%20+%20*,byte%2Dorder%20marks.%20%3C/

[ksugar]

@mkitti yes, we can assume UTF-8 encoding, which would be good enough in most use cases.

[mkitti]

So let's not do U then. Please encode your strings to bytes and store those. If you need details, let me know.

[ksugar]

@mkitti thanks, I will go for that strategy .

[bogovicj]

Won't need another dependency because
Charset charset = Charset.forName("UTF-32");
seems to work just fine.

It will be more portable to store bytes, but since U is a valid zarr2 dtype, I'd like to support it.

[ksugar]

@bogovicj thanks. Using bytes and encoding/decoding works good for now, but it would be good to have FixedLengthUTF32 for compatibility with zarr2.

[bogovicj]

@ksugar ,

I have an early prototype - if you want to try it, use the feat/zarr-unicode of this repo, as well as the feat/zarr-unicode branch of the n5 repo.

Edit: this code works for me with that code

final String dset = "tracks/nodes/ids";
try ( final N5ZarrReader zarr = new N5ZarrReader("<path-to>/Fluo-N2DL-HeLa-01.zarr") ) {

	final ZarrDatasetAttributes attrs = zarr.getDatasetAttributes(dset);
	final DataBlock<String[]> blk = (DataBlock<String[]>)zarr.readBlock(dset, attrs, 0);

	final String[] data = blk.getData();
	System.out.println(data[0]);
	System.out.println(data[1]);
}

[mkitti]

Won't need another dependency because\nCharset charset = Charset.forName("UTF-32");\nseems to work just fine.

The effective dependency there is a recent Java version (Java 17 at least I think)

[tpietzsch]

@bogovicj I was looking at the n5 feat/zarr-unicode branch and it took me a while to figure out that ZARR_UNICODE (respectively ZarrUnicodeStringDataCodec) is fixed length in contrast to ZARR_STRING (respectively ZarrStringDataCodec) which is variable length.

I wonder if we should make this more prominent in the naming somehow? Or is this something that most people are familiar with?

[bogovicj]

@tpietzsch

it took me a while to figure out that ZARR_UNICODE... is fixed length
we should make this more prominent in the naming somehow?

Agreed.