compress json data arrays

[marius311]

This compresses Float, Int, and String arrays inside the plot data to massively reduce html file size and improve performance.

• For numeric arrays, on the Julia side, we first Zlib compress, then base64 encode. On the Javascript side (via a new injected snippet), we base64 decode, Zlib uncompress, then leave as a typed array and pass directly into Plotly (this latter fact helps a ton with loading speed too).
• For string arrays, it concatenates / compresses to one long string and stores a lookup table of where the substrings are.

Recent Plotly versions have the ability to feed in base64 strings directly, but I chose to do it this way to allow for the Zlib compression and because we can also compress string arrays, which Plotly doesn't do. Zlib compressed string arrays are nice because in this domain its common for eg mesh.facecolor = [...] with hex strings and many repeated entries, so you get huge compression factors. Python Plotly uses the base64 feature but doesn't have compression or string arrays, so won't be as good as this package.

Open to feedback on this, lmk, I know there's some opionated choices. Anecdotally this code has been in production at @Atomic-Industries for ~1yr and is working great.

Here's an example code snippet that results in a 7.4Mb file on master and 1.2Mb file on this PR (and 3.4Mb with Python, I think the difference is Zlib)

using PlotlyLight, Downloads, Gmsh

# load sample mesh with ~100K triangles
gmsh.initialize()
Downloads.download("https://raw.githubusercontent.com/alecjacobson/common-3d-test-models/master/data/happy.obj", "happy.obj")
gmsh.merge("happy.obj")
node_tags, coords, _ = gmsh.model.mesh.getNodes()
(x, y, z) = eachrow(reshape(coords, 3, :))
tag2idx = Dict(tag => idx - 1 for (idx, tag) in enumerate(node_tags))
_, _, elem_node_tags = gmsh.model.mesh.getElements(2)
tri_node_tags = elem_node_tags[1]
(i, j, k) = eachrow(getindex.(Ref(tag2idx), reshape(tri_node_tags, 3, :)))

p = Plot(Config(;x, y, z, i, j, k, type="mesh3d"))
PlotlyLight.save(p, "happy.html")

import gmsh
import urllib.request
import plotly.graph_objects as go
import numpy as np

# load sample mesh with ~100K triangles
gmsh.initialize()
urllib.request.urlretrieve("https://raw.githubusercontent.com/alecjacobson/common-3d-test-models/master/data/happy.obj", "happy.obj")
gmsh.merge("happy.obj")
node_tags, coords, _ = gmsh.model.mesh.getNodes()
x = coords[0::3]
y = coords[1::3]
z = coords[2::3]
tag2idx = {tag: idx for idx, tag in enumerate(node_tags)}
_, _, elem_node_tags = gmsh.model.mesh.getElements(2)
tri_node_tags = elem_node_tags[0]
i = np.array([tag2idx[tag] for tag in tri_node_tags[0::3]])
j = np.array([tag2idx[tag] for tag in tri_node_tags[1::3]])
k = np.array([tag2idx[tag] for tag in tri_node_tags[2::3]])

fig = go.Figure(data=[go.Mesh3d(x=x, y=y, z=z, i=i, j=j, k=k)])
fig.write_html("happy_py.html", include_plotlyjs="cdn")

[joshday]

Very cool! Before merging, I think I'd want this to be implemented as an extension package and make it opt-in. Maybe add preset.display.compress!()? Or add compress::Bool to Settings

[marius311]

Thanks yea I can do the optional. Would you consider non-extension though? It just that its another step for the user and CodecZlib seems pretty light, the only leaf dependencies added are TranscodingStreams and CodecZlib_jll. Ok either way just putting out there.

[marius311]

Made it opt-in. Also added to readme, as well as removed the note about JSON3.write which sounded contradictory to the the compression (and I think was no longer true anyway). Lmk about the extension thing.

[joshday]

Would you consider non-extension though?

Yeah, that's cool. LGTM