Fix transforms & videos breaking when receiving new data in the background

[Wumpf]

Related

• Fixes RR-3329

What

This keeps all caches up to date.
We first tried discarding them which also worked fine, but decided to go with keeping things up to date since most likely those recordings are still in used. If it gets too much memory used, GC will do.

[github-actions]

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	7203596	https://rerun.io/viewer/pr/12452	+nightly +main

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

[emilk]

The original problem is that not all store events reach all the cashes.

For instance, it seems like the transform cache relies on knowing about all store additions:

rerun/crates/store/re_tf/src/transform_resolution_cache.rs

Lines 957 to 961 in fdbe7d3

	if event.kind == re_chunk_store::ChunkStoreDiffKind::Addition {
	// Since entity paths lead to implicit frames, we have to prime our lookup table with them even if this chunk doesn't have transform data.
	self.frame_id_registry
	.register_all_frames_in_chunk(&event.chunk);
	}

It seems like Cache::on_store_events were added in other to be able to selectively invalidate on store deletions, but some caches abuse it to also look as store additions, effectively becoming store subscribers rather than caches.

Unless we change the interface of trait Cache to explicitly only pass deletions, then we must face the possibility that some "caches" will need to get all store events, and not just some of them.

[Wumpf]

The original problem is that not all store events reach all the cashes.

so but if we don't have a cache to begin with it should work fine since upon initial creation of any of these caches we restore all the data from the store. So I do think this is a viable approach!

Relevant "examples":

• transform cache populates itself from the store on first access

  rerun/crates/viewer/re_viewer_context/src/cache/transform_database_store.rs

  Line 35 in a65732f

  .add_chunks(entity_db.storage_engine().store().iter_chunks());

• video stream cache doing a similar thing on a per video base

  rerun/crates/viewer/re_viewer_context/src/cache/video_stream_cache.rs

  Line 138 in a65732f

  load_video_data_from_chunks(store, entity_path, timeline)?;

I meant to draft this PR though since I didn't have a repro before/after yet as noted on Slack, that's an oversight on my part!

[emilk]

upon initial creation of any of these caches we restore all the data from the store

Oh I missed that! Then this should work.

I will try to repro and test

[emilk]

I created this issue, which may be related: https://linear.app/rerun/issue/RR-3342/debug-crash-video-santity-check

[emilk]

Me and @Wumpf reproed the problem like so:

diff --git a/docs/snippets/all/archetypes/transform3d_hierarchy_frames.py b/docs/snippets/all/archetypes/transform3d_hierarchy_frames.py
index 9aa3ab4acf..f2b03fd1f4 100644
--- a/docs/snippets/all/archetypes/transform3d_hierarchy_frames.py
+++ b/docs/snippets/all/archetypes/transform3d_hierarchy_frames.py
@@ -1,5 +1,6 @@
 """Logs a transform hierarchy using named transform frame relationships."""
 
+import time
 import numpy as np
 import rerun as rr
 
@@ -22,6 +23,10 @@ rr.log(
     rr.CoordinateFrame("sun_frame"),
 )
 
+print("Sleeping…")
+time.sleep(3.0)
+print("Slept")
+
 rr.log(
     "planet",
     rr.Ellipsoids3D(
(base) 

(switch away from the recording during the sleep).

And this PR fixes the problem!

[Wumpf]

as discussed: alternative approach of always updating the caches. Confirmed with above snippet