Feature Request: Autocapture Screenshots from Predefined Virtual Camera Poses

[shalins]

Summary

I'm writing to propose a new feature for the Rerun visualization tool: the ability to autocapture screenshots from fixed virtual camera poses for objects rendered in the 3D viewport (point clouds, 3d objects, etc.). This feature would significantly enhance our data visualization and debugging processes, especially in the context of machine learning model development and robotics.

Background

At our robotics company, we frequently work with multiple datasets to train and evaluate our machine learning models. Visualizing these datasets from various perspectives is crucial for in-depth analysis and effective debugging. Currently, Rerun allows us to visualize data on a per-need basis, but it lacks the capability to automatically capture and store visual snapshots from specific, predefined camera angles.

Feature Request

I would like to request the addition of an "Autocapture" functionality to Rerun. This feature should allow users to:

1. Define multiple virtual camera poses within the 3d viewport.
2. Automatically capture screenshots from these fixed camera poses.
3. Save these screenshots in a designated folder, with options to format file names for easy identification (e.g., dataset name, camera position, timestamp).

Use Case and Benefits

This feature would be immensely beneficial because it allows us to:

• Enhanced Debugging: Quickly log visual data from multiple datasets and capture snapshots for thorough examination.
• Consistent Visualization: Maintain uniformity in visual perspectives across different datasets and different model updates.
• Efficient Documentation: Easily attach before-and-after snapshots to PRs, which makes out development process far more effective and consistent over time.
• Streamlined Workflow: Automate the visual logging process, akin to scripts that generate metrics over multiple datasets, saving time and effort, especially if we can run this for hundreds of datasets.

Conclusion

Implementing this feature would greatly streamline our workflow and enhance our ability to analyze and debug complex datasets in robotics and machine learning applications. We believe it would also be a valuable addition for the broader Rerun user community engaged in similar tasks.

Thank you for considering this feature request.

[abey79]

Hello @shalins. Thanks for your input. Your proposal is legitimate and in-line with our roadmap.

The screenshot feature exists already via the command line. You can use rerun --screenshot-to /path/to/screenshot.png /path/to/file.rrd. It will however be of limited use until we build support for fine-grained control from code of the visualisation content (views, camera angles, etc.). It turns out that this is our next major focus over the coming weeks/months. We refer to this effort as "blueprint", and you can track progress with this label.

[nikolausWest]

To add to what @abey79 wrote though, I don't think headless rendering / screenshots is on our near term roadmap unfortunately so you'd still have to do this on a machine with a screen.

[julianschoep]

@nikolausWest curious, has there been an update on the headless rendering / screenshots front? Or is this not something we can really expect in the near future? I second that headless screenshotting would be very valuable for our workflows as well!

[Wumpf]

We have an experimental screenshot feature now, but it's unfortunately not yet headless and has a couple of limitations
See

• Experimental python API for taking viewer screenshots #12405
• Make programmatic screenshot taking more useful #12482