[aruco] Restore write access to objPoints and ids(or provide setters) for custom 3D reconstruction, curved markers, and projection-based calibration #4087
Description
Problem Description
After upgrading from OpenCV 4.5.5 to 4.11.0, I found that the objPoints and ids members of the cv::aruco::Board(or related) class have been changed from publicly modifiable to read-only. This breaks our existing workflow and prevents several advanced, legitimate use cases.
While Boardprovides a constructor that accepts objPoints and ids, this is insufficient. It forces a one-time, immutable initialization pattern, which is cumbersome and incompatible with dynamic applications where these parameters need to be updated after object creation (e.g., after computing new 3D points from an online reconstruction).
Furthermore, the design of GridBoard inherently assumes a planar layout, which is a specific case of a more general concept: a grid of IDs. For flexibility, the core abstraction should separate the logical grid of marker IDs from their physical 3D coordinates, allowing the latter to be customized.
Use Cases and Why This Change is Problematic
The inability to modify these members makes the following important applications impossible or unnecessarily complex:
Integration with External 3D Reconstruction: When using external SLAM, Structure-from-Motion (SfM), or lidar scanning to obtain more accurate 3D coordinates of marker corners, we need to feed these optimized objPointsback into the Aruco module for subsequent camera pose optimization that is consistent with the refined geometry.
Markers on Physical Curved Surfaces: For physical markers attached to complex surfaces like cylinders or spheres, the true 3D point cloud is not a planar grid. We require the ability to directly set these non-planar objPointsto compute the correct camera pose relative to the curved surface.
Virtual Production / Projection-Based Calibration (Critical - Updated): In virtual production/XR studios, a standard workflow is to project Aruco markers onto a large, curved LED wall for calibration.
The logical layout (ids) is still a regular grid, so using GridBoard to generate this layout is appropriate.
The physical 3D points (objPoints) must be derived from the CAD model of the curved LED wall and the known projection mapping. The standard planar objPoints generated by GridBoard are incorrect.
The efficient solution is to: create a GridBoard for the ID layout, then replace its objPoints with the precisely calculated 3D points on the curved surface. This is currently impossible with the read-only members, forcing inefficient workarounds like creating a custom Boardfrom scratch and manually aligning IDs.
Additional Complexity - Split & Discontinuous IDs: A single logical GridBoard is often split and projected onto multiple, separate screen regions. This results in a non-contiguous or re-ordered set of ids across the physical setup, while the objPointsare derived from the respective screen surface geometries. The current API, lacking write access to both ids and objPoints, makes it exceedingly cumbersome to construct such a composite board, as we cannot easily adapt a base GridBoardto match the fragmented physical layout.
In version 4.5.5, we could support all these scenarios elegantly by directly accessing and modifying these vectors. The read-only restriction in the new version forces us to implement complex, inefficient, and error-prone data management and duplication to “fit“ the library’s new API.
Desired Behavior
Provide public setters or restore public access: For classes like cv::aruco::Board, provide setObjPoints()and setIds()methods, or simply revert the members to being publicly writable. This is the most straightforward and flexible solution, enabling essential workflows like “create-gridboard-then-replace-objpoints” and adjusting IDs for composite projection setups.
Reconsider GridBoard abstraction: Ideally, the library should better separate the logical “grid of IDs” from the “3D geometry”. A more powerful design might allow a GridBoard to be initialized with custom 3D geometry and ID sequences. At a minimum, providing write access to the base Board data is essential for building such flexibility on top of the existing API.
Current Workaround and Its Limitation
The only current workaround is to fork OpenCV and revert the change locally or maintain a private patch. While this is technically possible, it defeats the purpose of using a common, widely-supported library. It creates maintenance burdens, isolates us from upstream fixes and improvements, and is not a viable solution for the broader community that likely faces similar challenges.
We strongly believe that for the sake of API consistency, flexibility, and supporting professional use cases, the OpenCV library itself should restore this functionality. This is a matter of library design catering to both simplicity for beginners and power for advanced users.
Environment
OpenCV: 4.11.0
Operating System: Windows 11
Compiler: MSVC 2022
Thank you for considering this request. This change is crucial for users pushing the boundaries of marker-based tracking in professional environments.