Tutorial: Teleoperation with OpenXR Hand Tracking

[tbirdso]

Addresses rows (32) and (33) in I4H v0.2 Execution Plan

Description

Integrates the CloudXR and Isaac teams' OpenXR hand tracking support in Isaac for Healthcare.

1. Adds support for Isaac Lab's OpenXR-based Se3HandTracking in the Robotic Ultrasound teleoperation workflow sample.
2. Adds tutorial for using Isaac Lab + CloudXR OpenXR components to run the robotic_ultrasound teleoperation sample with XR hand tracking inputs.

Notes:

• Changes are in draft stage. Feedback and suggestions for improvements are welcome.
• Isaac Lab OpenXR setup is largely informed by the Isaac Lab OpenXR tutorial, streamlined for Isaac for Healthcare and limited to v2.0.2 feature availability and API.
• Tested locally with Apple Vision Pro.
  • Last tested based on 929ed75. Rebased on main for PR submission.
• Observed limitations are noted in the tutorial, including:
  • Some XR features present in the v2.1.0 API are missing from the robotic_ultrasound pinned Isaac Lab version v2.0.2 (retargeting), limiting ease of use
  • Observed application error when room camera and XR device are rendering at the same time. External data limited to wrist camera + joint position during teleop

[Nic-Ma]

Hi @KumoLiu , @mingxin-zheng , could you please help take a look at this DDS issue?

Thanks.

[KumoLiu]

Thank you for the update! I've tried the latest code and overall it looks good and works.
There are some remain issues related to rotation and the field of view as displayed on the screen. I've left a few comments inline for your reference. Additionally, after starting the AR feature, the initial pose appears awkward. Do you know how we can adjust it to better display all the assets within the scene?

I haven't had the chance to review the DDS-related aspects yet, but I plan to look into them later.

[KumoLiu]

Setting zero_out_xy_rotation=True may prevent the probe from correctly orienting downward if it is not initially pointing in that direction. To address this, I developed a custom Retargeter that primarily utilizes the 6DOF from the palm. This adjustment resulted in significantly improved teleoperation of the probe within these degrees of freedom. Additionally, I observed that the Se3RelRetargeter offers options such as use_wrist_position and use_wrist_rotation. Do you think we might be able to apply this approach here? If you could test these settings—use_wrist_position=True, use_wrist_rotation=True, zero_out_xy_rotation=False—and see how they perform, I’d appreciate it. Otherwise, I'm also available to share the customized retargeter for your evaluation.

I am also happy to help verify when I back office.

[tbirdso]

Hi @KumoLiu , good point. I've updated the teleop sample to use wrist translation and rotation, in my experience I see only a minor difference in ease of control. However I've also reduced the rotation scale factor from the default value of 10.0 and the probe is now easier to control in 6DOF.

Please try out the latest update, if you are seeing significant improvement with your custom retargeter then please feel free to contribute the custom retargeter, thanks!

[KumoLiu]

Thanks, it works better now.
Let's use wrist pos and rot for now.

Also put my customized regarter here if you are interested in, The main difference is that in this regarter, I'm trying to use hand_data.get("palm") pos an rot.

class PalmRelRetargeter(RetargeterBase):
    """Retargets OpenXR palm tracking data to end-effector commands using relative positioning."""
    def __init__(
        self,
        bound_hand: OpenXRDevice.TrackingTarget,
        zero_out_xy_rotation: bool = False,
        delta_pos_scale_factor: float = 10.0,
        delta_rot_scale_factor: float = 10.0,
        alpha_pos: float = 0.5,
        alpha_rot: float = 0.5,
    ):
        """Initialize the palm relative motion retargeter.

        Args:
            bound_hand: The hand to track (OpenXRDevice.TrackingTarget.HAND_LEFT or OpenXRDevice.TrackingTarget.HAND_RIGHT)
            zero_out_xy_rotation: If True, ignore rotations around x and y axes, allowing only z-axis rotation.
            delta_pos_scale_factor: Amplification factor for position changes.
            delta_rot_scale_factor: Amplification factor for rotation changes.
            alpha_pos: Position smoothing parameter (0-1); higher values track input closely, lower values smooth more.
            alpha_rot: Rotation smoothing parameter (0-1); higher values track input closely, lower values smooth more.
        """
        if bound_hand not in [OpenXRDevice.TrackingTarget.HAND_LEFT, OpenXRDevice.TrackingTarget.HAND_RIGHT]:
            raise ValueError(
                "bound_hand must be either OpenXRDevice.TrackingTarget.HAND_LEFT or"
                " OpenXRDevice.TrackingTarget.HAND_RIGHT"
            )
        self.bound_hand = bound_hand

        self._zero_out_xy_rotation = zero_out_xy_rotation
        self._delta_pos_scale_factor = delta_pos_scale_factor
        self._delta_rot_scale_factor = delta_rot_scale_factor
        self._alpha_pos = alpha_pos
        self._alpha_rot = alpha_rot

        self._smoothed_delta_pos = np.zeros(3)
        self._smoothed_delta_rot_vec = np.zeros(3)

        self._position_threshold = 0.002  # Meters
        self._rotation_threshold = 0.02  # Radians

        self._previous_palm_data: np.ndarray | None = None # Stores [px, py, pz, qw, qx, qy, qz]


    def retarget(self, data: dict) -> np.ndarray:
        """Convert palm pose to robot end-effector command.

        Args:
            data: Dictionary mapping tracking targets to joint data.
                  The joint data for "wrist" is expected as [pos_x, pos_y, pos_z, quat_w, quat_x, quat_y, quat_z].

        Returns:
            np.ndarray: 6D array containing position delta (xyz) and rotation delta vector (rx,ry,rz)
                        for the robot end-effector.
        """
        hand_data = data.get(self.bound_hand)
        if hand_data is None:
            return np.zeros(6) # No data for the bound hand

        current_palm_data = hand_data.get("palm")
        if current_palm_data is None:
            # Wrist data not available, return zero command
            return np.zeros(6)

        if self._previous_palm_data is None:
            # First frame with valid wrist data, store and return zero command
            self._previous_palm_data = current_palm_data.copy()
            return np.zeros(6)

        # Extract current pose
        current_pos = current_palm_data[:3]
        # Scipy expects quaternion as [x, y, z, w]
        current_rot_scipy = Rotation.from_quat([current_palm_data[4], current_palm_data[5], current_palm_data[6], current_palm_data[3]])

        # Extract previous pose
        previous_pos = self._previous_palm_data[:3]
        previous_rot_scipy = Rotation.from_quat([self._previous_palm_data[4], self._previous_palm_data[5], self._previous_palm_data[6], self._previous_palm_data[3]])

        # Calculate delta pose
        delta_pos = current_pos - previous_pos
        delta_rot_obj_scipy = current_rot_scipy * previous_rot_scipy.inv()
        delta_rot_vec = delta_rot_obj_scipy.as_rotvec()

        # Store current pose for next iteration
        self._previous_palm_data = current_palm_data.copy()

        # Process deltas for command generation
        command_pos = delta_pos
        command_rot_vec = delta_rot_vec

        if self._zero_out_xy_rotation:
            command_rot_vec[0] = 0.0  # Zero out x-axis rotation
            command_rot_vec[1] = 0.0  # Zero out y-axis rotation

        # Smooth position and rotation
        self._smoothed_delta_pos = self._alpha_pos * command_pos + (1 - self._alpha_pos) * self._smoothed_delta_pos
        self._smoothed_delta_rot_vec = self._alpha_rot * command_rot_vec + (1 - self._alpha_rot) * self._smoothed_delta_rot_vec

        # Apply thresholds to ignore small movements
        if np.linalg.norm(self._smoothed_delta_pos) < self._position_threshold:
            self._smoothed_delta_pos = np.zeros(3)
        if np.linalg.norm(self._smoothed_delta_rot_vec) < self._rotation_threshold:
            self._smoothed_delta_rot_vec = np.zeros(3)

        # Scale to get final command
        final_pos_command = self._smoothed_delta_pos * self._delta_pos_scale_factor
        final_rot_command_vec = self._smoothed_delta_rot_vec * self._delta_rot_scale_factor


        return np.concatenate([final_pos_command, final_rot_command_vec])

[KumoLiu]

Could you clarify how we obtain the initial position and rotation? Does this pertain to the initial position of the headset? I observed that what I see on the Vision Pro does not accurately reflect on IsaacLab. Is this discrepancy expected, or do you have any suggestions on how we can adjust the settings so that what we view through the headset is accurately displayed on the screen?

[tbirdso]

These settings were experimentally chosen based on observation so that the initial headset position is adjacent to the sample workbench. We can adjust them to maximize user comfort. To my knowledge we do not have control over the screen display viewport during AR mode.

[KumoLiu]

To my knowledge we do not have control over the screen display viewport during AR mode.

Got it, thanks for the explaination.

[KumoLiu]

Why we need to matmul this transform matrix here? Does it related to the initial state of the probe?

[tbirdso]

Observed discrepancy between the hand tracking reference frame and the probe frame. Without the transformation applied, translation in one hand tracking relative axis moves the probe in a different, unexpected direction along its relative axis.

In the latest update I've included an additional mapping workaround so that wrist rotation deltas are also mapped to the expected probe direction in the probe's relative coordinate frame.

These are experimental workarounds based on observation. I am not certain at the moment why the hand tracking and probe coordinate frames appear to differ.

[KumoLiu]

Thanks for the update, I update the transform matrix in this PR: #161
Let me know if it make more sense to you. Thanks!

[mingxin-zheng]

Thanks for the PR.

Need to merge for rc1 tagging.

If there are any topic unresolved, feel free to open another PR to address.

[KumoLiu]

Hello @tbirdso,I discovered that this might be a known issue in Isaaclab: link. It seems that the camera and XR cannot function together. I have just joined their thread to see if the issue has been resolved.