Telesurgery Workflow

.gitignore

@@ -185,6 +185,7 @@ cython_debug/
 
 # VSCode
 .vscode/
+.idea/
 
 # Wandb
 wandb/

.pre-commit-config.yaml

@@ -17,6 +17,10 @@ repos:
     hooks:
       - id: ruff
         entry: ruff check
+        exclude: |
+            (?x)^(
+                workflows/telesurgery/cmake/pybind11/__init__.py
+            )$
   - repo: https://github.com/pycqa/isort
     rev: 6.0.0
     hooks:

NOTICE.txt

@@ -46,6 +46,10 @@ Licensed under GPL-3.0 (https://gcc.gnu.org/onlinedocs/libstdc++/manual/license.
 libxcb-cursor0 (https://launchpad.net/ubuntu/mantic/amd64/libxcb-cursor0)
 Licensed under MIT (https://gitlab.freedesktop.org/xorg/lib/libxcb-cursor/-/blob/master/COPYING)
 
+opencv-python (https://github.com/opencv/opencv-python)
+Copyright (c) Olli-Pekka Heinisuo
+License under MIT (https://github.com/opencv/opencv-python/blob/4.x/LICENSE.txt)
+
 openpi (https://github.com/Physical-Intelligence/openpi)
 Licensed under Apache 2.0 (https://github.com/Physical-Intelligence/openpi/blob/main/LICENSE)
 
@@ -81,6 +85,10 @@ torch (https://pypi.org/project/torch/)
 2016- Facebook, Inc (Adam Paszke); 2014- Facebook, Inc (Soumith Chintala); 2011-2014 Idiap Research Institute (Ronan Collobert); 2012-2014 Deepmind Technologies (Koray Kavukcuoglu); 2011-2012 NEC Laboratories America (Koray Kavukcuoglu); 2011-2013 NYU (Clement Farabet); 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston); 2006 Idiap Research Institute (Samy Bengio); 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz); 2016 Facebook Inc.; 2015 Google Inc.; 2015 Yangqing Jia; 2019-2020 Kakao Brain; 2022 Cruise LLC.; 2024 Tri Dao.
 Licensed under BSD-3-Clause (https://github.com/pytorch/pytorch/blob/main/LICENSE)
 
+v4l-utils (https://github.com/gjasny/v4l-utils)
+Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+Licensed under GPL-2.0 (https://github.com/gjasny/v4l-utils/blob/master/COPYING)
+
 x11-utils (https://launchpad.net/ubuntu/+source/x11-utils)
 2011 Gaetan Nadon; 1994-2003 The XFree86 Project, Inc. All Rights Reserved.; 1993
 The Regents of the University of California. All rights reserved.; 2003 The NetBSD Foundation, Inc.; 2001-2002 Damien Miller. All rights reserved.; 1998 Todd C. Miller <[email protected]>; 2001-2004 Thomas Winischhofer; 1991-2000 Silicon Graphics, Inc. All Rights Reserved.; 2001 by Bigelow & Holmes Inc. Luxi font; 2001 by URW++ GmbH. All Rights

README.md

@@ -62,6 +62,7 @@ For everything you need to get started, including detailed tutorials and step-by
 ### Workflows
 - [Robotic ultrasound](./workflows/robotic_ultrasound/README.md)
 - [Robotic surgery](./workflows/robotic_surgery/README.md)
+- [Telesurgery](./workflows/telesurgery/README.md)
 
 ### Tutorials
 - [Bring your own patient](./tutorials/assets/bring_your_own_patient/README.md)

pyproject.toml

@@ -12,10 +12,13 @@ known_third_party = ["openpi", "i4h_asset_helper"]
 [tool.ruff]
 line-length = 120
 target-version = "py310"
+exclude = [
+  "workflows/telesurgery/cmake/pybind11/"
+]
 
 [tool.ruff.lint]
 extend-select = ["E501"]
 extend-ignore = ["E402"]
 
 [tool.codespell]
-ignore-words-list = "solum"
+ignore-words-list = "solum, ptd"

tools/env_setup/install_haply.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+VERSION=3.3.2
+PLATFORM=amd64
+OS_VERSION=$(lsb_release -rs)
+
+# ---- Install Haply Inverse Service
+if command -v "haply-inverse-service" &> /dev/null; then
+    echo "Haply Inverse Service already installed, skipping."
+    exit 0
+fi
+
+if [[ "$(uname -m)" != x86_64 ]]; then
+    echo "Skipping for non-x86_64 arch."
+    exit 1
+fi
+
+# Download the deb package
+if [[ "$OS_VERSION" == "24.04" ]]; then
+    echo "Detected Ubuntu 24.04"
+    wget -O haply-inverse-service.deb https://cdn.haply.co/r/38736259/haply-inverse-service_3.3.2_amd64.deb
+elif [[ "$OS_VERSION" == "22.04" ]]; then
+    echo "Detected Ubuntu 22.04"
+    wget -O haply-inverse-service.deb https://cdn.haply.co/r/38736259/haply-inverse-service_3.3.2_2204_amd64.deb
+else
+    echo "Unsupported version of Ubuntu: $OS_VERSION"
+    exit 1
+fi
+
+
+# Install debian
+sudo apt-get install -y ./haply-inverse-service.deb
+rm haply-inverse-service.deb
+
+echo "Haply Inverse Service installed!"

tools/env_setup_telesurgery.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+# --- Setup Steps ---
+# Get the parent directory of the current script
+PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && cd .. && pwd)"
+source "$PROJECT_ROOT/tools/env_setup/bash_utils.sh"
+
+# Check if running in a conda environment
+check_conda_env
+
+# Check if NVIDIA GPU is available
+check_nvidia_gpu
+
+# Check if the third_party directory exists
+ensure_fresh_third_party_dir
+
+
+# ---- Install build tools (Common) ----
+echo "Installing build tools..."
+if [ "$EUID" -ne 0 ]; then
+    sudo apt-get install -y git cmake build-essential pybind11-dev v4l-utils lsb-release
+else
+    apt-get install -y git cmake build-essential pybind11-dev v4l-utils lsb-release
+fi
+
+# ---- Install IsaacSim and IsaacLab (Common) ----
+# Check if IsaacLab is already cloned
+if [[ "$(uname -m)" == x86_64 ]]; then
+    echo "Installing IsaacSim and IsaacLab... (only for x86)"
+    bash $PROJECT_ROOT/tools/env_setup/install_isaac.sh
+else
+    echo "Platform is not x86_64; Skip IsaacSim/IsaacLab"
+fi
+
+# ---- Install necessary dependencies (Common) ----
+echo "Installing necessary dependencies..."
+pip install numpy
+conda install -c conda-forge libstdcxx-ng -y
+
+# ---- Install Haply Inverse Service (Telesurgery) ----
+if [[ "$(uname -m)" == x86_64 ]]; then
+    echo "Installing Haply Inverse Service... (only for x86)"
+    bash $PROJECT_ROOT/tools/env_setup/install_haply.sh
+else
+    echo "Platform is not x86_64; Skip Haply Inverse Service Installation"
+fi
+
+
+# ---- Install necessary dependencies (Telesurgery) ----
+echo "Installing necessary dependencies..."
+pip install -r $PROJECT_ROOT/workflows/telesurgery/requirements.txt
+
+echo "=========================================="
+echo "Environment setup script finished."
+echo "=========================================="

workflows/telesurgery/README.md

@@ -0,0 +1,186 @@
+# Telesurgery Workflow
+
+![Telesurgery Workflow](../../docs/source/telesurgery_workflow.jpg)
+
+## Table of Contents
+- [System Requirements](#system-requirements)
+- [Quick Start](#quick-start)
+- [Running the Workflow](#running-the-workflow)
+- [Licensing](#licensing)
+
+## System Requirements
+
+### Hardware Requirements
+- Ubuntu 22.04
+- NVIDIA GPU with compute capability 8.6 and 32GB of memory
+   - GPUs without RT Cores, such as A100 and H100, are not supported
+- 50GB of disk space
+
+### Software Requirements
+- NVIDIA Driver Version >= 555
+- CUDA Version >= 12.6
+- Python 3.10
+- RTI DDS License
+
+## Quick Start
+
+### x86 & AARCH64 (IGX) Setup
+
+1. **Set up a Docker environment with CUDA enabled (IGX only):**
+   ```bash
+   cd <path-to-i4h-workflows>
+   xhost +
+   workflows/telesurgery/docker/setup.sh run
+
+   # Inside Docker
+   workflows/telesurgery/docker/setup.sh init
+   ```
+
+2. **Set up the x86 environment with CUDA enabled:**
+   ```bash
+   cd <path-to-i4h-workflows>
+   xhost +
+   workflows/telesurgery/docker/setup.sh init
+   ```
+
+3. **Create and activate a [conda](https://www.anaconda.com/docs/getting-started/miniconda/install#quickstart-install-instructions) environment:**
+   ```bash
+   source ~/miniconda3/bin/activate
+   conda create -n telesurgery python=3.10 -y
+   conda activate telesurgery
+   ```
+
+4. **Run the setup script:**
+   ```bash
+   cd <path-to-i4h-workflows>
+   bash tools/env_setup_telesurgery.sh
+   ```
+
+> Make sure your public key is added to the github account if the git authentication fails.
+
+### Obtain RTI DDS License
+
+RTI DDS is the communication package used by all scripts. Please refer to the [DDS website](https://www.rti.com/products) for registration. You will need to obtain a license file and set the `RTI_LICENSE_FILE` environment variable to its path.
+
+### NTP Server (Optional)
+
+An NTP (Network Time Protocol) server provides accurate time information to clients over a computer network. NTP is designed to synchronize the clocks of computers to a reference time source, ensuring all devices on the network maintain the same time.
+
+```bash
+# Run your own NTP server in the background
+docker run -d --name ntp-server --restart=always -p 123:123/udp cturra/ntp
+
+# Check if it's running
+docker logs ntp-server
+
+# fix server ip in env.sh for NTP Server
+export NTP_SERVER_HOST=<NTP server address>
+
+# To stop the server
+# docker stop ntp-server && docker rm ntp-server
+```
+
+### Environment Variables
+
+Before running any scripts, set up the following environment variables:
+
+1. **PYTHONPATH**: Set this to point to the **scripts** directory:
+   ```bash
+   export PYTHONPATH=<path-to-i4h-workflows>/workflows/telesurgery/scripts
+   ```
+   This ensures Python can find the modules under the [`scripts`](./scripts) directory.
+
+2. **RTI_LICENSE_FILE**: Set this to point to your RTI DDS license file:
+   ```bash
+   export RTI_LICENSE_FILE=<path-to-rti-license-file>
+   ```
+   This is required for the DDS communication package to function properly.
+
+3. **NDDS_DISCOVERY_PEERS**: Set this to the IP address receiving camera data:
+   ```bash
+   export NDDS_DISCOVERY_PEERS="surgeon IP address"
+   ```
+More recommended variables can be found in [env.sh](./scripts/env.sh).
+
+## Running the Workflow
+
+```bash
+cd <path-to-i4h-workflows>/workflows/telesurgery/scripts
+source env.sh  # Make sure all env variables are correctly set in env.sh
+
+export PATIENT_IP=<patient IP address>
+export SURGEON_IP=<surgeon IP address>
+```
+> Make sure the MIRA API Server is up and running (port: 8081) in the case of a physical world setup.
+
+### [Option 1] Patient in Physical World _(x86 / aarch64)_
+
+When running on IGX (aarch64), ensure you are in the Docker environment set up previously.
+
+```bash
+# Stream camera output
+python patient/physical/camera.py --camera realsense --name room --width 1280 --height 720
+python patient/physical/camera.py --camera cv2 --name robot --width 1920 --height 1080
+```
+
+### [Option 2] Patient in Simulation World _(x86)_
+
+```bash
+# Download the assets
+export ISAAC_ASSET_SHA256_HASH=8e80faed126c533243f50bb01dca3dcf035e86b5bf567d622878866a8ef7f12d
+i4h-asset-retrieve
+
+python patient/simulation/main.py [--encoder nvc]
+```
+
+### Surgeon Connecting to Patient _(x86 / aarch64)_
+
+# capture robot camera stream
+NDDS_DISCOVERY_PEERS=${PATIENT_IP} python surgeon/camera.py --name robot --width 1280 --height 720 [--decoder nvc]
+
+# capture room camera stream (optional)
+NDDS_DISCOVERY_PEERS=${PATIENT_IP} python surgeon/camera.py --name room --width 1280 --height 720 [--decoder nvc]
+
+# Connect to gamepad controller and send commands to API Server
+python surgeon/gamepad.py --api_host ${PATIENT_IP} --api_port 8081
+```
+
+### Using H.264/HEVC Encoder/Decoder from NVIDIA Video Codec
+
+Camera data can be streamed using either the H.264 or HEVC (H.265) codecs. To enable this for the Patient and Surgeon applications, use the `--encoder nvc` or `--decoder nvc` argument, respectively.
+
+Encoding parameters can be customized in the Patient application using the `--encoder_params` argument, as shown below:
+
+```bash
+python patient/simulation/main.py --encoder nvc --encoder_params patient/nvc_encoder_params.json
+```
+
+#### Sample Encoding Parameters for the NVIDIA Video Codec
+
+Here’s an example of encoding parameters in JSON format:
+
+```json
+{
+    "codec": "H264", // Possible values: H264 or HEVC
+    "preset": "P3", // Options include P3, P4, P5, P6, P7
+    "bitrate": 10000000,
+    "frame_rate": 60,
+    "rate_control_mode": 1, // Options: 0 for Constant QP, 1 for Variable bitrate, 2 for Constant bitrate
+    "multi_pass_encoding": 0 // Options: 0 to disable, 1 for Quarter resolution, 2 for Full resolution
+}
+```
+
+> [!NOTE]
+> H.264 or HEVC (H.265) codecs are available on x86 platform only.
+
+### Important Notes
+1. You may need to run multiple scripts simultaneously in different terminals or run in background (in case of docker)
+2. A typical setup requires multiple terminals running:
+   - Patient: Camera1, Camera2, Controller, etc.
+   - Surgeon: Camera1, Camera2, Controller, etc.
+
+If you encounter issues not covered in the notes above, please check the documentation for each component or open a new issue on GitHub.
+
+## Licensing
+
+By using the Telesurgery workflow and NVIDIA Video Codec, you are implicitly agreeing to the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [NVIDIA Software Developer License Agreement](https://developer.download.nvidia.com/designworks/DesignWorks_SDKs_Samples_Tools_License_distrib_use_rights_2017_06_13.pdf?t=eyJscyI6InJlZiIsImxzZCI6IlJFRi1zZWFyY2guYnJhdmUuY29tLyJ9). If you do not agree to the EULA, do not run this container.