|
| 1 | +# Telesurgery Workflow |
| 2 | + |
| 3 | + |
| 4 | + |
| 5 | +## Table of Contents |
| 6 | +- [System Requirements](#system-requirements) |
| 7 | +- [Quick Start](#quick-start) |
| 8 | +- [Running the Workflow](#running-the-workflow) |
| 9 | +- [Licensing](#licensing) |
| 10 | + |
| 11 | +## System Requirements |
| 12 | + |
| 13 | +### Hardware Requirements |
| 14 | +- Ubuntu 22.04 |
| 15 | +- NVIDIA GPU with compute capability 8.6 and 32GB of memory |
| 16 | + - GPUs without RT Cores, such as A100 and H100, are not supported |
| 17 | +- 50GB of disk space |
| 18 | + |
| 19 | +### Software Requirements |
| 20 | +- NVIDIA Driver Version >= 555 |
| 21 | +- CUDA Version >= 12.6 |
| 22 | +- Python 3.10 |
| 23 | +- RTI DDS License |
| 24 | + |
| 25 | +## Quick Start |
| 26 | + |
| 27 | +### x86 & AARCH64 (IGX) Setup |
| 28 | + |
| 29 | +1. **Set up a Docker environment with CUDA enabled (IGX only):** |
| 30 | + ```bash |
| 31 | + cd <path-to-i4h-workflows> |
| 32 | + xhost + |
| 33 | + workflows/telesurgery/docker/setup.sh run |
| 34 | + |
| 35 | + # Inside Docker |
| 36 | + workflows/telesurgery/docker/setup.sh init |
| 37 | + ``` |
| 38 | + |
| 39 | +2. **Set up the x86 environment with CUDA enabled:** |
| 40 | + ```bash |
| 41 | + cd <path-to-i4h-workflows> |
| 42 | + xhost + |
| 43 | + workflows/telesurgery/docker/setup.sh init |
| 44 | + ``` |
| 45 | + |
| 46 | +3. **Create and activate a [conda](https://www.anaconda.com/docs/getting-started/miniconda/install#quickstart-install-instructions) environment:** |
| 47 | + ```bash |
| 48 | + source ~/miniconda3/bin/activate |
| 49 | + conda create -n telesurgery python=3.10 -y |
| 50 | + conda activate telesurgery |
| 51 | + ``` |
| 52 | + |
| 53 | +4. **Run the setup script:** |
| 54 | + ```bash |
| 55 | + cd <path-to-i4h-workflows> |
| 56 | + bash tools/env_setup_telesurgery.sh |
| 57 | + ``` |
| 58 | + |
| 59 | +> Make sure your public key is added to the github account if the git authentication fails. |
| 60 | +
|
| 61 | +### Obtain RTI DDS License |
| 62 | + |
| 63 | +RTI DDS is the communication package used by all scripts. Please refer to the [DDS website](https://www.rti.com/products) for registration. You will need to obtain a license file and set the `RTI_LICENSE_FILE` environment variable to its path. |
| 64 | + |
| 65 | +### NTP Server (Optional) |
| 66 | + |
| 67 | +An NTP (Network Time Protocol) server provides accurate time information to clients over a computer network. NTP is designed to synchronize the clocks of computers to a reference time source, ensuring all devices on the network maintain the same time. |
| 68 | + |
| 69 | +```bash |
| 70 | +# Run your own NTP server in the background |
| 71 | +docker run -d --name ntp-server --restart=always -p 123:123/udp cturra/ntp |
| 72 | + |
| 73 | +# Check if it's running |
| 74 | +docker logs ntp-server |
| 75 | + |
| 76 | +# fix server ip in env.sh for NTP Server |
| 77 | +export NTP_SERVER_HOST=<NTP server address> |
| 78 | + |
| 79 | +# To stop the server |
| 80 | +# docker stop ntp-server && docker rm ntp-server |
| 81 | +``` |
| 82 | + |
| 83 | +### Environment Variables |
| 84 | + |
| 85 | +Before running any scripts, set up the following environment variables: |
| 86 | + |
| 87 | +1. **PYTHONPATH**: Set this to point to the **scripts** directory: |
| 88 | + ```bash |
| 89 | + export PYTHONPATH=<path-to-i4h-workflows>/workflows/telesurgery/scripts |
| 90 | + ``` |
| 91 | + This ensures Python can find the modules under the [`scripts`](./scripts) directory. |
| 92 | + |
| 93 | +2. **RTI_LICENSE_FILE**: Set this to point to your RTI DDS license file: |
| 94 | + ```bash |
| 95 | + export RTI_LICENSE_FILE=<path-to-rti-license-file> |
| 96 | + ``` |
| 97 | + This is required for the DDS communication package to function properly. |
| 98 | + |
| 99 | +3. **NDDS_DISCOVERY_PEERS**: Set this to the IP address receiving camera data: |
| 100 | + ```bash |
| 101 | + export NDDS_DISCOVERY_PEERS="surgeon IP address" |
| 102 | + ``` |
| 103 | +More recommended variables can be found in [env.sh](./scripts/env.sh). |
| 104 | + |
| 105 | +## Running the Workflow |
| 106 | + |
| 107 | +```bash |
| 108 | +cd <path-to-i4h-workflows>/workflows/telesurgery/scripts |
| 109 | +source env.sh # Make sure all env variables are correctly set in env.sh |
| 110 | + |
| 111 | +export PATIENT_IP=<patient IP address> |
| 112 | +export SURGEON_IP=<surgeon IP address> |
| 113 | +``` |
| 114 | +> Make sure the MIRA API Server is up and running (port: 8081) in the case of a physical world setup. |
| 115 | +
|
| 116 | +### [Option 1] Patient in Physical World _(x86 / aarch64)_ |
| 117 | + |
| 118 | +When running on IGX (aarch64), ensure you are in the Docker environment set up previously. |
| 119 | + |
| 120 | +```bash |
| 121 | +# Stream camera output |
| 122 | +python patient/physical/camera.py --camera realsense --name room --width 1280 --height 720 |
| 123 | +python patient/physical/camera.py --camera cv2 --name robot --width 1920 --height 1080 |
| 124 | +``` |
| 125 | + |
| 126 | +### [Option 2] Patient in Simulation World _(x86)_ |
| 127 | + |
| 128 | +```bash |
| 129 | +# Download the assets |
| 130 | +i4h-asset-retrieve |
| 131 | + |
| 132 | +python patient/simulation/main.py [--encoder nvc] |
| 133 | +``` |
| 134 | + |
| 135 | +### Surgeon Connecting to Patient _(x86 / aarch64)_ |
| 136 | + |
| 137 | +# capture robot camera stream |
| 138 | +NDDS_DISCOVERY_PEERS=${PATIENT_IP} python surgeon/camera.py --name robot --width 1280 --height 720 [--decoder nvc] |
| 139 | + |
| 140 | +# capture room camera stream (optional) |
| 141 | +NDDS_DISCOVERY_PEERS=${PATIENT_IP} python surgeon/camera.py --name room --width 1280 --height 720 [--decoder nvc] |
| 142 | + |
| 143 | +# Connect to gamepad controller and send commands to API Server |
| 144 | +python surgeon/gamepad.py --api_host ${PATIENT_IP} --api_port 8081 |
| 145 | +``` |
| 146 | +
|
| 147 | +### Using H.264/HEVC Encoder/Decoder from NVIDIA Video Codec |
| 148 | +
|
| 149 | +Camera data can be streamed using either the H.264 or HEVC (H.265) codecs. To enable this for the Patient and Surgeon applications, use the `--encoder nvc` or `--decoder nvc` argument, respectively. |
| 150 | +
|
| 151 | +Encoding parameters can be customized in the Patient application using the `--encoder_params` argument, as shown below: |
| 152 | +
|
| 153 | +```bash |
| 154 | +python patient/simulation/main.py --encoder nvc --encoder_params patient/nvc_encoder_params.json |
| 155 | +``` |
| 156 | + |
| 157 | +#### Sample Encoding Parameters for the NVIDIA Video Codec |
| 158 | + |
| 159 | +Here’s an example of encoding parameters in JSON format: |
| 160 | + |
| 161 | +```json |
| 162 | +{ |
| 163 | + "codec": "H264", // Possible values: H264 or HEVC |
| 164 | + "preset": "P3", // Options include P3, P4, P5, P6, P7 |
| 165 | + "bitrate": 10000000, |
| 166 | + "frame_rate": 60, |
| 167 | + "rate_control_mode": 1, // Options: 0 for Constant QP, 1 for Variable bitrate, 2 for Constant bitrate |
| 168 | + "multi_pass_encoding": 0 // Options: 0 to disable, 1 for Quarter resolution, 2 for Full resolution |
| 169 | +} |
| 170 | +``` |
| 171 | + |
| 172 | +> [!NOTE] |
| 173 | +> H.264 or HEVC (H.265) codecs are available on x86 platform only. |
| 174 | +
|
| 175 | +### Important Notes |
| 176 | +1. You may need to run multiple scripts simultaneously in different terminals or run in background (in case of docker) |
| 177 | +2. A typical setup requires multiple terminals running: |
| 178 | + - Patient: Camera1, Camera2, Controller, etc. |
| 179 | + - Surgeon: Camera1, Camera2, Controller, etc. |
| 180 | + |
| 181 | +If you encounter issues not covered in the notes above, please check the documentation for each component or open a new issue on GitHub. |
| 182 | + |
| 183 | +## Licensing |
| 184 | + |
| 185 | +By using the Telesurgery workflow and NVIDIA Video Codec, you are implicitly agreeing to the [NVIDIA Software License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and [NVIDIA Software Developer License Agreement](https://developer.download.nvidia.com/designworks/DesignWorks_SDKs_Samples_Tools_License_distrib_use_rights_2017_06_13.pdf?t=eyJscyI6InJlZiIsImxzZCI6IlJFRi1zZWFyY2guYnJhdmUuY29tLyJ9). If you do not agree to the EULA, do not run this container. |
0 commit comments