isaac-for-healthcare · mingxin-zheng · Jul 11, 2025 · Jul 9, 2025 · Jul 9, 2025 · Jul 9, 2025
diff --git a/workflows/robotic_surgery/README.md b/workflows/robotic_surgery/README.md
@@ -1,112 +1,283 @@
-# Robotic Surgery Workflow
+# 🔬 Robotic Surgery Workflow
 
 ![Robotic Surgery Workflow](../../docs/source/robotic_surgery_workflow.jpg)
 
+---
+
+## 🩺 Technical Overview
+
 The Robotic Surgery Workflow is a comprehensive solution designed for healthcare professionals and researchers working in the field of robotic-assisted surgery. This workflow provides a robust framework for simulating, training, and analyzing robotic surgical procedures in a virtual environment. It leverages NVIDIA's ray tracing capabilities to create highly realistic surgical simulations, enabling surgeons to practice complex procedures, researchers to develop new surgical techniques, and medical institutions to enhance their training programs. By offering a safe, controlled environment for surgical practice and research, this workflow helps improve surgical outcomes, reduce training costs, and advance the field of robotic surgery.
 
+The framework supports multiple surgical robots and tasks:
 
-## Table of Contents
-- [System Requirements](#system-requirements)
-- [Environment Setup](#environment-setup)
-  - [Prerequisites](#prerequisites)
-  - [Installation Steps](#installation-steps)
-  - [Asset Setup](#asset-setup)
-  - [Environment Variables](#environment-variables)
-- [Running the Workflow](#running-the-workflow)
+- **dVRK (da Vinci Research Kit)**: Patient Side Manipulator (PSM) for minimally invasive surgery.
+- **STAR (Smart Tissue Autonomous Robot)**: Autonomous surgical robot for tissue manipulation.
+- **Multi-arm coordination**: Dual-arm surgical robot synchronization.
+- **Surgical task simulation**: Needle lifting, suturing, peg transfer, etc.
 
-## System Requirements
+---
 
-### Hardware Requirements
-- Ubuntu 22.04
-- NVIDIA GPU with ray tracing capability
-    - GPUs without RT Cores (A100, H100) are not supported
-    - Minimum 8GB VRAM recommended
-- 50GB of disk space
-- 16GB RAM minimum
+## 📋 Table of Contents
 
-### Software Requirements
-- [NVIDIA Driver Version >= 555](https://www.nvidia.com/en-us/drivers/)
-- Python 3.10
+- [🚀 Quick Start](#-quick-start)
+- [⚡ Running Workflows](#-running-workflows)
+- [🔧 Detailed Setup Instructions](#-detailed-setup-instructions)
+- [🛠️ Troubleshooting](#️-troubleshooting)
 
-## Environment Setup
+---
 
-**Note**: The setup process takes approximately 30-40 minutes to complete, depending on your system and network connection.
+## 🚀 Quick Start
 
-### Prerequisites
+### ⏱️ Installation Timeline
+**Estimated Setup Duration:** 40-60 minutes (network-dependent asset downloads)
 
-The robotic surgery workflow is built on the following dependencies:
-- [IsaacSim 4.5.0](https://docs.isaacsim.omniverse.nvidia.com/4.5.0/index.html)
-- [IsaacLab 2.1.0](https://isaac-sim.github.io/IsaacLab/v2.1.0/index.html)
+### 🔍 System Prerequisites Validation
 
-### Installation Steps
+#### GPU Architecture Requirements
+- **NVIDIA GPU**: RT Core-enabled architecture (Ampere or later)
+- **Compute Capability**: ≥8.6
+- **VRAM**: ≥24GB GDDR6/HBM
+- **Unsupported**: A100, H100 (lack RT Cores for ray tracing acceleration)
 
-#### 1. Install NVIDIA Driver
-Install or upgrade to the latest NVIDIA driver from [NVIDIA website](https://www.nvidia.com/en-us/drivers/)
+   <details>
+   <summary>🔍 GPU Compatibility Verification</summary>
+
+   ```bash
+   nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader
+   ```
+
+   Verify output shows compute capability ≥8.6 (Ampere/Ada Lovelace/Hopper with RT Cores)
+   </details>
+
+#### Driver & System Requirements
+- **Operating System**: Ubuntu 22.04 LTS / 24.04 LTS (x86_64)
+- **NVIDIA Driver**: ≥535.129.03 (RTX ray tracing API support)
+- **Memory Requirements**: ≥8GB GPU memory, ≥32GB system RAM
+- **Storage**: ≥100GB NVMe SSD (asset caching and simulation data)
+
+   <details>
+   <summary>🔍 Driver Version Validation</summary>
 
-**Note**: The workflow requires driver version >= 555 for ray tracing capabilities.
+   ```bash
+   nvidia-smi --query-gpu=driver_version --format=csv,noheader,nounits
+   ```
+   </details>
+
+#### Software Dependencies
+- **Python**: 3.10 (exact version required)
 
-#### 2. Install Dependencies
+---
 
-##### Install Conda
+### 🐍 Conda Environment Setup
 
-[Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/main) is recommended.
+The robotic surgery workflow can be installed in a conda-based environment for dependency isolation and compatibility.
 
-##### Create Conda Environment
+**Prerequisites**: Ensure [Miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install) is installed on your system.
+
+#### 1️⃣ Repository Clone
+```bash
+git clone https://github.com/isaac-for-healthcare/i4h-workflows.git
+cd i4h-workflows
+```
 
+#### 2️⃣ Environment Creation & Dependency Installation
 ```bash
-# Create a new conda environment
 conda create -n robotic_surgery python=3.10 -y
-# Activate the environment
 conda activate robotic_surgery
+bash tools/env_setup_robot_surgery.sh
+```
+
+**⚠️ Expected Build Time**: The environment setup process takes 40-60 minutes. You may encounter intermediary warnings about conflicting library dependencies - these are non-critical and can be ignored.
+
+#### 3️⃣ Environment Variable Configuration
+```bash
+# Set PYTHONPATH for current session
+export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts
 ```
 
-##### Clone repository
+   <details>
+   <summary>💾 Persistent Environment Configuration (Recommended)</summary>
+
+   To avoid setting PYTHONPATH for every new terminal session:
+
    ```bash
-   git clone https://github.com/isaac-for-healthcare/i4h-workflows.git
-   cd i4h-workflows
+   # Add to your shell profile
+   echo "export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts:\$PYTHONPATH" >> ~/.bashrc
+   source ~/.bashrc
    ```
 
-##### Install All Dependencies
-The main script `tools/env_setup_robot_surgery.sh` installs all necessary dependencies:
+   This ensures the environment variables are automatically set when you open new terminals.
+
+   **Note:** If you have `robotic_ultrasound` workflow scripts or previous versions of `robotic_surgery` workflow scripts in your `PYTHONPATH`, you can reset it to include only the `robotic_surgery` scripts by running `export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts`
+   </details>
+
+
+✅ **Installation Complete** - Your robotic surgery simulation environment is ready.
+
+---
 
-###### Base Components
-- IsaacSim 4.5.0 (and core dependencies)
-- IsaacLab 2.1.0
-- Essential build tools and libraries
+## ⚡ Running Workflows
 
-Run the script from the repository root:
+### 🎯 Quick Start Examples
+
+#### 🤖 Basic Robot Control - dVRK PSM Reach
 ```bash
-cd <path-to-i4h-workflows>
+conda activate robotic_surgery
+python workflows/robotic_surgery/scripts/simulation/scripts/environments/state_machine/reach_psm_sm.py
+```
+
+**Expected Behavior:**
+- Isaac Sim launches with dVRK PSM simulation
+- PSM arm moves to predefined target poses
+
+> **⏳ First Run Loading Time**: Initial simulation loading takes 5-10 minutes for asset download and scene initialization. Isaac Sim may appear frozen with no console progress indication - this is normal behavior.
+
+#### 🔬 Surgical Task Simulation - Needle Lifting
+```bash
+conda activate robotic_surgery
+python workflows/robotic_surgery/scripts/simulation/scripts/environments/state_machine/lift_needle_sm.py
+```
+
+**Expected Behavior:**
+- Isaac Sim launches with robotic surgery simulation
+- Surgical simulation with needle manipulation
+- Precise grasping and lifting operations
+
+#### 🧠 Reinforcement Learning Training
+```bash
+conda activate robotic_surgery
+# Training (45+ minutes)
+python workflows/robotic_surgery/scripts/simulation/scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Reach-PSM-v0 --headless
+
+# Evaluation/Play mode
+python workflows/robotic_surgery/scripts/simulation/scripts/reinforcement_learning/rsl_rl/play.py --task Isaac-Reach-PSM-Play-v0
+```
+
+**Expected Behavior:**
+- RSL-RL agent training with GPU acceleration
+- TensorBoard logging for training progress monitoring
+- Trained model evaluation in simulation environment
+
+---
+
+### 🎯 Workflow Component Matrix
+
+| Category | Script | Usage Scenario | Purpose | Documentation | Key Requirements | Expected Runtime |
+|----------|--------|----------------|---------|---------------|------------------|------------------|
+| **🚀 Basic Control** | [reach_psm_sm.py](scripts/simulation/scripts/environments/state_machine/reach_psm_sm.py) | First-time users, basic robot control | Single-arm dVRK PSM reaching tasks | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#dvrk-psm-reach) | Isaac Lab | 2-5 minutes |
+| **🤖 Dual-Arm Control** | [reach_dual_psm_sm.py](scripts/simulation/scripts/environments/state_machine/reach_dual_psm_sm.py) | First-time users, basic robot control | Dual-arm dVRK PSM coordination | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#dual-arm-dvrk-psm-reach) | Isaac Lab | 2-5 minutes |
+| **⭐ STAR Robot** | [reach_star_sm.py](scripts/simulation/scripts/environments/state_machine/reach_star_sm.py) | First-time users, basic robot control | STAR robot reaching demonstrations | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#star-reach) | Isaac Lab | 2-5 minutes |
+| **🪡 Needle Manipulation** | [lift_needle_sm.py](scripts/simulation/scripts/environments/state_machine/lift_needle_sm.py) | First-time users, basic robot control | Suture needle lifting and manipulation | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#suture-needle-lift) | Isaac Lab | 3-7 minutes |
+| **🫁 Realistic OR Simulation** | [lift_needle_organs_sm.py](scripts/simulation/scripts/environments/state_machine/lift_needle_organs_sm.py) | Realistic surgical simulation in an operating room | Needle lifting | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#organs-suture-needle-lift) | Isaac Lab | 3-7 minutes |
+| **🧩 Peg Transfer** | [lift_block_sm.py](scripts/simulation/scripts/environments/state_machine/lift_block_sm.py) | First-time users, basic robot control | Peg transfer surgical training task | [State Machine README](scripts/simulation/scripts/environments/state_machine/README.md#peg-block-lift) | Isaac Lab | 2-5 minutes |
+| **🧠 RL Training** | [train.py](scripts/simulation/scripts/reinforcement_learning/rsl_rl/train.py) | AI model development | Reinforcement learning agent training | [RSL-RL README](scripts/simulation/scripts/reinforcement_learning/rsl_rl/README.md#training-and-playing) | Isaac Lab | 45+ minutes |
+| **🎮 RL Evaluation** | [play.py](scripts/simulation/scripts/reinforcement_learning/rsl_rl/play.py) | Model validation | Trained agent evaluation and visualization | [RSL-RL README](scripts/simulation/scripts/reinforcement_learning/rsl_rl/README.md#training-and-playing) | Isaac Lab, trained model | 5-10 minutes |
+| **🎮 MIRA Teleoperation Tutorial** | [teleoperate_virtual_incision_mira.py](../../tutorials/assets/bring_your_own_robot/Virtual_Incision_MIRA/teleoperate_virtual_incision_mira.py) | Interactive robot control | Virtual Incision MIRA keyboard teleoperation | [Virtual Incision MIRA README](../../tutorials/assets/bring_your_own_robot/Virtual_Incision_MIRA/README.md) | Isaac Lab | 5-10 minutes |
+
+---
+
+### 🎓 Understanding the Workflow Architecture
+
+```
+Isaac Sim Environment
+├── Robots (dVRK PSM, STAR)
+├── Surgical Instruments (needles, pegs, operating room equipment)
+├── Anatomical Models (organs)
+└── Control Systems (state machines, RL)
+```
+
+---
+
+## 🔧 Detailed Setup Instructions
+
+<details>
+<summary>📋 Advanced Configuration & Dependencies</summary>
+
+### 🏗️ Framework Architecture Dependencies
+
+The robotic surgery workflow is built on the following core dependencies:
+- [IsaacSim 4.5.0](https://docs.isaacsim.omniverse.nvidia.com/4.5.0/index.html)
+- [IsaacLab 2.1.0](https://isaac-sim.github.io/IsaacLab/v2.1.0/index.html)
+- [RSL-RL](https://github.com/leggedrobotics/rsl_rl) for reinforcement learning
+
+### 🐳 Docker Installation Procedures
+
+Please refer to the [Robotic Surgery Docker Container Guide](./docker/README.md) for detailed instructions on how to run the workflow in a Docker container.
+
+### 🔨 Conda Installation Procedures
+
+#### 1️⃣ NVIDIA Graphics Driver Installation
+Install or upgrade to the latest NVIDIA driver from [NVIDIA website](https://www.nvidia.com/en-us/drivers/)
+
+**Note**: The workflow requires driver version ≥535.129.03 for ray tracing capabilities.
+
+#### 2️⃣ Conda Installation
+Install Miniconda following the [official guide](https://www.anaconda.com/docs/getting-started/miniconda/install):
+
+```bash
+# Download and install Miniconda
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
+```
+
+#### 3️⃣ Environment Creation and Setup
+```bash
+# Create and activate environment
+conda create -n robotic_surgery python=3.10 -y
+conda activate robotic_surgery
+
+# Clone repository and install dependencies
+git clone https://github.com/isaac-for-healthcare/i4h-workflows.git
+cd i4h-workflows
 bash tools/env_setup_robot_surgery.sh
 ```
 
-### Asset Setup
+#### 4️⃣ Environment Variable Configuration
+```bash
+# Current session
+export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts:$PYTHONPATH
 
-**Note**: The assets can be automatically retrieved when running the workflows. Optionally, you can also download all the assets in advance. Please note that the assets are approximately 65 GB and may take some time to download depending on your internet connection.
+# Persistent configuration (recommended)
+echo "export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts:\$PYTHONPATH" >> ~/.bashrc
+source ~/.bashrc
+```
+
+### 📦 Asset Management
+
+#### Automated Asset Retrieval
+Assets are automatically downloaded when running workflows for the first time.
 
-Download the required assets using:
+#### Manual Asset Retrieval
 ```bash
+# Download all assets (65GB, 30-60 minutes)
 i4h-asset-retrieve
 ```
 
-This will download assets to `~/.cache/i4h-assets/<sha256>`. For more details, refer to the [Asset Container Helper](https://github.com/isaac-for-healthcare/i4h-asset-catalog/blob/main/docs/catalog_helper.md).
+**Asset Storage**: `~/.cache/i4h-assets/<sha256>/`
 
-### Environment Variables
+</details>
 
-Before running any scripts, you need to set up the following environment variables:
+---
 
-1. **PYTHONPATH**: Set this to point to the scripts directory:
-   ```bash
-   export PYTHONPATH=<path-to-i4h-workflows>/workflows/robotic_surgery/scripts
-   ```
-   This ensures Python can find the modules under the [`scripts`](./scripts) directory.
+## 🛠️ Troubleshooting
+
+### ⚠️ Common Issues
+
+#### 🐌 Long Loading Times
+**Symptoms**: Isaac Sim appears frozen during initial loading
+**Resolution**: This is expected behavior. Initial loading takes 5-10 minutes with no progress indication. Be patient and avoid force-closing the application.
+
+#### 🔗 Module Import Errors
+**Symptoms**: `ModuleNotFoundError` when running scripts
+**Resolution**: Ensure `PYTHONPATH` includes the scripts directory:
+```bash
+export PYTHONPATH=$(pwd)/workflows/robotic_surgery/scripts
+```
 
-## Running the Workflow
+#### 🔧 Environment Build Issues
+**Symptoms**: Conflicting library dependency warnings during setup
+**Resolution**: These warnings are non-critical and can be ignored. The environment will function correctly.
 
-The robotic surgery workflow provides several example scripts demonstrating different components:
+### 🆘 Support Resources
 
-- [Simulation](./scripts/simulation)
-  - Basic robot control
-  - Surgical task simulation
-  - State machine demonstrations
-  - Reinforcement learning examples
+- **Issue Tracking**: [GitHub Issues](https://github.com/isaac-for-healthcare/i4h-workflows/issues)
diff --git a/workflows/robotic_surgery/docker/README.md b/workflows/robotic_surgery/docker/README.md
@@ -59,7 +59,9 @@ docker run --name isaac-sim -it --gpus all --rm \
 
 ### 1. Interactive GUI Mode with X11 Forwarding
 
-For interactive development and debugging:
+The command to run the simulation is the same as [Running Workflows](../README.md#-running-workflows) section.
+
+For example,
 
 ```bash
 # Inside the container
@@ -69,11 +71,9 @@ conda activate robotic_surgery
 python workflows/robotic_surgery/scripts/simulation/scripts/environments/state_machine/reach_psm_sm.py
 ```
 
-Please refer to the [Simulation README](../scripts/simulation/README.md) for more details.
-
 ### 2. Headless Streaming Mode
 
-For remote access:
+For headless streaming server setup, you can use the WebRTC streaming mode, with `livestream` flag.
 
 ```bash
 # Inside the container