Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/markdown-link-check.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
},
{
"pattern": "^#%EF%B8%8F.*"
},
{
"pattern": "https://developer.nvidia.com/cuda-downloads"
}
],
"timeout": "20s",
Expand Down
8 changes: 6 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -212,11 +212,15 @@ wandb/
third_party/

# third-party files
**/clarius_cast/include
**/clarius_solum/include
holoscan_i4h/install
holoscan_i4h/build

# Raytracing Ultrasound Simulator
workflows/robotic_ultrasound/scripts/raysim

# Bring Your Own XR
openxr/

# Ignore Holohub CLI
cmake
utilities
66 changes: 23 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,74 +14,54 @@

## Table of Contents
- [Overview](#overview)
- [Key Features](#key-features)
- [Getting Started](#getting-started)
- [Project Structure](#project-structure)
- [Available Workflows](#available-workflows)
- [Tutorials](#tutorials)
- [Repository Structure](#repository-structure)
- [Contributing](#contributing)
- [Support](#support)
- [License](#license)
- [Acknowledgement](#acknowledgement)

## Overview

**[Nvidia Isaac for Healthcare](https://github.com/isaac-for-healthcare)** is a comprehensive 3-computer solution for healthcare robotics, enabling developers to leverage NVIDIA technologies for:
This repository contains **healthcare robotics workflows** - complete, end-to-end implementations that demonstrate how to build, simulate, and deploy robotic systems for specific medical applications using the [Nvidia Isaac for Healthcare](https://github.com/isaac-for-healthcare) platform.

- Accelerating development and testing of AI algorithms with synthetic data generated in digital twin
- Accelerating development, training and testing of robotic policies in digital twin environments
- Enabling development, training and testing of systems with hardware-in-the-loop (HIL)
- Seamlessly transitioning from simulation to physical systems (Sim2Real)

## Key Features
### What are Workflows?

![Key features](./docs/source/key_features.jpg)

Our framework provides powerful capabilities for healthcare robotics development:
Workflows are comprehensive reference implementations that showcase the complete development pipeline from simulation to real-world deployment. Each workflow includes digital twin environments, AI model training capabilities, and deployment frameworks for specific healthcare robotics applications.

### Digital Twin & Simulation
- **Digital prototyping** of next-gen healthcare robotic systems, sensors and instruments
- **Synthetic data generation (SDG)** for training AI models, augmented with real data
- **Hardware-in-the-loop (HIL)** testing and evaluation of AI models in digital twin environments
### Available Workflows
This repository currently includes three main workflows:

### AI & Robotics Development
- **Data collection** for training robotic policies through imitation learning
- **XR and haptics-enabled teleoperation** of robotic systems in digital twin
- **GPU-accelerated training** of reinforcement and imitation learning algorithms
- **Augmented dexterity training** for robot-assisted surgery
- **[Robotic Surgery](./workflows/robotic_surgery/README.md)** - Physics-based surgical robot simulation framework with photorealistic rendering for developing autonomous surgical skills. Supports da Vinci Research Kit (dVRK), dual-arm configurations, and STAR surgical arms. This workflow enables researchers and medical device companies to train AI models for surgical assistance, validate robot behaviors safely, and accelerate development through GPU-parallelized reinforcement learning. Includes pre-built surgical subtasks like suture needle manipulation and precise reaching tasks.
- **[Robotic Ultrasound](./workflows/robotic_ultrasound/README.md)** - Comprehensive autonomous ultrasound imaging system featuring physics-accurate sensor simulation through GPU-accelerated raytracing technology that models ultrasound wave propagation, tissue interactions, and acoustic properties in real-time. The raytracing ultrasound simulator generates photorealistic B-mode images by simulating acoustic wave physics, enabling synthetic data generation for training AI models without requiring physical ultrasound hardware. Supports multiple AI policies (PI0, GR00T N1), distributed communication via RTI DDS, and Holoscan deployment for clinical applications. This workflow enables medical imaging researchers, ultrasound device manufacturers, and healthcare AI developers to train robotic scanning protocols, validate autonomous imaging algorithms, and accelerate development through GPU-accelerated simulation before clinical deployment.
- **[Telesurgery](./workflows/telesurgery/README.md)** - Real-time remote surgical operations framework supporting both simulated and physical environments with low-latency video streaming, haptic feedback, and distributed control systems. This workflow features H.264/HEVC hardware-accelerated video encoding, RTI DDS communication, cross-platform deployment (x86/AARCH64), and seamless sim-to-real transition. Designed for surgical robotics companies, medical device manufacturers, and telemedicine providers to develop remote surgical capabilities, validate teleoperation systems, and deploy scalable telesurgery solutions across different network conditions.

### Deployment & Training
- **Sim2Real deployment** on Nvidia Holoscan
- **Continuous testing (CT)** of robotic systems through HIL digital twin systems
- **Interactive training** experiences for clinicians/users (pre-op planning, post-op evaluations/insights)
Each workflow provides complete simulation environments, training datasets, pre-trained models, and deployment tools to accelerate your healthcare robotics development.

Please see [What's New](./docs/source/whatsnew_0_1_0.md) for details on our milestone releases.

## Getting Started

For everything you need to get started, including detailed tutorials and step-by-step guides, follow these links to learn more about:

### Workflows
- [Robotic ultrasound](./workflows/robotic_ultrasound/README.md)
- [Robotic surgery](./workflows/robotic_surgery/README.md)
- [Telesurgery](./workflows/telesurgery/README.md)

### Tutorials

Get started with our comprehensive tutorials that guide you through key aspects of the framework:
- [Bring your own patient](./tutorials/assets/bring_your_own_patient/README.md)
- [Bring your own robot](./tutorials/assets/bring_your_own_robot)
- [Sim2Real Transition](./tutorials/sim2real/README.md)
- [Bring your own xr](./tutorials/assets/bring_your_own_xr)

## Project Structure
## Repository Structure

```
i4h-workflows/
├── docs/ # Documentation and guides
├── workflows/ # Main workflow implementations
│ ├── robotic_surgery/ # Robotic surgery workflow
│ └── robotic_ultrasound/ # Robotic ultrasound workflow
├── docs/ # Documentation and guides
├── tutorials/ # Tutorial materials
│ ├── assets/ # Asset-related tutorials
│ └── sim2real/ # Sim2Real transition tutorials
├── tests/ # Test suite
└── examples/ # Example implementations
│ ├── assets/ # Asset-related tutorials
│ └── sim2real/ # Sim2Real transition tutorials
├── workflows/ # Main workflow implementations
│ ├── robotic_surgery/ # Robotic surgery workflow
│ ├── robotic_ultrasound/ # Robotic ultrasound workflow
│ └── telesurgery/ # Telesurgery workflow
```

## Contributing
Expand Down
Binary file modified docs/source/telesurgery_workflow.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
33 changes: 33 additions & 0 deletions holoscan_i4h/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

cmake_minimum_required(VERSION 3.20)

project(holoscan-i4h)

set(CMAKE_INSTALL_PREFIX "${CMAKE_BINARY_DIR}/../install")

set(LIB_DIR "${CMAKE_BINARY_DIR}/lib")
set(INCLUDE_DIR "${CMAKE_BINARY_DIR}/include")

file(MAKE_DIRECTORY ${INCLUDE_DIR})
file(MAKE_DIRECTORY ${LIB_DIR})

add_subdirectory(operators)

message("install: ${CMAKE_INSTALL_PREFIX}")

install(DIRECTORY ${LIB_DIR}/ DESTINATION lib)
install(DIRECTORY ${INCLUDE_DIR}/ DESTINATION include)
51 changes: 51 additions & 0 deletions holoscan_i4h/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
## Holoscan Operators
The Holoscan operators for Isaac-for-Healthcare are located in the `holoscan_i4h/operators/` directory. These operators are modular components designed to be reused across multiple workflows and applications.

### Using the Operators
To import these operators in your code, ensure the root of the i4h-workflows repository is included in your PYTHONPATH. You can do this by running:

```bash
export PYTHONPATH=$PYTHONPATH:/path/to/i4h-workflows/
```

You can then import operators as follows:

```python
from holoscan_i4h.operators.realsense.realsense import RealsenseOp
```

### Directory Structure

The initial directory layout under `holoscan_i4h/` is straightforward and is shown below.
The primary directory of interest is the `operators/` folder where the Holoscan operators are located.

```
holoscan_i4h/
├── operators/
│ ├── realsense/
│ ├── clarius_cast/
│ ├── clarius_solum/
│ └── ...
├── CMakeLists.txt
├── README.md
└── __init__.py
```

### Build and Install Folders

Some operators — such as `clarius_solum` — may require a build step. This step is typically triggered automatically as part of the workflow setup process. When this occurs, additional `build/` and `install/` directories will appear:

* `build/`: A temporary directory used by CMake to generate and compile intermediate build artifacts (e.g., object files, build system metadata).
* `install/`: The final output directory for installed components, such as shared libraries (`.so`) and header files (`.h`), ready to be consumed by the operator.

Example layout after build:

```
holoscan_i4h/
├── operators/
├── build/
├── install/
├── CMakeLists.txt
├── README.md
└── __init__.py
```
14 changes: 14 additions & 0 deletions holoscan_i4h/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
17 changes: 17 additions & 0 deletions holoscan_i4h/operators/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

add_subdirectory(clarius_cast)
add_subdirectory(clarius_solum)
14 changes: 14 additions & 0 deletions holoscan_i4h/operators/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
Expand All @@ -16,10 +17,8 @@
cmake_minimum_required(VERSION 3.20)
project(clarius_cast)

set(CMAKE_CXX_STANDARD 17)

set(LIB_DIR "${CMAKE_CURRENT_SOURCE_DIR}/lib")
set(INCLUDE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/include/cast")
set(LIB_DIR "${CMAKE_BINARY_DIR}/lib")
set(INCLUDE_DIR "${CMAKE_BINARY_DIR}/include")

execute_process(
COMMAND lsb_release -sr
Expand All @@ -28,36 +27,31 @@ execute_process(
)

# Download Clarius library and header files
if(NOT EXISTS "${LIB_DIR}/libcast.so")
file(MAKE_DIRECTORY ${LIB_DIR})
if(NOT EXISTS "${LIB_DIR}/clarius_cast/libcast.so")
file(MAKE_DIRECTORY ${LIB_DIR}/clarius_cast)
file(DOWNLOAD

"https://github.com/clariusdev/cast/releases/download/v12.0.2/cast-12.0.2-linux.x86_64-gcc_ubuntu_${UBUNTU_VERSION}.zip"
"${LIB_DIR}/cast.zip"
"${LIB_DIR}/clarius_cast/cast.zip"
)
# Unzip the file
execute_process(
COMMAND ${CMAKE_COMMAND} -E tar xzf ${LIB_DIR}/cast.zip
WORKING_DIRECTORY ${LIB_DIR}
COMMAND unzip cast.zip
WORKING_DIRECTORY ${LIB_DIR}/clarius_cast
)

# Header files are included in the zip file
file(MAKE_DIRECTORY ${INCLUDE_DIR})
execute_process(
COMMAND mv "${LIB_DIR}/cast" "${CMAKE_CURRENT_SOURCE_DIR}/include"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
execute_process(COMMAND rm -rf "${INCLUDE_DIR}/cast")
message("mv ${LIB_DIR}/clarius_cast/cast ${INCLUDE_DIR}/")
execute_process(COMMAND mv "${LIB_DIR}/clarius_cast/cast" "${INCLUDE_DIR}/")

# Use python3.10 library
execute_process(
COMMAND mv "${LIB_DIR}/python310/pyclariuscast.so" "${LIB_DIR}/"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMAND mv "${LIB_DIR}/clarius_cast/python310/pyclariuscast.so" "${LIB_DIR}/clarius_cast"
)

# Cleanup
execute_process(
COMMAND bash -c "rm -rf ${LIB_DIR}/python* ${LIB_DIR}/cast.zip"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMAND bash -c "rm -rf ${LIB_DIR}/clarius_cast/python* ${LIB_DIR}/clarius_cast/cast.zip"
)

endif()
14 changes: 14 additions & 0 deletions holoscan_i4h/operators/clarius_cast/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Loading
Loading