Skip to content

Commit d34bf19

Browse files
mingxin-zhengmingxin-zheng
authored
Comprehensive documentation refractor for robotic ultrasound workflow (#181)
This PR addresses feedback on the robotic ultrasound experience by: - Restructure main README with quick start guide, detailed setup instructions, and comprehensive script catalog table - Split comprehensive simulation README into smaller individual files for references - Policy runner, simulation environments, teleoperation, evaluation - State machine, ultrasound raytracing, cosmos transfer1 integration - Imitation learning, utilities, and Holoscan apps - Clean up import statements in liver_scan_sm.py with proper isort formatting - Add reset.sh utility script for killing all workflow processes - Add missing __init__.py file for utils package - Improve documentation structure with jump-to-section navigation - Add troubleshooting section and system requirements clarification ### Description --------- Signed-off-by: Mingxin Zheng <mingxinz@nvidia.com>
1 parent bf36b30 commit d34bf19

16 files changed

Lines changed: 1009 additions & 688 deletions

File tree

‎workflows/robotic_ultrasound/README.md‎

Lines changed: 292 additions & 123 deletions
Large diffs are not rendered by default.

‎workflows/robotic_ultrasound/docker/README.md‎

Lines changed: 9 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -35,12 +35,12 @@ docker build --ssh default --no-cache -f workflows/robotic_ultrasound/docker/Doc
3535

3636
## Running the Container
3737

38-
```sh
38+
```bash
3939
# Allow Docker to access X11 display
4040
xhost +local:docker
4141

42-
# Run container with GUI support in background
43-
docker run --name isaac-sim -itd --gpus all --rm \
42+
# Run container with GUI support
43+
docker run --name isaac-sim -it --gpus all --rm \
4444
--network=host \
4545
--runtime=nvidia \
4646
--entrypoint=bash \
@@ -59,65 +59,23 @@ docker run --name isaac-sim -itd --gpus all --rm \
5959
-v ~/docker/isaac-sim/documents:/root/Documents:rw \
6060
-v ~/.cache/i4h-assets:/root/.cache/i4h-assets:rw \
6161
-v ~/docker/rti:/root/rti:ro \
62-
-v ~/raysim:/workspace/i4h-workflows/workflows/robotic_ultrasound/scripts/raysim:ro \
62+
-v $(pwd)/workflows/robotic_ultrasound/scripts/raysim:/workspace/i4h-workflows/workflows/robotic_ultrasound/scripts/raysim:ro \
6363
robotic_us:latest
6464
```
6565

6666
## Running the Simulation
6767

68-
> Multiple terminals are required for running the simulation.
69-
70-
### 1. Run Policy
71-
72-
First, start the policy runner in the background. This process will wait for simulation data and provide control commands.
68+
The command to run the simulation is the same as [Running Workflows](../README.md#running-workflows) section.
7369

74-
```bash
75-
docker exec -it isaac-sim bash
70+
For example,
71+
```sh
7672
# Inside the container
7773
conda activate robotic_ultrasound
78-
python workflows/robotic_ultrasound/scripts/policy_runner/run_policy.py
79-
```
8074

81-
**Note:** The policy runner should be started first since it will continuously run and communicate with the simulation via DDS.
82-
83-
### 2. Run Isaac Sim Simulation
84-
85-
In a separate terminal session, start the main simulation:
86-
87-
```bash
88-
docker exec -it isaac-sim bash
89-
# Inside the container, run the simulation
90-
conda activate robotic_ultrasound
91-
python workflows/robotic_ultrasound/scripts/simulation/environments/sim_with_dds.py --enable_camera
75+
# Run simulation with GUI
76+
(python -m policy_runner.run_policy --policy pi0 & python -m simulation.environments.sim_with_dds --enable_cameras & wait)
9277
```
9378

94-
95-
### 3. Ultrasound Raytracing Simulation (Optional)
96-
97-
For realistic ultrasound image generation, you can run the ultrasound raytracing simulator:
98-
99-
```bash
100-
docker exec -it isaac-sim bash
101-
# Inside the container, run the ultrasound raytracing simulator
102-
conda activate robotic_ultrasound
103-
python workflows/robotic_ultrasound/scripts/simulation/examples/ultrasound_raytracing.py
104-
```
105-
106-
This will generate and stream ultrasound images via DDS communication.
107-
108-
### 4. Visualization Utility (Optional)
109-
110-
To visualize the ultrasound images and other sensor data, you can use the visualization utility:
111-
112-
```bash
113-
docker exec -it isaac-sim bash
114-
# Inside the container, run the visualization utility
115-
conda activate robotic_ultrasound
116-
python workflows/robotic_ultrasound/scripts/utils/visualization.py
117-
```
118-
119-
This utility will display real-time ultrasound images and other sensor data streams.
120-
12179
## Troubleshooting
12280

12381
### GPU Device Errors

‎workflows/robotic_ultrasound/reset.sh‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/bin/bash
2+
3+
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
# Script to kill all robotic ultrasound workflow processes and their children
19+
20+
echo "Killing robotic ultrasound workflow processes..."
21+
22+
# Function to kill process and its children
23+
kill_process_tree() {
24+
local pattern="$1"
25+
local name="$2"
26+
27+
echo "Checking for processes matching: $pattern"
28+
29+
# Get PIDs of matching processes
30+
pids=$(pgrep -f "$pattern")
31+
32+
if [ ! -z "$pids" ]; then
33+
echo "Found PIDs: $pids for $name"
34+
35+
for pid in $pids; do
36+
echo "Killing process tree for PID $pid ($name)"
37+
38+
# Get all child processes
39+
children=$(pgrep -P $pid 2>/dev/null)
40+
if [ ! -z "$children" ]; then
41+
echo " Found child processes: $children"
42+
# Kill children first
43+
for child in $children; do
44+
echo " Killing child process $child"
45+
kill -TERM $child 2>/dev/null
46+
done
47+
fi
48+
49+
# Kill the parent process
50+
echo " Killing parent process $pid"
51+
kill -TERM $pid 2>/dev/null
52+
done
53+
54+
# Wait for graceful shutdown
55+
sleep 2
56+
57+
# Force kill any remaining processes and children
58+
for pid in $pids; do
59+
if kill -0 $pid 2>/dev/null; then
60+
echo "Force killing process tree for PID $pid"
61+
children=$(pgrep -P $pid 2>/dev/null)
62+
if [ ! -z "$children" ]; then
63+
for child in $children; do
64+
kill -9 $child 2>/dev/null
65+
done
66+
fi
67+
kill -9 $pid 2>/dev/null
68+
fi
69+
done
70+
71+
echo "Killed $name processes"
72+
else
73+
echo "No processes found for $name"
74+
fi
75+
}
76+
77+
# Kill specific processes and their children
78+
kill_process_tree "policy_runner.run_policy" "policy runner"
79+
kill_process_tree "simulation.environments.sim_with_dds" "simulation with DDS"
80+
kill_process_tree "simulation.examples.ultrasound_raytracing" "ultrasound raytracing"
81+
kill_process_tree "utils.visualization" "visualization"
82+
kill_process_tree "isaac-sim" "Isaac Sim"
83+
84+
# Double-check for any remaining processes
85+
echo ""
86+
echo "Double-checking for any remaining processes..."
87+
remaining=$(ps aux | grep -E "(policy_runner.run_policy|simulation.environments.sim_with_dds|simulation.examples.ultrasound_raytracing|utils.visualization|isaac-sim)" | grep -v grep | grep -v "kill_all_processes.sh")
88+
89+
if [ ! -z "$remaining" ]; then
90+
echo "Found remaining processes:"
91+
echo "$remaining"
92+
echo "Force killing any stragglers..."
93+
pkill -9 -f "policy_runner.run_policy" 2>/dev/null
94+
pkill -9 -f "simulation.environments.sim_with_dds" 2>/dev/null
95+
pkill -9 -f "simulation.examples.ultrasound_raytracing" 2>/dev/null
96+
pkill -9 -f "utils.visualization" 2>/dev/null
97+
pkill -9 -f "isaac-sim" 2>/dev/null
98+
else
99+
echo "No remaining processes found."
100+
fi
101+
102+
echo ""
103+
echo "All processes and their children should be terminated now."

‎workflows/robotic_ultrasound/scripts/policy_runner/README.md‎

Lines changed: 54 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -7,68 +7,68 @@ This script allows running different policy models (currently PI0 and GR00T N1)
77
* **PI0**: Based on the [openpi](https://github.com/Physical-Intelligence/openpi) library.
88
* **GR00T N1**: NVIDIA's foundation model for humanoid robots. Refer to [NVIDIA Isaac GR00T](https://github.com/NVIDIA/Isaac-GR00T) for more information.
99

10-
## Run PI0 Policy with DDS communication
10+
## Run Policy with DDS communication
1111

12-
### Prepare Model Weights and USD Assets and Install Dependencies
12+
### Dependencies
1313

14-
Please refer to the [Environment Setup](../../README.md#environment-setup) instructions specific to the **PI0** policy.
14+
You may need to install the dependencies for either pi0 (default installed) or gr00tn1.
1515

16-
### Ensure the PYTHONPATH Is Set
17-
18-
Please refer to the [Environment Setup - Set environment variables before running the scripts](../../README.md#set-environment-variables-before-running-the-scripts) instructions.
19-
20-
### Run Policy
21-
22-
Please move to the current [`policy_runner` folder](./) and execute:
23-
24-
```sh
25-
# Example for PI0
26-
python run_policy.py --policy pi0 [other arguments...]
16+
```bash
17+
# Install the dependencies for PI0, default installed
18+
bash tools/env_setup_robot_us.sh --policy pi0
19+
# Install the dependencies for GR00T N1
20+
bash tools/env_setup_robot_us.sh --policy gr00tn1
2721
```
22+
The environment for pi0 and gr00tn1 has conflicts with each other. You can only install one of them at a time.
2823

29-
## Run GR00T N1 Policy with DDS communication
24+
### Prepare Model Weights
3025

31-
### Prepare Model Weights and Dependencies
26+
The model weights will be downloaded automatically when you [run the policy runner](#run-policy).
3227

33-
Please refer to the [Environment Setup](../../README.md#environment-setup) instructions, ensuring you use the `--policy gr00tn1` option when running `tools/env_setup_robot_us.sh` to install GR00T N1 dependencies.
34-
For acquiring model weights and further details, consult the official [NVIDIA Isaac GR00T Installation Guide](https://github.com/NVIDIA/Isaac-GR00T?tab=readme-ov-file#installation-guide).
35-
Ensure the environment where you run `run_policy.py` has the GR00 TN1 dependencies installed.
28+
Optionally, you can also download the weights manually by running the following command:
29+
```bash
30+
# Download the model weights for PI0
31+
i4h-asset-retrieve --sub-path Policies/LiverScan/Pi0
3632

37-
### Ensure the PYTHONPATH Is Set
38-
39-
Please refer to the [Environment Setup - Set environment variables before running the scripts](../../README.md#set-environment-variables-before-running-the-scripts) instructions.
33+
# Download the model weights for GR00T N1
34+
i4h-asset-retrieve --sub-path Policies/LiverScan/GR00TN1
35+
```
4036

4137
### Run Policy
4238

43-
Please move to the current [`policy_runner` folder](./) and execute:
44-
4539
```sh
40+
# Example for PI0
41+
python -m policy_runner.run_policy --policy pi0
4642
# Example for GR00T N1
47-
python run_policy.py --policy gr00tn1 [other arguments...]
43+
python -m policy_runner.run_policy --policy gr00tn1
4844
```
4945

50-
## Command Line Arguments
51-
52-
Here's a markdown table describing the command-line arguments:
53-
54-
| Argument | Description | Default Value | Policy |
55-
|---------------------------|--------------------------------------------------------------------------|------------------------------------|-----------|
56-
| `--policy` | Policy type to use. | `pi0` | Both |
57-
| `--ckpt_path` | Checkpoint path for the policy model. | Uses downloaded assets | Both |
58-
| `--task_description` | Task description text prompt for the policy. | `Perform a liver ultrasound.` | Both |
59-
| `--chunk_length` | Length of the action chunk inferred by the policy per inference step. | 50 | Both |
60-
| `--repo_id` | LeRobot repo ID for dataset normalization (used for PI0). | `i4h/sim_liver_scan` | PI0 |
61-
| `--data_config` | Data config name (used for GR00T N1). | `single_panda_us` | GR00T N1 |
62-
| `--embodiment_tag` | The embodiment tag for the model (used for GR00T N1). | `new_embodiment` | GR00T N1 |
63-
| `--rti_license_file` | Path to the RTI license file. | Uses env `RTI_LICENSE_FILE` | Both (DDS)|
64-
| `--domain_id` | Domain ID for DDS communication. | 0 | Both (DDS)|
65-
| `--height` | Input image height for cameras. | 224 | Both (DDS)|
66-
| `--width` | Input image width for cameras. | 224 | Both (DDS)|
67-
| `--topic_in_room_camera` | Topic name to consume room camera RGB data. | `topic_room_camera_data_rgb` | Both (DDS)|
68-
| `--topic_in_wrist_camera` | Topic name to consume wrist camera RGB data. | `topic_wrist_camera_data_rgb` | Both (DDS)|
69-
| `--topic_in_franka_pos` | Topic name to consume Franka position data. | `topic_franka_info` | Both (DDS)|
70-
| `--topic_out` | Topic name to publish generated Franka actions. | `topic_franka_ctrl` | Both (DDS)|
71-
| `--verbose` | Whether to print DDS communication logs. | False | Both (DDS)|
46+
**Expected Behavior:**
47+
- Terminal messages will confirm that the policy has loaded and is running.
48+
- The policy will predict actions and publish them to DDS topics when image feeds are available.
49+
- When no image feeds are available on DDS, the model will not predict any actions.
50+
- You can run the [Simulation with Data Distribution Service (DDS)](../simulation/environments/README.md) to produce the data for the policy to consume in IsaacSim.
51+
52+
### Command Line Arguments
53+
54+
| Argument | Type | Default | Description | Policy Support |
55+
|----------|------|---------|-------------|----------------|
56+
| `--policy` | str | "pi0" | Policy type to use (choices: pi0, gr00tn1) | Both |
57+
| `--ckpt_path` | str | robot_us_assets.policy_ckpt | Checkpoint path for the policy model | Both |
58+
| `--task_description` | str | "Perform a liver ultrasound." | Task description text prompt for the policy | Both |
59+
| `--chunk_length` | int | 50 | Length of the action chunk inferred by the policy | Both |
60+
| `--repo_id` | str | "i4h/sim_liver_scan" | LeRobot repo ID for dataset normalization | PI0 only |
61+
| `--data_config` | str | "single_panda_us" | Data config name for GR00T N1 policy | GR00T N1 only |
62+
| `--embodiment_tag` | str | "new_embodiment" | The embodiment tag for the GR00T N1 model | GR00T N1 only |
63+
| `--rti_license_file` | str | $RTI_LICENSE_FILE | Path to the RTI license file | Both (DDS) |
64+
| `--domain_id` | int | 0 | Domain ID for DDS communication | Both (DDS) |
65+
| `--height` | int | 224 | Input image height for cameras | Both (DDS) |
66+
| `--width` | int | 224 | Input image width for cameras | Both (DDS) |
67+
| `--topic_in_room_camera` | str | "topic_room_camera_data_rgb" | Topic name to consume room camera RGB data | Both (DDS) |
68+
| `--topic_in_wrist_camera` | str | "topic_wrist_camera_data_rgb" | Topic name to consume wrist camera RGB data | Both (DDS) |
69+
| `--topic_in_franka_pos` | str | "topic_franka_info" | Topic name to consume Franka position data | Both (DDS) |
70+
| `--topic_out` | str | "topic_franka_ctrl" | Topic name to publish generated Franka actions | Both (DDS) |
71+
| `--verbose` | bool | False | Whether to print DDS communication logs | Both (DDS) |
7272

7373
## Performance Metrics
7474

@@ -89,3 +89,10 @@ Here's a markdown table describing the command-line arguments:
8989
> - GR00T N1 model: Predicts 16 actions in ~92ms
9090
>
9191
> You can choose how many of these predictions to utilize based on your specific control frequency requirements.
92+
93+
---
94+
95+
## Documentation Links
96+
97+
- [Simulation with Data Distribution Service (DDS)](../simulation/environments/README.md)
98+
- [IsaacLab Task Setting](../simulation/exts/robotic_us_ext/README.md)

0 commit comments

Comments
 (0)