Comprehensive documentation refractor for robotic ultrasound workflow

workflows/robotic_ultrasound/docker/README.md

@@ -35,12 +35,12 @@ docker build --ssh default --no-cache -f workflows/robotic_ultrasound/docker/Doc
 
 ## Running the Container
 
-```sh
+```bash
 # Allow Docker to access X11 display
 xhost +local:docker
 
-# Run container with GUI support in background
-docker run --name isaac-sim -itd --gpus all --rm \
+# Run container with GUI support
+docker run --name isaac-sim -it --gpus all --rm \
     --network=host \
     --runtime=nvidia \
     --entrypoint=bash \
@@ -59,65 +59,23 @@ docker run --name isaac-sim -itd --gpus all --rm \
     -v ~/docker/isaac-sim/documents:/root/Documents:rw \
     -v ~/.cache/i4h-assets:/root/.cache/i4h-assets:rw \
     -v ~/docker/rti:/root/rti:ro \
-    -v ~/raysim:/workspace/i4h-workflows/workflows/robotic_ultrasound/scripts/raysim:ro \
+    -v $(pwd)/workflows/robotic_ultrasound/scripts/raysim:/workspace/i4h-workflows/workflows/robotic_ultrasound/scripts/raysim:ro \
     robotic_us:latest
 ```
 
 ## Running the Simulation
 
-> Multiple terminals are required for running the simulation.
-
-### 1. Run Policy
-
-First, start the policy runner in the background. This process will wait for simulation data and provide control commands.
+The command to run the simulation is the same as [Running Workflows](../README.md#running-workflows) section.
 
-```bash
-docker exec -it isaac-sim bash
+For example,
+```sh
 # Inside the container
 conda activate robotic_ultrasound
-python workflows/robotic_ultrasound/scripts/policy_runner/run_policy.py
-```
 
-**Note:** The policy runner should be started first since it will continuously run and communicate with the simulation via DDS.
-
-### 2. Run Isaac Sim Simulation
-
-In a separate terminal session, start the main simulation:
-
-```bash
-docker exec -it isaac-sim bash
-# Inside the container, run the simulation
-conda activate robotic_ultrasound
-python workflows/robotic_ultrasound/scripts/simulation/environments/sim_with_dds.py --enable_camera
+# Run simulation with GUI
+(python -m policy_runner.run_policy --policy pi0 & python -m simulation.environments.sim_with_dds --enable_cameras & wait)
 ```
 
-
-### 3. Ultrasound Raytracing Simulation (Optional)
-
-For realistic ultrasound image generation, you can run the ultrasound raytracing simulator:
-
-```bash
-docker exec -it isaac-sim bash
-# Inside the container, run the ultrasound raytracing simulator
-conda activate robotic_ultrasound
-python workflows/robotic_ultrasound/scripts/simulation/examples/ultrasound_raytracing.py
-```
-
-This will generate and stream ultrasound images via DDS communication.
-
-### 4. Visualization Utility (Optional)
-
-To visualize the ultrasound images and other sensor data, you can use the visualization utility:
-
-```bash
-docker exec -it isaac-sim bash
-# Inside the container, run the visualization utility
-conda activate robotic_ultrasound
-python workflows/robotic_ultrasound/scripts/utils/visualization.py
-```
-
-This utility will display real-time ultrasound images and other sensor data streams.
-
 ## Troubleshooting
 
 ### GPU Device Errors

workflows/robotic_ultrasound/reset.sh

@@ -0,0 +1,103 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script to kill all robotic ultrasound workflow processes and their children
+
+echo "Killing robotic ultrasound workflow processes..."
+
+# Function to kill process and its children
+kill_process_tree() {
+    local pattern="$1"
+    local name="$2"
+
+    echo "Checking for processes matching: $pattern"
+
+    # Get PIDs of matching processes
+    pids=$(pgrep -f "$pattern")
+
+    if [ ! -z "$pids" ]; then
+        echo "Found PIDs: $pids for $name"
+
+        for pid in $pids; do
+            echo "Killing process tree for PID $pid ($name)"
+
+            # Get all child processes
+            children=$(pgrep -P $pid 2>/dev/null)
+            if [ ! -z "$children" ]; then
+                echo "  Found child processes: $children"
+                # Kill children first
+                for child in $children; do
+                    echo "  Killing child process $child"
+                    kill -TERM $child 2>/dev/null
+                done
+            fi
+
+            # Kill the parent process
+            echo "  Killing parent process $pid"
+            kill -TERM $pid 2>/dev/null
+        done
+
+        # Wait for graceful shutdown
+        sleep 2
+
+        # Force kill any remaining processes and children
+        for pid in $pids; do
+            if kill -0 $pid 2>/dev/null; then
+                echo "Force killing process tree for PID $pid"
+                children=$(pgrep -P $pid 2>/dev/null)
+                if [ ! -z "$children" ]; then
+                    for child in $children; do
+                        kill -9 $child 2>/dev/null
+                    done
+                fi
+                kill -9 $pid 2>/dev/null
+            fi
+        done
+
+        echo "Killed $name processes"
+    else
+        echo "No processes found for $name"
+    fi
+}
+
+# Kill specific processes and their children
+kill_process_tree "policy_runner.run_policy" "policy runner"
+kill_process_tree "simulation.environments.sim_with_dds" "simulation with DDS"
+kill_process_tree "simulation.examples.ultrasound_raytracing" "ultrasound raytracing"
+kill_process_tree "utils.visualization" "visualization"
+kill_process_tree "isaac-sim" "Isaac Sim"
+
+# Double-check for any remaining processes
+echo ""
+echo "Double-checking for any remaining processes..."
+remaining=$(ps aux | grep -E "(policy_runner.run_policy|simulation.environments.sim_with_dds|simulation.examples.ultrasound_raytracing|utils.visualization|isaac-sim)" | grep -v grep | grep -v "kill_all_processes.sh")
+
+if [ ! -z "$remaining" ]; then
+    echo "Found remaining processes:"
+    echo "$remaining"
+    echo "Force killing any stragglers..."
+    pkill -9 -f "policy_runner.run_policy" 2>/dev/null
+    pkill -9 -f "simulation.environments.sim_with_dds" 2>/dev/null
+    pkill -9 -f "simulation.examples.ultrasound_raytracing" 2>/dev/null
+    pkill -9 -f "utils.visualization" 2>/dev/null
+    pkill -9 -f "isaac-sim" 2>/dev/null
+else
+    echo "No remaining processes found."
+fi
+
+echo ""
+echo "All processes and their children should be terminated now."

workflows/robotic_ultrasound/scripts/policy_runner/README.md

@@ -7,68 +7,68 @@ This script allows running different policy models (currently PI0 and GR00T N1)
 *   **PI0**: Based on the [openpi](https://github.com/Physical-Intelligence/openpi) library.
 *   **GR00T N1**: NVIDIA's foundation model for humanoid robots. Refer to [NVIDIA Isaac GR00T](https://github.com/NVIDIA/Isaac-GR00T) for more information.
 
-## Run PI0 Policy with DDS communication
+## Run Policy with DDS communication
 
-### Prepare Model Weights and USD Assets and Install Dependencies
+### Dependencies
 
-Please refer to the [Environment Setup](../../README.md#environment-setup) instructions specific to the **PI0** policy.
+You may need to install the dependencies for either pi0 (default installed) or gr00tn1.
 
-### Ensure the PYTHONPATH Is Set
-
-Please refer to the [Environment Setup - Set environment variables before running the scripts](../../README.md#set-environment-variables-before-running-the-scripts) instructions.
-
-### Run Policy
-
-Please move to the current [`policy_runner` folder](./) and execute:
-
-```sh
-# Example for PI0
-python run_policy.py --policy pi0 [other arguments...]
+```bash
+# Install the dependencies for PI0, default installed
+bash tools/env_setup_robot_us.sh --policy pi0
+# Install the dependencies for GR00T N1
+bash tools/env_setup_robot_us.sh --policy gr00tn1
 ```
+The environment for pi0 and gr00tn1 has conflicts with each other. You can only install one of them at a time.
 
-## Run GR00T N1 Policy with DDS communication
+### Prepare Model Weights
 
-### Prepare Model Weights and Dependencies
+The model weights will be downloaded automatically when you [run the policy runner](#run-policy).
 
-Please refer to the [Environment Setup](../../README.md#environment-setup) instructions, ensuring you use the `--policy gr00tn1` option when running `tools/env_setup_robot_us.sh` to install GR00T N1 dependencies.
-For acquiring model weights and further details, consult the official [NVIDIA Isaac GR00T Installation Guide](https://github.com/NVIDIA/Isaac-GR00T?tab=readme-ov-file#installation-guide).
-Ensure the environment where you run `run_policy.py` has the GR00 TN1 dependencies installed.
+Optionally, you can also download the weights manually by running the following command:
+```bash
+# Download the model weights for PI0
+i4h-asset-retrieve --sub-path Policies/LiverScan/Pi0
 
-### Ensure the PYTHONPATH Is Set
-
-Please refer to the [Environment Setup - Set environment variables before running the scripts](../../README.md#set-environment-variables-before-running-the-scripts) instructions.
+# Download the model weights for GR00T N1
+i4h-asset-retrieve --sub-path Policies/LiverScan/GR00TN1
+```
 
 ### Run Policy
 
-Please move to the current [`policy_runner` folder](./) and execute:
-
 ```sh
+# Example for PI0
+python -m policy_runner.run_policy --policy pi0
 # Example for GR00T N1
-python run_policy.py --policy gr00tn1 [other arguments...]
+python -m policy_runner.run_policy --policy gr00tn1
 ```
 
-## Command Line Arguments
-
-Here's a markdown table describing the command-line arguments:
-
-| Argument                  | Description                                                              | Default Value                      | Policy    |
-|---------------------------|--------------------------------------------------------------------------|------------------------------------|-----------|
-| `--policy`                | Policy type to use.                                                      | `pi0`                              | Both      |
-| `--ckpt_path`             | Checkpoint path for the policy model.                                    | Uses downloaded assets             | Both      |
-| `--task_description`      | Task description text prompt for the policy.                             | `Perform a liver ultrasound.`      | Both      |
-| `--chunk_length`          | Length of the action chunk inferred by the policy per inference step.    | 50                                 | Both      |
-| `--repo_id`               | LeRobot repo ID for dataset normalization (used for PI0).                | `i4h/sim_liver_scan`               | PI0       |
-| `--data_config`           | Data config name (used for GR00T N1).                                     | `single_panda_us`                  | GR00T N1   |
-| `--embodiment_tag`        | The embodiment tag for the model (used for GR00T N1).                     | `new_embodiment`                   | GR00T N1   |
-| `--rti_license_file`      | Path to the RTI license file.                                            | Uses env `RTI_LICENSE_FILE`      | Both (DDS)|
-| `--domain_id`             | Domain ID for DDS communication.                                         | 0                                  | Both (DDS)|
-| `--height`                | Input image height for cameras.                                          | 224                                | Both (DDS)|
-| `--width`                 | Input image width for cameras.                                           | 224                                | Both (DDS)|
-| `--topic_in_room_camera`  | Topic name to consume room camera RGB data.                              | `topic_room_camera_data_rgb`       | Both (DDS)|
-| `--topic_in_wrist_camera` | Topic name to consume wrist camera RGB data.                             | `topic_wrist_camera_data_rgb`      | Both (DDS)|
-| `--topic_in_franka_pos`   | Topic name to consume Franka position data.                              | `topic_franka_info`                | Both (DDS)|
-| `--topic_out`             | Topic name to publish generated Franka actions.                          | `topic_franka_ctrl`                | Both (DDS)|
-| `--verbose`               | Whether to print DDS communication logs.                                 | False                              | Both (DDS)|
+**Expected Behavior:**
+- Terminal messages will confirm that the policy has loaded and is running.
+- The policy will predict actions and publish them to DDS topics when image feeds are available.
+- When no image feeds are available on DDS, the model will not predict any actions.
+- You can run the [Simulation with Data Distribution Service (DDS)](../simulation/environments/README.md) to produce the data for the policy to consume in IsaacSim.
+
+### Command Line Arguments
+
+| Argument | Type | Default | Description | Policy Support |
+|----------|------|---------|-------------|----------------|
+| `--policy` | str | "pi0" | Policy type to use (choices: pi0, gr00tn1) | Both |
+| `--ckpt_path` | str | robot_us_assets.policy_ckpt | Checkpoint path for the policy model | Both |
+| `--task_description` | str | "Perform a liver ultrasound." | Task description text prompt for the policy | Both |
+| `--chunk_length` | int | 50 | Length of the action chunk inferred by the policy | Both |
+| `--repo_id` | str | "i4h/sim_liver_scan" | LeRobot repo ID for dataset normalization | PI0 only |
+| `--data_config` | str | "single_panda_us" | Data config name for GR00T N1 policy | GR00T N1 only |
+| `--embodiment_tag` | str | "new_embodiment" | The embodiment tag for the GR00T N1 model | GR00T N1 only |
+| `--rti_license_file` | str | $RTI_LICENSE_FILE | Path to the RTI license file | Both (DDS) |
+| `--domain_id` | int | 0 | Domain ID for DDS communication | Both (DDS) |
+| `--height` | int | 224 | Input image height for cameras | Both (DDS) |
+| `--width` | int | 224 | Input image width for cameras | Both (DDS) |
+| `--topic_in_room_camera` | str | "topic_room_camera_data_rgb" | Topic name to consume room camera RGB data | Both (DDS) |
+| `--topic_in_wrist_camera` | str | "topic_wrist_camera_data_rgb" | Topic name to consume wrist camera RGB data | Both (DDS) |
+| `--topic_in_franka_pos` | str | "topic_franka_info" | Topic name to consume Franka position data | Both (DDS) |
+| `--topic_out` | str | "topic_franka_ctrl" | Topic name to publish generated Franka actions | Both (DDS) |
+| `--verbose` | bool | False | Whether to print DDS communication logs | Both (DDS) |
 
 ## Performance Metrics
 
@@ -89,3 +89,10 @@ Here's a markdown table describing the command-line arguments:
 > - GR00T N1 model: Predicts 16 actions in ~92ms
 >
 >  You can choose how many of these predictions to utilize based on your specific control frequency requirements.
+
+---
+
+## Documentation Links
+
+- [Simulation with Data Distribution Service (DDS)](../simulation/environments/README.md)
+- [IsaacLab Task Setting](../simulation/exts/robotic_us_ext/README.md)