Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion tools/env_setup/install_gr00tn1.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ else
# Ensure parent third_party dir exists
mkdir -p "$PROJECT_ROOT/third_party"
git clone https://github.com/NVIDIA/Isaac-GR00T "$GR00T_DIR"
pushd "$GR00T_DIR"
git checkout n1-release
popd
fi

pushd "$GR00T_DIR"
Expand All @@ -49,7 +52,8 @@ else
echo "pyav already updated to av or not found in Isaac-GR00T's pyproject.toml."
fi

pip install -e .
pip install -e .[base]
pip install transformers==4.45.2
popd

pip install --no-build-isolation flash-attn==2.7.1.post4
Expand Down
8 changes: 6 additions & 2 deletions tools/env_setup_robot_us.sh
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,13 @@ ensure_fresh_third_party_dir
# ---- Install build tools (Common) ----
echo "Installing build tools..."
if [ "$EUID" -ne 0 ]; then
sudo apt-get install -y git cmake build-essential pybind11-dev libxcb-cursor0
sudo apt-get install -y cmake
sudo apt-get update
sudo apt-get install -y git build-essential pybind11-dev libxcb-cursor0
else
apt-get install -y git cmake build-essential pybind11-dev libxcb-cursor0
apt-get install -y cmake
apt-get update
apt-get install -y git build-essential pybind11-dev libxcb-cursor0
fi


Expand Down
3 changes: 3 additions & 0 deletions tutorials/assets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ This directory contains tutorials for integrating and teleoperating your own rob

- [Replace Franka Hand with Ultrasound Probe](./bring_your_own_robot/replace_franka_hand_with_ultrasound_probe.md)
Step-by-step guide to replacing the Franka robot’s hand with an ultrasound probe in Isaac Sim, including CAD/URDF conversion, asset import, and joint setup for custom robotic ultrasound simulation.

- [Bring Your Own Head-Mounted Display with OpenXR](./bring_your_own_xr/)
Comprehensive guide for using OpenXR-enabled mixed reality devices (like Apple Vision Pro) for immersive robotic teleoperation in Isaac Lab. Learn to set up NVIDIA CloudXR runtime, configure hand tracking controls, and enable simulated robotic teleoperation in Isaac Lab
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ To convert URDF to USD for use in Isaac Sim:
- Select your URDF file and configure import settings
- Click "Import" to generate the USD representation

For detailed steps, refer to the [official documentation on importing URDF](https://docs.isaacsim.omniverse.nvidia.com/latest/robot_setup/import_urdf.html).
For detailed steps, refer to the [official documentation on importing URDF](https://docs.isaacsim.omniverse.nvidia.com/4.5.0/robot_setup/import_urdf.html).

## Replacing Franka Hand with Ultrasound Probe

Expand Down Expand Up @@ -106,9 +106,9 @@ For detailed steps, refer to the [official documentation on importing URDF](http

## Additional Resources

- [URDF to USD Conversion Documentation](https://docs.isaacsim.omniverse.nvidia.com/latest/robot_setup/import_urdf.html#tutorial-import-urdf)
- [URDF to USD Conversion Documentation](https://docs.isaacsim.omniverse.nvidia.com/4.5.0/robot_setup/import_urdf.html#tutorial-import-urdf)
- [Omniverse CAD Extension Documentation](https://docs.omniverse.nvidia.com/extensions/latest/ext_cad-converter/manual.html)
- [Omniverse Importers and Exporters Documentation](https://docs.isaacsim.omniverse.nvidia.com/latest/robot_setup/importers_exporters.html#)
- [Omniverse Importers and Exporters Documentation](https://docs.isaacsim.omniverse.nvidia.com/4.5.0/robot_setup/importers_exporters.html#)
- [Video Tutorial: Training a Robot from Scratch in Simulation, from URDF to OpenUSD](https://www.youtube.com/live/_HMk7I-vSBQ)

## Support
Expand Down
8 changes: 6 additions & 2 deletions workflows/robotic_ultrasound/scripts/simulation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ Ensure your `PYTHONPATH` is set up as described in the [Environment Setup - Set
Navigate to the [`simulation` folder](./) and execute:

```sh
python environments/state_machine/replay_recording.py --hdf5_path /path/to/your/hdf5_data_directory --task <YourTaskName>
python environments/state_machine/replay_recording.py /path/to/your/hdf5_data_directory --task <YourTaskName> --enable_camera
```

Replace `/path/to/your/hdf5_data_directory` with the actual path to the directory containing your `data_*.hdf5` files or single HDF5 file, and `<YourTaskName>` with the task name used during data collection (e.g., `Isaac-Teleop-Torso-FrankaUsRs-IK-RL-Rel-v0`).
Expand Down Expand Up @@ -216,12 +216,16 @@ Directly applying Cosmos-Transfer with various control inputs results in unsatis
This training-free guided generation approach by encoding simulation videos into the latent space and applying spatial masking to guide the generation process. The trade-off between realism and faithfulness can be controlled by adjusting the number of guided denoising steps. In addition, our generation pipeline supports multi-view video generation. We first leverage the camera information to warp the generated room view to wrist view, then use it as the guidance of wrist-view generation.

#### Download Cosmos-transfer1 Checkpoints
The cosmos-transfer1 dependency is already installed after completing the [Installation](#installation) section. Move to the third party `cosmos-transfer1` folder and run the following command to download the checkpoints:
The cosmos-transfer1 dependency is already installed after completing the [Installation](#installation) section. Please navigate to the third party `cosmos-transfer1` folder and run the following command to download the checkpoints:
```sh
cd third_party/cosmos-transfer1
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python scripts/download_checkpoints.py --output_dir checkpoints/
```

> **Note:** You need to be logged in to Hugging Face (`huggingface-cli login`) before running the download script. Additionally, for the `meta-llama/Llama-Guard-3-8B` model, you need to request additional access on the [Hugging Face model page](https://huggingface.co/meta-llama/Llama-Guard-3-8B) before you can download it.



#### Video Prompt Generation
We follow the idea in [lucidsim](https://github.com/lucidsim/lucidsim) to first generate batches of meta prompt that contains a very concise description of the potential scene, then instruct the LLM (e.g., [gemma-3-27b-it](https://build.nvidia.com/google/gemma-3-27b-it)) to upsample the meta prompt with detailed descriptions.
We provide example prompts in [`generated_prompts_two_seperate_views.json`](./environments/cosmos_transfer1/config/generated_prompts_two_seperate_views.json).
Expand Down
Loading