Port main to release branch as preparation for v0.2.0rc1 #163
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #132 ### Description --------- Signed-off-by: Mingxin Zheng <[email protected]>
part of #127 Add Playback Recording Feature --------- Signed-off-by: YunLiu <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
This PR introduces dockerfile and verified the IsaacLab livestreaming feature in the docker container setup --------- Signed-off-by: Mingxin Zheng <[email protected]>
Part of #127 ### Description Eval GRROT N1 model --------- Signed-off-by: YunLiu <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
Add cosmos transfer1 dependency refer to https://github.com/nvidia-cosmos/cosmos-transfer1/blob/main/INSTALL.md#inference Don't install `apex` which is for training part. Fix #135 --------- Signed-off-by: YunLiu <[email protected]>
Fixes #ISSUE_NUMBER ### Description --------- Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Julien Jomier <[email protected]>
Fixes #ISSUE_NUMBER ### Description --------- Signed-off-by: Mingxin Zheng <[email protected]>
This PR adds Docker support for the robotic ultrasound simulation, cleans up an unused ONNX utility, and refactors environment setup scripts for consistency in conda and docker env. --------- Signed-off-by: Mingxin Zheng <[email protected]>
### Description Integrate[ cosmos-transfer1](https://github.com/nvidia-cosmos/cosmos-transfer1/blob/861254a98a01d4e23f3e41d8a213d002a737e1d3/examples/inference_cosmos_transfer1_7b.md) with our proposed guided generation pipeline to the I4H workflow. --------- Signed-off-by: Pengfei Guo <[email protected]> Signed-off-by: YunLiu <[email protected]> Co-authored-by: YunLiu <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
Fixes #ISSUE_NUMBER ### Description This PR requires: ``` export ISAAC_ASSET_SHA256_HASH=8e80faed126c533243f50bb01dca3dcf035e86b5bf567d622878866a8ef7f12d ``` to ensure MIRA asset is included --------- Signed-off-by: Yiheng Wang <[email protected]> Signed-off-by: Yiheng Wang <[email protected]> Co-authored-by: Nic Ma <[email protected]>
Fixes duplicated install of cudnn introduced by #129 Signed-off-by: Mingxin Zheng <[email protected]> Co-authored-by: YunLiu <[email protected]>
… metrics evaluation pipeline. (#137) Enhance sim_with_dds.py to reset to the recorded initial state. If hdf5_path is None, it will reset to the initial state recorded in the recorded data. Add evaluation pipeline --------- Signed-off-by: YunLiu <[email protected]>
Part of #127 ### Description - upgrade Isaaclab to v2.1.0 - patch Isaaclab to support random texture generate and semantic segmentation generate - update data config used for GR00T-N1 training - Instroduce `BaseFeatureDict` to refactor the convert_hdf5_to_lerobot.py support both Pi0 and GR00T data format - Increase the default phantom randomness in the franka_manager_rl_env_cfg.py - Add training code --------- Signed-off-by: YunLiu <[email protected]>
Fixes #ISSUE_NUMBER ### Description I noticed the UltrasoundProbe class interface changed in raysim. Here is an update with two ultrasound probes that are available now. We can discuss the solution, and should probably add a version tag on the raysim repo, to make sure i4h-workflows is unaffected by changes to raysim.   --------- Signed-off-by: Maximilian Ofir <[email protected]> Signed-off-by: Maximilian Ofir <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
Fixes #151 ### Description --------- Signed-off-by: Yiheng Wang <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
Addresses rows (32) and (33) in [I4H v0.2 Execution Plan](https://docs.google.com/spreadsheets/d/1fEUVGnqZHRlmc08gH-tVyKPg071PJqzwJYrVa14Tv_4/edit?gid=1032227303#gid=1032227303) ### Description Integrates the CloudXR and Isaac teams' OpenXR hand tracking support in Isaac for Healthcare. 1. Adds support for Isaac Lab's OpenXR-based `Se3HandTracking` in the Robotic Ultrasound teleoperation workflow sample. 2. Adds tutorial for using Isaac Lab + CloudXR OpenXR components to run the `robotic_ultrasound` teleoperation sample with XR hand tracking inputs.  ### Notes: - Changes are in draft stage. Feedback and suggestions for improvements are welcome. - Isaac Lab OpenXR setup is largely informed by the [Isaac Lab OpenXR tutorial](https://isaac-sim.github.io/IsaacLab/main/source/how-to/cloudxr_teleoperation.html#setting-up-cloudxr-teleoperation), streamlined for Isaac for Healthcare and limited to `v2.0.2` feature availability and API. - Tested locally with Apple Vision Pro. - Last tested based on 929ed75. Rebased on `main` for PR submission. - Observed limitations are noted in the tutorial, including: - Some XR features present in the `v2.1.0` API are missing from the `robotic_ultrasound` pinned Isaac Lab version `v2.0.2` (retargeting), limiting ease of use - Observed application error when room camera and XR device are rendering at the same time. External data limited to wrist camera + joint position during teleop --------- Signed-off-by: YunLiu <[email protected]> Co-authored-by: YunLiu <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
…ng (#160) This PR enhances the test log printing with improved real-time output streaming, detailed logging, and configurable timeouts for test processes. --------- Signed-off-by: Mingxin Zheng <[email protected]>
- [x] Env setup (similar to other 2 workflows) - [x] IGX Support to run patient in real/physical world - [x] x86 Support to run patient in simulation world - [x] Surgeon to fetch/instruct command over Gamepad - [x] Haply device support for both SIM and Real - [ ] Dockerfile (needs to be verified; otherwise it's just a copy based on other 2 workflows) > HINT:: Reviewers Install and Try the setup > HINT:: Do direct [Code walk](https://github.com/isaac-for-healthcare/i4h-workflows/tree/sachi/telesurgery/workflows/telesurgery/scripts) --------- Signed-off-by: Sachidanand Alle <[email protected]> Signed-off-by: binliu <[email protected]> Signed-off-by: Victor Chang <[email protected]> Signed-off-by: binliunls <[email protected]> Co-authored-by: binliunls <[email protected]> Co-authored-by: Victor Chang <[email protected]> Co-authored-by: Victor Chang <[email protected]> Co-authored-by: wendell-hom <[email protected]> Co-authored-by: Wendell Hom <[email protected]> Co-authored-by: binliu <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
) ### Description Update the default policy_ckpt to use the new weights training in Isaacsim 4.5. Also update the evalutation readme to mention the latest 4 models path. --------- Signed-off-by: YunLiu <[email protected]> Signed-off-by: Mingxin Zheng <[email protected]> Co-authored-by: Mingxin Zheng <[email protected]>
Fixes #164 ### Description Workaround for #164 Move openpi related import after initialize SimulationApp Signed-off-by: YunLiu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_NUMBER
Description