Skip to content

Add Sim2Real pipeline for OMY robot with LeRobot dataset conversion #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 55 commits into from
Nov 6, 2025
Merged

Add Sim2Real pipeline for OMY robot with LeRobot dataset conversion #4

merged 55 commits into from
Nov 6, 2025

Conversation

@kimtaehyeong99
Copy link
Member

@kimtaehyeong99 kimtaehyeong99 commented Sep 24, 2025 •
edited
Loading

This PR introduces a complete sim2real pipeline for the OMY robot, enabling imitation learning experiments with both simulation and real hardware.

The pipeline includes:

  • Task recording in simulation: Record demonstrations for a plastic bottle pickup task.
  • Sub-task annotation: Annotate collected demonstrations into meaningful sub-tasks for better training.
  • Action representation conversion: Convert action commands from joint space control to IK-based end-effector pose commands.
  • Data augmentation: Apply augmentation techniques to expand the dataset and improve policy generalization.
  • Dataset conversion: Transform the augmented data into the LeRobot dataset format for training with LeRobot.

Additionally, this PR integrates the robotis_dds_python library, allowing the system to receive the leader’s /joint_trajectory values without any ROS2 dependency.
Through this interface, the real OMY robot’s leader device can control the simulated robot, effectively bridging the gap between simulation and reality.

This contribution provides a reproducible workflow for recording, annotating, augmenting, and training imitation learning policies with the OMY robot.

Related PR

kimtaehyeong99 and others added 30 commits August 13, 2025 13:59
@kimtaehyeong99
Applied the updated record demo from the Isaac Lab repository.
d3cd970 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
intermediate save
c9e78c8 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Separate tasks into real-world and simulation folders.
4b689a5 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Separate tasks into real-world and simulation folders.
aee839b 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
intermediate saving
52cb353 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Recorder: prevent save on ‘B’, export only on ‘N’ (mark episode succe…
14a5310 
…ss), clear cache on ‘R’; switch export modes accordingly.

Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
added isaaclab2lerobot data converter
6a732f2 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
delete cube_2, cube_3 env
879bc72 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Proceeding with adding the Robotis OMY table and integrating a camera…
0ae70d3 
… sensor.

Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Modify the USD files of robotis_omy_table and the bottle
37a8451 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
task name changed stack cube -> pickup bottle
3e1b717 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Add CLI arguments with default values (task name, repo ID, robot type…
d178bfd 
…, dataset path, FPS, etc.) for dataset conversion script

Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
top_cam -> cam_top, robot_cam -> cam_wrist
5d446e6 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Partial save
a5b2c14 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
always publish joint_states and images
c5f50a4 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
added omy gripper mimic joint
ed49151 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
changed OMY.usd collision
bbac6a3 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
deleted rh_l1, rh_l2, rh_r1 in isaac2lerobot.py
08f1881 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Change in stack task environment due to USD file modification
d9da1a1 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
update README file
425b79d 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
README update
30320d3 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Modify the omy_sdk code to utilize robotis_python_sdk.
0d5a3a6 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
added topic_manager in omy_sdk.py
49fea0f 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
code repacktoring
66f4155 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
OMY camera param update
2ea617c 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
intermediate save
2fc00c3 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Developing the data augmentation module
ed11e3d 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
added isaaclab licence
35727a0 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
Rename LICENSE to Apache-2.0 license
685ec83 
Signed-off-by: Taehyeong Kim <[email protected]>
@kimtaehyeong99
rollback license
d149574 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99
rsl_rl update
7e494e7 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99 kimtaehyeong99 self-assigned this Oct 28, 2025
@kimtaehyeong99 kimtaehyeong99 added the enhancement New feature or request label Oct 28, 2025
@github-project-automation github-project-automation bot moved this to 🌱 Todo in Platform Oct 28, 2025
@kimtaehyeong99
added robotis_aiworker_table
8d7517f 
Signed-off-by: kimtaehyeong99 <[email protected]>
@kimtaehyeong99 kimtaehyeong99 moved this from 🌱 Todo to 📝 Pull Request in Platform Oct 28, 2025
ola31
ola31 reviewed Oct 28, 2025
View reviewed changes
ola31
ola31 reviewed Oct 28, 2025
View reviewed changes
ola31
ola31 reviewed Oct 28, 2025
View reviewed changes
ola31
ola31 reviewed Oct 28, 2025
View reviewed changes
@kimtaehyeong99 kimtaehyeong99 requested a review from ola31 October 28, 2025 06:42
yun-goon
yun-goon reviewed Nov 3, 2025
View reviewed changes
@kimtaehyeong99 kimtaehyeong99 changed the title Add Sim2Real Pipeline for OMY Robot with LeRobot Dataset Conversion Add Sim2Real pipeline for OMY robot with LeRobot dataset conversion Nov 3, 2025
yun-goon
yun-goon reviewed Nov 3, 2025
View reviewed changes
Copy link
Member

@yun-goon yun-goon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right. Please fix one more thing

@kimtaehyeong99 @yun-goon
Update ReleaseNote.md
f357f8d 
Co-authored-by: Wonho Yun <[email protected]>
Signed-off-by: Taehyeong Kim <[email protected]>
yun-goon
yun-goon approved these changes Nov 6, 2025
View reviewed changes
Copy link
Member

@yun-goon yun-goon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good!

ola31
ola31 approved these changes Nov 6, 2025
View reviewed changes
Copy link
Member

@ola31 ola31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good

@kimtaehyeong99 kimtaehyeong99 merged commit e39e8bf into main Nov 6, 2025
1 check passed
@kimtaehyeong99 kimtaehyeong99 deleted the feature-omy-leader-follower branch November 6, 2025 09:35
@github-project-automation github-project-automation bot moved this from 📝 Pull Request to 🚩Done in Platform Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@ola31 ola31 ola31 approved these changes

@yun-goon yun-goon yun-goon approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@kimtaehyeong99 kimtaehyeong99

Labels

enhancement New feature or request

Projects

Platform
Status: 🚩Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kimtaehyeong99 @ola31 @yun-goon