Robot Augmentation Workflow #48
Conversation
|
@lynetcha-nv can we please review? @sophiahhuang for vis |
|
Hi @pjannaty , could you please help with the process or upload the new example videos here ( |
| @@ -0,0 +1,8 @@ | |||
| { | |||
| "prompt1": "a robotic grasp an apple from the table and move it to another place.", | |||
There was a problem hiding this comment.
Should vis, edge, depth, and seg control_weight maps pt be included in this json similarly to assets/robot_augmentation_example/example1/inference_cosmos_transfer1_robot_spatiotemporal_weights.json ?
There was a problem hiding this comment.
This JSON provides examples of candidate prompts as input. To use them, we could substitute the prompt in assets/robot_augmentation_example/example1/inference_cosmos_transfer1_robot_spatiotemporal_weights.json using the candidate prompts in this example1_prompts.json
| Your browser does not support the video tag. | ||
| </video> | ||
|
|
||
| You can run multiple times with different prompts (e.g., `assets/robot_augmentation_example/example1/example1_prompts.json`), and you can get different augmentation results: |
There was a problem hiding this comment.
Please include the full inference command to illustrate how this file is used at inference time.
There was a problem hiding this comment.
As mentioned in the comments above, this file is a reference file to provide candidate prompts.
cosmos_transfer1/auxiliary/robot_augmentation/spatial_temporal_weight.py
Outdated
Show resolved
Hide resolved
cosmos_transfer1/auxiliary/robot_augmentation/spatial_temporal_weight.py
Outdated
Show resolved
Hide resolved
cosmos_transfer1/auxiliary/robot_augmentation/spatial_temporal_weight.py
Show resolved
Hide resolved
|
Thank you @gyhandy. Can we please trigger the pipeline [internal]/nvidia-cosmos/cosmos-transfer1 following instructions? |
|
LGTM, Thank you @gyhandy! |
* add spatial-temporal weight adding code * add robot augmentation examples * update spatial temporal processing code and example * add readme * recover * recover * add prompts and update readme * fix comments * visualization of the mp4 * update readme * linting
|
@gyhandy Sir ,would please add how you prepare the segmentation folder and label like { |
4 participants
Implementation for a spatial-temporal robot augmentation pipeline:
Generating spatial-temporal weight matrices from robot segmentation masks.
Provides two augmentation settings: