Adds visual-based tactile sensor with shape sensing example #3420
Conversation
clean up license and configs; use full import path; add doc for visuo_tactile_sensor
JuanaDd
requested review from
Mayankm96,
jtigue-bdai,
kellyguo11 and
ooctipus
as code owners
| References | ||
| ~~~~~~~~~~ | ||
|
|
There was a problem hiding this comment.
@iakinola23 Could you help view this documentation here? Thanks!
There was a problem hiding this comment.
Hi @iakinola23 updated the documentation with your edits.
There was a problem hiding this comment.
@kellyguo11 Hi Kelly, I will upload the npz binaries, and the USD files to nucleus once the licensing concerns are resolved. Thanks!
Mayankm96
changed the title
github-actions
bot
added
documentation
kellyguo11
left a comment
kellyguo11
left a comment
There was a problem hiding this comment.
just a few high-level comments, haven't gone through the code in detail yet
| taxim_gelsight = gelsightRender("gelsight_r15") | ||
| import ipdb | ||
|
|
||
| ipdb.set_trace() |
There was a problem hiding this comment.
why is there a breakpoint here?
There was a problem hiding this comment.
deleted in the newest commit.
| self._camera_sensor = TiledCamera(self.cfg.camera_cfg) | ||
|
|
||
| # Initialize camera if not already done | ||
| # TODO: Juana: this is a hack to initialize the camera sensor. Should camera_sensor be managed by TacSL sensor or InteractiveScene? |
There was a problem hiding this comment.
maybe we can do something similar as the Raycaster camera?
There was a problem hiding this comment.
Hi Kelly, I checked the RayCasterCamera implementation, and it differs slightly from the tacsl sensor logic. The tacsl_sensor includes a camera sensor that can be enabled or disabled via the cfg, while RayCasterCamera only uses CameraData without having a camera or other sensor as a member. I’ve kept the current implementation and cleaned up the comments. Please let me know if you have any further concerns or suggestions.
There was a problem hiding this comment.
all changed to jpg format and documentation is updated accordingly.
| .. code-block:: bash | ||
| conda activate env_isaaclab | ||
| pip install opencv-python==4.11.0 trimesh==4.5.1 imageio==2.37.0 |
There was a problem hiding this comment.
do we need to add these to setup.py?
There was a problem hiding this comment.
Added opencv-python to setup.py; trimesh is already included, and imageio's dependency is not required in the newest commit.
| license agreement from NVIDIA CORPORATION is strictly prohibited. | ||
| ---------------- | ||
| Tensorized implementation of RGB rendering of gelsight-style visuo-tactile sensors |
There was a problem hiding this comment.
can you also add any licenses required for gelsight under docs/licenses?
There was a problem hiding this comment.
Updated the license header to be consistent with other files.
|
please also make sure to run ./isaaclab.sh -f for the linter checks |
| return dzdx, dzdy | ||
|
|
||
|
|
||
| def generate_normals(height_map): |
There was a problem hiding this comment.
generate_normals not needed and deprecated. only the tensorized version is used.
| self.background_tensor = torch.tensor(self.background, device=self.device) | ||
| print("Gelsight initialization done!") | ||
|
|
||
| def render(self, heightMap): |
There was a problem hiding this comment.
Delete render as it is not used. Only the tensorized version (render_tensorized ) is used.
There was a problem hiding this comment.
deleted render , generate_normals, and compute_image_gradient , padding as well.
|
Thank you for making TacSL available on IsaacLab! I have three quick questions:
Thanks! |
|
@ooctipus @Mayankm96 could you please help do a review of this one? |
|
@zxhuang97 Hi, thanks for your questions and for checking out TacSL on IsaacLab.
Hope this helps! |
There was a problem hiding this comment.
Only this diagram seems to be getting used. Please remove any unused images.
| conf_r15 = { | ||
| "data_dir": "gelsight_r15_data", | ||
| "background_path": "bg.jpg", | ||
| "calib_path": "polycalib.npz", | ||
| "real_bg": "real_bg.npy", | ||
| "h": 320, | ||
| "w": 240, | ||
| "numBins": 120, | ||
| "pixmm": 0.0877, | ||
| } | ||
| conf_gs_mini = { | ||
| "data_dir": "gs_mini_data", | ||
| "background_path": "bg.jpg", | ||
| "calib_path": "polycalib.npz", | ||
| "real_bg": "real_bg.npy", | ||
| "h": 240, | ||
| "w": 320, | ||
| "numBins": 120, | ||
| "pixmm": 0.065, | ||
| } |
There was a problem hiding this comment.
Following all other sensors, please make these into configclass objects with the correct doc-strings. We usually don't want to keep sensor-specific settings in the core codebase.
Sensors configuration live here: https://github.com/isaac-sim/IsaacLab/tree/main/source/isaaclab_assets/isaaclab_assets/sensors
There was a problem hiding this comment.
Please choose variable names that are intuitive to read and understand. "h" and "w" are insufficient.
There was a problem hiding this comment.
Thanks for the suggestion. A Config class has been added here and gelsight configs are put under isaaclab_asets/sensors
| tactile_margin: float = 0.003 | ||
| """Margin for tactile point generation (in meters).""" | ||
|
|
||
| contact_object_prim_path_expr: str | None = None |
There was a problem hiding this comment.
Unclear here what contact object means. How is this different from the prim_path that gets set?
There was a problem hiding this comment.
This parameter functions similarly to the filter_prim_paths_expr parameter in ContactSensor, helping pre-configure the SDF query in the tacsl sensor simulation. See the doc here
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
| """ | ||
|
|
||
| # Force field physics parameters | ||
| tactile_kn: float = 1.0 |
There was a problem hiding this comment.
Recommend using more understandable names like normal_contact_stiffness to make it clear from the variable name
There was a problem hiding this comment.
Updated to more understandable names for the three params in here
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor_cfg.py
Outdated
Show resolved
Hide resolved
| camera_cfg: TiledCameraCfg | None = None | ||
| """Camera configuration for tactile RGB/depth sensing. | ||
| If None, camera-based sensing will be disabled even if enable_camera_tactile is True. |
There was a problem hiding this comment.
| If None, camera-based sensing will be disabled even if enable_camera_tactile is True. | |
| If None, camera-based sensing will be disabled even if :attr:`enable_camera_tactile` is True. |
There was a problem hiding this comment.
Improved as suggested.
| sensor_data_dir_name: str = "gelsight_r15_data" | ||
| """Directory name containing the sensor calibration and background data.""" |
There was a problem hiding this comment.
Should explain in docstrings here w.r.t. where this directory is specified. If it is relative vs. absolute.
| Path Resolution: | ||
| - If ``base_data_path`` is ``None`` (default): Downloads from Isaac Lab Nucleus at | ||
| ``{ISAACLAB_NUCLEUS_DIR}/TacSL/{sensor_data_dir_name}/{filename}`` | ||
| - If ``base_data_path`` is provided: Uses custom path at | ||
| ``{base_data_path}/{sensor_data_dir_name}/{filename}`` |
There was a problem hiding this comment.
I think all this might be better described in the docstrings of individual parameters?
| image_height: int = 320 | ||
| """Height of the tactile image in pixels.""" |
There was a problem hiding this comment.
We prefer keeping MISSING for everything so users know form the config instance what params are set. Default valeus are only provided if users are seldom expected to modify them.
There was a problem hiding this comment.
Got it. Updated these params to MISSING in this commit.
| num_bins: int = 120 | ||
| """Number of bins for gradient magnitude and direction quantization.""" | ||
|
|
||
| mm_per_pixel: float = 0.0877 |
There was a problem hiding this comment.
Is this sensor resolution? Is it different from margin?
|
|
||
| VISUO_TACTILE_SENSOR_MARKER_CFG = VisualizationMarkersCfg( | ||
| markers={ | ||
| "debug_pts": sim_utils.SphereCfg( |
There was a problem hiding this comment.
| "debug_pts": sim_utils.SphereCfg( | |
| "taxels": sim_utils.SphereCfg( |
There was a problem hiding this comment.
Updated to "tacsl_pts"
| # Default to Isaac Lab Nucleus directory - download and cache | ||
| nucleus_path = os.path.join(ISAACLAB_NUCLEUS_DIR, "TacSL", data_dir, file_name) | ||
| cache_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), data_dir) |
There was a problem hiding this comment.
Instead of this, I recommend asking users to always provide the full path and not assume any hard coding with "TacSL". You could check if the file is local or not using this function: https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/utils/assets.py#L43
| return cache_path | ||
|
|
||
|
|
||
| def visualize_tactile_shear_image( |
There was a problem hiding this comment.
"visualize" always sounds like this is going to show the image too. Might be better to simply call this "compute"?
There was a problem hiding this comment.
Agree. Changed "visualize " to "compute".
| # Camera configuration | ||
| camera_cfg=TiledCameraCfg( | ||
| prim_path="{ENV_REGEX_NS}/Robot/elastomer_tip/cam", | ||
| update_period=1 / 60, # 60 Hz |
There was a problem hiding this comment.
How is sensor tick here affecting the sensor tick of the gelsight sensor too?
There was a problem hiding this comment.
Thanks for pointing out. The camera's update_period should be consistent with the sensor. Fixed here
There was a problem hiding this comment.
Should this file be called to match GelsightRender? For instance: `visuotactile_render.py'? Is this call very specific to gelsignt or other visual tactile sensors also work?
There was a problem hiding this comment.
Updated to visuotactile_render.py . Thanks!
There was a problem hiding this comment.
Renamed from "gelsight_utils.py" to "visuotactile_render.py"
| from .visuotactile_sensor_cfg import GelSightRenderCfg | ||
|
|
||
|
|
||
| def get_gelsight_render_data(base_data_path: str | None, data_dir: str, file_name: str) -> str | None: |
There was a problem hiding this comment.
Are these functions used outside this module? If not, can we move them to be helpers inside the class so developers don't concern themselves with understanding it?
There was a problem hiding this comment.
Moved to inside the class as suggested in this commit
…sor_cfg.py Co-authored-by: Mayank Mittal <[email protected]> Signed-off-by: Juana <[email protected]>
5 participants
Description
This is an implementation of TacSL integrated with Isaac Lab, which demonstrates how to properly configure and use tactile sensors to obtain realistic sensor outputs including tactile RGB images, force fields, and other relevant tactile measurements.
Fixes # (issue)
Type of change
Screenshots
The screenshots of added documentation and simulation outputs.
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there