## 0. Preface
In this tutorial, we walk through the step-by-step implementation of **keyboard teleoperation and demonstration data collection** for the **AgileX NERO (7-DoF) and PiPER** robotic arms within NVIDIA Isaac Lab.
By the end of this guide, you will be able to control the robotic arm in a simulated environment to perform a block-stacking (Pick and Place) task, collect human demonstration data (HDF5), and replay the dataset for imitation learning pipeline verification.
🚀 **Key Capabilities Supported**
- **Multimodal Teleoperation** : Direct keyboard control for end-effector SE(3) pose alignment.
- **Pick-and-Place (Stacking) Task**: Multi-block stacking simulation environment.
- **Demonstration Collection** : Seamless HDF5 data logging compatible with Robomimic.
- **Trajectory Replay** : Instant validation of collected human demonstrations.
- **Cross-Hardware Support** : Out-of-the-box support for AgileX NERO (7-DoF) and PiPER (6-DoF) arms.
📦 **Project Open-Source Repo**
- **GitHub:** [GitHub - szyzp/IsaacLab\_Data\_Collection · GitHub](https://github.com/szyzp/IsaacLab_Data_Collection.git)
## 1.Environment & Hardware Configuration
### 1.1 Generate the Unified URDF File
To import the NERO arm into Omniverse, we must first merge its separate XACRO description files into a single, clean URDF.
Navigate to your local repository directory:
`source/agx_teleop/agx_description/agx_arm_urdf/nero`
Merge `nero_description.urdf, nero_with_gripper_flange_description.xacro`, and `nero_with_gripper_description.xacro` into a single file named `nero_gripper.urdf`. Ensure all visual/collision mesh paths pointing to local ROS packages are resolved to relative directory paths (e.g., ../meshes/).
Here is the complete `nero_gripper.urdf`:
```xml
```
### 1.2 Compile URDF to USD Format
Convert the generated URDF into an NVIDIA USD (Universal Scene Description) file using Isaac Lab’s built-in exporter utility:
```bash
python ../IsaacLab/scripts/tools/convert_urdf.py
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/urdf/nero_gripper.urdf
/home/agilex/isaac-cosmos/agx_teleop/source/agx_teleop/agx_description/agx_arm_urdf/nero/usd/nero.usd
--fix-base
```
### 1.3 Create the Asset Configuration File
Create assets/nero.py to define the simulated joints, limits, PD controller gains, and stiffness/damping coefficients.
```python
import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets.articulation import ArticulationCfg
from agx_teleop.assets import AGX_TELEOP_DESCRIPTION_DIR
NERO_GRIPPER_CFG = ArticulationCfg(
spawn=sim_utils.UsdFileCfg(
usd_path=f"{AGX_TELEOP_DESCRIPTION_DIR}/agx_arm_urdf/nero/usd/nero.usd",
rigid_props=sim_utils.RigidBodyPropertiesCfg(
disable_gravity=False,
max_depenetration_velocity=5.0,
),
articulation_props=sim_utils.ArticulationRootPropertiesCfg(
enabled_self_collisions=False, solver_position_iteration_count=8, solver_velocity_iteration_count=0
),
),
init_state=ArticulationCfg.InitialStateCfg(
rot=(1.0, 0.0, 0.0, 0.0),
joint_pos={
"joint1": 0.0,
"joint2": 0.0,
"joint3": 0.0,
"joint4": 1.22,
"joint5": 0.0,
"joint6": 0.0,
"joint7": 1.31,
"gripper_joint1": 0.05,
"gripper_joint2": -0.05
},
joint_vel={".*": 0.0},
),
actuators={
"arm": ImplicitActuatorCfg(
joint_names_expr=["joint.*"],
effort_limit=25.0,
velocity_limit=1.5,
stiffness={
"joint1": 200.0,
"joint2": 170.0,
"joint3": 120.0,
"joint4": 80.0,
"joint5": 50.0,
"joint6": 20.0,
"joint7": 10.0
},
damping={
"joint1": 100.0,
"joint2": 60.0,
"joint3": 70.0,
"joint4": 24.0,
"joint5": 20.0,
"joint6": 10.0,
"joint7": 5,
},
),
"gripper": ImplicitActuatorCfg(
joint_names_expr=\["gripper_joint1","gripper_joint2"\],
effort_limit_sim=22,
velocity_limit_sim=1.5,
stiffness=800.0,
damping=20.0,
),
},
soft_joint_pos_limit_factor=0.9,
)
NERO_GRIPPER_HIGH_PD_CFG = NERO_GRIPPER_CFG.copy()
NERO_GRIPPER_HIGH_PD_CFG.spawn.rigid_props.disable_gravity = True
```
## 2.Task Environment Configuration
### 2.1 Configure Directory Tree Structure
Setup your task configurations under the standard Isaac Lab layout:
> Adapted from the official Isaac Lab stacking task implementation: **isaaclab/source/isaaclab\_tasks/isaaclab\_tasks/manager\_based/manipulation/stack**
```bash
cd agx_teleop
mkdir -p source/agx_teleop/agx_teleop/tasks/manager_based/manipulation/stack/config/nero/agents/robomimic
```
Create the required task files:
```bash
cd source/agx_teleop/agx_teleop/tasks/manager_based/
touch manipulation/stack/config/nero/ __init__.py
manipulation/stack/config/nero/agents/ __init__.py
manipulation/stack/config/nero/agents/robomimic/bc_rnn_low_dim.json
manipulation/stack/config/nero/nero_stack_ik_rel_env_cfg.py
manipulation/stack/config/nero/nero_stack_joint_pos_env_cfg.py
```
## 3. Implement Configurations
### 3.1 `bc_rnn_low_dim.json` (Robomimic Behavior Cloning Profile)
This file defines policy network structures (LSTM with Gaussian Mixture Models) for downstream imitation learning. Copy your target JSON config as-is without extra modifications:
```json
{
"algo_name": "bc",
"experiment": {
"name": "bc_rnn_low_dim_franka_stack",
"validate": false,
"logging": { "terminal_output_to_txt": true, "log_tb": true },
"save": {
"enabled": true,
"every_n_seconds": null,
"every_n_epochs": 100,
"epochs": \[\],
"on_best_validation": false,
"on_best_rollout_return": false,
"on_best_rollout_success_rate": true
},
"epoch_every_n_steps": 100,
"env": null,
"additional_envs": null,
"render": false,
"render_video": false,
"rollout": { "enabled": false }
},
"train": {
"data": null,
"num_data_workers": 4,
"hdf5_cache_mode": "all",
"hdf5_use_swmr": true,
"hdf5_normalize_obs": false,
"hdf5_filter_key": null,
"hdf5_validation_filter_key": null,
"seq_length": 10,
"dataset_keys": ["actions"],
"goal_mode": null,
"cuda": true,
"batch_size": 100,
"num_epochs": 2000,
"seed": 101
},
"algo": {
"optim_params": {
"policy": {
"optimizer_type": "adam",
"learning_rate": {
"initial": 0.001,
"decay_factor": 0.1,
"epoch_schedule": [],
"scheduler_type": "multistep"
},
"regularization": { "L2": 0.0 }
}
},
"loss": { "l2_weight": 1.0, "l1_weight": 0.0, "cos_weight": 0.0 },
"actor_layer_dims": [],
"gmm": {
"enabled": true,
"num_modes": 5,
"min_std": 0.0001,
"std_activation": "softplus",
"low_noise_eval": true
},
"rnn": {
"enabled": true,
"horizon": 10,
"hidden_dim": 400,
"rnn_type": "LSTM",
"num_layers": 2,
"open_loop": false,
"kwargs": { "bidirectional": false }
}
},
"observation": {
"modalities": {
"obs": {
"low_dim": ["eef_pos", "eef_quat", "gripper_pos", "object"],
"rgb": [], "depth": [], "scan": []
}
}
}
}
```
### 3.2 `nero_stack_joint_pos_env_cfg.py` (Joint Position Environment)
Sets up state configurations, randomizing object spawn spaces and assigning physical bounds to NERO.
```python
from isaaclab.assets import RigidObjectCfg
from isaaclab.managers import EventTermCfg as EventTerm
from isaaclab.managers import SceneEntityCfg
from isaaclab.sensors import FrameTransformerCfg
from isaaclab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from isaaclab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from agx_teleop.tasks.manager_based.manipulation.stack import mdp
from agx_teleop.tasks.manager_based.manipulation.stack.mdp import piper_stack_events
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import StackEnvCfg
from isaaclab.markers.config import FRAME_MARKER_CFG
from agx_teleop.assets.nero import NERO_GRIPPER_CFG
@configclass
class EventCfg:
init_nero_arm_pose = EventTerm(
func=piper_stack_events.set_default_joint_pose,
mode="reset",
params={
"default_pose": [0.0, 0.0, 0.0, 1.22, 0.0, 0.0, 1.31, 0.05, -0.05],
},
)
randomize_nero_joint_state = EventTerm(
func=piper_stack_events.randomize_joint_by_gaussian_offset,
mode="reset",
params={
"mean": 0.0,
"std": 0.02,
"asset_cfg": SceneEntityCfg("robot"),
},
)
randomize_cube_positions = EventTerm(
func=piper_stack_events.randomize_object_pose,
mode="reset",
params={
"pose_range": {"x": (0.3, 0.5), "y": (-0.10, 0.10), "z": (0.0203, 0.0203), "yaw": (-1.0, 1, 0)},
"min_separation": 0.12,
"asset_cfgs": [SceneEntityCfg("cube_1"), SceneEntityCfg("cube_2"), SceneEntityCfg("cube_3")],
},
)
@configclass
class NeroCubeStackEnvCfg(StackEnvCfg):
def __post_init__ (self):
super(). __post_init__ ()
self.events = EventCfg()
self.scene.robot = NERO_GRIPPER_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
self.scene.robot.spawn.semantic_tags = [("class", "robot")]
self.scene.table.spawn.semantic_tags = [("class", "table")]
self.scene.plane.semantic_tags = [("class", "ground")]
self.actions.arm_action = mdp.JointPositionActionCfg(
asset_name="robot", joint_names=["joint.*"], scale=0.5, use_default_offset=True
)
self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
asset_name="robot",
joint_names=["gripper_joint.*"],
open_command_expr={"gripper_joint1": 0.05, "gripper_joint2": -0.05},
close_command_expr={"gripper_joint1": 0.0, "gripper_joint2": 0.0},
)
self.gripper_joint_names = ["gripper_joint.*"]
self.gripper_open_val = [0.05, -0.05]
self.gripper_threshold = 0.005
cube_properties = RigidBodyPropertiesCfg(
solver_position_iteration_count=16,
solver_velocity_iteration_count=1,
max_angular_velocity=1000.0,
max_linear_velocity=1000.0,
max_depenetration_velocity=5.0,
disable_gravity=False,
)
self.scene.cube_1 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_1",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0.0, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/blue_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_1")],
),
)
self.scene.cube_2 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_2",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.55, 0.05, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/red_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_2")],
),
)
self.scene.cube_3 = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube_3",
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.60, -0.1, 0.0203], rot=[1, 0, 0, 0]),
spawn=UsdFileCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/green_block.usd",
scale=(1.0, 1.0, 1.0),
rigid_props=cube_properties,
semantic_tags=[("class", "cube_3")],
),
)
marker_cfg = FRAME_MARKER_CFG.copy()
marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
marker_cfg.prim_path = "/Visuals/FrameTransformer"
self.scene.ee_frame = FrameTransformerCfg(
prim_path="{ENV_REGEX_NS}/Robot/world",
debug_vis=False,
visualizer_cfg=marker_cfg,
target_frames=[
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_base",
name="end_effector",
offset=OffsetCfg(pos=[0.0, 0.0, 0.125\]),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link1",
name="tool_rightfinger",
offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),
),
FrameTransformerCfg.FrameCfg(
prim_path="{ENV_REGEX_NS}/Robot/gripper_link2",
name="tool_leftfinger",
offset=OffsetCfg(pos=(0.0, 0.0, 0.0)),
),
],
)
```
### 3.3 Teleoperation, Data Collection & Replay
#### 3.3.1 stack/config/nero/nero\_stack\_ik\_rel\_env\_cfg.py
Copy the corresponding Isaac Lab code and modify the following:
- Update imports and naming
- Modify the end-effector offset
Complete code for `nero_stack_ik_rel_env_cfg.py`:
```python
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from isaaclab.controllers.differential_ik_cfg import DifferentialIKControllerCfg
from isaaclab.devices.device_base import DeviceBase, DevicesCfg
from isaaclab.devices.keyboard import Se3KeyboardCfg
from isaaclab.devices.openxr.openxr_device import OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters.manipulator.gripper_retargeter import GripperRetargeterCfg
from isaaclab.devices.openxr.retargeters.manipulator.se3_rel_retargeter import Se3RelRetargeterCfg
from isaaclab.envs.mdp.actions.actions_cfg import DifferentialInverseKinematicsActionCfg
from isaaclab.managers import SceneEntityCfg
from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.utils import configclass
from agx_teleop.tasks.manager_based.manipulation.stack.stack_env_cfg import mdp
from . import nero_stack_joint_pos_env_cfg
##
# Pre-defined configs
##
from agx_teleop.assets.nero import NERO_GRIPPER_HIGH_PD_CFG # isort: skip
@configclass
class NeroCubeStackEnvCfg(nero_stack_joint_pos_env_cfg.NeroCubeStackEnvCfg):
def __post_init__ (self):
# post init of parent
super(). __post_init__ ()
# Set Nero as robot
# We switch here to a stiffer PD controller for IK tracking to be better.
self.scene.robot = NERO_GRIPPER_HIGH_PD_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
# Set actions for the specific robot type (nero)
self.actions.arm_action = DifferentialInverseKinematicsActionCfg(
asset_name="robot",
joint_names=["joint.*"],
body_name="gripper_base",
controller=DifferentialIKControllerCfg(command_type="pose", use_relative_mode=True, ik_method="dls"),
scale=0.5,
body_offset=DifferentialInverseKinematicsActionCfg.OffsetCfg(pos=[0.0, 0.0, 0.125]),
)
self.teleop_devices = DevicesCfg(
devices={
"handtracking": OpenXRDeviceCfg(
retargeters=[
Se3RelRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT,
zero_out_xy_rotation=True,
use_wrist_rotation=False,
use_wrist_position=True,
delta_pos_scale_factor=10.0,
delta_rot_scale_factor=10.0,
sim_device=self.sim.device,
),
GripperRetargeterCfg(
bound_hand=DeviceBase.TrackingTarget.HAND_RIGHT, sim_device=self.sim.device
),
],
sim_device=self.sim.device,
xr_cfg=self.xr,
),
"keyboard": Se3KeyboardCfg(
pos_sensitivity=0.05,
rot_sensitivity=0.05,
sim_device=self.sim.device,
),
}
)
@configclass
class NeroCubeStackRedGreenEnvCfg(NeroCubeStackEnvCfg):
def __post_init__ (self):
\# post init of parent
super(). __post_init__ ()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_2"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class NeroCubeStackRedGreenBlueEnvCfg(NeroCubeStackEnvCfg):
def __post_init__ (self):
# post init of parent
super(). __post_init__ ()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_2"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_1"),
},
)
@configclass
class NeroCubeStackBlueGreenEnvCfg(NeroCubeStackEnvCfg):
def __post_init__ (self):
# post init of parent
super(). __post_init__ ()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={"cube_1_cfg": SceneEntityCfg("cube_1"), "cube_2_cfg": SceneEntityCfg("cube_3"), "cube_3_cfg": None},
)
@configclass
class NeroCubeStackBlueGreenRedEnvCfg(NeroCubeStackEnvCfg):
def __post_init__ (self):
# post init of parent
super(). __post_init__ ()
self.terminations.success = DoneTerm(
func=mdp.cubes_stacked,
params={
"cube_1_cfg": SceneEntityCfg("cube_1"),
"cube_2_cfg": SceneEntityCfg("cube_3"),
"cube_3_cfg": SceneEntityCfg("cube_2"),
},
)
```
#### 3.3.2 stack/config/nero/ **init**.py\*\*
```python
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause
import gymnasium as gym
from isaaclab_tasks.manager_based.manipulation.stack.config.franka import agents
from agx_teleop.tasks.manager_based.manipulation.stack.config.nero import agents
\##
# Register Gym environments.
##
gym.register(
id="Isaac-Stack-Cube-Nero-IK-Rel-v0",
entry_point="isaaclab.envs:ManagerBasedRLEnv",
kwargs={
"env_cfg_entry_point": f"{ __name__ }.nero_stack_ik_rel_env_cfg:NeroCubeStackEnvCfg",
"robomimic_bc_cfg_entry_point": f"{agents. __name__ }:robomimic/bc_rnn_low_dim.json",
},
disable_env_checker=True,
)
```
## 4. Collect Demonstration Data via Teleoperation
```bash
# First, create a dataset directory
mkdir -p datasets
# Record demonstrations via teleoperation.
# This example uses keyboard teleoperation.
# Available devices: spacemouse, keyboard, handtracking
python scripts/tools/record_demos.py
--task Isaac-Stack-Cube-Nero-IK-Rel-v0
--device cpu --teleop_device keyboard
--dataset_file ./datasets/nero_dataset.hdf5
--num_demos 10
# Replay the recorded demonstrations
python scripts/tools/replay_demos.py
--task Isaac-Stack-Cube-Nero-IK-Rel-v0
--device cpu --dataset_file ./datasets/nero_dataset.hdf5
```
Keyboard teleoperation controls:
```plaintext
Keyboard Controller for SE(3): Se3Keyboard
Reset all commands: R
Toggle gripper (open/close): K
Move arm along x-axis: W/S
Move arm along y-axis: A/D
Move arm along z-axis: Q/E
Rotate arm along x-axis: Z/X
Rotate arm along y-axis: T/G
Rotate arm along z-axis: C/V
```
## FAQ
**Q1: How do I adapt this pipeline from the 7-DoF NERO arm to the 6-DoF PiPER arm (or any other custom manipulator)?**
To swap the robot asset, you need to update three core configuration files in your task directory:
1.**Asset Definition (assets/piper.py)**: Define the new USD file path, update the default joint positions (joint\_pos) to match a 6-DoF configuration, and re-tune the stiffness/damping parameters for the 6 active joints.
2.**Action Configuration (piper\_stack\_joint\_pos\_env\_cfg.py)**: Modify the joint regular expression to match the PiPER joint names (e.g., changing expression targets from 7 joints to 6). Update the gripper joint names and binary open/close limit values.
3.**Differential IK Config (piper\_stack\_ik\_rel\_env\_cfg.py)**:Update the body\_name parameter to point to the correct end-effector link (e.g., link6 or gripper\_base) and adjust the body\_offset to match the physical dimension of the PiPER gripper flange.
* * *
**Q2: The robotic gripper slips or fails to lift the blocks during teleoperation. How do I improve grap stability?**
Slipping or unstable grasping in Isaac Lab is usually a physics-tuning issue rather than an algorithmic one. Try the following adjustments:
- **Increase Gripper Stiffness** : In your asset configuration (assets/nero.py or piper.py), increase the gripper actuator stiffness (e.g., to 800.0 or 1000.0) and effort\_limit to ensure the gripper applies enough physical force.
- **Verify Collision Geometries** : Open your converted .usd asset in Isaac Sim and check that the collision meshes for the gripper fingers are set to “Convex Decomposition” (or Simplfied Box) rather than a loose bounding box.
- **Adjust Material Friction** : Add a high-friction physics material (e.g., static friction coefficient.
* * *
**Q3: Are the collected HDF5 datasets directly compatible with popular imitation learning codebases like ACT or Diffusion Policy?**
Yes, with minor formatting adjustments. The HDF5 files generated by our record\_demos.py script closely follow the standard Robomimic data structure, containing hierarchically organized datasets:
- data/demo\_x/obs/eef\_pos & eef\_quat (End-effector pose)
- data/demo\_x/obs/joint\_values (Joint angles)
- data/demo\_x/actions (User control inputs)
To feed this data into ACT (Action Chunking with Transformers) or Diffusion Policy, you can use Robomimic’s conversion scripts (convert\_robomimic\_to\_numpy.py) or write a simple Python wrapper to map the HDF5 dictionary keys to the observation/action expectations of your specific network.
* * *
**Q4: The simulation occasionally crashes or the robot “explodes” (joints jitter violently) when starting teleoperation. How do I fix this?**
“Exploding” physics usually happen when the control gains (PD) are too aggressive, or the solver parameters are too coarse.
- Increase Solver Iterations: In your environment configuration, increase the solver iterations on the robot asset:
```python
articulation_props=sim_utils.ArticulationRootPropertiesCfg(
solver_position_iteration_count=16, # Increase from 8 to 16
solver_velocity_iteration_count=4 # Increase from 0 to 4
)
```
- Disable Self-Collisions: Ensure that adjacent links on the arm do not have overlapping collision meshes by setting enabled\_self\_collisions=False.
- Lower Stiffness: If the arm vibrates violently upon contact, slightly lower the joint stiffness in your actuator configuration and add proportional damping (ideal ratio is roughly 10:1 to 20:1 for stiffness to damping).
