0.Introduction
In our previous article, we introduced a complete teleoperation and demonstration collection pipeline for the AgileX NERO and PiPER robotic arms in Isaac Lab.
The project demonstrated how to:
-
Import custom robot assets into Isaac Lab
-
Configure Differential IK control
-
Perform keyboard-based teleoperation
-
Record demonstrations into HDF5 datasets
-
Replay trajectories for verification
Previous article:
Isaac Lab Teleoperation & Data Collection: Controlling the AgileX NERO 7-DoF Robotic Arm
Repository: The full source code is available in the repository.
While teleoperation is an effective approach for collecting expert demonstrations, scaling datasets through manual operation quickly becomes time-consuming and difficult to maintain consistently.
In this follow-up work, we extend the original pipeline with an automated block stacking data generation framework that can autonomously complete manipulation tasks and continuously export successful demonstrations.
What’s New in this Tutorial ?
Compared with the previous teleoperation workflow, this version introduces:
Automated trajectory generation
Autonomous block stacking execution
Differential IK-based task completion
PCHIP trajectory smoothing
Automatic success detection
Continuous dataset generation
Zero human intervention during collection
The result is a scalable demonstration generation pipeline suitable for robot learning workflows.
1.Systerm Architecture
The overall workflow is shown below:
Cube Position Detection
↓
Trajectory Planner
↓
IK Stack Controller
↓
Differential IK (Isaac Lab)
↓
Robot Motion
↓
Recorder Manager
↓
HDF5 Dataset Export
Unlike the teleoperation workflow, no human operator is involved once the simulation starts.
The controller generates Cartesian target motions, while Isaac Lab’s built-in Differential IK module handles inverse kinematics computation internally.
2.Implementation Steps
Step 1.Automated Block Stacking Controller
A new script was introduced:
scripts/tools/record_ik_stack.py
The controller executes a complete stacking sequence:
Approach Cube
↓
Descend
↓
Grasp
↓
Lift
↓
Move To Target
↓
Place
↓
Retreat
Instead of commanding joint angles directly, the controller outputs Cartesian pose increments:
action = [ Δx,Δy,Δz,Δrx,Δry,Δrz,gripper ]
This design allows the same controller logic to be reused across different robot configurations.
The newly introduced block stacking controller is shown below.
sequence = [
(self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
(self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
]
Code shown below to realize this step
def plan(self) -> None:
"""plan the full pick-and-place trajectory."""
cube_height = self.cubes[0].data.root_pos_w[:, 2:3].clone(2.0
all_waypoints: list[torch.Tensor] = []
all_gripper_vals: list[float] = []
all_durations: list[int] = []
base_pos = self.cubes[0].data.root_pos_w.clone()
for i in range(1, len(self.cubes)):
src = self.cubes[i].data.root_pos_w.clone()
place_pos = self._shift_z(base_pos, cube_height)
sequence = [
(self._shift_z(src, self.Z_PRE), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_OPEN_VAL, 30),
(self._shift_z(src, self.Z_DOWN), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(src, self.Z_LIFT), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PRE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_CLOSE_VAL, 30),
(self._shift_z(place_pos, self.Z_PLACE), self.GRIP_OPEN_VAL, 20),
(self._shift_z(place_pos, self.Z_RETREAT), self.GRIP_OPEN_VAL, 30),
]
for wp, gv, dur in sequence:
all_waypoints.append(wp.squeeze(0))
all_gripper_vals.append(gv)
all_durations.append(dur)
all_durations[-1] += 30
base_pos = self._shift_z(base_pos, cube_height)
# Generate smooth trajectory
self._path = self._generate_smooth_trajectory(all_waypoinall_durations)
# Expand gripper timeline
self._gripper_timeline = []
for gv, dur in zip(all_gripper_vals, all_durations):
self._gripper_timeline.extend([gv] * dur)
self._total_steps = self._path.shape[0]
self._gripper_timeline = self._gripper_timeline[:s_total_steps]
self._step_idx = 0
self._post_traj_wait_count = 0
self._last_gripper_val = self.GRIP_OPEN_VAL
print(f"[IKStackController] Trajectory planned: {s_total_steps} steps")
Step 2.Differential IK-Based Control
Rather than solving inverse kinematics explicitly, the controller outputs Cartesian pose increments and delegates IK computation to Isaac Lab’s built-in Differential IK controller.
def compute_action(self, env: gym.Env) -> torch.Tensor:
"""Return the action for the current step.
Returns:
torch.Tensor of shape ``(1, 7)``:
``[Δx, Δy, Δz, Δrx, Δry, Δrz, gripper_binary]``
"""
# trajectory exhausted → hold position
if self._step_idx >= self._total_steps:
return torch.tensor(
[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, self_last_gripper_val]],
device=env.device,
)
# delta position = target − current
target_pos = self._path[self._step_idx] # [1, 3]
current_pos = self.ee_frame.data.target_pos_w[:, 0, :] # [1,3]
delta_pos = target_pos - current_pos # [1, 3]
# delta rotation = zero (keep orientation)
delta_rot = torch.zeros(1, 3, device=env.device)
# gripper binary command
gv = self._gripper_timeline[self._step_idx]
grip_tensor = torch.tensor([[gv]], device=env.device)
action = torch.cat([delta_pos, delta_rot, grip_tensor],dim=-1) # (1, 7)
self._step_idx += 1
self._last_gripper_val = gv
return action
Step 3.Trajectory Smoothing with PCHIP
One challenge during autonomous execution is trajectory discontinuity between waypoints.
To improve motion quality, the controller uses Hermite-based interpolation with PCHIP-style tangents to generate smooth Cartesian trajectories.
@staticmethod
def _generate_smooth_trajectory(
waypoints: list[torch.Tensor], durations: list[int]
) -> torch.Tensor:
"""Interpolate waypoints with PCHIP-style tangents.
Args:
waypoints: list of ``(3,)`` tensors.
durations: steps per segment (``len == len(waypoints) -1``).
Returns:
``torch.Tensor`` of shape ``(Total, 1, 3)``.
"""
n = len(waypoints)
wp = torch.stack(waypoints) # [N, 3]
tangents = torch.zeros_like(wp)
if n > 1:
tangents[0] = wp[1] - wp[0]
tangents[-1] = wp[-1] - wp[-2]
for i in range(1, n - 1):
tangents[i] = (wp[i + 1] - wp[i - 1]) / 2.0
segments = []
for i in range(n - 1):
p0, p1 = wp[i], wp[i + 1]
m0, m1 = tangents[i], tangents[i + 1]
steps = durations[i]
if torch.norm(p1 - p0) < 1e-4:
seg = p0.unsqueeze(0).repeat(steps, 1)
else:
s = torch.linspace(0, 1, steps, device=p0.device)unsqueeze(-1)
h00 = 2 * s**3 - 3 * s**2 + 1
h10 = s**3 - 2 * s**2 + s
h01 = -2 * s**3 + 3 * s**2
h11 = s**3 - s**2
seg = h00 * p0 + h10 * m0 + h01 * p1 + h11 * m1
segments.append(seg)
return torch.cat(segments, dim=0).unsqueeze(1) # [Total, 1, 3]
Step 4.Automatic Success Verification
To ensure dataset quality, demonstrations are exported only after the stacking task is successfully completed.
The workflow continuously evaluates the stacking success condition:
Failed trajectories are automatically discarded.
def process_success_condition(env: gym.Env, success_term: object | None, success_step_count: int) -> tuple[int, bool]:
if success_term is None:
return success_step_count, False
if bool(success_term.func(env, **success_term.params)[0]):
success_step_count += 1
if success_step_count >= args_cli.num_success_steps:
env.recorder_manager.record_pre_reset([0], force_export_or_skip=False)
env.recorder_manager.set_success_to_episodes(
[0], torch.tensor([[True]], dtype=torch.bool, device=env.device)
)
env.recorder_manager.export_episodes([0])
print("Success condition met! Recording completed.")
return success_step_count, True
else:
success_step_count = 0
return success_step_count, False
Step 5.Dataset Export
Successful demonstrations are exported automatically using Isaac Lab’s Recorder Manager.
Output format:
ik_dataset.hdf5
The exported datasets contain:
-
Observations
-
Actions
-
Episode Metadata
-
Success Labels
and can be directly integrated into:
-
Robomimic
-
LeRobot
-
ACT
-
Diffusion Policy
-
OpenVLA
with minimal conversion effort.
Step6. Running the Pipeline
- PiPER
python scripts/tools/record_ik_stack.py \
--task Isaac-Stack-Cube-Piper-IK-Rel-v0 \
--device cuda \
--dataset_file ./datasets/ik_dataset.hdf5 \
--num_demos 10
- NERO
python scripts/tools/record_ik_stack.py \
--task Isaac-Stack-Cube-Nero-IK-Rel-v0 \
--device cuda \
--dataset_file ./datasets/ik_dataset.hdf5 \
--num_demos 10
3.Why This Matters
This work moves the Isaac Lab workflow from:
Human Demonstration Collection
towards
Automated Demonstration Generation
which is an important step for building large-scale robot learning datasets.
Rather than spending hours manually collecting demonstrations, developers can now generate consistent trajectories automatically and focus on downstream policy training.
FAQ
Q1: Is this solution fully compatible with all Sogot desktop robotic arms?
This solution natively supports the two mainstream Sogot desktop robotic arm models: PiPER and NERO. It features dedicated optimizations for joint limits, movement speed, gripper parameters, and workspace of these two models. You can switch between models directly via corresponding task commands. It is not compatible with other arm models out of the box, but compatibility can be achieved by fine-tuning motion offset parameters and trajectory step lengths.
Q2: How to resolve environment mismatch errors during runtime?
First, confirm you have deployed the IsaacLab base environment with CUDA support, and verify matching versions for PyTorch, gymnasium, and Pinocchio dependencies.
Next, ensure the task parameter in your run command matches your Sogot arm model — PiPER and NERO task commands cannot be interchanged.
Finally, check that GPU acceleration is enabled. This project requires CUDA acceleration for robotic arm inverse kinematics (IK) solving and trajectory computation; running on CPU will throw runtime errors immediately.
Q3: Can the collected datasets be directly used for training physical Sogot robotic arms?
Yes. The datasets output by this solution replicate the physical characteristics of real Sogot arms across motion workspace, control logic, and movement trajectories. Simulation data aligns closely with real hardware control logic, so no extra format conversion is required. The datasets can be directly applied to imitation learning training for physical PiPER and NERO arms, effectively cutting the high cost and operational risks of real-world data sampling.
Q4: How to customize data sample volume and sampling frequency?
Flexible configuration is available via runtime command arguments:
-
num_demos: Sets the total number of successful demonstration samples to collect; set to 0 for infinite loop sampling.
-
step_hz: Adjusts the simulation sampling frequency (default: 30 Hz). Tune this value based on your Sogot arm’s movement speed to match different operation paces.
Q5: What causes stuttering trajectories or stacking failures on the robotic arm?
This issue most often stems from two root causes:
CUDA acceleration is not active, leading to delayed IK solving and choppy trajectory calculations.
Mismatch between simulation environment frame rate and robotic arm motion step length.
Troubleshooting steps: First check your GPU operational status. Then fine-tune PCHIP trajectory interpolation step length and Z-axis offset parameters to meet precise operation requirements for Sogot robotic arms.
Have Question?
If you encounter any issues with environment installation, parameter configuration, or RL training, feel free to leave your questions for further discussion.