0. Preface
Pika Sense provides an intuitive teleoperation interface for robotic manipulation and data collection. Combined with Pika Station, Pika Gripper, and the AgileX NERO robotic arm, it enables low-latency end-effector teleoperation for both single-arm and dual-arm systems.
This tutorial walks through the complete setup process, and serves as a reference for developers who want to integrate Pika Sense with their own robotic arms.
warning: Pika Sense currently supports:
- AgileX PiPER
- AgileX PiPER-X
- AgileX NERO
- xArm Lite 6
Support for additional robotic arms can be added by implementing the corresponding teleoperation interface. We welcome community contributions. If you successfully adapt another robotic arm, feel free to submit a Pull Request to our GitHub repository.
1.Hardware Preparation
This section describes the hardware setup required for teleoperating a single AgileX NERO robotic arm with its default NERO gripper.
- When assembling the robotic arm, align the red alignment marks on the arm connectors with the corresponding red marks on the cables.
- The textured sleeve on the aviation connector is the locking mechanism. During installation, align the red mark downward with the locating notch and push the connector straight in. To disconnect it, press the textured sleeve and then pull the connector out.
- If you are using a third-party robotic arm, you can follow the same hardware connection procedure described for the NERO arm.
- For third-party robotic arms, you must additionally verify that the power supply, communication interfaces, end-effector interfaces, and control protocols meet the requirements for teleoperation.
Step 1 – Connect the NERO Robotic Arm
Connect the NERO robotic arm as shown in the wiring diagram below. For detailed wiring instructions, please refer to the NERO User Manual.
When powering on the robot for the first time, complete the following steps in order (A–G):
- Connect connector A to the J2 port.
- Connect the CAN cable of aviation connector B.
- Connect the XT30 connector C.
- Align the red dots on aviation connector D, then plug it in with the red dot facing downward.
- Connect the plug of power adapter E.
- Verify that the AC power cord of power adapter E is properly connected, then power on the system. Wait until the indicator LED on the control panel starts flashing green.
- Connect the USB cable to your computer to begin using the device.
Step 2 - Connect Pika Sense
Next, connect the Pika Sense device.
For detailed information about the Pika Sense interfaces and wiring, refer to Section 1.1 of the Pika Positioner & Sense User Guide (Positioning and Calibration).
2.SoftWare Preparation
Step 1.Install System Dependencies
Before running the teleoperation software, install the required system packages:
sudo apt update && sudo apt install ethtool
sudo apt update && sudo apt install can-utils
Step 2.Install Conda
Install either Miniconda or Anaconda before proceeding.
If Conda is already installed, you can skip this step.
Step 3.Create the Pika Teleoperation Python Environment
Create a dedicated Conda environment for Pika teleoperation:
conda create -n pika python=3.10
conda activate pika
conda install pinocchio==3.2.0 casadi==3.6.7 -c conda-forge
pip install lark numpy==2.0.2 empy==3.3.4 meshcat pyyaml piper-sdk opencv-python ur-rtde netifaces catkin_pkg
3.Starting Teleoperation
3.1 Single-Arm Teleoperation
Step 1 – Activate the CAN Interface
Connect the robotic arm’s CAN cable to your computer, then activate the CAN interface by running:
cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts
bash can_activate.sh
Step 2 – Calibrate Pika Sense
Before starting teleoperation, calibrate the Pika Sense device.
For detailed calibration instructions, please refer to Sections 1.2 and 1.3 of the Pika Positioner & Sense User Guide (Positioning and Calibration).
Step 3 – Launch the Teleoperation Program
Choose the appropriate launch procedure according to your hardware configuration.
Option A – Using Pika Sense without the Pika Gripper
If you are not using a Pika Gripper, launch the following programs.
Terminal 1
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts
bash start_single_sensor_whit_teleop.bash
Terminal 2
source ~/pika_ros/install/setup.bash
conda activate pika
# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_single_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_single_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_single_nero.launch.py
After both terminals are running, double click the Pika Sense trigger to enable teleoperation.
Option B - Using Pika Sense with Pika Gripper
If a Pika Gripper is mounted on the robotic arm and controlled by Pika Sense, first bind one Pika Sense with one Pika Gripper by following the instructions in the Pika Positioner & Sense User Guide (Positioning and Calibration).
Terminal 1
conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts
bash start_sensor_gripper.bash
Terminal 2
source ~/pika_ros/install/setup.bash
conda activate pika
# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_single_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_single_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_single_nero.launch.py
Once the system is running, double click the Pika Sense trigger to start teleoperation.
3.2 Dual-Arm Teleoperation
Step 1 – Configure the CAN Interfaces
Before starting dual-arm teleoperation, both robotic arms must be assigned to the correct CAN interfaces.
First, connect the left robotic arm to your computer via the CAN adapter, then run:
cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts
bash find_all_can_port.sh
The terminal will display the USB port corresponding to the left arm.
Next, connect the right robotic arm and run the script again:
bash find_all_can_port.sh
The terminal will display the USB port corresponding to the right arm.
Open the can_config.sh file and copy the two detected USB port IDs into Lines 111 and 112, respectively.
For example, when using PiPER, the configuration should look like:
if [ "$EXPECTED_CAN_COUNT" -ne 1 ]; then
declare -A USB_PORTS
USB_PORTS["1-8.1:1.0"]="can_left:1000000" # Left arm
USB_PORTS["1-8.2:1.0"]="can_right:1000000" # Right arm
fi
After saving the configuration, activate both CAN interfaces:
cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts
bash can_config.sh
Step 2 – Calibrate Pika Sense and Bind Pika Devices
Calibrate the Pika devices before starting teleoperation. For detailed instructions, refer to the Pika User Manual, including the the Pika Positioner & Sense User Guide (Positioning and Calibration) section for base station deployment and Pika Sense calibration, followed by the Pika Device Binding section to bind the left and right Pika Sense devices.
Step 3 – Launch the Dual-Arm Teleoperation Program
Choose the launch procedure according to your hardware configuration.
Option A – Using Pika Sense without the Pika Gripper
If you are not using Pika Grippers, or you are using the robot’s original grippers, first bind two Pika Sense devices following the Pika Device Binding Guide.
Terminal 1
conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts && bash start_multi_sensor_whit_teleop.bash sensor
Terminal 2
source ~/pika_ros/install/setup.bash
conda activate pika
# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_double_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_double_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_double_nero.launch.py
Finally, double click the right Pika Sense trigger to start dual-arm teleoperation.
Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.
Option B – Using Pika Grippers
If Pika Grippers are installed on both robotic arms, first bind:
- Two Pika Sense devices
- Two Pika Grippers
using the Pika Device Binding section section 2.6, after that:
Terminal 1 —— Start Pika Sense
conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts
bash start_multi_sensor_whit_teleop.bash
Terminal 2 —— Start Pika Grippers
conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts && bash start_multi_gripper.bash #pika gripper
Terminal 3 —— Launch the Robot
source ~/pika_ros/install/setup.bash
conda activate pika
# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_double_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_double_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_double_nero.launch.py
Finally, double click the right Pika Sense trigger to start teleoperation.
Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.
4.Configuration Files
The configuration files for each supported robot are organized as follows:
config/
├── piper_rand_params.yaml
│ └── gripper_xyzrpy
│ Offset from Joint 6 to the gripper frame (m, rad)
│
├── xarm_params.yaml
│ ├── eff_position
│ │ Initial TCP pose (mm, rad)
│ └── pika_to_arm
│ Pika coordinate system → Robotic arm end-effector coordinate system (m, rad)
│
└── ur12e_params.yaml
├── pika_to_arm
│ Pika coordinate system → Robotic arm end-effector coordinate system (m, rad)
└── robot_ip
IP address of UR arm
5. Coordinate Frames and ROS Topics
To help developers integrate Pika Sense with their own robotic arms, this section describes:
- The Pika gripper coordinate frame
- The
/pika_poseROS topics
5.1 Pika Coordinate Frame
The Pika coordinate frame is defined at the center of the Pika gripper and is published through the /pika_pose topic.
When teleoperation is activated by double-clicking the Pika Sense gripper, the current pose of the Pika Sense is recorded as the reference (zero) pose. All subsequent position and orientation commands are expressed relative to this reference frame.
The coordinate axes of the /pika_pose frame are defined as follows:
- X-axis: Forward
- Y-axis: Left
- Z-axis: Up
5.2 ROS Topics
For single-arm teleoperation, Pika Sense publishes the robot target pose to: /pika_pose For dual-arm teleoperation, the left and right Pika Sense devices publish to separate topics: /pika_pose_l /pika_pose_r All of these topics use the standard ROS message type: geometry_msgs/PoseStamped Since most industrial robot controllers provide Cartesian end-effector control interfaces based on geometry_msgs/PoseStamped, the published pose messages can typically be used directly or converted into robot-specific motion commands with minimal adaptation.
For a complete implementation example, see:
FAQ
Q1 Which robotic arms are currently supported by Pika Sense?
Pika Sense currently supports teleoperation for the following robotic arms:
- AgileX PiPER
- AgileX PiPER-X
- AgileX NERO
- xArm Lite 6
Support for additional third-party robotic arms can be implemented by developers or users based on the robot’s control interface.
Q2. Can I use Pika Sense with a third-party robotic arm?
Yes.
To integrate a third-party robotic arm, the robot should support Cartesian end-effector pose control, or provide an interface that can convert the geometry_msgs/PoseStamped messages published by Pika Sense into robot-specific motion commands.
Q3.Why do I need to double-click the Pika Sense gripper before starting teleoperation?
Double-clicking initializes the teleoperation reference pose.
The current pose of the Pika Sense is recorded as the zero (reference) pose, and all subsequent position and orientation commands are calculated relative to this reference frame.
Q4.Is there anything I should pay attention to when double-clicking the Pika Sense gripper?
Yes.
Before double-clicking, ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector.
Doing so minimizes the initial pose offset and helps ensure that the robot moves intuitively in the same direction as the Pika Sense.
Q5. Which parameters should I pay attention to when integrating a third-party robotic arm?
The most important parameters are:
- The coordinate frame of the robot end-effector (TCP)
- The transformation between the Pika gripper center frame and the robot TCP frame
- The robot controller’s required pose or motion command format
In most cases, the pika_to_arm parameter in the configuration file defines the transformation between the Pika coordinate frame and the robot TCP, making it one of the key parameters to adjust when integrating a new robotic arm.