0.Introduction
This project demonstrates a vision-based smart ergonomic workstation built with an AgileX PiPER 6-DoF robotic arm, Orbbec RGB-D camera, and ROS 2.
Using YOLOv5-Face and depth perception, the system detects the user’s 3D position and controls the robotic arm to adjust the display orientation automatically.
Safety Considerations
- This prototype does not include certified safety functions, such as collision detection, human safety monitoring, speed monitoring, or emergency stop control.
- Keep all people away from the robot’s workspace during operation. Do not place any part of the body within the arm’s reachable or moving area.
- Perform real-world tests only with proper safety measures, including physical barriers, accessible emergency stops, and human supervision.
- For initial testing, use simulation, ROS 2 TF visualization, or non-human targets instead of tracking a person directly.
The overall perception-to-control workflow is illustrated below:
Orbbec RGB-D Camera
↓
YOLOv5-Face Detection & Landmark Extraction
↓
3D Face Position Estimation Using Depth Data
↓
Publish Face TF Frame
↓
Calculate Target Monitor Pose (arm_pose)
↓
Generate Smoothed Motion Commands
↓
Publish /pos_cmd
↓
PiPER Robotic Arm Moves the Display
Related Projects and Resources:
- Original project repository:
GitHub - lansijian/piper-x-smart-workstation · GitHub - Code repository organized for this article:
GitHub - smalleha/Adx_PiPER_Intelligent_workstation · GitHub - Demo video:
https://www.bilibili.com/video/BV1eZ336pEAr?t=1.4
1.How the Smart Workstation Works
1.1.Project Scope and Related Work
Using a robotic arm to control a display and adjust its viewing angle based on user position is not an entirely new research direction. For example, HoloArm has explored a similar concept by demonstrating a robot arm-driven display system that follows human faces.
The main contribution of this project is not the invention of a new interaction concept, but the implementation of a reproducible engineering pipeline integrating PiPER robotic arm, Orbbec RGB-D camera, YOLOv5-Face, and ROS 2 Humble.
Related Work:
HoloArm: A Face-Following 3D Display Using Autostereoscopic Display and Robot Arm
1.2.Development Environment
Software
- Ubuntu 22.04
- ROS 2 Humble
- CUDA 12.8
- Python 3.10
- Conda virtual environment
Hardware
- Orbbec RGB-D Camera
- AgileX PiPER Robotic Arm
2.Setup and Installation
2.1.Create ROS 2 Workspace
Create a ROS 2 workspace and download the required packages:
mkdir -p ~/agilex_ws/src
cd ~/agilex_ws/src
git clone https://github.com/smalleha/Adx_PiPER_Intelligent_workstation.git
2.2.Install Robot Arm Description
Clone the PiPER robot description package:
cd ~/agilex_ws/src
git clone https://github.com/agilexrobotics/agx_arm_sim.git
cd agx_arm_sim
git submodule update --init --recursive
2.3.Install Orbbec RGB-D Camera Driver
Install the Orbbec ROS 2 driver:
cd ~/agilex_ws/src
git clone https://github.com/orbbec/OrbbecSDK_ROS2.git -b main
2.4.Face Detection Environment Setup
Create a dedicated environment for YOLOv5-Face inference:
conda create -n yolo_face python=3.10 -y
conda activate yolo_face
pip install numpy==1.26.1 opencv-python onnxruntime-gpu
onnxruntime-gpu is recommended for GPU acceleration during inference.
2.5.Download YOLOv5-Face Model
mkdir -p "$HOME/yolov5-face/weights"
curl -L \
-o "$HOME/yolov5-face/weights/yolov5n_face.onnx" \
https://github.com/yakhyo/yolov5-face-onnx-inference/releases/download/weights/yolov5n_face.onnx
Update the model path in:
orbbec_face_landmark/config/orbbec_face_landmark.yaml
First, run the following command to confirm your actual home directory:
echo "$HOME"
Then, use the full absolute path in the YAML configuration file. For example:
model_path: /home/your_username/yolov5-face/weights/yolov5n_face.onnx
Do not directly use /home/$USER/.... Unless the program explicitly supports environment variable expansion, $USER may be interpreted as a literal string instead of being replaced with the actual username.
2.6.Build the ROS 2 Workspace
source /opt/ros/humble/setup.bash
cd ~/agilex_ws
colcon build --symlink-install
source ~/agilex_ws/install/setup.bash
3.Running the Demo
The demo follows the following execution order:
RGB-D Camera
↓
Face Detection
↓
TF Coordinate Transformation
↓
Target Pose Generation
↓
PiPER Motion Control
Step 1.Launch Orbbec Camera
Start the RGB-D camera node:
source /opt/ros/humble/setup.bash
source ~/agilex_ws/install/setup.bash
ros2 launch orbbec_camera dabai.launch.py \
depth_registration:=true
Check that:
- RGB stream is available
- Depth stream is published
- Camera node is running correctly
Step 2.Start Face Detection Node
Activate the inference environment:
conda activate yolo_face
Launch YOLOv5-Face detection:
source /opt/ros/humble/setup.bash
source ~/agilex_ws/install/setup.bash
ros2 launch orbbec_face_landmark orbbec_face_landmark.launch.py
At this stage, verify the following:
- Whether the ONNX model is loaded successfully.
- Whether the detected face bounding box and facial landmarks remain stable.
- Whether the depth values contain sudden jumps or invalid measurements.
- Whether the direction and distance of the
camera_link → faceTF transformation are correct.
Step 3: Visualize TF Transformation
ros2 launch agx_arm_description display.launch.py \
arm_type:=piper \
end_effector:=gripper \
with_camera_stand:=true \
with_camera:=true
ros2 launch face_tf_2_piper_link6 \
face_tf_2_piper_link6.launch.py
At this stage, the system only publishes coordinate transformations.
No robot motion command pos_cmd is sent.
Verify the following:
- The
base_link → facetransformation is correct. - The movement direction from
base_linktoarm_posematches the expected behavior. - The target pose does not jump unexpectedly when face tracking is temporarily lost.
- The target position remains within the robot arm’s safe operating workspace.
Step 4: Enable PiPER Motion Control
Before enabling robot motion:
Make sure the workspace is clear and emergency stop is accessible.
Verify the following:
- The robotic arm payload, monitor mount, fasteners, and cable routing have been checked.
- The monitor weight and center of gravity meet the requirements of the robotic arm and mounting structure.
- CAN communication, robot feedback, and emergency stop functions are working properly.
- The emergency stop button is easily accessible and can be triggered at any time.
- A physical safety boundary has been installed, and no person is inside the robot’s motion range.
- Only one
arm_posepublisher is active. - Only one
/pos_cmdpublisher is active. - The target pose is within the validated collision-free workspace.
- The maximum speed has been reduced, and initial tests are performed at the lowest velocity.
- Potential risks such as display detachment or robotic arm falling after power loss or program failure have been considered.
Launch the PiPER driver and initialize the robotic arm:
ros2 launch piper start_single_piper.launch.py
Start the TF-based motion controller:
ros2 launch piper_tf_control piper_tf_control.launch.py
This screenshot is taken from a demo video published by the Bilibili project team.
Based on this open-source project, the robotic arm can hold a display and continuously adjust its motion based on the position of a person’s head, allowing the screen to follow the user’s head movement in real time.
4.Safety Considerations
This prototype is intended for research and engineering validation. It does not include the following safety functions:
- Complete robotic arm and human collision detection;
- Certified human presence detection;
- Safety-rated speed and separation monitoring;
- Safety limits and protective stop mechanisms;
- Certified safety validation for the display, mounting structure, and cable system;
- Human approach area validation.
For initial testing, use:
- Simulation
- RViz TF visualization
- Non-human targets
before enabling physical robot motion.
FAQ
Q1：Can this project already be used as a commercial ergonomic workstation?
No. This project currently validates the complete pipeline from visual perception, 3D localization, TF transformation, to robotic arm motion control. It has not yet completed ergonomic performance evaluation, reliability testing, or human–robot collaboration safety certification.
Q2：Why does the display sometimes experience jitter during tracking?
The jitter may come from multiple factors, including face landmark detection errors, depth sensor noise, timestamp synchronization issues, TF estimation errors, and control frequency or velocity parameters.
Motion smoothing can reduce visual instability, but it cannot replace safety control mechanisms.
Q3：Can a person stand in front of the robotic arm while it is moving?
No. In the current demo configuration, users must not enter the robot arm’s reachable workspace or swept motion area.
Especially during initial testing, the person’s head and neck should not be used as the direct tracking target while the robotic arm is enabled.
Q4：Is this a completely original approach?
No. This project does not claim to introduce a new interaction concept.
The system was developed as an engineering prototype during AdventureX 2026, using PiPER robotic arm and ROS 2 to validate the integration pipeline. Similar research on robot-arm-driven display tracking and human-following interfaces has already been explored in previous work.
Project Origin and Acknowledgements
This project is based on the “Smart Ergonomic Workstation” track from AdventureX 2026 Hangzhou Station. We would like to thank the participating team “摆烂大队” for completing the initial concept design, prototype development, and on-site validation.
This article focuses on organizing and documenting the implementation workflow of the intelligent display control pipeline, providing a reference for reproduction and further development. It does not claim to introduce a completely independent or original solution.
We would also like to thank the original project contributors, repository maintainers, and subsequent documentation contributors for making the code and related resources publicly available.
License and Usage Notice
At the time of writing this article, the related project repositories do not contain a clearly specified LICENSE file in their root directories.
Public accessibility of a repository does not automatically grant permission for unrestricted copying, modification, or commercial use.
Before redistribution or commercial application, please contact the original authors to confirm the applicable licensing terms and usage permissions.