Think Outside the Box: Controlling a Robotic Arm with a Mobile Phone Gyroscope

ROS ROS General
, , , ,
1

:waving_hand: Hi ROS Community,

We’d like to share an experimental project where we explored controlling a robotic arm using the gyroscope and IMU sensors of a mobile phone. By streaming the phone’s sensor data via WebSocket, performing attitude estimation with an EKF, and mapping the results to the robotic arm, we managed to achieve intuitive motion control without the need for traditional joysticks or external controllers.

This post documents the setup process, environment configuration, and code usage so you can try it out yourself or adapt it to your own robotic applications. :rocket:

Abstract

This project implements robotic arm control using mobile phone sensor data (accelerometer, IMU, magnetometer).
The data is transmitted from the mobile phone to a local Python script in real time via WebSocket.
After attitude calculation, the script controls the movement of the robotic arm.

Tags

Mobile Phone Sensor, Attitude Remote Control, IMU, Attitude Calculation, EKF

Code Repository

GitHub: Agilex-College/piper/mobilePhoneCtl

Function Demonstration

Environment Configuration

  • Operating System: Ubuntu 18.04+ (recommended)
  • Python Environment: Python 3.6+
  • Clone the project:
git clone https://github.com/agilexrobotics/Agilex-College.git
cd Agilex-College/piper/mobilePhoneCtl/
  • Install dependencies:
pip install -r requirements.txt --upgrade
  • Ensure that piper_sdk and its hardware dependencies are properly installed and configured.

Mobile Phone App Installation

We recommend using Sensor Stream IMU+ (a paid app) for mobile phone–side data collection and transmission:

  • Install the app from the official website or app store.
  • Supports both iOS and Android.

App Usage Instructions

  1. Open the Sensor Stream IMU+ app.
  2. Enter the IP address and port of the computer running this script (default port: 5000), e.g., 192.168.1.100:5000.
  3. Select sensors to transmit: Accelerometer, Gyroscope, Magnetometer.
  4. Set an update interval (e.g., 20 ms).
  5. Tap Start Streaming to begin data transmission.

Python Script Usage

  1. Connect the robotic arm and activate the CAN module:
sudo ip link set can0 up type can bitrate 1000000
  1. Run main.py in this directory:
python3 main.py
  1. The script will display the local IP address and port—fill this into the app.
  2. Once the app starts streaming, the script performs attitude calculation and sends the results to the robotic arm via the EndPoseCtrl interface of piper_sdk.

Data Transmission & Control Explanation

  • The mobile phone sends 3-axis accelerometer, gyroscope, and magnetometer data in real time via WebSocket.
  • The script employs an Extended Kalman Filter (EKF) to compute Euler angles (roll, pitch, yaw).
  • Calculated attitudes are sent to the robotic arm via piper_sdk’s EndPoseCtrl for real-time motion control.

Precautions

  • Ensure the mobile phone and computer are on the same LAN, and that port 5000 is allowed through the firewall.
  • Safety first—be cautious of the robotic arm moving to avoid any collisions.
  • To modify port or initial pose, edit the relevant parameters in main.py .

Related Projects

Check out these previous tutorials on controlling the AgileX PIPER robotic arm:

:white_check_mark: That’s all for this demo!
If you’re interested in experimenting with mobile-sensor-based robot control, feel free to check out the GitHub repo (again) and give it a try.

We’d love to hear your thoughts, suggestions, or ideas for extending this approach—whether it’s integrating ROS2 teleoperation, improving the EKF pipeline, or adding gesture control.

Looking forward to your feedback and discussions! :raising_hands: