System Overview#

../../_images/vr_overview.png

System Overview#

This VR teleoperation system enables operators to control robotic arms and hands through real-time pose tracking while receiving stereo video feedback for immserive experience.

Architecture#

The system architecture consists of three main components:

  1. VR Interface: Captures human pose data from VR headset and controllers/hand tracking

  2. Control Server: Processes pose data, performs retargeting, and generates robot commands

  3. Environment: Executes commands and streams video feedback to the operator

The VR devices continuously stream human pose data to a central server, which retargets the human motions and sends joint commands to the robot. The environment streams stereo video back to the VR headset. While the visual feedback loop can run at 60 FPS, the control loop can operate at a lower frequency.

For simulation, we build based on ManiSkill3 a powerful unified framework for robot simulation and training. ManiSkill3 is powered by SAPIEN and provides comprehensive support for manipulation tasks and skill development.

Supported Robot Types#

The system supports two distinct categories of robots, each with specialized control tracking:

  1. Controller tracking for robots with gripper-based robots

    ../../_images/gripper_control.jpg

    Gripper control modules#

    • Pose Mapping: Controller pose data captured from the VR headset is mapped to arm joint positions through inverse kinematics (IK) solver.

    • Gripper Control: Controller button states determine gripper open/close commands.

    • System Control: Additional button inputs manage operational states, including: Teleoperation activation/deactivation, recording stop, and environment reset.

    Check out Gripper-Based Robot for more details about gripper-based robot control.

  2. Hand tracking for dexterous hand robots

    ../../_images/hand_control.jpg

    Hand control modules#

    • Arm Control: Wrist pose data from hand tracking is processed by the IK solver to generate arm joint positions.

    • Hand Control: Hand motions from hand tracking are mapped to robotic hand joint positions through retargeting.

    • System Control: A foot pedal provides hands-free control over operational states, allowing operators to maintain natural hand movements while managing system functions.

    Check out Dexterous Hand for more details about dexterous hand control.