Gripper-Based Robot#

With a VR headset connected via ALVR and tele.py running, you can start controlling a robot with gripper in virtual reality.

# example
./example.sh peg_insertion
# python3 tele.py --env configs/examples/maniskill_panda/peg_insertion_side.yaml --tele configs/agents/panda/tele_right.yaml
../../_images/peg_insertion_example.png

PegInsertionSide-v1 environment with Panda robot. The task is to pick up a orange-white peg and insert the orange end into the box with a hole in it. See ManiSkill3 PegInsertionSide-v1.#

How Control Works#

../../_images/controller_map.jpg

Controller Key Mapping definition#

The control system maps your hand position directly to the robot’s end-effector. Press the Bottom button on the right controller to start synchronization - the robot will follow your hand movements. Press it again to stop synchronization and freeze the robot in place.

Use the Upper button to control the gripper. Hold it down to close the gripper, release to open it. .. is used to control the gripper opening and closing.

Recording Your Movements#

For simulation tasks with predefined success conditions, recording happens automatically:

  • Recording starts when you begin synchronization with the robot.

  • Recording stops once you complete the task successfully, plus a few extra seconds for good measure.

  • The environment resets itself after each recording.

You also have manual controls available:

  • Press the A button to stop recording manually.

  • Press the B button to reset the environment whenever you want.

Visual Feedback#

../../_images/peg_insertion_example_vis.png

A small green shpere as visualization kit#

A small green sphere shows you important information. For example, in a “pick up the cube” task, the green sphere marks exactly where you need to place the cube.

The sphere changes color to tell you what’s happening:

  • Green: Ready to start.

  • Red: You’re synchronized with the robot (robot following your hand)

  • Blue: Success! You’ve completed the task

After the sphere turns blue, the environment will automatically reset after a few seconds and the sphere returns to green.

Workflow Example#

  1. Environment starts with green sphere

  2. Press bottom button → sphere turns red (synchronization active)

  3. Perform your demonstration with the robot following your hand and press the upper button to close the gripper

  4. Complete the task → sphere turns blue (success!)

  5. System automatically resets after a few seconds → sphere returns to green

Controller Reference#

  • Bottom button: Toggles synchronization between the controller and the robot’s end-effector.

    • Press once to enable (the robot will follow your hand).

    • Press again to disable (freeze the robot in place).

  • Upper button: Controls the gripper.

    • Hold down to close the gripper.

    • Release to open the gripper.

  • A button: Manually stop recording.

  • B button: Manually reset the environment.

  • Meta icon: Adjust your VR view.

    • Hold for 3 seconds to adjust the view.