Model Card for act_so101_pick_diverse_objects

This is an Action Chunking with Transformers (ACT) policy trained to pick up a wide variety of everyday objects with the SO101 robotic arm. The policy uses only a wrist-mounted camera as visual input.

The policy was trained with LeRobot and pushed to the Hub. See the full LeRobot documentation at LeRobot Docs.

Demo video

---

Model Details

• Policy class: ACT (Action Chunking with Transformers)
• Robot: SO101 (single arm)
• Observation: wrist camera only (640 × 480 @ 30 fps) + joint state
• Framework: LeRobot
• License: apache-2.0
• Paper: arXiv:2304.13705

---

Training Data

The policy was trained on a combination of two datasets collected on the same SO101 setup:

1. Expert teleoperation data

• Dataset: TakuyaHiraoka/so101_pick_diverse_objects
• Collected via standard teleoperation by a human expert.
• Each episode shows the arm moving from a resting pose to grasping a target object placed on a tabletop.
• See the dataset card for details on the objects and environment.

2. Human-in-the-Loop (HIL) correction data

• Dataset: TakuyaHiraoka/so101_pick_diverse_objects_hil_round1
• Collected by rolling out a prototype policy (trained on the expert data above) and having a human operator intervene to correct failure cases.

Combining both sources is intended to improve robustness on objects and configurations where pure expert imitation tends to fail.

---

Out-of-Scope / Limitations

• The policy was trained on a specific tabletop environment and lighting condition. Performance can degrade significantly under different backgrounds, lighting, or camera mountings.
• Only a wrist camera is used, so the policy has no global scene context; picking objects far outside the wrist camera's field of view is unlikely to work.
• The robot embodiment is fixed (SO101 with the Innomaker U20CAM-1080P-S1 wrist camera). Transferring to other arms or cameras will likely require retraining or fine-tuning.

---

How to Get Started with the Model

For a complete walkthrough, see the LeRobot training guide.

Train from scratch

lerobot-train \
  --dataset.repo_id=${HF_USER}/ \
  --policy.type=act \
  --output_dir=outputs/train/ \
  --job_name=lerobot_training \
  --policy.device=cuda \
  --policy.repo_id=${HF_USER}/ \
  --wandb.enable=true

Checkpoints are written to outputs/train/<desired_policy_repo_id>/checkpoints/.

Evaluate the policy / run inference

lerobot-record \
  --robot.type=so100_follower \
  --dataset.repo_id=/eval_ \
  --policy.path=TakuyaHiraoka/act_so101_pick_diverse_objects \
  --episodes=10

Prefix the evaluation dataset repo with eval_ and supply --policy.path pointing to this Hub checkpoint (or a local path).

---