Skip to content
Solutions · Physical AI & Robot Data

ACT and Diffusion Policy Training Data for Robots

ACT and Diffusion Policy training data from teleoperated bimanual arms and UMI handheld demonstrations, with wrist and scene views, joint states and actions on one clock, delivered in LeRobot format as ready-made sets or custom collection.

Book a Robot Policy Data Strategy Session Free 30-minute call · mutual NDA included
BimanualYAM and OpenArm-class arms, with ARX, PiPER and SO-101 on request.
UMIHandheld gripper demonstrations, ready-made or newly collected.
LeRobotDelivery format, with MCAP, HDF5 and RLDS on request.
ACT in the ALOHA Paper

What ACT Learns From

ACT, short for Action Chunking with Transformers, comes from the 2023 ALOHA paper by Zhao, Kumar, Levine and Finn (arXiv 2304.13705). It learns a generative model over action sequences from teleoperated two-arm demonstrations.

Four camera views

Four RGB cameras at 480×640, one on each follower wrist for a close-up of the grippers, one at the front and one on top.

Joint positions in and out

Observations are the follower arms' joint positions plus all four frames. Actions are the leader arms' joint positions, a 14-dimensional vector for the two arms.

Chunks of future actions

The policy predicts target joint positions for the next k timesteps, then averages overlapping chunks with an exponential weighting at execution.

50 Hz and 50 demonstrations per task

Teleoperation and recording run at 50 Hz, with leader arms mapped joint to joint onto the followers. The authors recorded 50 demonstrations per task, and 100 for Thread Velcro.

For your dataset, that means every camera view plus per-arm joint states and actions in each episode, on one clock at a fixed rate, so each predicted chunk lines up with its frames.

Chi et al. and UMI

What a Diffusion-Based Policy Learns From

Diffusion Policy, from Chi et al. at Columbia University, the Toyota Research Institute and MIT (arXiv 2303.04137), treats a robot's visuomotor policy as a conditional denoising diffusion process that predicts a sequence of actions for receding-horizon control.

Observation horizon

The policy reads the latest To steps of images and robot state. The paper's Push-T settings use 2 steps.

Action sequences

It predicts Tp steps of actions and runs Ta before planning again, 16 and 8 on Push-T. Position control, predicting the desired end-effector pose, outperformed velocity control.

Camera views

Two of the station's five cameras feed the single-arm UR5 policy at 320×240. The bimanual setup reads two scene and two wrist cameras plus both end-effector poses and gripper widths.

10 Hz commands

Commands are predicted at 10 Hz and interpolated to 125 Hz on the UR5, where operators teleoperated the demonstrations with a SpaceMouse.

UMI uses Diffusion Policy for all of its experiments, reading wrist-camera images, end-effector pose and gripper width and outputting relative end-effector trajectories (arXiv 2402.10329). Our UMI data collection page covers the ready-made UMI sets and new handheld collection.

Compare the Data

How the Two Models Compare on Training Data

Settings as each paper reports them for its own robots. We agree the camera set and action space for your training config before recording.

Data requirementACT (ALOHA paper)Diffusion Policy (Chi et al.)
CamerasFour RGB views at 480×640, two on the wrists, one front and one topTwo of the UR5 station's five cameras; two scene and two wrist cameras on the bimanual setup; wrist cameras only in UMI
ObservationFollower joint positions plus each camera's current frameThe latest To steps of images and robot state (2 on Push-T)
Action representationAbsolute joint positions for both arms (14 dimensions), predicted as a chunk of the next k stepsEnd-effector position sequences (16 predicted, 8 executed on Push-T); relative end-effector trajectories in UMI
Control rate50 Hz teleoperation and recording10 Hz commands, interpolated to 125 Hz on the UR5
Demonstrations per task, as reported50 per task, and 100 for Thread Velcro136 for real Push-T and 162 to 284 for the bimanual tasks
LeRobot and Delivery

Training Both Models in LeRobot

LeRobot, Hugging Face's robot learning library, stores datasets as synchronized video with action and state data, the format our episodes ship in.

ACT in LeRobot

The docs name ACT the first model they recommend for new users. It takes several RGB views, such as wrist and top cameras, plus current joint positions. Its config adapts to the cameras and motors in your dataset.

Both policy types

LeRobot's cheat sheet lists act and diffusion as policy types for lerobot-train, so one dataset trains both for a side-by-side comparison.

Formats and records

LeRobot by default, with MCAP, HDF5 and RLDS on request. Every episode carries provenance and consent records, with signed participant and site agreements permitting commercial training.

Teams fine-tuning a VLA on the same episodes can see our page on demonstrations for robot policies and VLAs, and our robot data collection services page covers every rig we run.

Ready-Made Sets

Ready-Made Sets and Custom Collection

Ready-made sets let your team train and compare both models before a custom program is scoped. What the sets cover:

1 Kitchen, bedroom and staged retail sets on UMI, YAM arms and the Unitree G1, with staged retail recorded in a mock store set.
2 An OpenArm bimanual set for two-arm policies.
3 Non-exclusive and sublicensable to your own client, in LeRobot format with MCAP, HDF5 and RLDS on request.
Custom Collection

A new task or a new embodiment.

Custom collection covers a new task on an existing robot or a new robot or embodiment, on non-exclusive or exclusive terms. Exclusivity covers the data and its resale rights, separate from the robot that records it. Exclusive terms are priced above non-exclusive, and term and scope are agreed once a pilot is defined.

A new task on an existing robot
A new robot or embodiment
Non-exclusive or exclusive terms
Scoping

What Shapes the Engagement

We scope every program against these factors with you, then quote it as a whole.

1

Model and training setup

One model or both, plus your observation and action settings.

2

Embodiment

YAM or OpenArm-class arms and UMI handheld capture, with other arms on request.

3

Camera set

The wrist and scene views your policy reads.

4

Action space

Joint positions for arm rigs, or the end-effector representation a UMI policy uses.

5

Tasks and environments

Kitchen, bedroom and staged retail families, or tasks written for your evaluation.

6

Add-on streams

Tactile or force-torque, and depth on bimanual arms and UMI.

7

Acceptance criteria

What counts as a reject, such as a dropped object, a stream out of sync or an incomplete task.

8

Pilot scope and licensing

How much of your task list the pilot covers, and ready-made or custom terms.

Start Today

Have a Policy to Train?

Send us the model, robot and tasks. We'll scope the episodes and send a representative sample pack with the same modalities, calibration and delivery format you'd get in production.

Book a Strategy Session →
Ask us about
Bimanual episodes on YAM and OpenArm-class arms
Ready-made kitchen, bedroom and staged retail sets
UMI handheld demonstrations
Camera sets matched to your policy config
Tactile, force-torque and depth add-ons
LeRobot, MCAP, HDF5 and RLDS delivery
Questions

Frequently Asked Questions on ACT and Diffusion Policy Training Data

The ALOHA paper recorded 50 demonstrations per task, and 100 for Thread Velcro. Chi et al. collected 136 for real Push-T and 162 to 284 for bimanual tasks. LeRobot's imitation learning tutorial suggests at least 50 episodes, with 10 per location. We size the pilot with you against your evaluation.

Yes. LeRobot lists <code>act</code> and <code>diffusion</code> as policy types for <code>lerobot-train</code>, so both train from one dataset and your team compares them on the same episodes.

Yes. The UMI paper trains Diffusion Policy on wrist-camera images, end-effector pose and gripper width, with relative end-effector trajectories as actions. We record UMI handheld demonstrations as ready-made kitchen, bedroom and staged retail sets or new collection, with depth on request.

Bimanual tabletop arms of the YAM and OpenArm class, with ARX, PiPER and SO-101 on request. Episodes record joint states and actions for both arms plus wrist and scene views on one clock.

LeRobot by default, with MCAP, HDF5 and RLDS on request. Camera keys, chunk size and horizons stay in your training config, set from each model's own docs.

Ready-made data is licensed non-exclusive, and you can sublicense it to your own client. Custom collection runs on non-exclusive or exclusive terms. Exclusivity covers the data and its resale rights, separate from the robot. Exclusive terms are priced above non-exclusive, and term and scope are agreed once a pilot is defined.

Scope Your ACT and Diffusion Policy Training Data

A strategy call on your model, robot and tasks. We'll scope a pilot with the camera views, action space and format your training run needs, and send a representative sample pack so your team can check the episodes first.

Discuss Your Project