ACT and Diffusion Policy Training Data for Robots
ACT and Diffusion Policy training data from teleoperated bimanual arms and UMI handheld demonstrations, with wrist and scene views, joint states and actions on one clock, delivered in LeRobot format as ready-made sets or custom collection.
What ACT Learns From
ACT, short for Action Chunking with Transformers, comes from the 2023 ALOHA paper by Zhao, Kumar, Levine and Finn (arXiv 2304.13705). It learns a generative model over action sequences from teleoperated two-arm demonstrations.
Four camera views
Four RGB cameras at 480×640, one on each follower wrist for a close-up of the grippers, one at the front and one on top.
Joint positions in and out
Observations are the follower arms' joint positions plus all four frames. Actions are the leader arms' joint positions, a 14-dimensional vector for the two arms.
Chunks of future actions
The policy predicts target joint positions for the next k timesteps, then averages overlapping chunks with an exponential weighting at execution.
50 Hz and 50 demonstrations per task
Teleoperation and recording run at 50 Hz, with leader arms mapped joint to joint onto the followers. The authors recorded 50 demonstrations per task, and 100 for Thread Velcro.
For your dataset, that means every camera view plus per-arm joint states and actions in each episode, on one clock at a fixed rate, so each predicted chunk lines up with its frames.
What a Diffusion-Based Policy Learns From
Diffusion Policy, from Chi et al. at Columbia University, the Toyota Research Institute and MIT (arXiv 2303.04137), treats a robot's visuomotor policy as a conditional denoising diffusion process that predicts a sequence of actions for receding-horizon control.
Observation horizon
The policy reads the latest To steps of images and robot state. The paper's Push-T settings use 2 steps.
Action sequences
It predicts Tp steps of actions and runs Ta before planning again, 16 and 8 on Push-T. Position control, predicting the desired end-effector pose, outperformed velocity control.
Camera views
Two of the station's five cameras feed the single-arm UR5 policy at 320×240. The bimanual setup reads two scene and two wrist cameras plus both end-effector poses and gripper widths.
10 Hz commands
Commands are predicted at 10 Hz and interpolated to 125 Hz on the UR5, where operators teleoperated the demonstrations with a SpaceMouse.
UMI uses Diffusion Policy for all of its experiments, reading wrist-camera images, end-effector pose and gripper width and outputting relative end-effector trajectories (arXiv 2402.10329). Our UMI data collection page covers the ready-made UMI sets and new handheld collection.
How the Two Models Compare on Training Data
Settings as each paper reports them for its own robots. We agree the camera set and action space for your training config before recording.
| Data requirement | ACT (ALOHA paper) | Diffusion Policy (Chi et al.) |
|---|---|---|
| Cameras | Four RGB views at 480×640, two on the wrists, one front and one top | Two of the UR5 station's five cameras; two scene and two wrist cameras on the bimanual setup; wrist cameras only in UMI |
| Observation | Follower joint positions plus each camera's current frame | The latest To steps of images and robot state (2 on Push-T) |
| Action representation | Absolute joint positions for both arms (14 dimensions), predicted as a chunk of the next k steps | End-effector position sequences (16 predicted, 8 executed on Push-T); relative end-effector trajectories in UMI |
| Control rate | 50 Hz teleoperation and recording | 10 Hz commands, interpolated to 125 Hz on the UR5 |
| Demonstrations per task, as reported | 50 per task, and 100 for Thread Velcro | 136 for real Push-T and 162 to 284 for the bimanual tasks |
Training Both Models in LeRobot
LeRobot, Hugging Face's robot learning library, stores datasets as synchronized video with action and state data, the format our episodes ship in.
ACT in LeRobot
The docs name ACT the first model they recommend for new users. It takes several RGB views, such as wrist and top cameras, plus current joint positions. Its config adapts to the cameras and motors in your dataset.
Both policy types
LeRobot's cheat sheet lists act and diffusion as policy types for lerobot-train, so one dataset trains both for a side-by-side comparison.
Formats and records
LeRobot by default, with MCAP, HDF5 and RLDS on request. Every episode carries provenance and consent records, with signed participant and site agreements permitting commercial training.
Teams fine-tuning a VLA on the same episodes can see our page on demonstrations for robot policies and VLAs, and our robot data collection services page covers every rig we run.
What NeuralChainAI Records for ACT and Diffusion Training
Demonstrations recorded with the streams both models read, on one clock.
YAM-class bimanual arms
Two tabletop arms with joint states and actions for both, a wrist view from each and scene cameras.
Learn more →OpenArm-class bimanual arms
The same two-arm streams on OpenArm-class hardware, plus a ready-made OpenArm bimanual set.
Learn more →ARX, PiPER and SO-101 on request
Other arms, scoped with you before the pilot.
UMI handheld demonstrations
A handheld gripper with a wrist camera records demonstrations away from the robot.
Learn more →Tactile or force-torque
Contact streams for tasks that depend on grasp force, on the cameras' clock.
Learn more →Depth on bimanual arms and UMI
Metric per-frame depth for policies that read it.
Ready-Made Sets and Custom Collection
Ready-made sets let your team train and compare both models before a custom program is scoped. What the sets cover:
A new task or a new embodiment.
Custom collection covers a new task on an existing robot or a new robot or embodiment, on non-exclusive or exclusive terms. Exclusivity covers the data and its resale rights, separate from the robot that records it. Exclusive terms are priced above non-exclusive, and term and scope are agreed once a pilot is defined.
What Shapes the Engagement
We scope every program against these factors with you, then quote it as a whole.
Model and training setup
One model or both, plus your observation and action settings.
Embodiment
YAM or OpenArm-class arms and UMI handheld capture, with other arms on request.
Camera set
The wrist and scene views your policy reads.
Action space
Joint positions for arm rigs, or the end-effector representation a UMI policy uses.
Tasks and environments
Kitchen, bedroom and staged retail families, or tasks written for your evaluation.
Add-on streams
Tactile or force-torque, and depth on bimanual arms and UMI.
Acceptance criteria
What counts as a reject, such as a dropped object, a stream out of sync or an incomplete task.
Pilot scope and licensing
How much of your task list the pilot covers, and ready-made or custom terms.
Have a Policy to Train?
Send us the model, robot and tasks. We'll scope the episodes and send a representative sample pack with the same modalities, calibration and delivery format you'd get in production.
Book a Strategy Session →Frequently Asked Questions on ACT and Diffusion Policy Training Data
The ALOHA paper recorded 50 demonstrations per task, and 100 for Thread Velcro. Chi et al. collected 136 for real Push-T and 162 to 284 for bimanual tasks. LeRobot's imitation learning tutorial suggests at least 50 episodes, with 10 per location. We size the pilot with you against your evaluation.
Yes. LeRobot lists <code>act</code> and <code>diffusion</code> as policy types for <code>lerobot-train</code>, so both train from one dataset and your team compares them on the same episodes.
Yes. The UMI paper trains Diffusion Policy on wrist-camera images, end-effector pose and gripper width, with relative end-effector trajectories as actions. We record UMI handheld demonstrations as ready-made kitchen, bedroom and staged retail sets or new collection, with depth on request.
Bimanual tabletop arms of the YAM and OpenArm class, with ARX, PiPER and SO-101 on request. Episodes record joint states and actions for both arms plus wrist and scene views on one clock.
LeRobot by default, with MCAP, HDF5 and RLDS on request. Camera keys, chunk size and horizons stay in your training config, set from each model's own docs.
Ready-made data is licensed non-exclusive, and you can sublicense it to your own client. Custom collection runs on non-exclusive or exclusive terms. Exclusivity covers the data and its resale rights, separate from the robot. Exclusive terms are priced above non-exclusive, and term and scope are agreed once a pilot is defined.
Related Physical AI Solutions
Robot Data Collection Services
Learn more →UMI Data Collection for Robot Learning Teams
Learn more →OpenArm Datasets and Bimanual Teleoperation Data
Learn more →YAM Bimanual Teleoperation Data for Robot Learning
Learn more →Embodied AI Training Data for Robot Policies and VLAs
Learn more →Physical AI and Robotics Consulting, Development and Training Data
Learn more →Physical AI and Robotics Consulting
Strategy, data capture and model training support for robot learning, from one team.
See physical AI services →AI Strategy Session
A scoping call on your model, robot, camera set and delivery format.
Book a session →Robot Data Collection Services
Teleoperation and egocentric capture on bimanual arms, the Unitree G1 and UMI.
See the services page →Scope Your ACT and Diffusion Policy Training Data
A strategy call on your model, robot and tasks. We'll scope a pilot with the camera views, action space and format your training run needs, and send a representative sample pack so your team can check the episodes first.