Skip to content
Physical AI

Robot Datasets for Manipulation: Comparing Open Datasets and Custom Collection

Robot datasets for manipulation compared: DROID, Open X-Embodiment, AgiBot World and ALOHA data, with scale, embodiments, licenses and when to collect more.

Robot datasets for manipulation give a model its episodes, camera views, actions and language instructions. The largest open ones now hold more than a million trajectories. This guide compares DROID, Open X-Embodiment, AgiBot World and the ALOHA datasets on scale, embodiments, sensors and licenses. It explains the LeRobot, RLDS and HDF5 formats they ship in and sets out when a team should collect its own robotics data or license off-the-shelf episodes for a specific robot and task.

Open corpora changed how physical AI teams start robot learning projects. A team can pretrain or co-train a model on public demonstrations, then fine-tune it on episodes from its own robot. The open datasets differ more than their headline trajectory counts suggest: one arm model or 22 embodiments, 15 Hz control or 50 Hz, a permissive license or a noncommercial one. Those differences decide whether a dataset helps a given model.

What a Robot Dataset for Manipulation Contains

A manipulation dataset is a set of episodes. Each episode is one attempt at a task from the first frame to the last. Each step inside it pairs what the robot observed with the action it was commanded to take. Most of these corpora hold human demonstrations, recorded while a person teleoperates the robot, so an imitation learning policy or a vision-language-action (VLA) model learns to reproduce the operator’s actions from the observations.

DROID shows the typical contents. Its paper states that each episode contains three camera views, depth information, camera calibration and language annotations. The RLDS release stores wrist and exterior camera images, gripper, Cartesian and joint positions, the commanded actions and the language instruction for every step. All DROID data is recorded at 15 Hz, with actions logged in both joint space and end-effector space.

Four parts of an episode decide how well a robot dataset transfers to your model:

  • Observations: which cameras record, where they sit on the robot or in the scene and whether depth, tactile or force signals come with them.
  • Actions: the action space (joint positions, end-effector poses or gripper commands) and the control rate the policy will run at.
  • Language: the instruction attached to each episode, which becomes the conditioning text for a language-conditioned policy.
  • Metadata: robot type, calibration and the license that governs training use.

Open Robot Datasets Compared: DROID, Open X-Embodiment, AgiBot World and ALOHA

The table sets the four side by side. Every cell comes from the dataset’s own project page, paper, repository or Hugging Face card, checked in November 2026.

DatasetTrajectoriesEmbodimentsScenes or tasksSensorsLicense
DROID76,000 demonstration trajectories (350 hours)Franka Panda 7-DoF arm564 scenes, 86 tasksTwo exterior stereo cameras and one wrist-mounted stereo camera, depth, calibration, language annotations; 15 HzCC-BY 4.0 (stated in the paper)
Open X-EmbodimentMore than 1 million real robot trajectories22, from single arms to bimanual robots and quadrupeds527 skills (160,266 tasks), pooled from 60 datasetsVaries by source dataset; every dataset stored as RLDS episodesRepository: Apache 2.0 for software, CC-BY 4.0 for other materials; check each source dataset
AgiBot World (Beta)More than 1 million trajectories from 100 robots (2,976.4 hours)Mobile dual-arm robots with whole-body control; 14 arm degrees of freedom217 tasks across 100+ real-world scenarios in five domainsVideo from all camera perspectives, depth (PNG), proprioception (HDF5); hardware lists visual tactile sensors and 6-DoF dexterous handsCC BY-NC-SA 4.0 plus a community license agreement; gated access
ALOHA datasets (LeRobot Hub)One dataset per task, such as 50 episodes in aloha_static_candy and 55 in aloha_mobile_chairALOHA bimanual leader-follower rigs, static and mobile; 14-value state and actionOne task per datasetStatic: a high view, a low view and two wrist views. Mobile: a high view and two wrist views; 50 fpsMIT on the dataset cards checked

DROID

The DROID dataset (Distributed Robot Interaction Dataset) holds 76,000 demonstration trajectories, or 350 hours of interaction, collected across 564 scenes and 86 tasks by 50 collectors at 13 institutions over 12 months. Every episode comes from a Franka Panda 7-DoF arm with two adjustable exterior stereo cameras on tripods and a stereo camera on the wrist, teleoperated with an Oculus Quest 2 headset and controllers. The team distributes it in RLDS (1.7 TB) and as raw stereo HD video in MP4 (8.7 TB), and the paper releases it under CC-BY 4.0. For a manipulation model, its strength is scene breadth on one arm.

Open X-Embodiment

Open X-Embodiment pools more than one million real robot trajectories from 60 existing datasets contributed by 34 research labs. It spans 22 robot embodiments (from single arms to bimanual robots and quadrupeds) and 527 skills across 160,266 tasks. Each dataset is a sequence of episodes in the RLDS episode format that loads with tfds.load. The project’s GitHub repository licenses its software under Apache 2.0 and its other materials under CC-BY 4.0, and it keeps a spreadsheet that lists each source dataset with its citation. Camera setups, control rates and action spaces change from one source to the next, so a team training for one robot can filter to the embodiments closest to its own.

AgiBot World

AgiBot World’s Beta release holds more than one million trajectories from 100 robots, a total of 2,976.4 hours. AgiBot’s Hugging Face page lists 217 tasks, 87 skills and more than 3,000 objects across over 100 real-world scenarios in five domains. The robots are mobile dual-arm platforms with whole-body control, and the hardware list includes visual tactile sensors and 6-DoF dexterous hands. Proprioception ships in HDF5 and depth as PNG images, and the joint data covers 14 arm degrees of freedom (seven per arm). Access requires agreeing to share your Hugging Face account information, and the data and code are released under CC BY-NC-SA 4.0 with an AgiBot World community license agreement. The Alpha release is a subset of Beta.

The ALOHA Datasets

ALOHA is the low-cost bimanual leader-follower system from the 2023 ACT paper by Tony Zhao and colleagues (arXiv 2304.13705). The paper records 50 demonstrations per task (100 for Thread Velcro) with four Logitech C922x webcams. Two of them sit on the follower wrists, and teleoperation and recording both run at 50 Hz. LeRobot’s Hugging Face organization hosts the real-robot ALOHA recordings as one dataset per task. The aloha_static_candy card lists 50 episodes with a high view, a low view and two wrist views. The aloha_mobile_chair card lists 55 episodes with a high view and two wrist views. Both record at 50 fps with a 14-value state and action vector, and both cards carry an MIT license.

Formats: LeRobot, RLDS and HDF5

The four datasets arrive in three formats. Converting between them is a routine ingestion step, and the choice shapes how your training code loads episodes.

RLDS

RLDS stands for Reinforcement Learning Datasets, a Google Research ecosystem of tools to store, retrieve and manipulate episodic data. A dataset loads as a tf.data.Dataset of episodes, and each episode holds a tf.data.Dataset of steps with is_first and is_last flags plus optional observation, action, reward and is_terminal fields. Open X-Embodiment and DROID both use the RLDS episode format, so either one loads through TensorFlow Datasets.

Need training data for your models? Scope a data-collection or labeling pipeline in a strategy session with our team.
Book a strategy session →

LeRobot

LeRobotDataset v3.0 from Hugging Face stores states, actions and timestamps in Apache Parquet. Camera frames go into MP4 shards per camera, and metadata records the schema, frame rates and each episode’s start and end offsets. Many episodes share one Parquet or MP4 file, and StreamingLeRobotDataset reads data straight from the Hub. LeRobot’s guide to porting large datasets uses DROID 1.0.1 as its worked example, starting from the original TensorFlow Records and RLDS files. A v3.0 port of DROID now sits on the Hub, and AgiBot World’s own tooling builds on the LeRobot library with conversion scripts.

HDF5

HDF5 keeps arrays in one hierarchical file, which suits fixed-rate joint and action logs. The ACT codebase behind ALOHA includes a script that saves videos from an .hdf5 dataset, and AgiBot World stores its proprioception in HDF5. Recording HDF5 on the robot and converting to LeRobot or RLDS for training keeps the capture code simple and the training loader standard.

Pick the format your training code already reads, then convert the open data once at ingestion so every episode in the training mix shares one schema.

Matching an Open Robot Dataset to Your Robot, Scenes and Sensors

An open corpus earns its place in pretraining and co-training. A model for one robot doing one job also needs episodes that match that robot. Four checks show how close a dataset sits to your deployment.

  • Embodiment: DROID covers one arm model, AgiBot World covers its own mobile dual-arm platforms and the ALOHA sets cover one bimanual rig. A humanoid, a different arm or a different gripper needs episodes in its own action space or a retargeting step.
  • Scenes and tasks: DROID’s 564 scenes and AgiBot World’s five domains are broad, and a specific retail backroom, kitchen layout or factory cell can still sit outside them. Count the episodes that resemble your target scenes before you plan a fine-tune around a corpus.
  • Sensors: camera count and placement change what a policy sees. The ALOHA datasets include two wrist views. DROID adds depth and calibration, and AgiBot World lists visual tactile sensors in its hardware. A model that expects a stream at deployment needs that stream in training.
  • Rate and license: DROID records at 15 Hz and ALOHA at 50 Hz, so check the control rate your model expects and resample when you mix sources. Licenses run from MIT and CC-BY 4.0 to CC BY-NC-SA 4.0, and the noncommercial term in the last one matters for a product team.

When to Collect Your Own or License Off-the-Shelf Robot Data

Open data gives a model breadth. Episodes recorded on your embodiment, in your settings and with your sensor stack give it the behavior you deploy. A practical program combines the two, and the open question is where the task-specific episodes come from.

Collecting in-house suits a team that owns the robot, controls the site and has operators to run sessions. Licensing suits a team that needs volume across several tasks and settings, or episodes from hardware outside its own lab.

NeuralChainAI offers both routes. Teams that want to license off-the-shelf robot data can pick from ready-made sets recorded on UMI handheld grippers, the Unitree G1 and YAM bimanual arms in kitchen, bedroom and staged retail task families, plus an OpenArm bimanual set. Staged retail episodes are real demonstrations recorded in a mock store set. The sets ship in LeRobot format with MCAP, HDF5 and RLDS on request. The license is non-exclusive and lets you sublicense the data to your own client.

Custom collection records your tasks on non-exclusive or exclusive terms, either as a new task on an existing robot or on a new robot or embodiment. Exclusivity covers the data and its resale rights; the robot hardware sits outside it. Exclusive terms are priced above non-exclusive ones, and duration and scope are agreed once a pilot is defined. Streams are agreed per program, from RGB, depth and wrist views to joint states and actions, with tactile and force-torque where the task needs them. Every episode carries signed participant and site agreements and a per-episode chain of custody you can audit.

Cost depends on the embodiment, the task and episode counts, the streams and the license term. We scope and quote each program through our robot data collection services, starting with a representative sample pack your team can inspect before any commitment.

Weighing open robot datasets against custom collection for one robot and one task? NeuralChainAI’s robot data collection services cover off-the-shelf real-robot data and custom programs delivered in LeRobot format. Bring your embodiment, task list and target model to an AI strategy session and we will map which episodes to license and which to record.

Frequently Asked Questions on Robot Datasets

DROID (Distributed Robot Interaction Dataset) is an open manipulation dataset of 76,000 demonstration trajectories, or 350 hours of interaction, recorded on Franka Panda arms across 564 scenes and 86 tasks by 50 collectors at 13 institutions. Each episode carries three camera views, depth, camera calibration and language annotations, recorded at 15 Hz. The paper releases the full dataset under CC-BY 4.0, distributed in RLDS and as raw stereo MP4 video.
The project's GitHub repository licenses its software under Apache 2.0 and its other materials under CC-BY 4.0. The corpus pools 60 datasets from 34 labs, and the repository's spreadsheet lists each source with its citation. Read the release terms of each source dataset you plan to train on before commercial use, and keep that record with your training data.
RLDS (Reinforcement Learning Datasets) is a Google Research format and tool set for episodic data. A dataset loads as a tf.data.Dataset of episodes, and each episode holds a sequence of steps with is_first and is_last flags plus fields such as observation, action and reward. Open X-Embodiment and DROID both use it. Teams training with LeRobot code convert RLDS episodes to LeRobotDataset v3.0, and LeRobot's porting guide walks through that conversion for DROID.
The AgiBot World Beta page states that all data and code in the repository are under CC BY-NC-SA 4.0, a Creative Commons license with noncommercial and share-alike terms. It also sets out an AgiBot World community license agreement. Access requires agreeing to share your Hugging Face account information. Read both documents in full and take legal advice before any training run that feeds a commercial product.
Cost follows the embodiment and hands, the number of tasks and episodes, the streams recorded per episode, the settings and the license term, with exclusive terms priced above non-exclusive ones. We scope and quote each program as a whole through our robot data collection services, starting with a sample pack in the same format you would receive in production.

Leave a Comment

Build the Dataset Your Models Need.

30 minutes with a senior consultant to scope multimodal capture, labeling, or a physical-AI data pipeline.

Book Your Session
Discuss your Physical AI project Discuss your project