Skip to content
Physical AI

LeRobot Dataset Format: Recording Pick and Place Episodes

A LeRobot dataset stores a pick and place demonstration as three synchronized parts: frame-by-frame Parquet rows holding joint states and actions, MP4 shards holding every camera view and metadata that maps episode boundaries onto both. This guide covers what the v3.0 format contains, how to record episodes with lerobot-record, how many demonstrations a pick and […]

LeRobot Datasets

A LeRobot dataset stores a pick and place demonstration as three synchronized parts: frame-by-frame Parquet rows holding joint states and actions, MP4 shards holding every camera view and metadata that maps episode boundaries onto both. This guide covers what the v3.0 format contains, how to record episodes with lerobot-record, how many demonstrations a pick and place task needs, the storage and throughput numbers to plan against and the quality gates to run before a single training step.

LeRobot has become the default recording format for low-cost manipulation research, and the training stacks followed. LeRobot trains ACT, Diffusion Policy and SmolVLA directly and GR00T fine-tuning reads a LeRobot variant, which means the recording decisions you make on day one set the ceiling for every policy you train afterward.

What a LeRobot Dataset Contains?

The format separates storage from the access API. Low-dimensional, high-frequency signals such as joint positions and actions go into Apache Parquet. Camera frames go into MP4 files grouped by camera. Metadata describes the schema, the frame rate, the task text and where each episode starts and ends.

Loading a sample returns a dictionary of PyTorch tensors with keys including observation.state, action, observation.images.<camera_name> and timestamp. The task text you type at record time becomes the language condition your policy trains against, so treat it as a schema field rather than a label. A dataset recorded with the task string “Grab the black cube” teaches a different conditioning signal than one recorded with “Pick the black cube and place it in the bin,” even when the demonstrations are identical.

The v3.0 Layout and What Changed From v2.1

Version 3.0 reorganized storage to handle datasets at Open X-Embodiment scale.

Aspectv2.1v3.0
Tabular dataOne Parquet file per episodeMany episodes per Parquet shard under data/chunk-000/
VideoOne MP4 per episode per cameraConsolidated MP4 shards under videos/<camera>/chunk-000/
Episode metadataepisodes.jsonlChunked Parquet under meta/episodes/
Task definitionstasks.jsonlmeta/tasks.parquet
Episode boundariesResolved by filenameResolved through metadata offsets
Hub accessDownload then loadStreamingLeRobotDataset iterates without local copies

LeRobotDataset v3.0 ships in lerobot 0.4.0 and later. The practical benefit is file system pressure: a 5,000-episode dataset under v2.1 produced tens of thousands of small files, which slowed initialization and broke on some cluster filesystems. Fewer, larger shards fix that.

Setting Up the SO-101 Leader and Follower Arms

The SO-101, which the Hugging Face LeRobot docs call their flagship robot, pairs a follower arm that performs the task with a leader arm you move by hand. TheRobotStudio designed it in collaboration with Hugging Face. The follower runs six STS3215 servos at 1/345 gearing, and the leader mixes 1/191, 1/345 and 1/147 ratios so it holds its own weight and still moves with little force.

The LeRobot SO101 guide prepares each SO-101 arm in three steps. lerobot-find-port finds its USB port and lerobot-setup-motors writes the motor IDs and baud rates. lerobot-calibrate then gives leader and follower the same position values in the same physical pose. A lerobot-teleoperate run with so101_follower and so101_leader drives the follower from the leader, which checks the pairing before the first recorded episode.

Recording Pick and Place Episodes With lerobot-record

Calibrate before the first session, and reuse the same robot.id across teleoperation, recording and evaluation, since the identifier controls which calibration file loads. A representative SO-101 recording command:

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=follower_arm \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=leader_arm \
  --dataset.repo_id=${HF_USER}/pick_place_bin \
  --dataset.single_task="Pick the red cube and place it in the bin" \
  --dataset.num_episodes=50 \
  --dataset.episode_time_s=30 \
  --dataset.reset_time_s=15 \
  --display_data=true

Three keys control the session while it runs. Right arrow (or n) ends the current episode early and saves it. Left arrow (or r) discards and re-records the episode. Escape (or q) stops the session. On Wayland, headless machines or macOS without accessibility permission, these read from the terminal instead of a global keyboard hook, so keep the launching terminal focused.

The re-record key does more work than any other setting. A demonstration where the gripper fumbles and the operator recovers teaches recovery behavior. Re-record any episode where the operator loses the object and stops, so the policy learns from complete attempts.

Recording Flags That Change Dataset Quality

FlagControlsSetting for pick and place
--dataset.single_taskLanguage conditioning textFull verb phrase naming object and destination
--dataset.fpsSampling rate for state and video30, matching the camera fps
--dataset.episode_time_sMaximum episode durationRoughly double your median demonstration
--dataset.reset_time_sPause between episodes for scene resetLong enough to reposition objects deliberately
--robot.camerasWhich views enter the datasetWrist plus one third-person view minimum
--dataset.streaming_encodingVideo encoding during capturetrue, when frame rate drifts under load

Settle the camera choice before recording starts. A wrist camera gives the policy the close-range visual feedback that grasping depends on, and a third-person view gives it scene context for locating the target. Community SO-101 pick and place sets commonly record front, wrist and top views.

How Many Episodes a Pick and Place Task Needs?

Published work gives usable anchors. The LeRobot learning materials and community quickstarts converge on 50 demonstrations as a first real-robot task target.

Spread episodes across task variants such as object positions, since variation is what a policy generalizes from.

Designing Variation Into the Episode Set

Decide what varies before recording, and hold out some of the variation for evaluation. Object positions that never appear during recording give a clean measure of whether the policy generalized or memorized.

A workable variation plan for a warehouse pick and place task covers object start position, target container position, lighting, distractor objects and operator. Vary one axis at a time within a recording session so you can trace a later gap to its cause. Record a few labeled recovery episodes as well, so the policy learns what to do when a grasp slips.

Single Task or Multi Task: How to Split the Recording?

The --dataset.single_task flag names one task for the whole recording session, and each distinct task string becomes its own task index in meta/tasks.parquet. That gives you two structuring choices.

Need training data for your models? Scope a data-collection or labeling pipeline in a strategy session with our team.
Book a strategy session →

Record separate datasets per task when the tasks share no objects or workspace, when you want to evaluate each independently or when different operators handle different tasks. Record one dataset spanning several task strings when you are fine-tuning a language-conditioned policy and want the model to learn which instruction maps to which behavior. VLA fine-tuning generally benefits from the second shape, since the language conditioning is the thing you are training.

Keep task strings mechanically consistent whichever you choose. “Pick the red cube and place it in the bin” and “pick the red cube and place it in the bin” register as two tasks and split your data across two indices. Write the string once, store it in a shell variable and reuse it across every session for that task.

Storage and Throughput Planning

QuantityReference figure
Multi-camera recording at 30 fpsAround 1 GB per 10 minutes of capture
Timesteps per pick and place episode300 to 600 at 30 Hz
Wall-clock per episode including resetDemonstration time plus the reset window you set
Hub uploadRuns after all episodes finish, not per episode

Two operational notes follow from that upload behavior. Record in batched sessions of a size you can afford to lose, and confirm disk headroom before a long session rather than during one. When you write a dataset through the Python API instead of the CLI, call dataset.finalize() to close the Parquet writers properly, or the metadata will not match the shards.

Quality Gates to Run Before Training

Hugging Face’s LeRobot team ran a comparable cleanup on its own community collection. Starting from 851 public LeRobot datasets contributed by 235 users, they worked through missing video files, inconsistent data types across batches, broken sequential indexing from deleted files and mismatched action or state dimensions, fixing or dropping datasets at each stage. The cleaned community_dataset_v3 release holds 791 datasets, 50,622 episodes, roughly 26 million frames and 251.5 hours of interaction.

Run the equivalent on your own data before you train:

  • Confirm every episode carries the full feature schema and no NaN values in the state or action tensors.
  • Check episode length distribution and inspect anything far outside your median, since outliers are usually a mis-stopped recording.
  • Play back a random sample with the dataset visualizer and watch the wrist view specifically.
  • Verify the task string is identical across episodes intended as one task, since a typo creates a second task index.
  • Separate evaluation episodes from training episodes at the dataset level rather than by convention.

Add an operator consistency check when more than one person records. Two teleoperators solving the same task at different speeds or from different angles produce a bimodal action distribution that can push the policy toward averaging two valid strategies. Standardize the approach in a short written protocol before recording, or record enough episodes per operator that the policy sees each strategy as a complete mode.

Loading and Streaming for Training

The dataset object returns PyTorch tensors and drops into a standard DataLoader. The delta_timestamps argument requests temporal windows in seconds relative to the current frame, which is how you feed frame stacks to policies that condition on recent history.

Because LeRobot wraps several training pipelines, the same recorded dataset feeds ACT and Diffusion Policy training and SmolVLA fine-tuning directly. GR00T reads a LeRobot variant that adds a meta/modality.json file, and OpenVLA trains from RLDS data, so those two need a conversion step.

Migrating v2.1 Datasets to v3.0

Existing datasets convert with the bundled script, which aggregates per-episode Parquet and MP4 files into shards and writes the episode offsets into the new metadata:

python -m lerobot.scripts.convert_dataset_v21_to_v30 \
  --repo-id=${HF_USER}/pick_place_bin

Verify episode count and total frame count on both sides before deleting the original. If you buy episodes from a vendor, ask which version they write, since a v2.1 delivery in 2026 means a conversion step you did not price.

Converting Rosbag, MCAP and HDF5 Recordings Into LeRobot Episodes

Trajectory data recorded outside LeRobot converts through the same dataset API LeRobot uses for recording. Its porting guide works through DROID as the example and says to install the reader for your source format.

On the read side, a ROS 2 rosbag records topics as timestamped messages, and MCAP is the default log format in ROS 2, according to the MCAP project. Its ROS 2 guide reads an MCAP file as a stream of topic, message and timestamp tuples. HDF5 organizes data in groups, and each HDF5 dataset holds raw data values, so a converter maps each one (camera frames, joint states, actions) to a LeRobot feature.

On the write side, LeRobotDataset.create takes a repo ID, the fps and a features dict that maps each feature name to its type and shape. Each frame goes in through add_frame, which must include a task key. save_episode writes each episode to disk and finalize writes the Parquet footers after the last one. Timing needs care. A bag logs every topic at its own rate while a LeRobot dataset stores frames at one fps, so resample each stream onto the dataset clock before calling add_frame, then compare episode and frame counts against the source.

Recording Your Own Episodes Versus Buying Them

Recording in-house makes sense when the task is specific to your cell, the objects are yours and one operator can cover the variation you need in a few sessions. Buying episodes makes sense when you need operator diversity, scene diversity across sites or sensor modalities your rig does not carry. Our physical AI training data page covers what labeling and ground-truth layers add on top of raw capture.

NeuralChainAI delivers LeRobot-ready episode sets through our embodied AI training data practice, and runs multi-site capture programs through robot data collection services.

Planning a pick and place dataset and unsure how many episodes or which camera setup to commit to? Talk to our team about physical AI and robotics consulting and development, and we will size the recording plan against the policy you intend to train.

Frequently Asked Questions on LeRobot Datasets

Fifty demonstrations is the standard starting target for a first real-robot task, and published research uses 50 for single-stage pick and place and around 100 for multi-stage tasks such as opening a drawer before placing an object inside it. At 30 Hz with episodes running 9 to 18 seconds, 50 episodes gives roughly 15,000 to 30,000 training frames. Add task variants rather than repetitions once you pass that point, since variation across object positions generalizes further than more episodes from one fixed pose.
Version 2.1 wrote one Parquet file and one MP4 per episode per camera, with episode boundaries resolved by filename. Version 3.0 packs many episodes into larger Parquet and MP4 shards, moves episode and task metadata into chunked Parquet under meta/, and resolves boundaries through metadata offsets instead of filenames. It also adds StreamingLeRobotDataset for iterating straight from the Hugging Face Hub. v3.0 ships in lerobot 0.4.0 and later, and existing datasets convert with the bundled conversion script.
Plan on roughly 1 GB per 10 minutes of multi-camera recording at 30 fps. A 50-episode pick and place session with two cameras and a 15-second reset between episodes lands in the low single-digit gigabytes. Confirm headroom before a long session rather than during one, because the Hub upload runs after all episodes finish rather than after each one, so a disk-full failure mid-session costs the whole batch.
Record separate datasets when tasks share no objects or workspace, when you evaluate them independently, or when different operators handle different tasks. Record one dataset spanning several task strings when you are fine-tuning a language-conditioned policy, since the model needs to learn which instruction maps to which behaviour. Whichever you choose, keep the task string byte-identical across sessions. A capitalization difference creates a second task index and splits your data.
SmolVLA, ACT and Diffusion Policy train on a LeRobot dataset directly through LeRobot's own training command. GR00T reads a LeRobot variant with an added meta/modality.json file that describes the state, action and video layout, and OpenVLA's training code reads RLDS data, so both need a conversion step. Check which dataset version your training stack expects, since some older forks still assume the v2.1 layout.

Leave a Comment

Build the Dataset Your Models Need.

30 minutes with a senior consultant to scope multimodal capture, labeling, or a physical-AI data pipeline.

Book Your Session
Discuss your Physical AI project Discuss your project