World Model Training Data: Real Video for World Foundation Models
World model training data captured from the real world: grounded egocentric video with synced metric depth, camera pose, action labels and full provenance. The physically faithful footage a world foundation model needs, not scraped mono clips. Delivered LeRobot-ready.
What's in Our World-Model Dataset
Every episode is real, physically faithful capture: synchronized stereo video and metric depth, camera pose and ego-motion, action labels and provenance, far beyond the mono clips most video corpora ship.
Long Unbroken Takes
Continuous first-person footage, not chopped highlight clips, so dynamics stay intact.
Metric Depth, Not Just RGB
True-scale geometry per frame — the signal a world model needs to predict physics.
Real Physics Interactions
Contact, deformables and slip on real floors — the hardest things to simulate.
Camera Pose and Ego-Motion
Head and wrist IMU at 200Hz, so motion is measured, not inferred from pixels.
Action and Intent Labels
Verb-noun action segments and 21-point hand pose for action-conditioned training.
Clear Licence and Provenance
Signed consent and SHA256 chain of custody on every episode.
Why Metric Depth, Not Just RGB
Monocular video is scale-ambiguous, so a model cannot recover true geometry from it. Our stereo capture ships metric depth per frame — the exact signal a world model needs to predict how a scene will move. Stereo remains underexplored in world and VLA models, which is where the gap is. Three things scraped or synthetic video structurally can't give you:
We ground the model on real physics, not inferred pixels.
Real video, metric depth and action labels on one clock — the grounded signal a world foundation model needs, and the one scraped mono video can't provide. Simulation is unbeatable for resets and perfect labels, but real capture grounds the model on the physics it will actually face. We treat sim and real as complements, not rivals.
How You Receive and Use It
Every episode ships LeRobot-ready, with RLDS/TFDS and ROS 2 / Foxglove MCAP on request, so it drops into a Cosmos-style or Isaac Lab post-training workflow. One honest caveat: this is human egocentric capture — it grounds and post-trains a world model, and it does not replace your simulator or ship robot action labels out of the box.
Action-conditioned video prediction
Genie-style next-frame prediction, trained on real footage with real action labels — verb-noun segments and 21-point hand pose synced to the video.
Grounding Cosmos-style world models
Post-train an open world foundation model on the environments you actually care about, drawing on the same multimodal stack as our physical AI training data.
Scene geometry and 3D reconstruction
Metric depth plus ego-motion reconstructs scenes at true scale for occupancy and mapping — geometry a monocular corpus cannot recover.
Custom capture for your world model
Environments, objects and failure modes captured to spec, so the data you post-train on is your data, not a generic corpus everyone else already has.
LeRobot-ready, drops in without conversion
Every episode ships as a native LeRobotDataset, so it drops into a Cosmos-style or Isaac Lab post-training workflow without a conversion project.
RLDS/TFDS and ROS 2 / Foxglove MCAP
RLDS/TFDS and ROS 2 / Foxglove MCAP are available on request, so the data lands in the exact format your training and playback tooling already expects.
Grounds real physics where simulation breaks
Human egocentric capture grounds and post-trains your world model on friction, deformables and slip — exactly where sim-to-real breaks — without replacing your simulator or shipping robot action labels out of the box.
Representative sample pack to evaluate first
A small but fully representative delivery with all modalities and calibration, so your team can inspect the footage and sensor alignment before any commitment.
Grounding a World Model on Real Footage?
Tell us the environments and dynamics you need. We'll map the capture and send a representative sample pack — the same modalities, calibration and delivery format you'd get in production, so your team can inspect the footage and sensor alignment before any commitment.
Book a Strategy Session →When You Need Custom Capture to Your Environments, Not a Generic Corpus
Scraped and synthetic video cover common footage well. But teams training real world foundation models need capture that a scraped or synthetic dataset structurally can't deliver:
Send the environments, dynamics and edge cases you need and we scope a capture program against them — sites, objects, failure modes, episode counts and acceptance criteria — so the footage you post-train on is your data, measured on real floors, not a generic corpus everyone else already has.
Frequently Asked Questions
Every episode is captured under a signed release that permits commercial model training, with a per-episode chain of custody you can audit. Licensing is per-dataset and non-exclusive by default; exclusive and custom terms are available.
Native LeRobotDataset, plus RLDS/TFDS and ROS 2 / Foxglove MCAP on request. It drops into Cosmos-style or Isaac Lab post-training without a conversion project.
Scraped clips are mono and scale-ambiguous; synthetic stalls on contact and deformables. Ours is real, grounded, physically faithful capture with metric depth and action labels.
Yes. Send the environments, dynamics and edge cases you need and we scope a capture program against them, with exclusive licensing available on custom episodes.
Start with the sample pack — a small but fully representative delivery with all modalities and calibration, so your team can inspect it before any commitment.
Related Solutions in the Physical-AI Cluster
Physical AI Training Data — Multimodal Field Capture for Robot Learning
Learn more →Embodied AI Training Data — Demonstrations for Robot Policies & VLAs
Learn more →Tactile Manipulation Data — The Modality Simulation Can't Fake
Learn more →Robot Data Collection Services — Teleoperation & Egocentric Capture
Learn more →Physical AI & Robotics — Consulting, Development & Training Data
Learn more →Private & On-Premise AI — Self-Hosted AI Deployment
Learn more →AI Transformation Workshop
Half-day strategy workshop to map the environments and dynamics you need and identify the right capture program for your world model.
Book a workshop →AI Strategy Session
30-minute scoping call. We'll talk through your environments, target world model and delivery format, then scope a representative sample pack.
Book a session →AI Consultant vs In-House Team
Honest tradeoffs on standing up real-world video capture in-house versus engaging a partner for a scoped, delivered capture program.
Read the comparison →See the Data Before You Decide
A 30-minute strategy call. We'll walk through the environments and dynamics you need, your target world model and delivery format — then scope a representative sample pack so your team can inspect the footage and sensor alignment before any commitment.