The Data Problem in Physical AI
From reading Physical Intelligence’s π*0.6 recent research breakthrough, and tracking recent robotics launches, I had a lightbulb moment recently: we’re watching robotics have its ImageNet moment in real-time. These VLA models work. But there’s a bottleneck everyone’s dancing around: data.
Collecting robot demonstrations costs $100-500 per hour when you factor in hardware, operators, and environment setup. Current datasets contain maybe 1,000 hours total, mostly staged demos in controlled lab environments. Meanwhile, foundation models in other domains trained on millions of hours of data.
The economics don’t work. We can’t manually collect our way to the data diversity that Physical AI needs.
But here’s what does exist: 10,000+ hours of human work captured on video. Factory floors, warehouses, construction sites, kitchens — all documented from the worker’s perspective through egocentric cameras. Build AI’s Egocentric-10K dataset alone contains footage from 85 manufacturing facilities. Ego4D captured 3,600 hours of daily activities. EPIC-KITCHENS documented 100 hours of cooking.
This data is abundant, authentic, and diverse. It’s also completely unusable for robot learning in its current form.
The gap isn’t data availability. It’s data infrastructure — the conversion layer that transforms raw human video into robot-trainable formats. I realized this after spending a week trying to load Egocentric-10K in any usable format. The data exists. The tooling doesn’t. So I built it.
What I Built: A Video-to-Dataset Pipeline
Over two weeks, I developed ego2robot: an end-to-end pipeline that converts egocentric human demonstrations into LeRobot-compatible episodes. The result is 50 curated episodes from real factory work, now available on Hugging Face.
Why factory video? Partly because it’s available - earlier this month, Build Ai released their Egocentric-10K dataset with 10,000 hours. But also because I wanted to test a hypothesis: if automated conversion works for the messiest, least-structured video (factory floors are chaos), it’ll work for anything.
More importantly, I built it as reusable infrastructure. The same pipeline that processed factory videos can handle warehouse footage, kitchen tasks, or any egocentric video. The architecture is modular. The format is standard. The tools are open.
This is Phase 1 of building data infrastructure for Physical AI. Let me show you how it works.
Technical Architecture: Five-Stage Pipeline
Figure 1: ego2robot Pipeline: 5 Stage Architecture
Stage 1: Intelligent Curation
Raw egocentric video is mostly noise. A 433-second factory clip contains maybe 60 seconds of actual manipulation - the rest is idle time, or poor camera angles.
I built quality filters that automatically identify valuable sequences:
Motion scoring using frame differencing (OpenCV) detects active manipulation vs. static scenes
Hand visibility detection with MediaPipe tracks when hands are engaged with objects
Threshold filters (motion >0.1, hand visibility >30%) keep only manipulation-dense clips
Result: 433 seconds → 60 seconds of usable data. 7x efficiency gain without manual annotation.
Automated quality filtering is infrastructure that enables scale. You can’t manually review every video and expect to process 10,000 hours. I almost built a supervised classifier here. Then I realized, that doesn’t scale. The whole point is to process video faster than humans can label it.
Stage 2: Semantic Understanding
Foundation models excel at transfer learning. I leveraged pretrained models to extract semantic meaning without dataset-specific training:
VideoMAE embeddings (768-dimensional vectors) capture temporal patterns in manipulation sequences. These embeddings cluster naturally — similar actions produce similar representations, even across different objects or contexts.
CLIP zero-shot classification maps video frames to action descriptions. I defined 10 action prompts (”picking up an object,” “inspecting quality,” “tightening a screw”) and let CLIP assign labels based on visual similarity. No manual annotation. No fine-tuning.
The combination produces rich semantic signals: embeddings for clustering, labels for interpretability. Foundation models trained on general video/images transfer remarkably well to industrial footage — VideoMAE and CLIP weren’t trained on factories, yet they extract meaningful representations.
Stage 3: Unsupervised Skill Discovery
Figure 2: t-SNE skill cluster visualization from sample dataset
Figure 3: CLIP assigned many of the clusters as “Quality Inspection”.
I ran K-means clustering (k=10) on VideoMAE embeddings and discovered something interesting: hierarchical skill structure.
CLIP assigned “Quality Inspection” to 6 different clusters. At first glance, this seems redundant. But examining the clusters revealed meaningful subtypes:
Cluster 3: High hand visibility (0.67) — close-up tactile inspection
Clusters 5-8: Medium hand visibility (0.42-0.58) — visual inspection variants
Cluster 0: Mixed inspection + machine operation
The clusters discovered execution variations that semantic labels can’t capture. Two workers performing “inspection” might approach the task differently — one through touch, one visually. Those differences matter for robot learning.
This hierarchical structure (coarse semantic labels + fine-grained visual patterns) is exactly what foundation models need. It’s the difference between “inspect a part” (task specification) and “here are 6 ways humans actually do it” (execution diversity).
The discovered distribution also validates the pipeline: 50% quality inspection, 34% assembly/fastening, 8% machine operation. This matches real manufacturing operations where QC is continuous. The pipeline found ground truth by analyzing patterns, not by being programmed with prior knowledge.
This was the most surprising result. I expected the clusters to be noise. Instead, they found real patterns I wouldn’t have thought to annotate manually.
Stage 4: Robot-Compatible Export
Converting human video to robot format requires honest framing. I’m not claiming to generate robot joint trajectories from human hand motion; that’s a hard kinematics problem. Instead, I generate pseudo-actions suitable for representation learning:
Hand tracking (MediaPipe) extracts wrist positions across frames, producing 2D motion trajectories. I compute frame-to-frame deltas, normalize to image dimensions, and clip to [-1, 1] range. These become the “actions” in the dataset.
LeRobot v3 format structures episodes as sequences of (observation, action) pairs:
Observations: RGB frames (360×640 @ 6fps) + hand bounding boxes
Actions: 2D hand motion vectors [Δx, Δy]
Metadata: Skill cluster IDs, quality scores, zero-shot labels
Why LeRobot format? Interoperability. Any tool that reads LeRobot datasets can use this data. Physical Intelligence’s π₀ can load it. Researchers can experiment with it. Future models can pretrain on it.
Format standardization is infrastructure. When everyone uses the same data structure, tools become composable. That’s how ecosystems scale.
Stage 5: Validation
I built a simple CNN to predict actions from observations and trained it on the dataset. After 10 epochs: 0.01 MSE loss (roughly 90% accuracy in predicting hand motion direction and magnitude). Validation loss was slightly lower than training loss, indicating good generalization.
This proves the dataset contains learnable patterns. The pipeline didn’t just shuffle frames around; it extracted meaningful structure that models can learn from.
Why This Matters Beyond One Dataset
The Data Flywheel for Physical AI
Foundation model ecosystems scale through flywheels:
Open infrastructure enables community contribution
Diverse data improves model performance
Better models attract more developers
More developers generate more applications
More applications produce more usage data
Loop closes: better data → better models
Right now, Physical AI is stuck at step 1. We don’t have the infrastructure layer that enables community data contribution at scale.
ego2robot demonstrates what that infrastructure looks like:
Automated curation (no manual review bottleneck)
Semantic extraction (pretrained models transfer well)
Standard formats (interoperability across tools)
Validation methods (prove dataset quality)
This pattern is reusable. The same architecture that processed factory videos can handle warehouse operations, kitchen tasks, construction work, or medical procedures. Each domain adds diversity. Each dataset compounds in value.
Platform Economics vs. Data Collection
Consider two approaches to scaling robot training data:
Approach A: Centralized Collection
One organization collects all demonstrations
Cost: $100-500/hour × 10,000 hours = $1-5M
Timeline: Years to collect diverse data
Result: Single organization owns data
Approach B: Infrastructure-Enabled Distribution
Build tools that let anyone convert video → datasets
Cost: Tool development (one-time) + compute (marginal)
Timeline: Immediate access to existing video archives
Result: Community ownership, exponential scaling
Approach B requires infrastructure. ego2robot is that infrastructure for egocentric video. But the pattern generalizes: tools that reduce conversion costs by 10-100x unlock orders of magnitude more data.
The value isn’t in owning one dataset. It’s in enabling a thousand datasets to exist.
Where the Ecosystem Is Headed
Three trends are converging:
1. Foundation models need diverse pretraining data
Physical Intelligence’s π₀ paper shows that visual diversity improves generalization. But diverse robot data doesn’t exist at scale. One path forward: pretrain on diverse human video, fine-tune on robot-specific demonstrations.
2. Spatial intelligence unlocks 3D understanding
World Labs launched Marble to generate 3D worlds from 2D images. That capability bridges human video → robot simulation. Feed egocentric clips into spatial intelligence models → generate 3D training environments → simulate robot policies before deployment.
3. Developer platforms determine market structure
The robotics companies that win won’t just build models. They’ll build platforms where developers can convert their domain data to standard formats, pretrain on diverse datasets, fine-tune on task-specific demonstrations, and deploy policies to hardware.
LeRobot is building this for open-source. Physical Intelligence might build it commercially. Either way, data infrastructure is the foundation layer.
Key Insights from Building This
Foundation model transfer works better than expected. VideoMAE and CLIP weren’t trained on factory footage, yet they extract meaningful representations. This suggests general-purpose visual pretraining is more valuable than domain-specific fine-tuning for many applications.
Quality filters compound returns. Automated curation reduced data volume 7x while improving relevance. Manual review would have been a bottleneck. Investing in quality scoring infrastructure pays dividends at scale.
Format standardization enables composition. By exporting to LeRobot format, this dataset immediately works with existing tools. That interoperability is more valuable than any custom optimization.
Infrastructure scales differently than content. Building one dataset has linear value. Building tools that enable hundreds of datasets has exponential value. Always ask: “Am I creating a single artifact or reusable infrastructure?”
Execution velocity matters. I built this in two weeks because I used pretrained models, streamed data from Hugging Face, and focused on the 20% of work that delivers 80% of value. Starting with the target format (LeRobot v3) forced architectural decisions early and prevented rework. Validating continuously at each stage — motion scoring on 3 clips, embeddings on 5 videos, clustering on 60 samples — caught issues before they compounded.
Where This Goes Next
I built ego2robot to solve a specific problem: converting factory video into robot datasets. The architecture, however, is general-purpose. The quality filters, semantic extraction, and format conversion patterns apply to any egocentric video.
The goal was to “build the infrastructure layer that makes video-to-dataset conversion 10x cheaper and 100x more accessible.”
What that infrastructure enables:
Researchers can port new domains (warehouses, kitchens, hospitals) without reimplementing the pipeline
Companies can convert their proprietary video into training data using standard tools
The community can contribute diverse datasets in compatible formats
Foundation models get the data diversity they need to generalize
This project showed that automated video-to-dataset conversion works at quality sufficient for representation learning.
For Researchers and Builders
If you’re working on robot foundation models, VLA architectures, or embodied AI:
Use this dataset: Hugging Face Hub
50 episodes of factory manipulation, LeRobot v3 format, ready for pretraining experiments.
Extend this pipeline: GitHub
Open-source tools for video curation, semantic extraction, and format conversion. Contributions welcome for:
Additional domains (Ego4D, EPIC-KITCHENS, Assembly101)
Improved features (depth estimation, 3D trajectories)
Better evaluation (dataset quality metrics, benchmarks)
Share your results: If you train on this data or build on these tools, I’d love to hear what works. The fastest way to validate this approach is through real usage.
If you’re working on this stuff and think I’m missing something obvious, tell me. I’m building in public specifically to get that feedback.
Contact:
Email: michelle@aetherone.xyz
LinkedIn: linkedin.com/in/sunmichelle
Twitter: @michellelsun
GitHub: github.com/msunbot/ego2robot
Conclusion
Physical AI is where NLP was in 2018 — foundation models are emerging, but data infrastructure is still nascent. The companies that build the best data pipelines, developer tools, and ecosystem primitives will shape how the field develops.
ego2robot is one piece of that infrastructure. It demonstrates that automated video-to-dataset conversion is feasible, cost-effective, and valuable. More importantly, it shows the pattern: build reusable tools, use standard formats, enable community contribution.
That pattern scales. The question is: what infrastructure does the ecosystem need next?
I’m figuring that out by building, shipping, and listening to what creates value. This is Phase 1.
Built with: PyTorch, Transformers, MediaPipe, OpenCV, scikit-learn
Timeline: 2 weeks focused work
Open source: MIT License (code), Apache 2.0 (dataset)