When the robot doorway navigation had regressed between model versions, the question wasn’t “did performance drop?”, it was “which specific failure mode emerged, and when?
Temporal abstraction as infrastructure allowed this to be answered in a query: filter episodes where "doorway_approach" succeeded but "threshold_crossing" failed, compare model v2.1 vs. v2.2, show me those exact sequences. In seconds, the answer came back: 23 episodes, all involving narrow doorways with partial occlusion. We could click into episodes, inspect frames where contact was detected, and see exactly what changed.
This is what temporal abstraction looks like as infrastructure. Recent work on failure diagnosis notes that researchers typically address these questions "using heuristic approaches, which are labor intensive and costly" (RoboMD, 2025). Structured episodes with temporal events offer a more systematic path.
Introducing ConceptOps
ConceptOps is a Physical AI data engine that implements temporal abstraction as infrastructure. It takes raw robot demo videos and turns them into structured episodes, with detected events, making robot experience queryable and inspectable.
Core capabilities in v1:
Ingests raw egocentric video
Runs perception normalization (SAM-3 segmentation or fallback)
Detects temporal events using heuristics and simple learned components
Creates structured episodes with time-stamped event markers
Generates timeline visualization where you can click events to inspect frames
Exports to standard formats (RLDS, LeRobot, COCO) with preserved provenance
This is a reference implementation that makes episodes, events and provenance concrete, to instantiate the framework of temporal abstraction.
VLA research emphasizes that "curating robot training datasets is critical...mere data volume does not guarantee generalization." ConceptOps explores whether structured episodes with temporal events can enable more efficient curation workflows.
Episodes as First-Class Objects
An episode in ConceptOps is a complete interaction sequence with temporal bounds, semantic context, and outcome metadata. It’s the fundamental unit of robot experience.
Each episode contains the full temporal scan (start/end timestamps), video segments, detected events with precise timing, and a provenance chain tracking which processing pipeline created this episode version.
RLDS stores episodes as flat step sequences; ConceptOps adds temporal event structure within those sequences.
Design decision: We made episodes immutable with versioned metadata, rather than mutable records. Once created, an episode doesn’t change. New analysis or reprocessing creates a new episode version. This provides provenance clarity: when you query “all failed grasping episodes from model v3”, you get exactly the episodes as they existed when that model ran, not episodes that have been modified by subsequent processing.
Example: A “pick and place” episode spans 34 seconds from approach through final placement. It includes events for “approach_started” (0:02), “grasp_attempted” (0:08), “grasp_confirmed” (0:11), “transport_started” (0:12), “placement_attempted” (0:29), “placement_failed” (0:31). The episode is tagged with failure mode “unstable_grasp” and tracks that it came from model v2.3, processed with segmentation pipeline v1.2.
You can now retroactively query “show me all episodes where grasp succeeded but placement failed” or “ The structure enables questions you couldn’t easily ask with video files and spreadsheets.
Events as Temporal Markers
Events mark semantically meaningful moments within episodes: object contacted, grasp confirmed, motion phase changed, placement attempted. They add queryable structure to what would otherwise be continuous video.
For v1, events are detected using a combination of heuristics (motion analysis, segmentation change detection) and simple learned components. The event taxonomy is intentionally small - pick, move, place, contact, please a few task-specific markers - because we’re testing whether this structure approach is useful before claiming a universal taxonomy.
This shows detected events across a 90-second manipulation sequence. Each event has precise frame boundaries - the first manipulation phase spans frames 0-6, the second frames 6-12, and the third frames 12-18. This temporal structure enables filtering episodes by event presence and comparing event timing across model versions.
Design decision: Events are detected autonomically, not manually annotated. The goal is to build infrastructure that works on thousands of episodes. The tradeoff is that v1 event detection is limited: it handles basic manipulation events well, struggles with task-specific semantics (is this “drawer_opened” or just “motion_detected”?), and doesn’t generalize easily across different robot morphologies.
Event detection is still an open problem. v1 shows one approach; other teams might use vision-language models, learned temporal segmentation, or domain-specific heuristics. What matters is that events become first-class objects with timestamps and labels, enabling filtering and inspection.
In the pick and place example, events for “approach_started”, “grasp_attempted”, “transport_started” let us filter episodes by which phase succeeded or failed - impossible without temporal markers.
Provenance and Inspectability
Provenance tracks which model version generated this data, which perception pipeline processed it, which dataset version it belongs to and when these operations happened. In ConceptOps, provenance is immutable metadata attached directly to episodes and events.
When an episode exports to RLDS or LeRobot format, provenance metadata exports alongside it. This enables tracing data lineage even after it’s left ConceptOps and entered the training pipeline.
Why this matters: When performance regressed in the doorway navigation example, we could trace back to find that 847 training episodes came from model v2.1 during a specific deployment window. Without provenance as infrastructure, this requires manual log correlation and guesswork.
Inspectability is the counterpart to provenance: structured data only helps if you can actually inspect it. Episode timelines show detected events with timestamps, enabling filtering by event presence and comparing timing across processing versions.
Integration and Interoperability
ConceptOps sits between raw data collection and model training. It’s the structured layer where robot experience becomes queryable.
Input: Ingests raw egocentric video. Can also import existing RLDS datasets. ConceptOps adds temporal event structure to RLDS's episode and step format. The goal is to work with data teams already have, not force migration to a new format.
Output: Exports episodes to RLDS, LeRobot, and COCO formats with preserved episode structure and provenance metadata. Your training pipeline consumes standard formats; the temporal structure comes along for free.
Where ConceptOps fits in the stack: Most teams use ConceptOps as an evaluation and dataset curation layer. Episodes flow in from deployed robots or simulation, events get detected, you can filter and inspect to understand failure modes, then export curated subsets for training. Episodes can be queried via API for batch evaluation, regression detection or automated dataset curation workflows. It’s the dataset system of record that makes robot experience structured and queryable.
Design Decisions and Tradeoffs
Immutability: We chose immutable episodes with versioned metadata. Each reprocessing creates a new version of episodes. This trades update flexibility for provenance clarity. We prioritized the ability to support evaluation and query across model versions, eg, you can easily query “show me data as it existed for model v3”.
Autonomic Event detection: We built automatic detection using heuristics and simple models rather than requiring manual annotation. This trades detection accuracy for scalability. We chose imperfect detection to prove a structured approach works.
Generic primitives: Episodes and events are intentionally generic, designed to work across different manipulation tasks, not optimized for any single tasks. This trades task-specific performance for generality. A grasping-only system could have richer grasp-specific event types; we chose breadth over depth for v1.
Batch vs. streaming: ConceptOps is optimized for batch evaluation workloads, not real-time streaming. Processing takes seconds to minutes depending on video length. We prioritized analysis over real-time performance.
Small taxonomy: The v1 event taxonomy is deliberately minimal. We are testing whether structured episodes with basic events are useful before claiming to have the right universal taxonomy.
Open Questions
Several questions remain open:
Should event taxonomies standardize across the ecosystem, or should they remain task-specific? There’s trade-offs between interoperability (shared vocabulary helps tools compose) and flexibility (every task has unique semantics).
What is the right granularity for events? “Grasp” vs. “finger_contact” vs. “Force_threshold_exceeded”. Too coarse you lose useful structure; too fine and the taxonomy explodes.
How should episode boundaries be defined for continuous operation? v1 assumes discrete demonstrations with clear start/end, while product robots operate continuously.
If you’re building Physical AI systems and working through these problems, I would love to hear how you’re approaching them.