The Stack We’re Building On
The Physical AI stack in 2025 is converging around a powerful idea: train generalist models directly on sensorimotor data - video in, actions out.
This approach has produced real progress. Take Generalist AI’s GEN-0, trained on 270K hours of real world manipulation data and Physical Intelligence’s recent breakthrough, showing robot foundation models improve from human videos after sufficient robot data pretraining. Large-scale manipulation datasets and cross-domain pretraining have unlocked levels of generalization that seemed out of reach just a few years ago.
But as these systems scale, across more data versions, more model iterations, more deployment context, a different class of problems emerges. Teams can see that behavior changes, but struggle to explain where, when, or why those changes occur.
Progress stops being gated by model capacity. It becomes gated by understanding.
The question becomes “what changed, and where did it change?”
In software and LLM systems, regressions are caught with explicit evaluation suites. Physical AI adds a harder dimension: behavior unfolds over time, across variable physical contexts. Regression doesn’t always show up as outright failures. They appear in when or how a behavior unfolds.
A grasp may fail only during the approach phase. A navigation policy may improve on short routes but degrade once trajectories extend past 30 seconds.
As robots move to real-world deployment, the inability to confidently answer “what changed and why?” becomes a bottleneck of reliable iteration.
Where the Stack Shows Strain
Let me make this concrete with 3 scenarios.
Scenario 1: The Doorway Regression (Localized Temporal Failure)
You’ve trained model v3. Success rates are up 8%. Three weeks post-deployment, the robot hesitates at doorways.
Roll back to v2: the behavior returns. But what changed: New perception? Dataset composition? Where in the temporal sequence did it break?
Watching rollout videos doesn’t give systematic answers.
Scenario 2: Dataset Growth without Behavioral Visibility
Your dataset grew from 100 to 10,000 hours. Coverage improves - But of what? How much of that data reflects successful manipulation versus partial attempts? When manipulation fails, does it happen during approach, contact, or execution?
Without structured temporal units, dataset growth is measurable only in hours, not in behavior.
Scenario 3: Debugging Without Temporal Handles
A grasp fails. You have a 45-second rollout. The behavior clearly has phrases - approach, alignment, contact, lift - but those phases aren’t explicit or queryable.
You can watch the video, but you can’t systematically compare this failure to others or determine where the divergence occurred.
This is the pattern I kept hitting when building data pipelines for robot learning. Once behavior changed, I couldn’t localize where in time it mattered. Improvements and regressions blurred inside long trajectories.
The failure wasn’t lack of signal - it was lack of structure.
The more I tried to evaluate across dataset versions, the clearer it became: the system was missing a representation of time that supports comparison.
Large, well-funded teams likely solve versions of this internally - just as companies built feature stores long before Tecton made them a recognized architectural layer.
When feature stores became explicit, they didn’t introduce a new idea; they gave teams shared vocabulary and structure.
Today, temporal abstraction is treated as an internal data preprocessing problem - built ad-hoc, hack together in the pipeline. Physical AI needs a similar shift: recognizing it as a first-class architectural layer that fundamentally shapes how we evaluate, debug, and iterate on embodied systems.
This is what this essay explores: what temporal abstraction is, why it matters architecturally, and what it looks like when built systematically.
III: The Missing Abstraction - Temporal Structure as a First-Class Concept
Software systems have functions, classes, and modules. ML systems have datasets, features, and metrics. Physical AI systems have raw video and model weights.
What is missing: No first-class representation of temporal units with behavioral semantics.
On one hand, raw videos are too fine-grained; frames tell you little about behavior. On the other hand, full episodes are too coarse; sub-behaviors cannot be isolated. Models learn temporal weights, but those patterns live in weights. You can’t compose them into new training sets, compare them across model versions, or debug them when behavior changes.
A parallel: when databases became first-class, applications moved beyond flat files. Instead of just reading and writing, databases provided a structured way to think about data, query systematically and reason about relationships.
Feature stores followed the same evolution. Teams had features living in ad-hoc scripts. Tecton made them first-class: versioned, queryable, reusable. That shift enabled systematic feature management and sparked an ecosystem.
Temporal abstraction needs the same elevation. Instead of being thought of as a preprocessing step, they are architectural primitives that enable systematic evaluation and debugging of embodied behavior.
It lets you answer questions like: “Why did doorway navigation regress in v3?”, “Show me all manipulation attempts where contact succeeded but grasping failed”.
So what does this layer actually look like?
IV: What Temporal Abstraction Looks Like
Take a robot navigating through a doorway - the scenario from earlier. Instead of treating it as an opaque video stream, represent it as structured temporal units:
Episodes as units of behavior - Not just “video files” but semantic containers: “hallway approach”, “threshold alignment”, “doorway crossing”
Events as behavioral boundaries - Meaningful state transitions that mark phase changes: “obstacle detected”, “speed reduction initiated”, “threshold crossed”, “collision avoidance triggered”.
Slices as temporal queries - “Show me all doorway crossings where the robot hesitated”, “compare threshold detection behavior between model versions”, “find episodes where approach succeeded but crossing failed”.
These architectural primitives enable:
Evaluation at behavioral granularity: Compare performance at the episode/event level, not just aggregate success rates
Temporal debugging: Isolate where in the sequence behavior diverged between model versions
Dataset provenance: Track which episodes contributed to which capabilities
Compositional construction: Build new training sets from episodes with specific temporal properties
The primitives - episodes, events, slices - are what make temporal structure queryable, versionable, and composable. Any system that takes temporal abstraction seriously as an architectural layer needs these primitives
V: The Counterarguments: Why not just better models? Does Structure Limit Discovery?
“Don’t bigger models solve this?”
Models do learn temporal patterns as they scale. But learned representations aren’t queryable, versionable or composable.
You can’t ask: “Show me where behavior changed between model versions.
You can’t construct “a dataset of only successful grasps in specific contexts”
The distinction: Implicit representation (in weights) vs. explicit abstraction (in the data layer). If you only care about task performance, implicit might suffice. But to understand, debug, and improve systematically, you need explicit structure.
“Doesn’t structure limit discovery?
The concern is real. What if the “right” segmentation emerges only after training, not before?
Premature abstraction can be worse than no abstraction. But the alternative, no explicit structure at all, has its own cost: you can’t query, compare, or compose behavior systematically.
The key is making temporal abstraction revisable, not fixed:
Versioned: so you can refine them as understanding improved
Queryable: so models can still process raw data if needed
Compositional: so new boundaries can be defined without rebuilding everything
The goal isn’t rigid ontology. It’s making temporal semantics explicit enough to support engineering, while remaining flexible as understanding evolves.
Structure makes discovery legible and reproducible.
VI: What Temporal Abstraction Enables
When temporal structure becomes first-class, a powerful shift happens:
Evaluation becomes comparative, not absolute
Dataset growth becomes additive and managed, not chaotic
Model changes become diagnosable, not opaque
More concretely, immediate applications include:
Behavioral regression testing: “Did this change break doorway navigation?”
Dataset construction: “Give me episodes where X succeeded but Y failed”
Temporal debugging: “At what point in this sequence did the robot lose track?”
Provenance tracking: “Which subset of training data led to this capacity?”
Slice-level evaluation: Compare performance across behaviorally meaningful segments.
This changes how datasets grow. New data stops being “more footage” and becomes additions to specific behavioral capabilities. You can see what coverage you’re gaining, what gaps remain, and what regressions emerge.
For example, you might discover:
A navigation model that improved overall success by 8% but now hesitates specifically at doorway thresholds (the “crossing” phase from Scenario 1)
Threshold detection that works reliably in wide doorways but fails when width drops below 90cm
Approach behavior that improved in hallways but regressed at the moment of alignment with the door frame
None of these patterns show up in aggregate metrics. They only become visible with episode and event-level evaluation.
That made iteration safer, and regressions easier to detect.
Open Questions and Design Space
I would be curious to hear readers weigh in on:
Granularity: How fine-grained should temporal units be?
Automation: How much can be inferred vs. must be specified?
Standardization: Should the ecosystem converge on shared primitives?
Model integration: Should models consume explicit temporal structure, or just data pipelines?
Tradeoffs: Explicit structure adds complexity - when is it worth it? When does structure help vs. hurt exploration?
Where We Go from Here
This is not about abandoning end-to-end learning. It’s about recognizing where in the stack temporal abstraction belongs.
Different teams will implement this differently, and should. The important thing: making temporal structure first-class, rather than implicit.
As more robots are being deployed to homes, factories, and fields, the need for reliable evaluation and confident iteration will only increase. Temporal abstraction is one step toward that reliability.
The specifics, however, are yet to be sketched out: What are the right primitives? How do they compose? What does infrastructure look like that takes this seriously?
The next phase of embodied AI is not just better models - it’s better understanding of behavior over time.
If you’re building Physical AI systems, I’m curious whether this resonates - or where you think this framing breaks down.