When I evaluate a robotics company, I start with three questions:
What task does the robot perform?
Which part of the system is holding it back?
What evidence proves it works repeatedly at the required economics?
I built the task-first Physical AI stack as my framework for answering these questions.
It maps 11 functional layers, from the part that touches the work, through actuation, power, intelligence. It can be used to locate a bottleneck in a system, to understand where a company competes, and to follow where the value accrues.
What is the physical AI stack
The physical AI stack is the set of hardware, software, data and operating systems that turns a goal into a reliable action in the physical world.
Most technology stacks are drawn from the enabling technology outward: chips, compute, models, applications. For robots, I find it more intuitive to begin with the task. I call this the task-first stack.
Start with a job such as placing a product in a bin. A gripper touches the product, an actuator moves the gripper, the power system supplies the required energy, the control software regulates the motion, planning finds a feasible path, perception locates the object and bin, a policy selects the action, and data trains the model.
There’s also compute, simulation and evaluation, which we’ll cover in this piece.
These 11 layers map the working system behind a robot task. Beneath them is the industrial foundation: energy, critical materials, manufacturing capacity and physical infrastructure.
The task-first stack changes the starting question. Instead of asking how advanced a model has to be, ask what has to go right for the task to succeed. The answer then points us to where the bottleneck is.
One task through all 11 layers
Recently Stanford student and OpenAI Robotics intern Thijs Simonian gave GPT-6 Astra an SO-101 robot arm, a paintbrush and a camera, then asked it to paint the Golden Gate Bridge. Astra planned one minute of actions at a time, reviewed the result and adjusted across attempts with feedback.
This is a good example of how the stack works and identifies the bottleneck. We will also go over 3 additional examples at the end of the piece.
For this Golden Gate Bridge painting task, the bottleneck is the control and actuation interface between Astra and the robot. The model (GPT-6 Astra) could recognize what needed improvement turn after turn, but the robot’s control interface, actuation and limited feedback limited how precisely the arm could execute the model’s judgment.
Don’t worry if the table doesn’t make sense to you right now. We’ll walk down the stack one layer at a time. After that, refer back to the table and it will make more sense.
1. What is a robot end effector? Task meets the world
A robot’s end effector is the part that touches the work. Common forms are a parallel-jaw gripper, vacuum cup, magnetic tool and multi-finger hand. Its geometry, sensing and materials determine which objects and processes the robot can physically handle.
The market spans task-specific industrial tooling and generalized dexterous hands. SCHUNK, Robotiq and OnRobot sell grippers and end-of-arm tooling. Shadow Robot, Wuji Hand and Sharpa develop multi-finger, or dexterous hands. I wrote about the designs and economics in Dexterous Hand Primer:
An end effector needs to manipulate objects at the required speed, force, tolerance, uptime and price. A simple gripper is sufficient for pick-and-place tasks. A dexterous hand widens the task set, but also adds joints, weight and cost.
Tactile and force-torque sensing sit close to this layer, or you can think of sensing as layer 1b. These sensors tell the robot whether contact is secure, excessive or beginning to slip.
What matters: what touches the work, and what tasks does that choice open up or rule out?
2. What are robot actuators? Turn command into force
Actuators convert electrical commands into movement. The actuator shapes the force, speed, precision, efficiency and thermal performance a robot can achieve.
Humanoid joints have to reconcile high torque, low weight, compactness, low backlash, reliability, cost and manufacturability.
I saw this layer up close in a precision gear factory in Songgang, Shenzhen. 26 wire-cutting machines ran around the clock. Staff inspected parts to tight tolerances. The joint’s eventual force, precision and reliability began in processes like these.
An actuator that is heavy, expensive or thermally constrained can limit the system even as the software improves.
What matters: can every joint deliver the torque, precision and thermal margin the task requires, at a weight and cost the robot can carry?
3. What powers a robot? Store and deliver energy
The power system provides energy to the actuators, compute, sensors and auxiliary systems. For mobile robots, this usually begins with a battery pack and extends through the battery-management system, power electronics, wiring, protection and charging interface.
Capacity determines how long the robot can work between charges. Pack weight competes directly with payload, while charging time and cycle life shape fleet utilization and replacement cost.
The power system can constrain runtime through charging frequency, battery swaps and thermal limits.
What matters: can the power system sustain the task’s peak load? What is the runtime of the robot?
4. What is robot control? Move toward the target
Control is the software layer that converts a desired motion into the continuous stream of motor commands that produces it. The control loop recalculates the required position, velocity, torque or motor current hundreds to thousands of times a second on a real-time processor. It has to absorb a late measurement, a small collision, a heavy object or a battery-voltage change while keeping the robot stable and within its limits.
Two common approaches are (1) PID, proportional-integral-derivative, the classical feedback loop: it measures the gap between where a joint is and where it should be, then corrects in proportion to that gap; and (2) model predictive control, or MPC, which simulates the effect of potential commands and picks one that satisfies constraints such as torque limits, balance, collision clearance or energy use.
Much of this layer is available through open-source libraries and vendor tools, including ros2_control, Drake and NVIDIA Isaac. Robot companies still have to integrate and tune the loops for their mechanics, tasks and safety limits. Companies such as Applied Intuition and Intrinsic also sell parts of the broader robotics software and tooling stack.
What matters: when something unexpected happens, what keeps the robot safe and on task?
5. What is robot motion planning? Choose a feasible path
Motion planning is software that selects a feasible sequence of movements, from the robot’s current state to a goal. For a manipulator, that may include choosing a grasp, or placing the object without colliding with the shelf. For a mobile robot, it may include route planning and task sequencing.
Classical methods include graph search, inverse kinematics, sampling-based planners such as rapidly exploring random trees, and trajectory optimization. MoveIt is an open-source manipulation framework. MoveIt Pro packages a commercially supported workflow. NVIDIA’s cuMotion library accelerates trajectory optimization on GPUs.
Planning and control are adjacent but serve different purposes. Planning decides what movement to attempt. Control makes the robot follow it in real time. This guide traces the dependencies from the object, so control appears first. During the task, information generally flows from perception to planning to control.
What matters: can the robot find a safe, efficient path for the current scene, and recover when that path fails?
6. What is robot perception? Estimate what is happening now
Robot perception turns sensor readings into an estimate of the robot and its environment. Cameras, depth sensors, LiDAR, force-torque sensors, tactile sensors, encoders and IMUs all contribute different information. A camera may identify a package. A wrist force sensor may reveal that it is snagged. Tactile sensing may show that it has begun to slip.
Perception answers a present-tense question: where are the object, robot and relevant contacts now? A model can use that information, together with instruction and goal, to decide what to do next.
Some perception functions are available as libraries, including object detection, pose estimation, visual odometry, SLAM and sensor fusion. NVIDIA Isaac ROS is a packaged example. For deployment, the robot still depends on sensor calibration, latency and edge cases of the target environment.
Control, planning and perception usually run on the robot. Perception and planning commonly run on the main onboard computer, while the fastest control loops may run closer to the joints.
What matters: what does the robot need to observe, measure and infer before it can act reliably?
7. What are VLMs, VLAs, world models in robotics?
Models interpret observations, reason about a goal, predict outcomes or generate actions.
A vision-language model, or VLM, processes visual and language inputs. In robotics, it can interpret a scene and an instruction and produce a high level plan.
A vision-language-action model, or VLA, combines vision and language inputs with an action output. Physical Intelligence’s π models, Google DeepMind’s Gemini Robotics and NVIDIA’s GR00T are examples.
A world model learns how a scene may evolve. Given a current state and a possible action, it predicts the future state. NVIDIA Cosmos and World Labs’ Marble are different approaches in this broad category.
A world action model, or WAM, is a world model that also outputs actions. Dyna Robotics’ DYNA-2 is one example. NVIDIA’s Cosmos 3 also includes policy variants that connect video prediction with robot actions.
Perception estimates the present, a world model predicts possible futures and a world action model ties those futures to executable choices.
The model is absorbing the stack. These models are increasingly taking on work that previously sat in the perception and planning layers. For example, a VLA may generate a high-level action chunk directly from camera input and an instruction, bypassing several hand-built perception and planning interfaces. It may also generate lower-level actions, but fast motor control still sits underneath it. A world action model can predict how candidate actions change the scene, which can support or replace part of a planner’s work.
What matters: which decision does the model make, and which decisions remain with conventional software and control?
8. What data trains physical AI systems?
There are 5 main types of robot-learning data:
Robot demonstrations (or teleoperation data): A human controls a robot and records observations and actions together.
Human demonstrations (or egocentric data): Humans record themselves completing tasks.
Simulation: A virtual environment generates trajectories, labels and failures under controlled conditions.
Internet and video data: YouTube and other online videos provide demonstrations at much larger scale than robot fleets.
Deployment data: success, failure and recovery data from robots doing the actual work
I wrote about robot data here.
Volume is only part of the equation. The data also need to be relevant to the task, embodiment, sensor and environment. Failure data and variety of data are also key.
A whole universe of companies is forming around this layer. XDOF builds robotics data infrastructure and datasets. Mecka describes itself as a data, evaluation and deployment layer for physical AI, with a focus on egocentric human activity.
What matters: who can produce data that is relevant to this robot’s work, and how does that data improve the next release?
9. Where does physical AI compute run?
Robotics uses at least three kinds of compute:
(1) Cloud or datacenter compute trains models, processes fleets of data and runs large simulation workloads away from the robot.
(2) Onboard accelerated compute handles latency-sensitive perception, model inference and sometimes planning. NVIDIA Jetson and Jetson Thor are examples of this tier. Qualcomm, AMD and specialized edge-compute vendors also make processors for this tier. These computers need to fit the robot’s power, heat, weight and cost budgets.
(3) Real-time embedded compute executes fast control and safety functions on motor drives, microcontrollers or real-time processors. Texas Instruments, NXP, Infineon, STMicroelectronics and motor-drive vendors make processors and control components for this tier.
What matters: which computation must happen on the robot, and can the hardware deliver it inside the latency, power and thermal budget?
10. What is robotics simulation used for?
Simulation creates a controlled environment for design, training, integration and stress testing before a policy reaches the real world.
Simulation can generate variation that is expensive or unsafe to capture on hardware. It can also test an integrated stack before a physical robot is available at scale.
The sim-to-real gap is the mismatch between the simulated environment and the physical one. Contact physics, material wear, lighting, sensor noise are common sources of mismatch. A policy can perform well in simulation, and still fail in a real deployment.
MuJoCo is a physics engine widely used in research. Gazebo supplies open-source physics, rendering and sensor simulation. NVIDIA Isaac Sim connects simulation with NVIDIA’s broader robot-development workflow.
What matters: what did the team validate in simulation, and which real-world conditions remain outside that environment?
11. How should physical AI systems be evaluated?
Evaluation measures the gap between reality and expectation. For example, whether the system works across changing conditions, recovers from errors and makes economic sense. A useful scorecard includes task success, intervention rate, recovery behavior, cycle time, latency, energy consumption, safety and cost per successful task.
Evaluation also closes the learning loop. Failure helps inform the model training, data collection, hardware or workflow adjustments.
What matters: how do we tell that this system works in the target environment and business case?
Where does value accrue in the physical AI stack?
Companies create the most value when they solve the bottleneck holding back an important task.
Better models reduce the burden on hand-built perception and planning. Better sensing, end effectors, actuators and power systems make more behaviors possible or keep them running longer. More deployment produces more interaction data. That data then improves the model.
The cycle is better robot → more deployment → more data → better model → better robot.
The flywheel turns only as fast as its slowest layer. For one company that slowest layer is data collection. For another it is the hand, the actuator, the power system, edge compute, safety validation or the cost of running the robot at all.
The most valuable part of the stack can change as technology improves. Today, the main limitation may be the model and data pipeline. As models improve, the bottleneck may move to the actuator, sensor, or deployment network.
How can investors and operators use the physical AI stack?
Start with a specific task and ask 3 questions:
a. Where does the system learn? Trace who generates interaction data, who owns it and how it reaches the next version.
b. What limits the task today? Locate the constraint in the model, sensing, mechanics, power, reliability or unit economics.
c. What evidence would establish deployment? Ask for the operating conditions, task success, recovery behavior, cycle time, energy use and the economics of repeated operation.
The task-first stack breaks down a robot demo into a chain of parts that can be examined individually. Trace the task from physical contact back through the stack. It is then easier to see what the system depends on, where a company may have an advantage and where the robot is most likely to fail.
For paid readers, below I apply the stack to three tasks and surface three different bottlenecks: warehouse bin picking, inserting a connector and loading a dishwasher.
