Physical AI: The Convergence of Foundation Models, Sim-to-Real Transfer, and Embodied Intelligence

Robot learning in the real world necessarily confronts incomplete information. Friction coefficients, object mass distributions, and sensor noise cannot be directly observed. Therefore, Physical AI problems are formulated as Partially Observable Markov Decision Processes (POMDPs) [4-5].

Definition 1 (POMDP): A POMDP is defined by the tuple ℳ=(S,A,T,R,Ω,O,γ) where:

The robot’s objective is to find a policy π(a_t|o_1:t) that maximizes expected cumulative reward:

where τ = (s₀), a₀), o₁, s₁, a₁), …) is a trajectory.

The optimal policy must depend only on the history of observations o_1:t and actions a_1:t−1 since the true state s_t is not observable. Thus, the optimal policy operates in belief space: π*(a_t|o_1:t).

By the Markov property and information structure, conditioning on the full history is sufficient and necessary for optimal decision-making. Any policy conditioning on fewer observations would theoretically violate optimality [4].

However, integrating a World Model as a separate learning module enables the system to predict future observations based on current observations and actions. This approach provides three key advantages. First, it enhances data efficiency by allowing the system to simulate future outcomes without requiring expensive or dangerous real-world robot interactions. Second, it facilitates imagination-based planning, enabling the agent to perform lookahead planning in imagination space to evaluate consequences before committing real actions. Finally, it acts as a Sim-to-Real bridge, where the model can be trained on synthetic rollouts and then effectively transferred to control real robots.

Formally, the world model learns:

where ${\widehat{o}}_{t+1}$ is the predicted next observation. This decouples perception (learning W) from control (learning π), enabling data-efficient multi-task learning.

VLA models reformulate robot control as conditional text generation [6]. However, continuous action spaces A ∈ ℝ^N must be discretized for transformer-based sequence modeling.

Action Tokenization: Each action dimension $a_t^{\left(i\right)}$ is mapped to discrete bins:

where B = 256 (standard), mapping to 8-bit integers (0–255). This quantization introduces precision loss ${ϵ}_q=\frac{{\text{max}}_i-{\text{min}}_i}{B}$ and can be mitigated by adaptive action scaling per task, multi-scale loss functions, post-hoc continuous refinement.

Autoregressive Modeling: The VLA model predicts action tokens sequentially:

where:

Analysis: This discretization reduces the action space size from continuous ℝ^N to discrete B^N, enabling efficient transformer modeling. The quantization error is bounded by ${ϵ}_q\le \frac{{\text{max}}_i-{\text{min}}_i}{256}$, typically <0.1° for rotation and <1 mm for translation—acceptable for most manipulation tasks [6].

Diffusion Policy [7] generates robot actions through probabilistic denoising. This sub-section contrasts it with Flow Matching.

Forward Process (Noise Injection):

where ${\overline{\alpha }}_t={\displaystyle {\prod }_{s=1}^t{\alpha }_s}$ gradually adds Gaussian noise to clean actions a₀. At t = T, we have q(a_T|a₀)≈N(0, I) (pure noise).

Reverse Process (Denoising):

The reverse diffusion process iteratively removes noise, requiring 50–1,000 timesteps (Number of function evaluations, NFE).

Training Objective:

where ε_θ is the noise predictor network. This is equivalent to denoising score matching [10].

Inference: Sample a_T ∼ N(0, I) and iteratively denoise:

Computational Cost Analysis: Each inference step requires a neural network forward pass. With 50–1,000 steps and 2 ms per pass:

This is incompatible with real-time control requirements (10–100 ms target). The multimodal action distribution is a key advantage (multiple valid actions), but the speed cost is prohibitive [7].

Rather than stochastic reverse diffusion, Flow Matching models deterministic paths from noise to data via Optimal Transport [8,11]. Flow Matching learns a velocity field v_θ defining deterministic trajectories. Define ϕ_t (x) as the trajectory through latent space under parameter t ∈ [0, 1]:

This is a standard ODE [12]. The velocity field learns straight-line paths via optimal transport from the noise to the data distribution. The training objective becomes:

where x₀ ∼ p(x) (data), x₁ ∼ p(N) (noise), and ϕ_t (x) = (1 − t)x₁ + tx₀ is linear interpolation.

Integrating the learned ODE from x₀ (noise) to x₁ (action) supports fast inference as described in (12).

Table 1 shows the performance comparison of diffusion policy vs. Flow matching.

Flow Matching (π₀) [13] specifically uses NFE=16 with DOPRI5 solver, then the latency of flow is calculated as follows:

Through the integration of action chunking—predicting a multi-step trajectory in a single forward pass—this inference speed supports continuous, asynchronous execution by a low-level controller, enabling highly responsive real-time control at frequencies of up to 50 Hz, which is essential for dexterous manipulation.

Unlike RT-2 and other specialized VLA models that require task-specific fine-tuning [6], π₀ is a prominent generalist robot policy. Distinguished by a Hybrid VLA architecture with Flow Matching, it is trained on a massive scale comprising over 10 million real robot trajectories and 150 million internet videos. This dataset enables cross-embodiment learning across seven robot platforms, including Boston Dynamics Spot and Tesla Optimus. The model handles 30–60 second manipulation sequences without task-specific fine-tuning. The architecture of π₀ is formulated as follow:

where:

The morphology ρ_k is provided as explicit input, enabling zero-shot transfer to unseen morphologies. The cross-embodiment loss is as follows:

where K={Spot, Optimus, UR10, …} (7+ robots) and D_k is robot k’s dataset.

The key innovation is conditioning the policy explicitly on morphology parameters ρk. By doing so, the model effectively learns to disentangle morphology-specific kinematics from morphology-agnostic task semantics, allowing the network to understand the task logic independently of the specific robot body. This approach yields significant performance gains, achieving an over 80% success rate on novel, long-horizon tasks even when deployed on unseen robot morphologies. In terms of computational efficiency, the model is capable of running at over 100 Hz on edge hardware such as the NVIDIA Jetson Thor, supporting real-time decision cycles of 10 ms. Furthermore, it functions as a truly robot-agnostic controller that requires no morphology-specific fine-tuning to adapt to new hardware. Despite these strengths, the model is currently optimized primarily for manipulation, where it reaches an 85% success rate, whereas performance lags in locomotion tasks with a lower success rate of 50%.

The observed performance discrepancy of the π₀ model—achieving an 85% success rate in manipulation tasks compared to a 50% success rate in locomotion—highlights a critical boundary in current robot foundation models. This gap can be attributed to several fundamental factors. First, there is a severe dataset imbalance; large-scale, standardized datasets predominantly feature manipulation tasks, providing significantly richer supervision for these skills compared to locomotion. Second, the inherent control dynamics differ substantially. Manipulation tasks often assume a stable, fixed base and focus on kinematic trajectories, whereas locomotion requires managing continuous, contact-rich dynamic balancing and underactuated physical states.

Real robot data is expensive and dangerous. Sim-to-Real transfer bridges the physics gap, but simulation inaccuracy remains the bottleneck [14].

3.2.1. DrEureka

Physics engine inaccuracies in friction, damping, and mass create a reality gap that causes sim-trained policies to fail on real hardware. For instance, a mere 4% error in the friction coefficient can result in a 30% increase in task failure rates. To address this, DrEureka [14] employs Vision-language models (VLMs) to automate the calibration process. When a policy fails during real-world deployment, the system analyzes the failure video to diagnose parameter mismatches and automatically adjusts the simulator settings to align with reality. Fig. 1 shows how DrEureka works. Correspondence loss of this algorithm is presented as follows:

Fig. 1. Automated Calibration Process of DrEureka.

Download Original Figure

$$L_{\text{S2R}}=\parallel v_{\text{sim}}-v_{\text{real}}{\parallel }_2^2+\lambda \parallel {\nabla }_t{v}_{\text{sim}}-{\nabla }_t{v}_{\text{real}}{\parallel }_2^2,$$

(16)

where v is optical flow (motion field) and λ weights velocity smoothness. The first term enforces spatial alignment and the second enforces temporal consistency. This approach significantly reduces manual parameter tuning from hours to minutes, achieving over 92% sim-to-real success on manipulation tasks while generalizing effectively to unseen robot platforms with minimal performance degradation.

3.2.2. GenSim

Creating physics simulations for new environments requires manual CAD modeling and parameter tuning. A typical CAD process takes 1–2 days per environment. GenSim [15] generate 3D physics simulations directly from video or text prompts using neural rendering. Fig. 2 describes overall process of GenSim.

Fig. 2. GenSim Simulation Generation Pipeline.

Download Original Figure

The technical pipeline begins with Video Input Processing, where approximately 30 frames are extracted from real-world manipulation footage, and camera intrinsics and poses are estimated utilizing COLMAP’s structure-from-motion techniques. This foundation facilitates 3D Reconstruction through a combination of NeRF [16] and Gaussian Splatting [17], where the neural radiance field captures view-dependent appearance. Following reconstruction, the system advances to Physics Simulation Generation. In this stage, object bounding boxes are extracted from the NeRF model, while object poses (position, rotation) and velocities are inferred via optical flow. This data enables the automatic generation of Unified robot description format (URDF) files and the creation of a physics engine simulation populated with parameters, such as mass and friction, derived directly from video cues. The process concludes with a Validation loop, where the predicted motion is simulated using the inferred parameters and compared against the original video using optical flow exceeds. If the mismatch exceeds a predefined threshold, the system iterates to refine the parameters. This automated pipeline dramatically accelerates development, converting a 15-minute video into a full physics simulation in just 2 hours compared to the 1–2 days required for manual modeling.

As a transformer engine Jetson Thor includes dedicated matrix multiplication units optimizing attention operations:

Hardware accelerates this at 1200 TFLOPS, enabling transformer inference without CPU bottleneck. This is critical for real-time π₀ execution.

Table 2 summarizes the specifications of Jetson T4000 Blackwell Architecture [18].

Cosmos is designed to generate video predictions that strictly adhere to physical laws, a capability critical for ensuring safety and enabling long-horizon planning [9]. The system’s architecture comprises four primary components: an Encoder that maps video frames to latent codes via learned vector quantization; a Prediction Head that forecasts future latent states based on the current frame and action; Physics Constraints incorporated implicitly through inductive biases; and a Decoder that reconstructs RGB video frames from the latent representations.

The system enforces four physical constraints:

Foundation models achieve strong performance on 30–60s tasks but success rates drop to 40% for sequences exceeding 5 minutes. Multi-hour operations like industrial cleaning lack training data. Of 150k robot hours collected, most sequences are under one minute; over 1 million hours are estimated for robust long-horizon performance. At ~$1,000/hour, data collection is impractical. Approaches such as hierarchical RL, meta-learning, curriculum learning, and self-supervised learning are promising alternatives.

Simulation-trained policies fail to transfer reliably due to mismatches in physics, sensing, and environment. Even small physics errors (e.g., a 4%–5% friction mismatch) can raise failure rates by 30%. Sensor simulations lack realistic noise and latency, while tactile signals are hard to model. Environmental variations further degrade performance. Domain randomization helps but increases training time severalfold with limited benefit [15].

Despite claims of robot-agnostic capability, π₀ shows uneven performance: 85% success on manipulation versus 35%–70% on tool use, navigation, and locomotion. The imbalance arises from data skew—95% manipulation vs. 5% locomotion. True generalization requires balanced, multi-domain data and diversified architectures.

Open issues include whether models reason or memorize, how much data is required per task class, how transferable policies are across robot types, and how to enable mid-task recovery rather than restart after errors.

Table 3 provides performance tradeoffs of representative models. The π₀ model mitigates the speed-generality tradeoff often observed in previous robot foundation models. By leveraging Flow Matching’s deterministic ODE paths rather than stochastic diffusion processes, π₀ achieves real-time inference speeds exceeding 100 Hz while simultaneously demonstrating robust cross-morphology generalization across seven distinct robot embodiments [13].

Systematic data scaling reveals clear emergence patterns: increasing training data from RT-2’s 10K robot hours to π₀’s 150 K hours (a 15× scale-up) produces three critical capabilities: (1) emergent long-horizon reasoning supporting 30+ second manipulation sequences without intermediate supervision; (2) zero-shot transfer to unseen robot morphologies including Boston Dynamics Spot and Tesla Optimus variants; and (3) enhanced robustness to distribution shift across lighting conditions, surface textures, and camera viewpoints [13].

World models and foundation policies exhibit complementary strengths that enable synergistic integration. While Cosmos provides physics-consistent forward prediction capabilities, it lacks direct motor control interfaces; conversely, π₀ excels at action generation but cannot anticipate long-term consequences [9]. Their integration forms a closed-loop verification system: π₀ proposes candidate actions, Cosmos simulates prospective trajectories, and discrepancy analysis between predicted and expected outcomes enables preemptive error detection and correction prior to physical execution.

The true potential of Physical AI emerges not from the isolated advancement of Foundation Models, Sim-to-Real Transfer, or Embodied Intelligence, but from their convergence. Foundation models provide the broad semantic reasoning and zero-shot generalization capabilities required for complex tasks. However, these models inherently lack grounding in real-world physical dynamics. Sim-to-Real transfer acts as a critical bridge, offering a scalable environment to inject physical constraints and fine-tune these models safely. Consequently, this convergence culminates in Embodied Intelligence: agents capable of understanding abstract human instructions and robustly executing them in dynamic, unconstrained environments. Looking forward, it is conceivable to anticipate a stage where the components of Physical AI automatically generate physical devices on demand, integrating and adapting the most optimal artificial intelligence for these new embodiments. This prospect underscores the necessity for further in-depth analysis of the technologies driving this future convergence of Physical AI.