Actuator Reality Shaping
for Zero-Shot Sim-to-Real Robot Learning

Reinforcement Learning Sim-to-Real Actuator Reality Shaping
Paper (PDF) Humanoid Video Moonbot Video

Abstract

Sim-to-real transfer in robot learning is often limited by discrepancies between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors. Rather than closing this gap by increasing simulator fidelity through system identification, domain randomization, or learned actuator models, we introduce an alternative paradigm: actuator reality shaping. We shape the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation. Each joint is equipped with a two-degree-of-freedom feedforward-feedback controller that decouples reference-response shaping from robust stabilization, providing a standardized actuator interface for reinforcement learning policies. Policies trained only with the prescribed reference model can be deployed zero-shot on real hardware without task-level fine-tuning or learned actuator models. We validate the approach on a single-joint testbench, a 7-DOF arm reaching task, a wheeled-legged robot driving over a slope, and a humanoid robot walking, suggesting that actuator reality shaping can serve as a reusable interface for robot learning across diverse hardware platforms.

Key Idea

Actuator reality shaping concept
Actuator reality shaping. We make the real actuator behave like the simulator's ideal model, rather than the reverse. (Top) simulation with the ideal second-order reference model; (Bottom) zero-shot hardware deployment, where a per-joint 2-DoF driver shapes the response so that θ ≈ θref.

Existing sim-to-real methods try to identify the hardware so the policy can adapt to it. We take the dual perspective: we shape the hardware's closed-loop response at the driver level so the policy perceives a fixed, simulator-defined plant on every platform. Even high-gear-ratio joints emulate the ideal reference plant, with residual error absorbed at runtime by a disturbance observer and integral feedback.

Contributions

  1. We introduce the actuator reality shaping paradigm and derive a 2-DoF control architecture that shapes the closed-loop actuator response to match a prescribed simulator reference model.
  2. We validate the architecture on single-joint tracking with disturbance injection and demonstrate zero-shot sim-to-real transfer on a 7-DOF arm reaching task, comparing against the factory cascaded servo, a tuned PID, and a learned Delta-Action baseline (ASAP).
  3. We further demonstrate generality via zero-shot transfer on a wheeled-legged robot and a full humanoid robot performing forward walking.

Method: 2-DoF Controller + Disturbance Observer

Per-joint 2-DoF architecture
A cascaded 2-DoF controller (position outer loop around a velocity inner loop) augmented with a disturbance observer (DOB) drives the real motor so that its closed-loop response tracks the simulator's reference model.

The feedforward path specifies the desired closed-loop dynamics seen in simulation; the feedback path (PID with integral action) compensates for disturbances and residual actuator-model mismatch. The DOB attenuates the residual model error ΔP below its cutoff, absorbing unmodeled friction, cogging, and external loads.

Zero-Shot Demonstrations

The same per-joint 2-DoF driver layer transfers to platforms with very different actuators and dynamics, without task-level tuning or learned actuator models.

Humanoid: Forward Walking

22-DOF humanoid using PH54 servos. Trained with the BeyondMimic framework in simulation, deployed zero-shot — the only change is the actuator model swapped for the ideal reference plant.

Moonbot-mini: Slope Traversal

31-DOF wheeled-legged robot climbing a slope with poorly backdrivable YM070 servos. The 2-DoF controller reproduces the simulator's compliant spring-damper behavior, enabling zero-shot transfer.