LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

Introduction: The Hidden Risks of Embodied Intelligence

Vision-Language-Action (VLA) models show strong manipulation capabilities in laboratory settings, but Cui et al. argue that their operational safety under strict constraints remains largely unverified. Existing evaluation leans on “success-only” metrics—benchmarks such as the original LIBERO, RT-1, or RT-2—which measure whether a robot can move a block in a static, deterministic environment. The authors’ contention is that this leaves safety behaviour untested rather than demonstrated.

A robot may appear successful while committing what the paper calls covert failures: violations of latent safety constraints that standard success metrics do not capture but that matter in human-centric spaces. To address this gap, the authors introduce LIBERO-Safety, a parametric benchmark designed to test the physical and semantic limits of embodied models.

---

The LIBERO-Safety Framework: A Multi-Dimensional Stress Test

Standard benchmarks rely on rigid templates that capture little of the variability of real environments. LIBERO-Safety uses the Unified Behavior Domain Definition Language (UBDDL) to procedurally generate 7,603 unique scenes. This isn’t just about changing colors; UBDDL randomizes 953 objects and 462 hand-object pairs with variable lighting, textures, and physical parameters to ensure models aren’t just memorizing a specific pixel-to-joint mapping.

The framework categorizes risks into a five-dimensional safety taxonomy:

1. Affordance-Aware Grasping (AAG): Identifying safe interaction regions and synthesizing geometry-aware grasps.
2. Human-Robot Interaction (HRI): Maintaining safety profiles while performing tasks around human proxies.
3. Tabletop Spatial Avoidance (TSA): Maneuvering through workspace clutter without collisions.
4. Free-Space Hand-Object Avoidance (FSHOA): This track moves beyond simple “box” obstacle proxies by using MANO and GrabNet to simulate the high-dimensional complexity of human hand kinematics. This forces models to respect the nuanced geometry of human interaction.
5. Semantic Safety Reasoning (SSR): A check on whether a model refuses hazardous instructions—for instance, identifying that “striking the beer mug with the hammer” violates safety logic.

These dimensions are scaled across three tiers: L0 (nominal), L1 (spatial/dynamic perturbations), and L2 (out-of-distribution challenges).

---

Solving the Scaling Crisis: Keypose-Driven Data Generation

A central bottleneck in safety research is the data itself: human teleoperation is slow and inconsistent. The authors’ Keypose-Driven Pipeline shifts the burden from the human to the motion planner.

Metric	Human Teleoperation	Keypose-Driven Pipeline
Human Effort	7.4 min/task	1.8 min/task
Data Yield	1:1	1:M (Scalable)
Collision Guarantee	Human-dependent	CuRobo-enforced
Spatial Representation	World-centric	Object-centric

By defining sparse human intent as object-centric keyposes, the pipeline decouples the demonstrated expertise from the scene’s specific coordinate frame. Because these keyposes are relative to the object, the resulting 19,664 collision-free demonstrations generated via CuRobo are portable — a “1:M yield” in which a single expert effort transfers across the diverse spatial layouts UBDDL generates.

---

Cross-Paradigm Results: How Do Modern VLAs Rank?

The authors evaluated several VLA paradigms against the benchmark. Their results in the Embodied Physical Safety Track (L2 scenarios):

Model Paradigm	Example Model	AAG (L2)	HRI (L2)	TSA (L2)	FSHOA (L2)
Standard VLA	OpenVLA	4.0%	0.7%	12.7%	0.0%
Standard VLA	$\pi$0.5	35.3%	83.3%	56.7%	51.3%
WM-based VLA	UniVLA	10.7%	22.0%	34.7%	19.3%
Dual-System VLA	GR00T N1.6	19.3%	73.3%	52.7%	50.0%

Key Research Findings:

• Pre-training is not enough: OpenVLA scores near zero on several L2 tracks (0.0% FSHOA, 0.7% HRI). On this evidence, large-scale pre-training alone does not confer reactive safety; the authors argue it must be explicitly aligned.
• The role of data diversity: The paper reports spatial reasoning that generalises in models trained on high-diversity data. Testing with Moka pots (Figure 4), models moved from the non-linear avoidance seen in training to more direct trajectories at test time — which the authors read as evidence of learned geometry rather than memorised paths.
• The Semantic Gap: In the SSR track, RoboBrain2.0 showed a high Refusal Rate (80%) at L0 but collapsed under the “Semantic Traps” of L1 and L2. In contrast, RynnBrain-CoP’s “Chain-of-Thought” (CoP) approach proved far more resilient to deceptive instructions.

---

Anatomy of a Failure: Why VLAs Still Struggle

The paper identifies a persistent “Generalization-Safety Tension”: while data diversity helps a robot stay safe, it creates new bottlenecks for task success. Two primary failure modes stand out:

1. Sub-optimal Trajectory Synthesis:

   • Temporal Overflows: The robot exhibits erratic avoidance maneuvers that exceed the time limit.
   • Kinematic Deadlocks: The robot enters a safe but restricted configuration where it can no longer complete the task.
   • Crucially, these erratic motions are captured by the LDLJ (Log-Dimensionless Jerk) metric. High jerk indicates a lack of smoothness that would cause physical damage to real-world actuators, even if a collision is avoided.

2. Semantic Misalignment: Perceptual errors often cause a model to execute a “stable but task-irrelevant” interaction. For instance, a model might perform a perfect, collision-free grasp on an Akita black bowl when the instruction clearly specified the bowl “next to the cookies box” (Figure 6). It’s safe, but it’s wrong.

---

Conclusion: The Path to Reliable Autonomy

The authors’ central argument is that safety cannot be a plug-in module or a secondary metric — it belongs in the training loop. Their takeaways for the safety community:

1. Explicit alignment: Safety does not emerge from scale alone; it requires diverse, collision-averse demonstration data.
2. Decoupled evaluation: Task failure and safety violations should be measured separately, using metrics such as LDLJ and Collision Rate.
3. Advanced reasoning: Surviving semantic traps points toward Dual-System architectures that use Chain-of-Thought reasoning for instruction grounding.

For failure-first purposes, the most useful contribution is the decoupling itself. A benchmark that scores a collision-free but task-irrelevant grasp as a task failure rather than a safety failure is measuring the distinction this repository’s unsafe_action_elicitation class also depends on: what a model does with an actuator is a different question from whether it said the right thing. Worth noting the results here are the authors’ own, on their own benchmark, and have not been independently replicated.

Read the full paper on arXiv · PDF