ASR LEADERBOARD

Which VLA policies were measured, and by how much.

How often each attack family pushed a policy out of its safe zone, the attack-success rate, with its n, a 95% interval and the benign control each row must be read against. Simulation only. Every row is reproducible from a published recipe, and families that did not beat the control stay in at their measured counts.

4 rowssmolvla × liberoSim-only · evidence, not certification

4 rows, not a survey. Every row is one policy on one suite, smolvla × libero (checkpoint HuggingFaceVLA/smolvla_libero), so the table compares attack families with each other, not policies.

OLDER TOOL VERSIONMeasured with an older tool version — not a current result

Measured with 0.41.2. The product's own verdict, the one the banner carries, is that these rows were measured with v0.41.2, 7 releases ago, past this project's own 2-release window; the current release is v0.48.0. The board was assembled on 2026-11-11 from commit 89fe74f; rebuilding a board re-stamps that date and re-measures nothing.

Attack-success rate by policy, suite and attack family, with 95% Wilson confidence intervals and the benign control.
Policy	Suite	Attack family	ASR	95% CI	Benign control	Successes	Attempts	Transfer	Measured with
smolvla	libero	`instruction`	33.3%	[26–41%]	2.0% (1/50) [0–10%]	50	150	real model	0.41.2
smolvla	libero	`injection`	4.0%	[1–13%]	2.0% (1/50) [0–10%]	2	50	real model	0.41.2
smolvla	libero	`visual`	2.0%	[1–7%]	2.0% (1/50) [0–10%]	2	100	real model	0.41.2
smolvla	libero	`baseline`control	2.0%	[0–10%]	2.0% (1/50) [0–10%]	1	50	real model	0.41.2

Calibrated: no — a default keep-out box, not one fitted per task, so a success is divergence out of that box rather than a hazard rate
Reproducible: no — the policy sampler is not fully seeded, so these are one draw, not a constant
Outside this board: A second architecture has a committed preliminary leg on a different lineage: π0.5 (lerobot/pi05_libero_finetuned_v044) on the same ten libero_object tasks, roleplay 1/30 against none 0/30, 3 seeds, 2 arms of 8. It is not in this board because every row here must share one measurement, and that run is another policy on another run; it is not a transfer claim either. The card on /findings carries it with its status.
Not applicable: mcp_tool_desc produced episode records but zero applicable episodes. Excluded from every denominator — not measured, which is a different claim from measured zero.

2 of 3 adversarial families stayed at the benign floor. The injection and visual families fired no more often than the untouched instruction did on this policy — inside the benign control's own 95% interval. They stay in the table at their measured rate.
The payload is signed, and you should check it rather than trust this page. Ed25519 over the whole board minus the signature field, verifiable offline with no network:provael leaderboard verify --in leaderboard.json --pubkey leaderboard.pubBoth files are in the product repo. Altering a single success count returns INVALID and exits non-zero. That covers origin and integrity; it says nothing about whether four rows are enough, which is the point above.
Simulation, not a hazard rate. An attack that moves a policy out of its envelope in sim is a robustness signal. It is not a real-robot incident probability, and nothing here is a certification.

Quoting a row? How to cite this work carries the BibTeX and — separately — the release and commit these numbers were produced by, which is not the same version as the current release.

The published contract carries attempts per row, not a separate episode or seed count. There is still no PER-ROW tool version — measured_with is one value for the whole file — so the “measured with” column repeats it only while that list has a single entry, and says mixed rather than picking one when it does not. This table shows what the artifact actually contains rather than inferring the rest. Provenance for the whole file: schema v7 · generated 2026-11-11T06:18:26Z · commit 89fe74f · digest cbb8d6fbf5feab2d… · signature ed25519 by key 8d62aa33ed5162f3 (HPUioSKPq/6SbsPL…, 88 chars — full value in the artifact).

The write-up behind the headline row →Source leaderboard.json ↗Interactive Space ↗

Submissions

Get your policy on it.

Open to any policy anyone can re-run. The bar is reproducibility, not permission: we do not need your weights, we need a run someone else can repeat.

Wondering how a rate on this board compares to the published attack literature? Mostly it does not — see published attack baselines, which gives every published figure a verdict on whether it can be compared to a Provael ASR at all, and names one row it declined to print.

To add a row yourself, for free, follow /submit: a local run with the benign control, the check CI runs, and a pull request to the product repository. To have it run for you, with the scorecard and the SARIF rather than a table row, use the paid option at the foot of this page.

What gets published, and what does not

Published

Your report.json, in the public repo where /submit says to put it.
The row: ASR with its 95% Wilson interval, and the benign (none) control rate beside it — never the rate alone.
The checkpoint identity, suite and tasks, seeds and horizon, so someone else can re-run it.
A real-transfer or stub-scaffolding label, so a fixture run is never quietly mixed in with a real one.

Not published, and not asked for

Your weights. The bar is that someone else can repeat the run, which needs a checkpoint they can fetch — not a model you hand over.
Any observation, image or trajectory from your run. The report carries outcomes and statistics, not the episode contents.
Anything about you beyond the name you put on the directory.

Two things are refused rather than published, and both are refusals of the method rather than of the result: a run nobody can reproduce, and an unseeded run from a stochastic policy — one draw is not a measurement, and a board of single draws cannot support the comparison it exists for.

A result that makes Provael look bad gets published unchanged. If your policy survives the sweep, if an attack family we publish a transfer for does nothing to your model, or if the benign control fires so often that our predicate is obviously drawn in the wrong place — that is the row, and it goes up as measured, under your name and ours. We do not get an edit. The only grounds for declining a submission are the two method failures above, and a decline says which one and why, in the PR. How we’d be caught if that were untrue →

What gets checked. That the run is deterministic at the stated seed, that the benign control is present and its false-positive rate is reported alongside the ASR, that every interval is computed rather than asserted, and that the checkpoint is identified precisely enough for someone else to fetch it. A submission that cannot be re-run is declined — including ours. Full method: docs.provael.com ↗

Want your own policy measured, with the scorecard and the SARIF rather than a table row?

Get your policy measured →

Which VLA policies were measured, and by how much.

Leaderboard

How to read every row

Get your policy on it.

What gets published, and what does not