Nobody has reproduced this yet. Here is how to.
No third party has published a reproduction of any result, and no notified body, certifier, insurer or standards committee has reviewed this work. This page says what counts as a reproduction, what each published number is, and where your result would be listed.
Everything below is pinned to commit 4e75173 (release v0.48.0), verified 2026-11-12.
Three ways to check, and only the third is a reproduction
Two of the three need no GPU, and they catch the failure a reader worries about most: a number edited after it was measured. Only the third re-runs the policy, so only the third goes in the register. The commands for each, with what to expect, are on /reproduce.
- Check the published board has not been alteredNo GPU · under a minute
- Rebuild the board from the committed run reportsNo GPU · a few minutes
- Re-run the measurement against the real policyGPU required · hours
What each published row is, exactly
The board carries 3 measured adversarial families plus the benign control, all from one committed run: one policy (smolvla), one suite (libero), task libero_object (all 10 tasks), 5 seeds, horizon 280.
| Family | Published rate (successes / trials) | Reproduce with |
|---|---|---|
| visual | 2/100 | --attacks none,visualcovers decoy_object, patch |
| instruction | 50/150 | --attacks none,instructioncovers goal_substitution, paraphrase, roleplay |
| injection | 2/50 | --attacks none,injectioncovers scene_text |
| baseline | 1/50benign control — not an attack | --attacks none,instructionthe control ships in every run; it is the none arm of any --attacks list |
The raw artifact behind the headline SmolVLA and LIBERO result, unrendered and at the pinned commit, is aggregate.json. If your number differs, open a pull request against results/ with it.
The immutable report behind these rows: report.json at 4e75173. The predicate is uncalibrated — it counts trials driven out of the configured benign envelope, not a certified hazard rate — and that limit travels with every number here, including a reproduced one. The same rows, laid out in the ISO/IEC 17025 clause-7.8 test-report shape (not an accredited report), are on the sample evidence pack, rendered from the product's own output; a reproduction can be laid out the same way with provael report --format test-report.
What counts as a reproduction, and what you get for it
0 independent reproductions
Nobody outside this project has reproduced a published Provael result. The register is empty as of , and it is empty because the count is 0, not because this page failed to load it.
Each entry will list its author, exactly what they reproduced, the result they got and a link to their artifact. Every published run is seed-pinned and its artifacts are committed, so a reproduction is a checkable claim rather than a favour. Three things are recorded separately and never added together: a signature verification (the artifact is what it says it is), a regeneration from stored traces (the numbers follow from the committed shards) and a fresh execution of the policy in the simulator. Only the last is an independent measurement.
The product keeps the same three-level register: see the reproduction request (what to run, what to expect, hardware and time, known limits) and the reproduction-result issue form, which takes an agree, disagree or null outcome at any of the three levels without running provael submit. As of 15 November 2026: requested 0 · attempted 0 · completed 0 · agrees 0 · disagrees 0.
Putting your result on the board
One command validates your run against the submission schema, signs it so the numbers are tamper-evident in transit, and opens the pull request. Every row it produces carries your handle and is marked as a third-party submission.
provael submit runs/repro --submitted-by <your-handle>provael submit ships on PyPI, so pip install provael is all you need. It has been in a published release since 0.32.0 (8 August 2026).
If you measure a policy you did not train, the 14-day rule applies to you too
Provael sends a result about someone else's policy to that policy's authors 14 days before publishing it, and hands over the whole artifact up front: the signed report.json, the exact command, checkpoint revision, seeds, horizon and simulator versions, the predicate stated as uncalibrated where it is, the benign control arm, and the draft caveat in the wording we intend to publish. If we cannot hand over enough for them to re-run it themselves, it is not ready to publish to anyone.
The clock starts when the notice is sent, not when it is read. If they find an error we re-run and correct or withdraw before publication. If they disagree and we are not persuaded, both positions are printed on the same page, theirs in their words. If they do not respond we publish after 14 days and the entry records that, so silence cannot become a veto.
The same protection binds us when we publish about your policy, and binds you when you publish about someone else's.
The full rule, including what happens on a correction and why the board carries no safety ranking: Measuring someone else’s policy ↗
Prefer to open the PR by hand, or hit a problem? The submission guide is CONTRIBUTING-leaderboard.md, and /contact reaches a human.
To quote this work, see how to cite: the release you cite is not the pinned commit above. Corrections already issued are on errata. Protocols filed before their runs, and the dated commitments this project is still behind on, are on studies.