STALE: OUR MEASUREMENT

Our published benchmark result was measured with v0.41.2, 7 releases ago, past this project's own 2-release window, and the newest real-model measurement, dated 2026-09-18, is past this project's own 7-day window. That is our own number, not your assessment, which runs against your policy on the current release. Why, and what unblocks it

RESEARCH REGISTER · PROTOCOLS BEFORE RESULTS

Studies, and the ones that have not run yet.

Not one measured study here is a closed-loop real-policy result: three are stub-validated on the deterministic CPU fixture and say so in their own first paragraph, and one replayed recorded frames open-loop and returned a null. Four measured, four pre-registered (three not yet run), and two of three dated commitments still open.

A pre-registered study states its protocol, predicate and stopping rule before the run, so a null is a result rather than something to reframe. One marked not yet run has no numbers behind it, by design. Read the evidence class on each card, not the word “measured”: the one measured real-policy transfer this project has is the SmolVLA × LIBERO instruction result, on /results.

Measured

Run, and reported with what the run can support

Each of these has run and its outcome is published. The evidence class says how far that outcome reaches: a stub-validated result shows that a measurement protocol works end to end. It is not evidence about a real policy.

  • STUB-VALIDATED

    Instruction canonicalization

    Defense, deterministic CPU fixture

    stub-validated-scaffolding — this is not a real-model result. A credited verdict is evidence that the pipeline position and the measurement protocol work end to end, not that the mitigation would hold up on a real policy.

    Protocol filed

  • STUB-VALIDATED

    Action envelope

    Defense, deterministic CPU fixture

    credited on stub and reach, not-credited on humanoid — and most of the credit carries no information. A magnitude clamp cannot address a balance predicate, which is not a magnitude at all. No real-model transfer is claimed.

    Protocol filed

  • STUB-VALIDATED

    EAI04 action-space-integrity transfer

    Attack transfer, deterministic CPU fixture

    Measured on the CPU reach fixture only. The SmolVLA × LIBERO row is empty: no real-policy EAI04 transfer is claimed.

    Protocol filed

  • OPEN-LOOP NULL

    Offline real-observation (recorded SO-101 frames × SmolVLA)

    Recorded real frames, replayed open-loop

    OPEN-LOOP. NO ROBOT MOVES. Recorded frames of a real SO-101 dataset are replayed through the policy and what it would do under a benign instruction is compared against what it would do under an attack. Nothing is executed and no trajectory is produced. This is not a closed-loop real-robot attack-success rate and must never be reported as one. The result was a null.

    Protocol filed · run

Pre-registered: not yet run, or a preliminary leg only

Protocols published before the run

These carry a protocol and, at most, a preliminary leg with no headline claimed. Do not cite them as evidence of anything except intent. The date beside each is when the protocol first landed in the product repository, which you can check in its git history.

  • PRELIMINARY LEG

    π0 / π0.5 cross-architecture instruction transfer

    π0.5 through LeRobot first (Amendment 1, 14 Sep 2026); π0 via openpi deferred — both flow-matching policies

    PRE-REGISTERED, preliminary leg run 18 September 2026 (results below the amendment; no headline claimed). Amended 14 September 2026, before any attack arm ran; the amended protocol is deposited on Zenodo as 10.5281/zenodo.22751558, a public timestamp that precedes the π0.5 attack arms.

    Pre-registered

  • NOT YET RUN

    Meta-World second-suite instruction transfer

    SmolVLA × Meta-World — a different task distribution

    PRE-REGISTERED — protocol only, no results claimed.

    Pre-registered

  • NOT YET RUN

    Sim-to-real correlation (SO-ARM101 + SmolVLA)

    A physical SO-ARM101 arm, supervised table-top trials

    PRE-REGISTERED — protocol only, no results claimed. Provael ships no physical-robot control code and no real-world-harm payloads; the physical eval would use LeRobot’s own evaluation loop under direct human supervision, with a physical inline cut on the DC supply as the e-stop — the servos’ own over-current protection re-arms on the next command.

    Pre-registered

  • NOT YET RUN

    Humanoid whole-body / locomotion transfer (GR00T-N1)

    NVIDIA GR00T-N1 via the gated GrootAdapter

    PRE-REGISTERED — protocol only, no results claimed.

    Pre-registered

Dated commitments

Where this project is behind, with a date attached

Each item below is a gap. Each has a named action and a date by which an outcome is published: the number if it exists, and the reason if it does not. They stay on this page if they slip.

  • OPEN

    Zero real-robot results.

    By this date the studies index states one of two things: the completed trial count with its measured sim-to-real correlation, or the specific blocker and its cost.

    Outcome published by .

  • OPEN

    Zero results against any flow-matching policy.

    By this date, either the first instruction-family rate against π0 with its Wilson interval and benign control, or the reason the served path did not come up.

    Outcome published by .

  • MET

    Zero third-party submissions and zero forks.

    By this date the release containing provael submit is published to PyPI, so reproducing a result and submitting it needs pip install provael and one command.

    Due , met . provael submit first shipped in 0.32.0 (8 August 2026); the commitment was confirmed met on 18 August with v0.34.0 on PyPI: pip install provael then provael submit --help resolves. Whether anyone then submits is not something this project put a date on, and the submission register is still empty.

Transcribed from the product repository’s own studies index, which is the maintained source: docs/studies/index.md ↗. A met commitment keeps its original due date rather than being deleted, so the record stays legible.