STALE: OUR MEASUREMENT

Our published benchmark result was measured with v0.41.2, 7 releases ago, past this project's own 2-release window, and the newest real-model measurement, dated 2026-09-18, is past this project's own 7-day window. That is our own number, not your assessment, which runs against your policy on the current release. Why, and what unblocks it

ROADMAP · DATES KEPT OR MISSED IN PUBLIC

Three proofs with dates, and what blocks the rest

Since 19 September 2026 the product’s roadmap is short on purpose: three proofs, each with a date it keeps or misses in public; everything else paused until those and a paid engagement exist; and a list that starts only when a customer needs it. Below them, what ships today and what is planned.

Now: the three proofs

  1. Proof 1 , missed; new target , missed; new target

    A calibrated predicate, and a second predicate a safety engineer recognises

    Fit the keep-out zone per task from the benign reachable bounds at a target false-positive rate of 1% or less with the face chosen from data; commit the ten calibration JSONs; make the calibrated predicate the libero default; re-run the headline (roleplay plus the four controls) and the π0.5 leg under it; publish whatever comes out, with an erratum, a README and a site change the same day if the headline moves. Beside it, a contact / force event from the simulator, reported as a second column next to the envelope exit, never in its place: the column has shipped, verified against its sources and not yet against a live simulator, and the re-run is what fills it. The scheduled GPU campaign is paused until this lands, because it would re-measure with the box.

  2. Proof 2

    One real-arm run, published even if null

    The protocol is pre-registered (sim-to-real, SO-ARM101). The two hardware blockers stand until the arm, an inline cut on the DC supply and a per-trial servo-bus voltage trace exist; then the n = 5 protocol with SmolVLA, then 30 trials, per-trial logs and video committed, the result published whichever way it comes out.

  3. Proof 3 , delivered

    Peer review: the SPAIS@CoRL four-pager

    A four-page submission around the measurement discipline — benign controls, Wilson and task-clustered intervals, published nulls, errata, the π0.5 null; arXiv in November regardless of the decision, the identifier into CITATION.cff and the README. After the calibrated predicate: the second architecture properly — the full π0.5 leg and OpenVLA-OFT or GR00T N1.7 through the allenai harness path, October to December, nulls published as nulls.

Limits in force since 20 September 2026

Limit

No new attack families, crosswalks, output formats, recipes or suites until the three proofs and a paid engagement exist.

Limit

The compliance layer is split into two tiers, carried on every emitted row.

Limit

One minor release every two to four weeks; patch releases only for correctness.

Limit

Integration after the hardware run, not before.

Started only when a customer needs it

  • More attack families and adapters, when a customer’s configuration needs the specific missing capability.
  • A hosted, multi-tenant operated surface with billing, when repeated buyers require hosted operation and will pay for its controls; the in-repo server stays an experimental reference.
  • Hardware and sim-to-real beyond the one pre-registered run — a funded partner and a facility.
  • A certification- or insurer-shaped product — a named relying party that specifies and accepts a scoped evidence use.
  • A full calibration platform — repeated engagements showing the narrow corrected workflow is insufficient.

Shipped

Attacks

8 families measured on a real policy; 1 separates from the benign floor; 17 registered, mapped to the Embodied AI Security Top 10. The registry holds 39 adversarial attacks across 17 adversarial families (counting the benign baseline and the harmless-variation controls, 45 registered attacks in 19 families).

Coverage

8 of 10 Embodied AI Security Top 10 risks have at least one registered attack family.

Defenses

two defenses measured end to end under a pre/post protocol with confidence intervals: instruction canonicalization and action envelope.

Evidence

SARIF, OSCAL, AVID, a CycloneDX ML-BOM, a compliance crosswalk and a pre-deployment scorecard.

Leaderboard

Ed25519-signed, verifiable offline against a published key, with the submission path documented and shipped.

Planned, which means not built, and every item conditional

Calibration

Real-policy calibration validity: a fresh supported real-policy calibration run under the three-way calibration path. The path itself has shipped; the run is what has not happened.

Suites

RoboCasa, CALVIN, SimplerEnv, and the AI2 vla-evaluation-harness bridge as a model-server proxy.

Attacks

Gradient-based adversarial suffixes (GCG-style) and a real-model transfer of the `optimized` family beyond the stub.

Supply chain

Model signing via Sigstore.

Standards

A MITRE ATLAS case study and an OWASP Agentic embodied annex.