Text-layer red-teaming stops at the sentence. The robot doesn’t.
garak, PyRIT, andpromptfoo are excellent at scanning what a model says. But a vision-language-action policy turns language into motion — so the failure is a trajectory, not a paragraph. That action layer is the part text-only tools structurally can’t reach, and it’s the whole reason Provael exists. These tools aren’t rivals — they’re the other half of your stack.
| Tool | What it tests | Success = | Taxonomy | Evidence |
|---|---|---|---|---|
| garak NVIDIA | LLM text & prompt I/O | Jailbroken / toxic / leaked text | OWASP LLM Top 10 · MITRE ATLAS | Vulnerability-scan report |
| PyRIT Microsoft | Generative-AI & agent I/O | Harmful / policy-violating text | OWASP · MITRE ATLAS | Red-team eval logs |
| promptfoo OpenAI (acq. 2026) | LLM evals & prompt red-teaming | Failed eval / jailbreak | OWASP LLM Top 10 | Eval reports |
| Provael the action layer | VLA robot policy — the commanded action | Policy driven to an unsafe action (keep-out), as a calibrated ASR | Embodied AI Security Top 10 | SARIF + conformity evidence (EU AI Act / Machinery Reg / ISO) |
Comparison reflects each tool’s primary, publicly-documented scope as of July 2026; capabilities evolve — check each project. Tool names are trademarks of their respective owners; Provael is independent and not affiliated with them.
Questions people actually ask
Can garak or PyRIT test a robot policy?
They test the language and tool-call layer brilliantly — but they do not model the action space. A vision-language-action policy turns an instruction into a trajectory, and that trajectory is where the embodied failure lives. Text-only tools have nothing to score there. Use them for the language layer; use Provael for the action layer.
Is Provael a replacement for LLM red-teaming?
No — it is complementary. If your robot has an LLM in the loop, red-team the text with garak or PyRIT and red-team the resulting motion with Provael. The two cover different halves of the same system.
What makes Provael’s number different from a jailbreak demo?
Every Provael result is a calibrated attack-success rate with a 95% Wilson confidence interval and a matched benign-control false-positive rate — plus published null results when an attack does not transfer. That is a measurement you can file, not a one-off "we broke it."
Do I need a GPU?
No. Provael is CPU-first: the deterministic core and the full attack suite run without a GPU. Real-model transfer tests are optional and gated behind an extra.
Red-team the layer no one else covers.
Keep your LLM red-teaming. Add the action layer — a calibrated, reproducible attack-success rate for your robot’s policy.