license: apache-2.0
tags:
- provael
- vla
- red-teaming
- leaderboard
Provael leaderboard — submission queue
Incoming results for the Provael VLA policy ASR leaderboard.
This dataset is a queue, not the board. A maintainer validates each submission and promotes it;
until then a file here is a request, not a published result.
How a result gets here
Either route works and both end in the same review:
- The Submit a result tab on the
Space — uploads your JSON and opens
a pull request against this dataset. - A GitHub issue with the JSON attached. Needs no
Hugging Face account, and is the route the Space falls back to when this queue is unavailable.
What a submission must contain
The output of provael leaderboard build — which carries the ASR, its 95% Wilson interval, the
benign control, the attempt count, and the transfer status of every row. Validate before sending:
pip install provael
provael submit --dry-run --in <your-run-dir>
That signs and prints exactly what would be submitted, touching no network.
What gets rejected, so it is not a surprise
- A result with no benign control. The control is what makes an ASR mean anything.
- A row that reports an attack success rate without its denominator.
- A stub-policy run presented as a real-model result. Stub runs are welcome and are labelled as
stub — the labelling is the requirement, not the policy.
Null results are wanted. The published board already carries measured zeros, and a family that
does not transfer is a finding.