pretty_name: AIREP Embedded Evaluation Profile v0.1
license: other
license_name: airep-dual-license
license_link: https://github.com/halvrenofviryel/ai-runtime-evidence-protocol/blob/85f99b79fbe78377a53f03d2fafc5331eab42f71/LICENSE-CC-BY-4.0.txt
tags:
- ai-evaluation
- ai-safety
- ai-governance
- audit
- provenance
- reproducibility
- airep
AIREP Embedded Evaluation Profile v0.1
This is not a training dataset or benchmark. It is a Hugging Face distribution mirror of an
experimental evaluation-evidence profile, its schema basis, registry and fixtures. The canonical
specification history lives in the AIREP GitHub repository. Byte identity between this mirror and
its source commit is a distribution-integrity property, not independent scientific verification.
Experimental evaluation-evidence contract for AIREP v0.2.
Profile identity
| Profile identifier | airep.embedded-evaluation |
| Profile version | 0.1 |
| Carrier | AIREP wire version 0.2 |
| Schema basis digest | sha256:ce57bd493d1a3166051bc9029972d6be215793edcd3f1b314534959659b3da8b |
| Canonical GitHub source | https://github.com/halvrenofviryel/ai-runtime-evidence-protocol/tree/85f99b79fbe78377a53f03d2fafc5331eab42f71/spec/airep/v0.2/profiles/embedded-evaluation |
| Source commit of this mirror | 85f99b79fbe78377a53f03d2fafc5331eab42f71 — fixed source snapshot for the files in this publication (including its historical README) |
| First publication commit | 2b62580aa05302cc978c2cb5a2a55d8a0a215b55 — schema, registry and both fixtures are byte-identical to that commit; only PROFILE_README.md changed since |
| Status | Experimental companion profile — not a ratified standard, not a fifth AIREP artifact family |
The five files here are byte-identical to the source commit's spec/airep/v0.2/profiles/embedded-evaluation/
directory (the profile README is published as PROFILE_README.md); MANIFEST.sha256 lists their digests.
Publication policy and current exporter scope
Frozen, source-pinned publication; not a continuously updated mirror of canonical main.
The schema, registry, fixtures and PROFILE_README.md retain the exact source pin above.
Card clarifications do not refresh those source bytes. A new source edition requires an explicit
new publication revision and pin.
PROFILE_README.md is historical source text, not a current parser-support statement.
The current exporter parses LightEval results_*.json; native Inspect .eval and arbitrary
OpenEvals formats are not currently supported. Explicit timezone-aware start/end timestamps
are required; a timezone-naive filename is never silently labelled UTC. Profile validation is
not an assurance result or safety certification. Declared context is recorded, not established.
What it binds
Evaluator and evaluated-party context with a declared independence basis; target model/system/
checkpoint identity; evaluator access tier, capabilities and limitations; harness/task/environment/
safeguard configuration; explicit measurement execution and observed states; content-addressed
evidence with visibility/withholding state; redactions, deviations, retries and incidents; and
optional verification references.
Three negative-state guards are enforced by the schema: NOT_RUN can only pair withNOT_MEASURED/NOT_APPLICABLE; PASS/FAIL require RAN/PARTIAL; INVALIDATED never yieldsPASS/FAIL. example.not-measured.json preserves NOT_RUN → NOT_MEASURED explicitly.
Claim limitations
A profile PASS means only that the profiles["airep.embedded-evaluation"] payload validates
against the exact digest-pinned schema basis. It does not establish:
- evaluator independence or competence (
independent-declaredis a declaration, not proof); - that the named model, revision or configuration was the one actually executed;
- truth, representativeness, completeness or reproducibility of observations;
- model safety, alignment, security, regulatory conformity or deployment suitability;
- any AIREP assurance class — Core/Authenticated/Witnessed semantics are unchanged;
- validity of a platform
verifyToken, which the profile carries only asplatform-verificationevidence.
Without an accepted basis for the exact identifier, the correct verifier result is NOT_EVALUATED,
never an implicit pass.
Files
embedded-evaluation.schema.json— self-contained Draft 2020-12 schema (basis)registry.json— digest-pinned registry for the AIREP v0.2--profile-basesmechanismexample.measured.json— synthetic measurement that ran and failed its declared criterionexample.not-measured.json— synthetic planned evaluation that did not runPROFILE_README.md— the canonical profile README at the source commitMANIFEST.sha256— digests of the mirrored files
Related
- Exporter Space (LightEval results → profile payload + evidence manifest; current parser LightEval
results_*.jsononly, Inspect/OpenEvals not parsed): https://huggingface.co/spaces/phionyx/airep-evaluation-evidence - Exporter source (LightEval results → profile): https://github.com/halvrenofviryel/ai-runtime-evidence-protocol/tree/main/integrations/lighteval
- Collection: https://huggingface.co/collections/phionyx/phionyx-runtime-evidence-and-measurement-6aa65830536927640b962c5a
- Technical note, Access Is Not Yet Verifiability: https://huggingface.co/blog/phionyx/access-is-not-yet-verifiability
Licensing
The canonical AIREP repository is dual-licensed by file type: code under Apache-2.0
(LICENSE), specification text and schemas under CC BY 4.0 (LICENSE-CC-BY-4.0.txt), with a
patent non-assertion statement. The mirrored files are specification/schema/example material and
follow the CC BY 4.0 terms; this card declares license: other so those file-specific terms are
not collapsed into a single repository-wide SPDX claim.
Paper
AIREP: A Protocol for Per-Decision Evidence in AI Runtime Governance
https://arxiv.org/abs/2608.21363
The Embedded Evaluation Profile is an experimental AIREP v0.2 companion profile.
Its presence does not create an AIREP assurance class or constitute independent validation.