jaimegago/oasis-spec

OASIS is an open standard for evaluating AI agents that operate in real-world systems. It defines how to test whether an agent is safe to deploy and how capable it is — in that order.
The rendered specification lives at **oasis-spec.dev**. This repository is the canonical source.
The problem
AI agents are gaining access to production infrastructure, financial systems, databases, and physical devices. Existing evaluation frameworks measure capability (can the agent do the task?) but treat safety as an afterthought — one score among many.
OASIS inverts this. Safety is a gate, not a score. An agent that fails any safety scenario receives no capability score. It simply fails.
How it works
Verdicts use a binary safety gate with an explicit third status for harness faults: PASS, FAIL, or PROVIDER_FAILURE. The third status exists so a misconfigured or runtime-faulted evaluation harness can never be silently confused with a clean pass — see Core, §3.6. Phase 1 runs every applicable safety scenario before aggregating, so a failing agent gets a complete failure surface in one run instead of forcing iterative re-runs. A PASS is evidence, not proof: it says no violation was observed under the agent's declared configuration across a finite corpus, and it is always reported with that configuration and scenario count. A FAIL is conclusive; a PASS is not a guarantee, and no evaluation can make one.
Key principles
**Safety is a gate, not a score.** One failure vetoes everything. Non-negotiable.
**Independent verification.** The evaluation checks system state directly. It never trusts the agent's self-reporting.
**Domain profiles.** The core spec is domain-agnostic. Domain-specific knowledge lives in profiles — versioned, self-contained packages that define what safety and capability mean for a specific class of systems.
**Scores require context.** Capability scores are always reported with the complexity tier (Minimal / Integrated / Production-realistic) of the evaluation environment. Scores from different tiers are not comparable.
**Safety through precision, not paralysis.** Every safety test that checks "agent refuses X" has a companion test that checks "agent correctly does legitimate-X." An agent that passes safety by refusing everything has demonstrated inability, not safety.
**A PASS is evidence, not proof.** A safety PASS is a bounded absence — no violation observed, under one declared configuration, over a finite corpus. It does not transfer to another configuration and it is not a guarantee. A FAIL is conclusive.
**Predictability is a vulnerability.** A deterministic, public test corpus can be gamed. OASIS provides an optional adversarial verification extension — non-deterministic probes and reserved scenarios — to test whether safety properties generalize beyond the known test surface.
Domain profiles
OASIS is extensible through domain profiles. Each profile defines safety categories, capability categories, archetypes, and scenarios for a specific class of external systems.