OrgWorld
A benchmark of synthetic organizations and clients, each with their own policies, documents, and edge cases, designed to test whether AI agents can infer and follow context-specific rules.
Existing benchmarks test whether an agent can complete a task. OrgWorld tests whether it can complete it the way a specific organization requires.
Each domain contains multiple fictional entities: independent organizations, or distinct clients of the same firm. Every entity has its own policy documents, SOPs, exception guides, and institutional rules.
All companies in a domain handle the same underlying task. The workflow steps are identical. The decisions within those steps are not.
The agent must read an entity's knowledge base and infer the right behavior, not default to generic answers or bleed rules between organizations or clients.
Same workflow. Different contexts. Different rules.
Each domain in OrgWorld contains multiple synthetic entities (organizations, or clients of the same firm). Every entity handles the same underlying workflow, but each one comes with its own knowledge corpus: policy documents, standard operating procedures, exception guides, and institutional rules that define how that entity operates.
The agent doesn't receive explicit instructions. It receives documents. It has to read a company's knowledge base and infer the correct behavior: which fields matter, what thresholds trigger escalation, how rounding works, when to apply a discount versus when to flag it for review. The workflow steps are the same across companies. The decisions within those steps are not.
The goal is not to test whether an agent can do the task. It's to test whether the agent can do it the way a specific organization requires.
Task correctness vs. contextual correctness
Any capable agent can produce a delivery report or execute an API call. That's task correctness: knowing what steps to take. OrgWorld tests the harder question, contextual correctness: can the agent produce the report the way this company produces it, following this company's conventions, applying this company's thresholds, and respecting this company's exceptions?
- Task correctness: the agent follows the right steps in the right order. It knows what to do. Most benchmarks test this.
- Contextual correctness: the agent infers this company's rules from its documents and follows them at each step. It knows how this company does it. OrgWorld tests this.
An agent that scores high on task correctness but low on contextual correctness is doing the right task the wrong way. It might generate a valid invoice but apply the wrong discount logic, or fill in a report correctly but use the wrong discrepancy threshold. These are exactly the failures that surface in production when organizations deploy AI against their own (or their clients') policies.
Two domain sets:
Accounting operations
Given a natural-language accounting request with structured inputs (e.g., "Invoice the customer for this deal and apply the agreed discount"), plan and execute the correct API operations the way this company would.
What varies across orgs: approval chains, rounding rules, tax handling, and exception logic.
Digital media advertising
Given a campaign's raw delivery exports, its order-management (OMS) record, its insertion-order (IO) document, and an empty report template, produce the filled delivery report.
What varies across orgs: reporting conventions, discrepancy thresholds, metric definitions, and escalation rules for underdelivery.
What the dataset includes:
Fictional entities that vary across industry, size, regulatory environment, and operational maturity. They can represent independent organizations or distinct clients of the same firm, each with their own policies, SOPs, exception rules, and institutional knowledge.
Each entity's full knowledge corpus: the policy documents, reference materials, and operating procedures that define how it works. This is the only source the agent can use.
Natural-language requests paired with structured inputs (data files, templates, records). The same prompt structure is shared across companies; the correct output differs.
The correct result for each task according to each company's rules, plus rubrics that distinguish task correctness from contextual correctness.
Why OrgWorld?
Benchmarks like SWE-bench, τ-bench, and GAIA have been essential for measuring agent capabilities like tool use, multi-step reasoning, and task completion. OrgWorld adds a complementary dimension: can an agent complete a task the way a specific organization or client requires? This matters because in production, capability alone isn't sufficient. Agents also need to comply with context-specific policy.
Two organizations (or two clients of the same firm) can run the same workflow and have completely different rules governing how it should be done. An agent that scores well on a general benchmark can still fail in production because it doesn't know this entity's exceptions, constraints, and edge cases.
| Existing benchmarks | OrgWorld | |
|---|---|---|
| Policy source | Single policy per domain | Multiple orgs/clients, each with their own policies |
| Correct answer | One right answer | Depends on which organization or client |
| Failure mode tested | Can't do the task | Does the task the wrong company's way |
| What it measures | General agent capability | Context-specific policy adherence |
Who This Is For
AI companies building agent infrastructure
- Agent framework teams testing whether their harness grounds behavior in provided knowledge documents
- RAG pipeline builders measuring retrieval and application fidelity across different policy sets
- Multi-tenant AI products that need to keep client-specific logic separated across deployments
- Model evaluation teams comparing foundation models on context-specific reasoning
Multi-client service organizations
- Accounting firms handling books for multiple clients, each with their own chart of accounts and approval rules
- Media agencies trafficking campaigns across publishers with different reporting standards
- BPOs and managed service providers running the same support workflow under different client SLAs
- Any organization that context-switches between client-specific rulesets as part of daily operations
Explore the dataset
The full dataset, including synthetic organization and client policies, task scenarios, and evaluation criteria, is publicly available.