AI Reliability: Eval-Driven Development & Maker-Checker

1. The Evidence Problem: What the Agent Factory Book Calls "Trusting the Checker"

The AI Agent Factory framework dedicates an entire crash course to one of the hardest problems in agentic AI engineering: Trusting the Checker.

The fundamental challenge: language models can produce fluent, confident, grammatically correct output that is factually wrong, procedurally incomplete, or subtly broken. In a simple Q&A context, this produces a bad answer. In a production AI Worker context, this produces a bad outcome in a real system.

The book's principle: do not judge success by the output. Judge success by the outcome.

The difference is enormous:

Output-level verification: "Did the model generate valid-looking JSON?" → easily passes
Outcome-level verification: "Was the correct record actually written to the database? Does the ledger balance? Was the approval actually sent?" → requires checking the real world

2. The Eval-Driven Development (EDD) Methodology

The Agent Factory's crash course on Eval-Driven Development establishes that production AI systems must be built with the same rigor as traditional software: every new capability needs tests, every important failure creates a new test case, and the system must improve through measurable evidence.

Eval-Driven Development means:

Define evaluation criteria before writing prompts — know what correct looks like
Write deterministic evals first — schema checks, status codes, required field validation
Add model-based evals — use a second LLM to evaluate reasoning quality
Run evals on every deployment — treat eval failures like test failures
Capture production failures as new eval cases — every real bug becomes a regression test

graph TD
    Build[Build AI Worker] --> DefineEvals[Define Eval Suite]
    DefineEvals --> Run[Run Evals on Test Cases]
    Run --> Pass{All Evals Pass?}
    Pass -->|Yes| Deploy[Deploy to Production]
    Pass -->|No| Fix[Fix Worker / Prompts / Context]
    Fix --> Run
    Deploy --> Monitor[Monitor Production Traces]
    Monitor --> Failure[Production Failure Detected]
    Failure --> NewEval[Add New Eval Case]
    NewEval --> Run

3. The Maker-Checker Pattern: Autonomous Verification

The Agent Factory framework introduces the Maker-Checker Pattern as the primary architecture for autonomous verification in high-stakes workflows.

The principle: no single agent should both produce and verify its own work.

The two agents:

Maker Agent: Performs the work — plans, executes tools, produces output, writes to systems.
Checker Agent: Independently evaluates the work against the Definition of Done. Has tighter permissions, a different system prompt focused on verification, and no access to the Maker's reasoning trace.

from agents import Agent, Runner
from typing import Optional

# The Maker: focused on completing the task
maker_agent = Agent(
    name="invoice_processor",
    instructions="""
    You process supplier invoices according to the company's AP procedures.
    Extract line items, validate amounts, match against purchase orders, and
    stage approved invoices for payment.
    """,
    tools=[query_purchase_orders, extract_invoice_data, stage_payment],
)

# The Checker: focused on verifying the work
checker_agent = Agent(
    name="invoice_verifier",
    instructions="""
    You verify that invoice processing work meets the Definition of Done:
    1. Every line item was extracted correctly (compare to source document)
    2. Total amount matches the sum of line items
    3. PO match was verified (not assumed)
    4. Payment staging record exists with correct amount and vendor
    5. Audit trail is complete
    
    Return: VERIFIED, REJECTED (with specific failures), or ESCALATE (if unclear).
    """,
    tools=[query_staging_records, query_audit_trail, compare_documents],
    # Note: Checker has NO tools that write to production systems
)

async def process_invoice_with_verification(invoice_id: str) -> dict:
    # Step 1: Maker processes the invoice
    maker_result = await Runner.run(maker_agent, input=f"Process invoice {invoice_id}")
    
    # Step 2: Checker independently verifies the work
    checker_result = await Runner.run(
        checker_agent,
        input=f"Verify invoice {invoice_id} processing. Maker output: {maker_result.final_output}"
    )
    
    return {
        "invoice_id": invoice_id,
        "maker_output": maker_result.final_output,
        "verification_result": checker_result.final_output,
        "status": "COMPLETE" if "VERIFIED" in checker_result.final_output else "FAILED"
    }

4. Building an Eval Suite: Three Types of Evals

The Agent Factory book classifies evals into three types:

Type 1: Deterministic Evals (Schema & Logic)

Fast, cheap, binary. Run on every output.

interface DeterministicEvalResult {
  name: string;
  passed: boolean;
  failureDetail?: string;
}

async function runDeterministicEvals(output: AgentOutput): Promise<DeterministicEvalResult[]> {
  return [
    {
      name: "output_schema_valid",
      passed: OutputSchema.safeParse(output).success,
      failureDetail: OutputSchema.safeParse(output).error?.message,
    },
    {
      name: "no_pii_in_output",
      passed: !containsPII(output.text),
      failureDetail: containsPII(output.text) ? "PII detected in output" : undefined,
    },
    {
      name: "required_fields_present",
      passed: ["invoiceId", "amount", "vendorId", "stagingRecordId"].every(
        (field) => field in output.data
      ),
    },
    {
      name: "amounts_balance",
      passed: Math.abs(output.data.totalAmount - output.data.lineItemSum) < 0.01,
      failureDetail: `Discrepancy: ${output.data.totalAmount} vs ${output.data.lineItemSum}`,
    },
  ];
}

Type 2: Model-Based Evals (Reasoning Quality)

Use a second LLM to judge the quality of reasoning. More expensive, but catches subtle errors.

async def run_model_eval(task: str, agent_response: str, rubric: str) -> dict:
    """Use a judge model to evaluate agent output quality."""
    judge_prompt = f"""
    Task: {task}
    Agent Response: {agent_response}
    
    Evaluation Rubric:
    {rubric}
    
    Rate this response on a scale of 1-5 for each criterion in the rubric.
    Provide a JSON response with: {{"scores": {{}}, "overall": 1-5, "concerns": []}}
    """
    
    result = await openai.chat.completions.create(
        model="gpt-4.1",  # Use a capable model for judging
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"}
    )
    
    return json.loads(result.choices[0].message.content)

Type 3: Human Evals (Spot Checks & Edge Cases)

Periodic human review of sampled outputs. Used for novel scenarios and policy compliance.

5. Distributed Tracing for AI Workers

The Agent Factory book specifies that every production AI Worker must emit structured traces that enable operators to inspect exactly what happened in each run.

Every trace should include:

span.agent_id: Which agent executed this step
span.run_id: Unique identifier for this workflow execution
span.step_id: Which step in the workflow
span.tool_name + span.tool_args: What tool was called and with what parameters
span.input_tokens + span.output_tokens: Token usage per step
span.latency_ms: How long each step took
span.verification_result: Pass/Fail from the checker
span.human_reviewed: Whether a human reviewed this step

The Agent Factory book uses the OpenAI Agents SDK's tracing module as the baseline tracing implementation, extended with custom spans for MCP tool calls.

6. The "Leaving the Laptop" Principle

The Agent Factory crash course "Leaving the Laptop" establishes the ultimate test for a production AI Worker: can the system run reliably without you watching it?

A worker that requires constant human supervision is not a worker — it is an assistant. The goal of AI reliability engineering is to build workers that can be trusted to operate overnight, over weekends, and across time zones, with humans reviewing outcomes rather than monitoring execution.

This requires:

Comprehensive evals that catch failures automatically
Maker-Checker verification that doesn't rely on human spot-checks for routine work
Escalation policies that know precisely when to pause and call a human
Audit-grade traces that provide complete post-hoc accountability

7. Key Takeaways

Judge success by outcomes in real systems, not by output plausibility.
Use Eval-Driven Development: define what correct looks like before writing any prompts.
Implement the Maker-Checker pattern for high-stakes workflows.
Build three types of evals: deterministic (schema), model-based (reasoning quality), and human (spot-check).
Emit structured distributed traces so every run is inspectable after the fact.
Work toward the "Leaving the Laptop" standard: workers that earn trust without constant supervision.

AI Reliability Engineering: The Agent Factory's Eval-Driven Development, Maker-Checker Pattern, and 'Leaving the Laptop' Standard