Context Engineering: The Agent Factory's System of Context, SKILL.md Pattern, and AI-Searchable Knowledge Architecture

1. The System of Context: Agent Factory's Canonical Framework

The AI Agent Factory framework introduces one of the most important concepts in production AI engineering: the System of Context. This is distinct from, but complementary to, the System of Record.

The Agent Factory defines them precisely:

• System of Record (SoR): The authoritative, durable database where enterprise state lives — structured data, financial records, user accounts, audit logs. This is the source of truth for facts.
• System of Context (SoC): The curated, dynamically assembled layer of knowledge, domain procedures, rules, and operational memory that is structured for model consumption. This is the source of truth for how to think about and operate within the domain.

Most developers only build the System of Record. The System of Context is what separates a chatbot from a reliable AI Worker.

graph TD
    SoR[System of Record: Postgres + APIs] --> SoC[System of Context]
    DomainK[Domain Knowledge] --> SoC
    Skills[Skill Procedures / SKILL.md] --> SoC
    Rules[Policies & Rules] --> SoC
    History[Task History & Outcomes] --> SoC
    SoC --> Agent[AI Agent Reasoning Step]
    Agent --> Output[Typed Output + Trace]

2. The Five Layers of a System of Context

The Agent Factory book specifies that a complete System of Context has five distinct layers:

3. The SKILL.md Pattern: Portable Domain Intelligence

One of the most practical tools in the Agent Factory framework is the SKILL.md file pattern. Instead of hardcoding domain expertise into enormous, brittle system prompts, you encode each operational skill into a standalone, portable Markdown file.

Structure of a SKILL.md

---
name: customer_refund_processor
description: Procedures for processing customer refund requests following SOX compliance rules.
applicable_when: task involves refund, reimbursement, or credit reversal
tools_required: [stripe_api, ledger_query, audit_log_writer]
requires_approval: true
---

# Customer Refund Processing Procedure

## Pre-conditions
1. Customer account must be in ACTIVE or SUSPENDED state.
2. Refund request must reference a valid transaction ID from the last 90 days.
3. Refund amount must not exceed original transaction amount.

## Processing Steps
1. Retrieve the original transaction from Systems of Record using `ledger_query`.
2. Validate refund eligibility against the three pre-conditions above.
3. If eligible: stage the Stripe refund. Do NOT execute. Write to audit_log.
4. Escalate staged refund to human operator for approval.
5. Only after approval: execute via `stripe_api.create_refund()`.
6. Write final confirmation and refund ID to audit log.

## Failure Handling
- If transaction not found: return NOT_FOUND with transaction_id.
- If ineligible: return REJECTED with specific rule violated.
- If Stripe API fails: checkpoint current state, retry up to 3 times, then escalate.

The harness loads this file into context dynamically when the current task matches the applicable_when trigger — keeping the working context lean and preventing context rot.

4. AI-Searchable Context: Neon Postgres + pgvector

The Agent Factory crash course on AI Searchable Context establishes Neon Postgres with pgvector as the canonical backend for production Systems of Context:

• Relational tables: Store structured state, agent history, tool invocation logs, and human review records.
• pgvector extension: Store high-dimensional vector embeddings for semantic similarity retrieval.
• text-embedding-3-small: The recommended embedding model (OpenAI) for generating context vectors.

# Context retrieval with Neon Postgres + pgvector
import asyncpg
import openai

async def retrieve_context(task_description: str, limit: int = 5) -> list[dict]:
    """Retrieve the most relevant context chunks for a given task."""
    
    # 1. Generate embedding for the task
    embedding_response = await openai.embeddings.create(
        model="text-embedding-3-small",
        input=task_description
    )
    task_embedding = embedding_response.data[0].embedding
    
    # 2. Query pgvector for nearest neighbors
    conn = await asyncpg.connect(NEON_DATABASE_URL)
    results = await conn.fetch("""
        SELECT content, metadata, 1 - (embedding <=> $1) as similarity
        FROM system_of_context
        WHERE similarity > 0.75
        ORDER BY embedding <=> $1
        LIMIT $2
    """, task_embedding, limit)
    
    return [dict(row) for row in results]

5. Context Engineering: The 4-Layer Memory Stack

The Agent Factory framework defines four memory layers that every agent must manage:

1. Working context: The active token window. Trim aggressively. Keep only what the current step needs.
2. Short-term memory: Active task state in Redis. TTL-bounded. Holds current step, variables, and pending actions.
3. Long-term memory: Cross-session knowledge in Postgres + pgvector. Domain facts, past outcomes, reusable patterns.
4. Skill procedures: SKILL.md files loaded on-demand. Domain expertise encoded as reusable instructions.

The context layer crash course in the book defines this as "Building the Context Layer" — the engineering discipline of deciding what goes into each layer and when.

6. The Connector-Native App Pattern

The Agent Factory book introduces the concept of Connector-Native Apps: applications designed from the ground up to connect AI Workers to external systems through standard interfaces.

A connector-native app:

• Exposes its functionality through MCP servers (not raw function calls)
• Maintains a System of Record that the AI Worker can query and write to
• Provides typed schemas for all inputs and outputs
• Emits audit-grade traces for every operation

This is the engineering pattern behind apps like Hermes (message routing) and OpenClaw (coding harness) described in the Agent Factory ecosystem.

7. Key Takeaways

• Build a System of Context in addition to your System of Record.
• Encode domain expertise in modular SKILL.md files, not monolithic system prompts.
• Use Neon Postgres + pgvector as the canonical durable context store.
• Implement 4-layer memory: Working → Short-term → Long-term → Skills.
• Design applications as connector-native apps that expose MCP-compatible interfaces.