The Evolutionary Engine: Deterministic Swarm Management

Thesis: RAG and probabilistic retrieval are brittle. The autonomous enterprise requires a Deterministic, Evolutionary Engine. This system treats Work, Knowledge, and Automation as distinct pipelines that evolve through Wardley Stages, managed via file-system locks (Queues) and rigorous distillation.

---

1. The Knowledge Lifecycle (The Upstream)

Knowledge is not retrieved; it is refined. It moves from raw noise to crystallized doctrine.

Stage	Definition	Artifact Type	Evolution Trigger
1. Conception	Raw logs of what worked and what failed. The “exhaust” of work.	execution_logs/, notion_scratchpad/	Review: An agent or human reviews the logs for new patterns.
2. Emerging	Sample entries and hypotheses. “This worked twice.”	knowledge/drafts/, experiments/	Validation: The pattern is tested proactively in a new context.
3. Converging	Growing bodies of evidence on specific topics. Playbooks start to form.	knowledge/playbooks/, docs/guides/	Standardization: The playbook is stress-tested against edge cases.
4. Accepted	Core principles and universal doctrine. Invariant truths.	knowledge/procedure/, knowledge/doctrine/, policy/	Stabilization: It becomes a rule (linter, barrier, or hard constraint).

---

2. The Work Engine: Queues & Git

Deterministic coordination without the “Meeting.”

Structure: A simple, file-based project management system stored in a Git Repository. Core Mechanic: File System Locks = Coordination.

The Queues Protocol

1. The Board: A directory structure representing the state of work.
   • /projects/active/
   • /projects/backlog/
   • /projects/archive/
2. The Bead (Task File): A Markdown file defining the unit of work.
   • Contains: Spec, Context, Definition of Done, Link to Knowledge.
3. The Lock:
   • When an Agent (or Swarm) picks up a Task, it creates a .lock file (e.g., task_001.md.lock).
   • Rule: No other agent edits a locked file.
   • Benefit: Prevents collision without complex databases.
4. The Output:
   • Work Artifacts: The code/doc created.
   • Learnings Artifact: A separate file (task_001_learnings.md) containing the “diff” between expected and actual outcome. This feeds the Conception stage of Knowledge.

---

3. The Automation Pipeline (The Downstream)

Matching the Agent Type to the Wardley Evolutionary Stage of the work.

Evolutionary Stage	Work Type	The Executor (Agent Type)	Methodology
1. Genesis	Novel, High Uncertainty. “Figure it out.”	Ideation Skill / Researcher	Exploratory: Large context window, high temperature, generates options.
2. Custom-Built	Implementing a specific solution. “Build the MVP.”	AI Architect	Lean Startup: Build proof-of-concept, validate against spec, iterate fast.
3. Product	Standardized feature work. “Scale this.”	Skill Swarm (Worker Agents)	Agile/Kanban: Pull from backlog, execute known pattern, test, merge.
4. Commodity	Repeatable, known volume. “Run the pipes.”	Deterministic Script / IaC	Hard Automation: Terraform, Cron jobs, CLI tools. Zero hallucination.

---

4. The Grand Loop: The Flywheel

How the system improves itself.

1. Execute: Agents pick up Queues (Work) using their current Skills.
2. Record: As they work, they record Learnings (Success/Failure/Novelty) into the Conception bucket.
3. Distill: A separate “Gardener Process” (Human or Agent) reviews Conception logs and promotes valid patterns to Emerging -> Converging -> Accepted.
4. Upgrade: Use the Accepted Doctrine to refactor the Skills, updating the System Prompts and Tools the Agents use in Step 1.

Comparison to RAG

• RAG: “Search a pile of messy documents and hope the LLM finds the right answer.” (Probabilistic, Prone to Noise).
• Evolutionary Engine: “Distill experience into Procedures, then hard-code those Procedures into the Agent’s skills.” (Deterministic, Improving Quality).