Crystal-Agents — System Internal Design & Architecture

Version: v0.3.0
Date: 2026-06-22
Purpose: Smallpreneur agent stack for structured business automation on a single VPS.

1. High-Level Architecture

Crystal-Agents is a workflow-driven multi-agent orchestration system. It accepts tasks through four entry points (CLI, web dashboard, Microsoft Teams webhook, Telegram bot), routes them to defined workflows, executes steps via LLM-backed agents with tool access, gates risky actions behind human approval, and surfaces everything through a local dashboard.

┌──────────────┬──────────────┬──────────────┬──────────────┐
│   CLI        │  Dashboard   │  Teams       │  Telegram    │
│  (argparse)  │  (HTTP/HTML) │  (Webhook)   │  (Polling)   │
└──────┬───────┴──────┬───────┴──────┬───────┴──────┬───────┘
       │              │              │              │
       └──────────────┴──────────────┴──────────────┘
                          │
                   ┌──────▼──────┐
                   │ Runtime     │  build_runtime() wires everything
                   │ Bundle      │  AppConfig + PolicyConfig + Notifier
                   └──────┬──────┘
                          │
                   ┌──────▼──────┐
                   │ CEO Router  │  Keyword match → fallback to LLM
                   │ (classifier)│  classification → workflow name
                   └──────┬──────┘
                          │
                   ┌──────▼──────┐
                   │ Orchestrator│  route() → canonical workflow name
                   │             │  run()  → delegates to WorkflowRunner
                   └──────┬──────┘
                          │
                   ┌──────▼──────┐
                   │ Workflow    │  Iterates steps: agent + prompt + tool
                   │ Runner      │  Checks policy.json before tool calls
                   └──────┬──────┘
                          │
              ┌───────────┼───────────┐
              │           │           │
        ┌─────▼─────┐ ┌──▼──┐ ┌─────▼─────┐
        │ LLM Call  │ │Tool │ │ Approval  │
        │ (DeepSeek)│ │Exec │ │ Store     │
        └───────────┘ └─────┘ └───────────┘

Key Design Principle

Every workflow is a sequence of steps. Each step is a (agent, prompt_template, optional_tool) triple. The runner feeds the prompt to the agent’s LLM, captures the response, optionally invokes a tool, and feeds output into the next step. There are two execution models:

The runner auto-detects which model to use via hasattr(workflow_obj, "run_direct").

2. Layer-by-Layer Walkthrough

2.1 Configuration Layer (config/)

AppConfig (config/app_config.py) is a @dataclass(slots=True) tree. Load path:

config/config.json  →  env var overrides  →  dataclass defaults
     (base)              (secrets)             (fallback)

Env var naming: CRYSTAL_{SECTION}_{KEY} (e.g., CRYSTAL_TEAMS_WEBHOOK_URL, CRYSTAL_B2_APPLICATION_KEY). A false string value in config JSON explicitly disables a channel even if credentials exist.

PolicyConfig (config/policy.json) defines:

• Per-agent tool allowlists (agent_tools)
• Per-tool approval requirements (tool_approval_required)
• Default policy for unknown agents/tools

2.2 Entry Points

All converge on build_runtime() → Orchestrator.run().

2.3 Orchestrator Layer (orchestrator/)

orchestrator/
├── cli.py              # CLI entry point
├── orchestrator.py     # Core Orchestrator: route() + run()
├── runner.py           # WorkflowRunner: step execution loop
├── ceo_router.py       # LLM-based natural language → workflow classifier
├── runtime.py          # build_runtime(): dependency injection / wiring
├── types.py            # TaskRequest, RunRecord dataclasses
├── approvals.py        # ApprovalStore (JSONL-based pending/approved/denied)
├── policy.py           # PolicyConfig loader from policy.json
├── llm_providers.py    # ModelProviderClient (OpenAI-compatible API)
├── context_builder.py  # Builds LLM context from memory files
├── memory_loader.py    # Loads company/product knowledge from memory/
├── agent_generation.py # Generates agent definitions from memory
├── agent_loop.py       # Agent execution loop (single-turn)
├── audits.py           # Audit orchestration
├── geo.py              # GEO scan orchestration
├── publishes.py        # Content publishing orchestration
└── telegram_bot.py     # Telegram polling bot

Routing Flow

1. CEORouter.route_request(prompt) — first checks keyword patterns (~20 if-elif blocks for slash commands and known phrases), then falls back to LLM classification if no keyword matches. Returns {"action": "workflow|scheduler_list|reports|scheduler_run", "workflow": "...", "prompt": "..."}.

2. Orchestrator.route(task) — maps the task type string to a canonical workflow name. Handles aliases (feature-spec → feature_spec, content → content_drafting). Falls through to the raw task type if no mapping exists.

3. Orchestrator.run(task) — creates a run directory under runs/<run_id>/, writes manifest.json, delegates to WorkflowRunner.run().

Step Execution (WorkflowRunner)

For each step in the workflow definition:

1. Resolve agent by name from AgentRegistry
2. Build prompt from template + previous step outputs
3. Call LLM via ModelProviderClient
4. Parse response for tool calls (parse_edit_instructions for repo_writer, or tool name extraction)
5. If tool call found: check PolicyConfig — if requires_approval, block and wait (up to 5 min polling ApprovalStore)
6. Execute tool via ToolExecutor
7. Feed tool output into next step
8. On completion: write result.md, notify via MultiChannelNotifier

2.4 Agent Layer (agents/)

Each agent is a named role with a system prompt template:

All agents inherit from BaseAgent which provides parse_edit_instructions() for extracting file edits from LLM responses.

2.5 Workflow Layer (workflows/)

18 workflow definitions, each a WorkflowDefinition with a sequence of WorkflowStep objects:

2.6 Tool Layer (tools/)

Tools are concrete capabilities. Each is registered in ToolRegistry and gated by policy.json:

Approval flow: When a tool marked requires_approval: true is invoked, the ApprovalStore creates a pending entry. The workflow pauses (polling every few seconds, up to a 5-minute timeout). An operator approves or denies via the dashboard /approvals page or Teams /approve <key> command.

2.7 Scheduler Layer (scheduler/)

scheduler/
├── cron.py       # CronExpression parser (5-field cron)
├── service.py    # SchedulerService: poll loop, job execution
├── history.py    # SchedulerHistoryStore (JSONL)
└── models.py     # SchedulerEvent, SchedulerJobPlan

The scheduler runs on a configurable poll interval (default 60s). Each tick:

1. Parse current time
2. Find jobs whose cron expression matches
3. Execute the allowlisted bash script as a subprocess
4. Jobs marked requires_approval: true pause until approved
5. Log results to logs/scheduler_history.jsonl

The integrated CLI command runs dashboard + scheduler loop + telegram bot as child subprocesses with a supervisor that monitors for failures.

2.8 Notification Layer (notifications/)

notifications/
├── multi.py      # MultiChannelNotifier: fans out to all enabled channels
├── teams.py      # TeamsNotifier: Adaptive Cards to MS Teams webhook
└── telegram.py   # TelegramNotifier: messages to Telegram chat

The MultiChannelNotifier checks each channel’s enabled() method. If enabled: false in config, the channel is silently skipped. Teams notifications use Microsoft Adaptive Card format (v1.4) with color-coded titles (error=red, success=green, neutral=blue) and structured FactSet sections.

2.9 Dashboard Layer (dashboard/)

A zero-framework HTTP server. DashboardHandler extends BaseHTTPRequestHandler:

• Auth: HTTP Basic Authentication on every request (except Teams webhook endpoint). No sessions, no CSRF tokens.
• Routing: do_GET/do_POST dispatch to route handlers via lazy imports (to avoid circular dependencies).
• Rendering: All HTML is generated inline via _page(title, body) which wraps content in a dark-themed CSS grid layout with sidebar navigation.
• State: DashboardState holds references to AppConfig, ApprovalStore, SchedulerService, ReportGenerator, CEORouter, and ActionStore.

Route handlers live in dashboard/routes/:

• home.py — Main page with run form + recent runs table
• teams.py — Teams webhook handler (no auth, HMAC verification)
• actions.py — Action feed view
• approvals.py — Approval queue with approve/deny buttons
• runs.py — Run detail view
• scheduler.py — Scheduler status + manual trigger
• reports.py — Daily/weekly report views
• audits.py — Audit results
• issues.py — Issue tracker
• publishes.py — Publish history
• geo.py — GEO scan results
• docs.py — Document center (B2 integration, work-in-progress)

2.10 Storage Pattern

The system uses JSONL files (one JSON object per line) as its primary storage. At least 8 stores follow this identical pattern:

Each implements append()/log(), list_all(), and latest_state() independently with no shared base class.

2.11 Memory & Context

memory/ contains static knowledge files loaded by memory_loader.py:

• soul.md — Company identity/voice
• company.md — Company information
• product.md — Product catalog
• pricing.md — Pricing information
• skills.md — Team skills/capabilities
• history.md — Company history
• coding_standards.md — Code conventions

These are injected into LLM context by context_builder.py before each agent call. Files containing “TODO:” are skipped.

3. Data Flow: End-to-End Example

User types in dashboard: “Write a blog post about our new AI features”

1. POST /run → DashboardHandler.do_POST() → handle_runs_post()
2. DashboardState.trigger_workflow("", "Write a blog post about...")
3. No workflow specified → CEORouter.route_request(prompt) classifies via LLM → {"action": "workflow", "workflow": "content_drafting"}
4. TaskRequest(task_type="content_drafting", prompt="...") created
5. Orchestrator.run(task) → route("content_drafting") → confirms canonical name
6. WorkflowRunner.run():
   • Step 1: ResearchAgent + “Research the topic: AI features” → LLM call → research notes
   • Step 2: ContentAgent + “Draft a blog post based on: {research}” → LLM call → draft
   • Step 3: ReviewerAgent + “Review this draft: {draft}” → LLM call → review feedback
7. Result written to runs/<run_id>/result.md
8. MultiChannelNotifier.send() → Teams Adaptive Card with run summary
9. Dashboard shows run in recent runs table

4. Deployment Architecture

Production runs via Docker on a single VPS:

docker-compose.yml
├── dashboard    (port 8000:8000) — standalone dashboard service
├── telegram     (no ports)       — standalone telegram bot
├── scheduler    (no ports)       — standalone scheduler loop
└── integrated   (port 8001:8000) — dashboard + scheduler + telegram in one container

The integrated service is the primary production target. It runs all three components as subprocesses inside a single container with a supervisor that monitors for failures. Deployment uses deploy-oneclick.ps1 which:

1. Builds Docker image locally
2. Pushes to AWS ECR
3. SSHs into VPS, syncs config + scripts
4. Runs docker compose up -d --force-recreate integrated

SSH keys and AWS credentials are mounted into containers from the host (~/.ssh, ~/.aws).

5. Known Design Decisions & Trade-offs

1. No web framework — Dashboard uses stdlib http.server to avoid external dependencies. Trade-off: manual HTML generation, no templating, no session management.

2. JSONL over SQLite — All persistent state is JSONL files. Simple to inspect/debug (tail -f logs/approvals.jsonl), no schema migrations, but no query capabilities and risk of corruption on concurrent writes.

3. Keyword-first routing — CEO router checks keyword patterns before calling the LLM. Reduces latency and cost for common commands, but the keyword list can drift from actual workflow names.

4. Two execution models — Most workflows use declarative build() with steps; chat and audit_fix_loop use run_direct() with custom Python. The runner detects which at runtime. Flexible but means two code paths to maintain.

5. OpenAI-compatible API with DeepSeek — The config labels the provider "openai" but points to api.deepseek.com. DeepSeek implements the OpenAI API format, so the same client code works. The label is misleading.

6. Basic Auth without sessions — Simple to implement, no session storage needed. Trade-off: credentials sent on every request, no logout, no brute-force protection.

7. One container, three processes — The integrated service runs dashboard, scheduler, and telegram as subprocesses. Simpler than orchestrating three separate containers. Trade-off: one process crash can take down all three (mitigated by supervisor restart).