Crystal-Agents: Okara-Inspired Sprint Plan

Context

Crystal-Agents is a horizontal multi-agent workflow engine (11 agents, 11 workflows, 6 tools) with a file-based architecture. The Okara analysis revealed that while our architecture is sound, we’re missing depth in marketing intelligence and proactive UX that surfaces value before the user asks. The primary high-impact gaps and new design constraints are:

1. Action Feed: Turn the dashboard from a passive log viewer into a decision-making surface.
2. Autonomous Onboarding: Populate memory files automatically from a URL or repository scan.
3. Integrated Containerization: Combine the 3 current containers (dashboard, telegram, scheduler) into a single, unified container process manager (crystal-integrated) for simpler deployment and resource optimization.
4. AWS ECR Deployment: Move the container build phase from the VPS to the local environment, building and pushing to AWS ECR, and pulling the pre-built image on the VPS (mirroring the Erpcrystal_mfg deployment model).
5. Safe Coexistence & Fallback: Run the new integrated container parallel to the existing three containers. Keep the old containers (particularly the critical scheduler running production workloads) running undisturbed. Stop only the old telegram container for testing, with the ability to instantly roll back if any issues occur.
6. Rich Newsletter & Follow-Up Email Campaigns: Enable the marketing/content agent to draft HTML newsletters and follow-ups containing rich media (screenshots, inline graphics/infographics, and video preview links) and schedule sending to existing users and prospects.

This plan covers 8 sprints (~2 weeks each) building incrementally on the existing foundation.

Scheduler Safety Rule & Coexistence Strategy

The existing scheduler/service.py cron-backed job executor runs production workloads (checking disk space, backup status, web health checks, scheduled reports) and must not be disrupted. The safety rules are:

• Additivity: All scheduler additions are additive. New jobs are registered alongside existing ones.
• Coexistence Setup: The new crystal-integrated container is added to docker-compose.yml but runs in parallel, disabled/stopped by default.
• Incremental Testing: During integration testing, only the old telegram container is stopped. The new integrated container is started to handle Telegram and Dashboard requests, while the old scheduler container continues to run production cron jobs.
• Rollback Path: If the integrated container encounters issues during testing, it is stopped, and the standalone telegram container is instantly restarted.
• ECR Deployment: All image builds are performed locally, pushed to AWS ECR, and pulled on the VPS. Code is no longer compiled or built on the VPS.

Dual-Purpose Design: Marketing + Coding

Unlike Okara (a commercial vertical AI CMO), Crystal-Agents is an internal tool used for both marketing and software development. Every feature built in these sprints must serve both domains:

This dual-purpose approach means each sprint delivers value to both the marketing and engineering workflows that Crystal-Agents already handles.

Sprint 1: Action Feed + Dashboard Modularization + Single Container Integration

Theme: Turn the dashboard from passive log viewer into an action surface, configure the single integrated container, and set up AWS ECR local deployment.

Why first: Every subsequent sprint enqueues action items, and having a pre-built ECR image deployment pipeline ensures we can safely push and test updates.

Files to Create

• dashboard/action_store.py — ActionItem dataclass + ActionStore (JSONL, follows ApprovalStore pattern at orchestrator/approvals.py:27-110). Action types: marketing, coding, approval, system. Fields: action_id, domain (marketing/coding), type, priority (1-5), title, description, suggested_workflow, suggested_prompt, target_url, status, created_at
• dashboard/routes/ — extract route handlers from the monolithic 928-line DashboardHandler
• deploy-oneclick.ps1 — local PowerShell deploy script modeled after deploy-mfg-oneclick.ps1:
  1. Authenticates local Docker daemon with AWS ECR registry (540065258656.dkr.ecr.us-east-1.amazonaws.com).
  2. Builds the consolidated Docker image locally and tags it as 540065258656.dkr.ecr.us-east-1.amazonaws.com/erpcrystal-agents:latest.
  3. Pushes the image to AWS ECR.
  4. SCPs the .env and docker-compose.yml configs to the VPS at /opt/crystal-agents.
  5. SSHs into VPS, logs in to ECR on the VPS, pulls the new image, and recreates only the target container.

Files to Modify

• dashboard/app.py — refactor DashboardHandler into thin router delegating to route modules; add /actions page as new default landing with domain filter tabs (All / Marketing / Coding); render prioritized action cards with “Run” buttons
• orchestrator/approvals.py — wire ApprovalStore.request() to auto-enqueue an action feed item
• orchestrator/cli.py — add an integrated command that acts as a process supervisor, spawning dashboard, telegram, and scheduler loop as Python subprocesses, logging their outputs with tags, and terminating clean/failing-fast if any of them exits.
• docker-compose.yml — add a new service integrated pointing to the pre-built erpcrystal-agents ECR image and using the integrated CLI command. Ensure existing services (dashboard, telegram, scheduler) are untouched.

Dual-Purpose Design

Action items are tagged with a domain field. The dashboard shows filter tabs so the user can switch between marketing actions (“12 pages missing meta descriptions”) and coding actions (“3 dependencies are outdated”, “PR #45 open for 5 days”, “test coverage dropped 4% in /utils”).

Deliverables

• ActionStore with enqueue(), list_pending(), mark_done(), dismiss()
• Dashboard /actions page: domain-filterable feed with severity badges and one-click “Run” buttons
• Approval requests auto-create action feed items
• Existing dashboard routes unchanged (refactor is structural only)
• deploy-oneclick.ps1 script successfully builds locally and deploys ECR images (erpcrystal-agents) to VPS
• CLI integrated command successfully runs dashboard, telegram, and scheduler processes concurrently
• docker-compose.yml updated with a standalone integrated service for safe co-existence testing

Sprint 2: Autonomous Onboarding (Business + Codebase)

Theme: Self-populating memory from a URL or repo — no manual markdown editing.

Why second: All content/SEO/marketing workflows need brand context. All coding workflows need architectural context. Currently both are hand-written stubs.

Files to Create

• tools/website_scanner.py — crawls a URL, extracts about page, products, value props, social links
• tools/competitor_mapper.py — uses web_search to identify 3-5 competitors + positioning
• tools/brand_voice_analyzer.py — analyzes existing content to extract tone, vocabulary, audience
• tools/repo_scanner.py — reads a local repo: directory tree, language breakdown, entry points, dependency graph, test structure, existing AGENTS.md/CLAUDE.md
• workflows/business_onboarding.py — 4-step: scan website → map competitors → analyze voice → write memory/*.md
• workflows/codebase_onboarding.py — 3-step: scan repo → analyze architecture → write memory/coding_standards.md, memory/architecture.md, AGENTS.md

Files to Modify

• orchestrator/runtime.py — register both workflows
• orchestrator/ceo_router.py — route “onboard business”, “onboard codebase”, “learn my repo”, “onboard my project”
• dashboard/app.py — “Run Onboarding” section with two options: Business / Codebase; enqueue action items on first load

Dual-Purpose Design

Two separate workflows sharing the same pattern:

• Business onboarding: Takes a URL → populates memory/company.md, memory/product.md, memory/pricing.md, memory/marketing.md, memory/soul.md
• Codebase onboarding: Takes a local path → populates memory/architecture.md, memory/coding_standards.md, updates AGENTS.md with project-specific context

Deliverables

• BusinessOnboardingWorkflow populates marketing memory files from a URL
• CodebaseOnboardingWorkflow populates coding memory files from a local repo
• CEO Router catches both onboarding intents
• Dashboard shows domain-specific onboarding action items on first run
• CLI: crystal-agent run --task business_onboarding --prompt "https://example.com"
• CLI: crystal-agent run --task codebase_onboarding --prompt "D:/path/to/repo"

Sprint 3: Audit Intelligence (SEO + Code Quality)

Theme: Replace the 9-line seo.py stub with real audit capability, and add a parallel code audit workflow.

Why third: Audit findings drive the Audit→Fix→PR loop (Sprint 5) and feed the action feed. Both marketing sites and codebases need auditing.

Files to Create

• tools/site_auditor.py — crawls sitemap + pages, checks meta, headings, schema, alt text, broken links
• tools/keyword_classifier.py — classifies keywords by funnel stage (TOFU/MOFU/BOFU)
• tools/code_auditor.py — scans repo: missing tests for new code, TODO/FIXME accumulation, dependency freshness, file sizes, lint violations, security patterns (hardcoded secrets, eval usage)
• workflows/seo_audit.py — 3-step: crawl → classify keywords → write report to generated/seo_audits/
• workflows/code_audit.py — 3-step: scan repo → analyze findings → write report to generated/code_audits/
• logs/audits/ — AuditStore (JSONL) with domain field (marketing/coding)

Files to Modify

• agents/seo.py — rewrite with comprehensive system prompt + allowed_tools
• orchestrator/policy.py — register site_auditor, keyword_classifier, code_auditor
• orchestrator/runtime.py — register SEOAuditWorkflow, CodeAuditWorkflow
• orchestrator/ceo_router.py — route “audit site”, “audit code”, “seo audit”, “code audit”, “code quality”
• dashboard/app.py — “Audits” page with domain tabs (SEO / Code); enqueue audit action items periodically

Dual-Purpose Design

• SEO Audit: Technical site crawl → meta, schema, heading, link checks → marketing action items
• Code Audit: Repo scan → missing tests, stale deps, lint violations, security smells → coding action items
• Both share the same AuditStore JSONL pattern, differentiated by domain field

Deliverables

• Full technical site audit (sitemap, meta, headings, schema, links)
• Code quality audit (deps, tests, lint, security)
• Keyword classification by funnel stage
• SEO agent with real system prompt and tool access
• Dashboard audit history page with domain filter
• Each finding enqueued as a domain-tagged action item

Sprint 4: Platform-Specific Content + Publish Pipeline (Social, Docs, and Rich Email/Newsletter campaigns)

Theme: Replace generic content.py with platform-aware agents (social + docs), add a dedicated email/newsletter marketing agent, and add approval-gated publishing (including rich HTML email delivery with SMTP credentials and S3-based rich assets).

Files to Create

• agents/content/linkedin.py — hook-first, whitespace, professional tone, hashtag strategy
• agents/content/twitter.py — character limits, thread structure, engagement hooks
• agents/content/reddit.py — subreddit-aware, value-first, authentic voice
• agents/content/docs_writer.py — API docs, changelogs, README updates, AGENTS.md maintenance
• agents/content/newsletter.py — system prompt for drafting engaging, customer-centric newsletters and follow-ups based on product features, metrics, and case studies
• memory/audiences.json — local JSON mailing list defining campaign targets segmented into existing_clients and prospects
• tools/s3_assets.py — S3 assets connector using boto3/awscli to list, verify, and reference rich media (screenshots, infographics, and MP4 video previews) stored in the crystal-marketing-assets S3 bucket.
• tools/email_sender.py — HTML email compiler and sending tool using SMTP credentials configured in .env (mirroring the Blazor/API ErpCrystal_MFG Serilog setup with env variables: EMAIL_HOST, EMAIL_PORT, EMAIL_USERNAME, EMAIL_PASSWORD, EMAIL_FROM). Built-in template compiler that accepts markdown/HTML and embeds rich content items (screenshots, play-button-overlay video preview links, and inline infographics) pulled from S3. requires_approval = True.
• tools/publisher.py — adapters for webhook, WordPress REST API, file output, docs_repo (commits to a docs repo via git_ops), and email (using tools/email_sender.py); requires_approval = True
• tools/publish_history.py — PublishHistoryStore (JSONL)
• workflows/content_publish.py — 3-step: research/gather → platform/newsletter draft + rich media insertion (resolving S3 assets from crystal-marketing-assets and compiling HTML templates) → publish/email send to segments from audiences.json

Files to Modify

• agents/content.py — detect platform/medium hints in prompt, delegate to appropriate agent’s system prompt; add docs and email as options
• orchestrator/policy.py — register publisher, s3_assets, and email_sender
• orchestrator/runtime.py — register ContentPublishWorkflow
• dashboard/app.py — “Published Content” page with domain filter (Social / Docs / Email); render email drafts with rich media preview and action buttons

Dual-Purpose Design

• Social track: LinkedIn, X, and Reddit agents for marketing content.
• Docs track: docs_writer agent generates API documentation, changelogs, README updates, and AGENTS.md content — published to a docs repo via the git_ops adapter.
• Email track: newsletter agent writes HTML emails to targets in audiences.json.
• Publisher tool supports all targets: file target for social drafts, docs_repo target (wraps existing git_ops) for docs, and email target (wraps email_sender with SMTP and S3 assets) for newsletters.

Deliverables

• Platform-specific content agents (LinkedIn, X, Reddit, Docs, Newsletter)
• s3_assets tool for listing and pulling screenshots, infographics, and MP4 video previews from crystal-marketing-assets S3 bucket
• email_sender tool with SMTP delivery matching MFG credentials and approval-gate verification
• memory/audiences.json mailing list structured for existing and prospect segments
• Rich email template engine (markdown support, inline S3 image placeholders, play-button overlay for S3 videos)
• Publisher tool with approval gate and multiple adapters (web, file, git, email)
• Publish workflow: research → draft → compile rich S3 media → publish/send to segment
• Publish history dashboard page with domain filter

Sprint 5: Audit → Fix → PR Loop (Dual-Domain)

Theme: Close the loop — audits find issues, coding agent fixes them, PR is created. Works for both SEO fixes and code fixes.

Files to Create

• tools/issue_tracker.py — IssueTrackerStore (JSONL): tracks audit findings + fix status + PR URLs. Fields: issue_id, domain (marketing/coding), audit_run_id, severity, category, description, fix_suggestion, status (open/fixing/pr_created/fixed/dismissed), pr_url
• workflows/audit_fix_loop.py — run audit (SEO or code) → identify fixable issues → invoke git_workflow per issue → update tracker

Files to Modify

• tools/site_auditor.py — add auto_fix_suggestion field to each finding
• tools/code_auditor.py — add auto_fix_suggestion field to each finding
• orchestrator/runtime.py — register AuditFixLoopWorkflow
• orchestrator/ceo_router.py — route “audit and fix”, “fix issues”, “fix code issues”
• dashboard/app.py — “Issues” page with domain tabs (Marketing / Coding) and “Fix This” / “Dismiss” buttons

Dual-Purpose Design

• Marketing issues: “Missing meta description on /pricing” → Fix it → PR updates the HTML
• Coding issues: “3 functions in /utils missing tests” → Fix it → PR adds test coverage
• Same IssueTrackerStore, same audit_fix_loop workflow, same “Fix This” button — different domain

Deliverables

• Issue tracker with full lifecycle (open → fixing → pr_created → fixed)
• AuditFixLoopWorkflow: audit → fix → PRs (works for both domains)
• Dashboard issues page with domain tabs and one-click “Fix This”
• Action feed populated with critical findings from both domains

Sprint 6: GEO Agent + Citation Monitoring

Theme: Track whether AI models (ChatGPT, Claude, Perplexity, Grok) cite your brand.

Files to Create

• agents/geo.py — system prompt on AI citation factors (schema, backlinks, entity definitions)
• tools/citation_tracker.py — queries multiple AI platforms for brand mentions
• workflows/geo_scan.py — 3-step: analyze site GEO factors → scan AI platforms → generate report
• logs/geo/ — GEOScanStore (JSONL)

Files to Modify

• orchestrator/policy.py — register citation_tracker, geo agent
• orchestrator/runtime.py — register GEOScanWorkflow
• orchestrator/ceo_router.py — route “geo”, “citation”, “ai citation”
• dashboard/app.py — “GEO” page with citation trends

Deliverables

• GEOAgent with domain expertise in AI citation mechanics
• Citation tracking across ChatGPT, Claude, Perplexity, Grok
• GEO scan reports with optimization recommendations
• Dashboard GEO page with citation history

Sprint 7: llms.txt + AGENTS.md Generator + Daily Brief

Theme: Generate AI-readable files for both websites and repos, plus a unified daily briefing system.

Files to Create

• tools/llmstxt_generator.py — crawls site, generates llms.txt + llms-full.txt
• tools/agentsmd_generator.py — reads repo structure, generates/updates AGENTS.md / CLAUDE.md with project context, build commands, coding patterns
• workflows/llmstxt_generation.py — crawl → generate → write
• workflows/agentsmd_generation.py — scan repo → generate → write AGENTS.md
• workflows/consolidated_brief.py — aggregates runs, approvals, audits, publishes, issues (from both domains) into one markdown brief
• logs/briefs/ — directory for generated daily briefs

Files to Modify

• orchestrator/runtime.py — register all three workflows
• orchestrator/ceo_router.py — route “llms.txt”, “agents.md”, “claude.md”, “daily brief”
• dashboard/app.py — consolidated brief page; “Generate llms.txt” and “Generate AGENTS.md” buttons
• reports/summary.py — add consolidated brief aggregation method

Dual-Purpose Design

• llms.txt: For company/product websites — helps AI models understand the site
• AGENTS.md: For code repositories — helps AI coding agents understand the project. This is the coding equivalent of llms.txt
• Daily Brief: Aggregates activity from BOTH domains in one report with sections: “Marketing Activity”, “Development Activity”, “Pending Approvals”, “Action Items”

Deliverables

• llms.txt generator tool + workflow
• AGENTS.md generator tool + workflow
• Consolidated daily brief aggregating marketing + coding activity
• Dashboard brief page as optional default landing
• CLI: crystal-agent run --task llmstxt_generation --prompt "https://example.com"
• CLI: crystal-agent run --task agentsmd_generation --prompt "D:/path/to/repo"

Sprint 8: Analytics Integration + Social & Newsletter Scheduling

Theme: Real analytics (GA/GSC for marketing, GitHub/CI for coding) + scheduled content/newsletter distribution for both domains.

Scheduler safety: The existing cron job executor is untouched. New job types (post_publisher, newsletter_sender_job, weekly_audit, daily_brief) are added as new SchedulerJobPlan entries in config.json. The scheduler’s run_due() loop picks them up automatically with zero changes to its core logic. Verify existing jobs still run with crystal-agent scheduler list.

Files to Create

• tools/ga_connector.py — Google Analytics Data API; graceful fallback if not configured
• tools/gsc_connector.py — Google Search Console API; graceful fallback
• tools/github_stats.py — GitHub API: open PRs, review queue depth, CI failure rate, contribution stats
• tools/post_scheduler.py — queues posts and email campaigns by platform/audience + datetime; requires_approval = True
• workflows/social_calendar.py — research trends → draft N posts/emails → schedule across the calendar
• workflows/analytics_report.py — pull GA4 + GSC OR GitHub stats (domain parameter) → generate report

Files to Modify

• agents/analytics.py — add real system prompt + allowed_tools for both marketing and coding analytics
• orchestrator/runtime.py — register new workflows
• dashboard/app.py — “Analytics” page with domain tabs (Marketing / Engineering); “Scheduled Content” calendar view (includes queued social posts and drafted newsletter campaigns)
• config/app_config.py — add AnalyticsConfig dataclass; add new job entries for post_publisher, newsletter_sender_job, weekly_code_audit, weekly_seo_audit, daily_brief — as additional entries alongside existing jobs (never replacing)

Dual-Purpose Design

• Marketing analytics: GA4 traffic, GSC keywords, social post and email campaign click-through rates
• Engineering analytics: GitHub PR stats, CI pass rate, code churn, review turnaround
• Scheduled content: Social posts and newsletter email campaigns for marketing; changelog/doc updates for coding
• Both share the same analytics_report workflow with a domain parameter

Deliverables

• GA4 and GSC connectors (graceful fallback if unconfigured)
• GitHub stats connector for engineering analytics
• Post/Newsletter scheduler with due-date polling via new post_publisher / newsletter_sender_job scheduler jobs
• Social & email campaign calendar workflow
• Dashboard analytics page with domain tabs (Marketing / Engineering) and scheduled campaign manager
• Existing scheduler jobs verified untouched via crystal-agent scheduler list

Dependency Graph

Sprint 1 (Action Feed) ──────────────────────> ALL subsequent sprints
     │
Sprint 2 (Onboarding: Business + Codebase)
     │
     ├──> Sprint 3 (Audits: SEO + Code Quality)
     │         │
     │         ├──> Sprint 5 (Audit→Fix→PR: Marketing + Coding issues)
     │         │
     │         └──> Sprint 6 (GEO: marketing-only)
     │
     ├──> Sprint 4 (Content: Social + Docs)
     │         │
     │         └──> Sprint 8 (Analytics + Scheduling: Marketing + Engineering)
     │
     └──> Sprint 7 (llms.txt + AGENTS.md + Daily Brief)
               │
               └──> Sprint 8 (brief feeds into analytics dashboard)

• Sprint 1 is prerequisite for all (action feed items)
• Sprint 2 needed before 3 & 4 (brand + codebase context)
• Sprint 3 needed before 5 & 6 (audit findings drive fixes + GEO analysis)
• Sprint 4 needed before 8 (publisher used by social scheduler)
• Sprints 5, 6, 7 are independent of each other
• Sprint 8 is the final integration sprint

Patterns to Follow (from existing codebase)

New Dependencies (Minimal)

• Sprint 3: beautifulsoup4 + lxml for HTML parsing (current regex parsing in web_search.py is too fragile for full audits)
• Sprint 8: google-analytics-data + google-auth-oauthlib for GA4/GSC (graceful fallback if unconfigured)

Everything else uses stdlib only (json, pathlib, datetime, uuid, subprocess, http.server, urllib).

Verification

After each sprint, verify by:

1. Sprint 1:
   • Start dashboard, confirm /actions page loads with domain filter tabs.
   • Run deploy-oneclick.ps1 to build and push the new integrated container image to ECR.
   • Start the new integrated container on the VPS alongside the old services.
   • Stop the old telegram container, run a request via Telegram, and verify the integrated service handles it correctly.
   • Stop the integrated service, restart the old telegram container, and verify the fallback is clean.
2. Sprint 2: Run business onboarding with a real URL → verify memory/company.md etc. populated. Run codebase onboarding with a local repo path → verify AGENTS.md and memory/architecture.md populated.
3. Sprint 3: Run SEO audit → verify generated/seo_audits/ report. Run code audit → verify generated/code_audits/ report. Confirm both create domain-tagged action items.
4. Sprint 4:
   • Run content publish for LinkedIn post → verify draft + publish history.
   • Run content publish for API docs → verify docs generated and published.
   • Draft and compile an HTML newsletter using the new email agent, verify embedded rich media elements (screenshots, video play button preview link, inline infographics) render correctly in drafts.
   • Verify SMTP/SES credentials by successfully sending a test HTML email.
5. Sprint 5: Run SEO audit → click “Fix This” on a finding → verify PR created. Run code audit → click “Fix This” on a code issue → verify PR created.
6. Sprint 6: Run crystal-agent run --task geo_scan --prompt "<brand name>", verify citation report.
7. Sprint 7: Run llms.txt generation → verify output. Run AGENTS.md generation on a repo → verify AGENTS.md updated. Run consolidated brief → verify dashboard renders it with both marketing and coding sections.
8. Sprint 8:
   • Configure GA/GSC credentials → run analytics report → verify data.
   • Configure GitHub token → run engineering analytics → verify data.
   • Schedule a social post and a rich newsletter campaign → verify both are published automatically on schedule.
   • Run crystal-agent scheduler list → verify ALL existing jobs still appear unchanged.