MAF — Architecture

The technical map. Every component is linked to its source so you can verify the described behavior actually exists in the code.

Table of contents

1. Layered view
2. Arenas — phases, agents, replan loop
3. The 4-layer data plumbing
4. Outbox routing
5. Lifecycle events
6. Decision memory + outcome harvesting
7. LLM model selection
8. Channel discovery + preview
9. Dashboard internals — routers, state, freshness
10. Config save safety — pydantic + ETag + atomic
11. Refresher heartbeats
12. Trigger templates library
13. Alpaca data via trtools2 — not directly

Layered view

                         ┌─────────────────────────────────────┐
                         │             Dashboard (FastAPI)      │
                         │   /  /live  /channels  /data  /docs  │
                         │                                       │
                         │   WS /ws/events   /ws/arenas/{name}   │
                         └────────────────┬─────────────────────┘
                                          │
        ┌─────────────────────────────────┼─────────────────────────────────┐
        │                                 │                                 │
        │   maf:control:in     ┌──────────┴──────────┐    maf:actions:out   │
        │   maf:control:out    │       MAFApp        │    maf:decisions:out │
        │   maf:events         │  (arena lifecycle)  │    arena env streams │
        │                       └────┬────────┬─────┘                       │
        │                            │        │                             │
        │                ┌───────────┘        └──────────────┐              │
        │                ▼                                    ▼              │
        │       ┌─────────────────┐                ┌─────────────────────┐  │
        │       │  Arena.run()    │                │  Workers (service)  │  │
        │       │  PhaseGraph     │                │   ControlInbox      │  │
        │       │   • phases      │                │   KronosRefresher   │  │
        │       │   • specialists │                │   MirofishRefresher │  │
        │       │   • synthesis   │                │   TriggerDispatcher │  │
        │       │   • replan      │                │                     │  │
        │       └────────┬────────┘                └──────────┬──────────┘  │
        │                │                                     │             │
        │                ▼                                     ▼             │
        │   ┌─────────────────────┐              ┌─────────────────────────┐ │
        │   │  source adapters    │◀────reads────│   Redis Streams (32+)   │ │
        │   │  + memory + LLMs    │              │   trtools2:*  fomo2:*   │ │
        │   │                     │              │   kronos:*    mirofish:*│ │
        │   └─────────────────────┘              │   maf:arena:* …         │ │
        │                                         └─────────────────────────┘ │
        └──────────────────────────────────────────────────────────────────┘

The MAFApp (src:src/maf/app.py#L124) is the entry point. It owns the loaded arenas, installs the EventBus, and (in service mode) launches the worker tasks alongside.

Arenas

An Arena is a small graph of agents over a shared state dict. The construction is in Arena.__init__. The execution loop is Arena.run:

state ─▶ Phase 1 (analysis)       parallel: specialists ▶ AgentSignals
       ─▶ Phase 2 (synthesis)     sequential: synthesis agent
       ─▶ Phase 3 (replan_check)  sequential: replan controller (loops back?)
       ─▶ Phase 4 (emit)          sequential: empty phase, MAFApp publishes after

Phase patterns

Phase supports three execution patterns:

• parallel — every agent gets its own deep-copied state, runs concurrently, results merged afterwards. Used for fan-out specialists.
• sequential — agents run one after the other, each sees the latest state.
• debate — N-way round-robin between debater agents for max_rounds, then a judge synthesises.

Specialists, synthesis, replan

• SpecialistAgent — runs a ReAct loop calling configured source adapters as tools, emits a structured AgentSignal (signal + confidence + key_factors) plus a narrative.
• SynthesisAgent — reads all agent signals, computes a confidence-weighted ensemble score, makes one LLM call for the verdict + reasoning.
• ReplanAgent — confidence-gated re-run controller. Detects data gaps (parse-fail, no_, stale_ markers) and loops back to the analysis phase up to max_iterations times.

Dynamic transitions

The PhaseGraph follows configured transitions by default, but a phase can set state["_next_phase"] to override for one hop. The replan controller uses this to loop back.

The 4-layer data plumbing

The goal: near-realtime decisions without paying expensive costs on every arena run. Achieved by separating what arenas need (cached artifacts) from when those artifacts get computed (proactive refreshers driven by a watch list).

Layer 1 — Watch list

WatchList is a Redis sorted-set keyed by opaque (target_id, kind) tuples with TTL-decayed scores. Anything expensive keys off this set, so cold targets cost zero.

Kinds today: "symbol" (Kronos refresher reads), "document" (MiroFish refresher), "question" (reserved for future research-debate triggers).

API: - add(target_id, kind=..., ttl_seconds=..., attrs=...) — ZADD GT. Later interest extends TTL but can't shorten it. - members(kind=...) — current watched entries. - decay() — evict expired members.

Layer 2 — Refreshers (proactive cache fills)

Both refreshers are failure-tolerant — per-target exponential backoff on sidecar errors, atomic INCR/DECR refund on the MiroFish daily budget. They emit only when something meaningful changed (prob_up_delta > 0.05 or direction flip for Kronos) so the emit streams are signal-rich for trigger rules.

Layer 3 — Trigger dispatcher

TriggerDispatcher reads the triggers: block of every arena YAML, tails the configured input streams, evaluates when: predicates via a tiny safe-eval mini-language (safe_eval), applies per-(arena, target) cooldown, and XADDs to maf:control:in when a rule fires.

The dispatcher reuses the existing control plane — it doesn't run arenas directly. Every trigger leaves an audit trail on the events bus and on maf:control:in.

Example rule from market_pulse.yaml:

triggers:
  - on_stream: kronos:forecasts:emitted
    when: "abs(payload.prob_up_delta) > 0.15 or payload.direction_flipped"
    target: { ticker: "{payload.symbol}" }
    cooldown_s: 60
    action_mode: semi

Layer 4 — Arena consumption

Specialists read from the caches via standard source adapters. MAF itself stays torch-free, never speaks Neo4j directly, and only HTTP-s to fomo2's request stream when an agent explicitly demands it.

Key adapters: - KronosForecastSource — reads cached Redis key, sets stale_kronos_forecast marker when old. - MirofishSimSource — reads cached sim by report_id or scans by symbol. - Trtools2BarsSource — direct stream tail of trtools2:bars:{tf}. - Trtools2ApiSource — HTTP client for trtools2's dashboard (port 8888 by default). Use this when the agent needs historical bars / snapshots / news that aren't on the live Redis stream.

ReplanAgent reads the gap markers and forces a re-run whenever a cache is stale or missing — so the system self-corrects without manual intervention.

Adapter freshness — declared, not hard-coded

Each adapter ships its own freshness spec via the freshness_spec() classmethod on BaseSource. The Setup tab's freshness badges call this method to learn which Redis stream / key / scan-pattern the adapter consumes, without needing a parallel registry in dashboard code.

Supported spec types: - stream — single Redis Stream; report XLEN + last-id age. - key — TTL'd cache keys; cross-reference an emit_stream for accurate "last refreshed" age (never invents an age from TTL). - scan — pattern scan; report key count + optional emit-stream age. - external — HTTP / SQL / live API; no caching. - request_response — round-trips through Redis Streams.

A small _LEGACY_FRESHNESS_FALLBACK map handles adapters that haven't migrated yet — each row is a candidate for moving to a classmethod with no behaviour change.

Outbox routing

Two outbox streams, picked by the arena's target_key:

The router is MAFApp._publish_arena_output. It reads arena.config.target_key and dispatches accordingly. Trading arenas don't pollute deliberation streams; deliberation arenas don't trigger order routers.

TradingAction details

TradingAction(
  arena, arena_id, correlation_id, published_at,
  target = ActionTarget(ticker, exchange, asset_class, trade_date),
  verdict = "BUY" | "HOLD" | "SELL",
  mode = "auto" | "semi" | "manual",
  sizing = ActionSizing(confidence, ensemble_score, size_fraction, horizon),
  reasoning,
  meta,
)

Consumed by ActionConsumer which runs every action through RiskGate and publishes its decision to maf:executions:out as an ExecutionEnvelope.

GenericDecision details

GenericDecision(
  arena, arena_id, correlation_id, published_at,
  target = dict (opaque),
  target_key,                        # "question_id" / "pr_id" / …
  verdict = str,                     # arena-specific vocabulary
  confidence, reasoning,
  contributors = [{agent, signal, confidence, summary}, …],
  meta,
)

Lifecycle events

Every arena run emits structured events to maf:events via EventBus. The dashboard's Live page pumps them over WebSocket via /ws/events.

Event kinds emitted today:

arena.start         arena.complete         arena.error
phase.start         phase.complete         phase.error
agent.start         agent.complete         agent.error
agent.signal
decision.emit       action.emit
source.fetch        source.error
llm.call            llm.error
control.command     control.ack
system.status

Every event carries kind, arena, arena_id, phase, ts, correlation_id, and a kind-specific payload. See EVENT_KINDS for the full list.

Failures in publishing never block arena execution — the bus is best-effort by design, errors are swallowed-and-logged once-per-streak.

Decision memory + outcome harvesting

For the mastermind arena and any arena that opts in, decisions are persisted into a DecisionMemory backed by HybridMemory (BM25 + ChromaDB). Each Decision carries arena_id so outcomes can be correlated back later.

When the engine fills/closes a position, it publishes to maf:outcomes:out. The ExecutionHarvester tails that stream, finds the matching Decision by arena_id (via add_outcome), and backfills outcome + auto-generated lesson stub. The next mastermind run's recall pass finds the outcome — feedback loop closed.

LLM model selection

pick_model chooses an Ollama Cloud model based on the task profile passed by the caller (or a fallback "quick"/"deep" tier). Profiles map to specific models:

• synthesis, judge, json_strict → gpt-oss:120b (heavy reasoning)
• quick, narrative, classification → glm-4.7 (fast, clean JSON)
• coding, long_context → qwen3-coder:480b
• research → deepseek-v3.1:671b

When MAF_USE_RANKINGS=1, the picker consults RankingsCache — a TTL-cached join of OpenRouter's usage leaderboard + Artificial-Analysis benchmark scores against your Ollama Cloud catalog. The picker then prefers the highest-ranked Ollama-available model for the relevant benchmark dimension.

Channel discovery + preview

The Channels page is backed by two pure functions:

• discover_channels — merges three sources of truth (static catalogue, StreamsConfig, arena output_streams) with a live Redis SCAN over known prefixes, then hydrates length + last_id.
• preview_stream — XREVRANGE the N most-recent entries, JSON-decode field-by-field, unwrap the common {"data": "<json>"} envelope.
• infer_schema — walk decoded entries, count types per key, gather sample values, flag nested fields.

These functions are reusable from any caller — they're not tied to the dashboard. A future "what does this channel emit?" CLI command would reuse them verbatim.

Dashboard internals — routers, state, freshness

The dashboard is a FastAPI app split into ~12 router modules, each owning one concern. The split was a pure refactor — same routes, same behaviour, easier to extend.

src/maf/dashboard/
├── api.py              — owns `app = FastAPI(...)`, includes routers,
│                         keeps a handful of /api/data/* + helpers.
├── state.py            — get_maf_app() / get_trail() / get_redis_url()
│                         / require_maf_app(). Single source of truth
│                         for runtime mutables, no circular imports.
├── ws.py               — WebSocket: /ws/events + /ws/arenas/{name}.
├── docs_render.py      — markdown + source viewer renderers.
└── routers/
    ├── arenas.py       — /api/arenas/* + / + /arenas/{name}
    ├── channels.py     — /api/channels{,preview} + /channels
    ├── docs.py         — /docs, /docs/{slug}, /source/{path}, /docs.css
    ├── llm.py          — /api/llm/{rankings,picker} + /llm
    ├── pages.py        — /live, /data, /modules, /sources, /wizard,
    │                     /oracle, /mastermind, /more (redirect), /favicon
    ├── sources.py      — /api/sources*, /api/modules*, /api/config*
    ├── specialised.py  — /api/oracle/envelopes, /api/mastermind/decisions
    ├── system.py       — /api/system/status (cached 5s),
    │                     /api/streams/health,
    │                     /api/arenas/{name}/freshness
    ├── triggers.py     — /api/triggers/library, /api/triggers/validate
    ├── watch.py        — /api/watch (GET/POST/DELETE)
    └── wizard.py       — /api/wizard

Routers use APIRouter and are registered into the shared app at import time so TestClient(app) works without calling create_dashboard(). Each router reaches runtime singletons through state.get_maf_app() rather than module-level imports — that's what keeps circular imports out of the picture.

Live JS-syntax check

Every HTML page route has a CI test (test_html_page_inline_js_parses) that renders the page, extracts every <script> block, and runs node --check on it. Catches the "stray ternary breaks all onclick handlers" class of bug before it ships.

Config save safety — pydantic + ETag + atomic

PUT /api/arenas/{name}/config (code) has three guard rails before touching the YAML:

1. Pydantic validation — round-trips through ArenaConfig. Returns 422 with a structured error if any field is wrong type / missing / has an invalid role / etc. The file is not touched on a 422.
2. ETag / If-Match — GET /config sets an ETag header (first 16 chars of sha256 of the on-disk YAML). PUT honours If-Match: if it doesn't match the current ETag, returns 412 Precondition Failed so concurrent editors can't silently overwrite each other. ETag is optional — calls without it skip the check.
3. Atomic write — write to a tempfile next to the YAML, then os.replace(). Either the new content fully replaces the old or nothing changes. No half-written YAML, even on crash mid-write.

The response includes the new ETag so the client can chain subsequent edits without re-fetching.

Refresher heartbeats

Both refreshers write a JSON heartbeat to Redis on every tick:

Payload shape: {ts, watched, min_cadence_s, timeframes} for Kronos; {ts, reports_stream, daily_budget} for MiroFish.

/api/system/status reads these keys to render the kronos_refresher / mirofish_refresher status pills. Green when the heartbeat is younger than the TTL window; red with "not running" hint when the key is missing.

This makes the difference between "configured" and "actually running" visible in the status bar — no SSH required to know whether the worker is alive.

Trigger templates library

config/trigger_templates.yaml ships 7 prebuilt trigger rules (Kronos prob shift, Kronos 1h flip, high-confidence call, MiroFish tickers, trtools2 strategy event, news sentiment spike, fomo2 report). Each entry has:

• id, title, summary, needs — metadata for the picker UI.
• rule — the actual trigger record matching TriggerConfig shape.

The dashboard's Setup tab loads them via triggers_library, which caches by file mtime so edits hot-reload without a server restart. Adding a new template is a YAML append — no Python change.

Validation is via triggers_validate — it runs the user's when: expression through the actual safe_eval parser with a stream-appropriate sample payload, returning the result or the parser error.

Alpaca data via trtools2 — not directly

MAF does not call Alpaca directly. trtools2 owns the Alpaca integration end-to-end (live feed engine, news ingester, backfill, QuestDB persistence). MAF consumes that data via two routes:

The trtools2_api adapter sends X-API-Key from TT2_API_KEY env var (or TRTOOLS2_API_KEY alias). Default base URL http://localhost:8888 overridable via TRTOOLS2_API_URL or per-binding config. Gracefully degrades on connection failures, returning {data: [], error: "..."} so the arena keeps running with an empty payload rather than crashing.

The legacy direct AlpacaSource (code) is kept as a stub for backward-compatibility with old YAML, but it's no longer registered in the adapter dropdown — binding it returns a guidance message pointing at the trtools2 path.

See alpaca_live.yaml for a complete example arena that uses both routes (4 stream-based bindings + 3 HTTP bindings + 2 prewired smart triggers).