Skip to content

Architecture overview

The Smart LLM Supervisor Gateway is a single FastAPI process that makes per-call model-routing decisions and proxies the chat completion request to OpenRouter. It is intentionally small: the intelligence lives in the policy table, not in the code that loads it.

Top-level diagram

flowchart LR
    subgraph Client
        C["Coding client<br/>(Aider, Cline, Roo-Code,<br/>OpenAI SDK, custom)"]
    end

    subgraph Edge
        N["nginx<br/>TLS termination<br/>smart-openrounter.bee1x.one"]
    end

    subgraph App["app container (loopback)"]
        F["FastAPI<br/>/v1/chat/completions<br/>/v1/route/decision<br/>/healthz /metrics"]
        P["Policy engine<br/>src/smart_gateway/policy.py"]
        H["Model health registry<br/>src/smart_gateway/health.py"]
        I["Identity resolver<br/>src/smart_gateway/identity.py"]
        PG["Privacy gate<br/>src/smart_gateway/privacy_gate.py"]
        B["Budget ledger<br/>(PostgreSQL)"]
    end

    subgraph DB
        D[("PostgreSQL 16<br/>budget_ledger<br/>budget_reservations")]
    end

    subgraph Upstream
        OR["OpenRouter<br/>chat completions"]
        M1["DeepSeek V4 Flash"]
        M2["DeepSeek V4 Pro"]
        M3["Nemotron 3 Ultra"]
        M4["GLM-5.2"]
    end

    C -->|HTTPS| N -->|127.0.0.1:9600| F
    F --> I
    F --> PG
    F --> P
    F --> H
    F -->|HTTPS| OR
    OR --> M1
    OR --> M2
    OR --> M3
    OR --> M4
    B --> D

Components

Public layer

  • nginx (host process). TLS termination, ACME renewal, public vhost. The only process that binds a public address. See ops/nginx/smart-openrounter.bee1x.one.conf.
  • app container (smart-supervisor-production-gateway-1). FastAPI on port 8080 inside the container, exposed on the host as 127.0.0.1:9600. Never public.

Application modules

Module Purpose
app.py FastAPI routes. The only HTTP layer.
routing.py / policy.py / supervisor.py Decision logic. policy.decide is the canonical implementation.
identity.py 4-tier session identity resolver.
privacy_gate.py Secret redaction and sensitive-content detection.
health.py Per-model health state machine.
runtime_extract.py Normalizes runtime evidence from client messages.
scorer.py Weighted complexity score (0..100).
budget_state.py Reservation state machine.
db.py PostgreSQL-backed budget ledger with optimistic concurrency.
observability.py Prometheus metrics and structured JSON logs.
clients/openrouter.py Async HTTP client with retry policy.
clients/litellm.py Optional LiteLLM client (not used in production profile).

Configuration

  • frozen-v3/config/models.yaml — alias → real OpenRouter ID mapping.
  • frozen-v3/config/routing-policy.yaml — rule chain and thresholds.

These are mounted read-only into the gateway container (docker-compose.production.yml::volumes). The gateway refuses to start if the freeze ID is changed.

Storage

  • PostgreSQL 16 in a separate container (smart-supervisor-production-db-1). Holds the budget ledger and reservation rows. Volume sgw_production_data on a local Docker volume. No other persistent state lives in the gateway.

Upstream

  • OpenRouter (OPENROUTER_BASE_URL=https://openrouter.ai/api/v1). The gateway forwards the upstream's payload unchanged except for the model ID and a few of OpenRouter's outbound headers.

Request lifecycle

sequenceDiagram
    participant CL as Client
    participant NG as nginx
    participant GW as Gateway
    participant ID as Identity resolver
    participant PG as Privacy gate
    participant PO as Policy engine
    participant MH as Model health
    participant OR as OpenRouter

    CL->>NG: HTTPS POST /v1/chat/completions
    NG->>GW: HTTP POST 127.0.0.1:9600/v1/chat/completions
    GW->>ID: resolve session/identity
    GW->>PG: scan messages for secrets
    PG-->>GW: redacted messages, sensitivity flag
    GW->>PO: decide(task, runtime, policy)
    PO->>MH: model health lookup
    PO-->>GW: RoutingDecision(action, model, cause)
    alt action == DENY
        GW-->>NG: HTTP 403
        NG-->>CL: HTTP 403 + body
    else action == STOP_FOR_HUMAN
        GW-->>NG: HTTP 403 + detail
        NG-->>CL: HTTP 403 + body
    else action in {ROUTE, SCOUT}
        GW->>OR: POST /chat/completions (real model ID)
        OR-->>GW: chat completion (or stream)
        GW-->>NG: HTTP 200 (or SSE)
        NG-->>CL: HTTP 200 (or SSE)
    end

State machine summary

The state machine is in frozen-v3/02-state-machine.md. In short:

INGEST → RESOLVE_SESSION_TASK → EXTRACT_RUNTIME_SIGNALS
       → POLICY_DECISION
         ├─ NEED_SCOUT → SCOUT_RESULT → FINAL_ROUTE
         ├─ HUMAN_REQUIRED
         └─ FINAL_ROUTE
       → FORWARD_MODEL_REQUEST → OBSERVE_RESPONSE_METADATA
       → PERSIST_EVENT → RETURN_TO_CLIENT

What is intentionally absent

  • No agent loop. The gateway never invokes tools, runs shell commands, or modifies the filesystem. The coding client is the agent.
  • No multimodal. Vision inputs are rejected by the policy.
  • No multi-tenant. A single API key, a single production user.
  • No UI. All configuration is YAML files in the deployment directory or environment variables.