Skip to content

Budget

The budget subsystem enforces per-goal and per-day spending caps. It is implemented as a reservation ledger in PostgreSQL with optimistic concurrency control.

Caps

The defaults from frozen-v3/config/routing-policy.yaml:

Scope Soft Hard Soft behavior Hard behavior
Per goal $2.00 $5.00 Pro disabled; Flash and Scout still allowed. STOP_FOR_HUMAN.
Per day $10.00 $25.00 Same. Same.

These are deliberately small. The frozen design (00-architecture-decisions.md) calls them out as "deliberately review-required values". To change them you must edit the YAML, freeze the policy, and release.

Reservation lifecycle

stateDiagram-v2
    [*] --> RESERVED: reserve(key, amount)
    RESERVED --> ACTIVE: mark_active()
    RESERVED --> RELEASED: release() / sweep_orphans()
    ACTIVE --> SETTLED: settle(actual_cost)
    ACTIVE --> ORPHANED: stale heartbeat
    ORPHANED --> RELEASED: sweep_orphans()
    SETTLED --> [*]
    RELEASED --> [*]
State Meaning
RESERVED Funds locked, not yet spent.
ACTIVE A worker is using the reservation.
SETTLED Actual cost recorded; remainder (if any) released.
RELEASED Funds returned to the ledger.
ORPHANED Worker crashed or heartbeat stopped. Eligible for sweep.

The state machine is enforced in src/smart_gateway/budget_state.py.

Database layout

CREATE TABLE budget_ledger (
    id        SERIAL PRIMARY KEY,
    available BIGINT NOT NULL,
    hard_cap  BIGINT NOT NULL,
    version   BIGINT NOT NULL
);

CREATE TABLE budget_reservations (
    key              TEXT PRIMARY KEY,
    amount_reserved  BIGINT NOT NULL,
    amount_settled   BIGINT NOT NULL,
    state            TEXT NOT NULL,
    version          BIGINT NOT NULL,
    last_heartbeat   TIMESTAMP
);

The ledger row is id=1 (a single global ledger). Reservations reference it via key and rely on PostgreSQL's transactional semantics plus an explicit version column to detect concurrent modifications.

Concurrency

src/smart_gateway/db.py::BudgetDB.reserve uses SELECT … FOR UPDATE-style behavior via a transaction and an optimistic version check:

UPDATE budget_ledger
SET available = available - $1, version = version + 1
WHERE id = $2 AND version = $3

If the rowcount is 0, a ConcurrentModificationError is raised and the caller is expected to retry.

Degraded mode

When PostgreSQL is unreachable, the gateway enters SAFE_DEGRADED mode (frozen-v3/10-degraded-mode-spec.md):

  • Privacy hard gate: retained.
  • Explicit route rules: retained.
  • Security / risk hard rules: retained.
  • Cross-worker state: unavailable.
  • Global atomic budget: unavailable.
  • Cross-request retry history: degraded / unavailable.

For a high-risk or Pro request that requires the shared hard budget while the shared DB is unavailable:

MUST FAIL CLOSED or REQUIRE HUMAN APPROVAL. Do not silently downgrade to Flash. Do not claim FULL_STATEFUL guarantees while using worker-local memory.

In practice, the gateway returns STOP_FOR_HUMAN with cause="degraded_mode" for such requests. Low-risk requests that do not require the shared hard budget may still proceed.

What the user sees

There is no per-request budget counter exposed to clients. The gateway enforces the caps internally and emits routing decisions whose cause indicates when a soft or hard cap fired.

The /v1/route/decision dry-run endpoint always assumes budget_spent_usd = 0. To exercise the hard-cap code path, point ROUTING_POLICY_PATH at a policy with per_goal_hard_usd: 0.0001 and re-run the request.

What the operator sees

  • Structured log line: Routing decision made with decision.cause = budget_hard_cap_exceeded or decision.cause = budget_degrade_to_flash.
  • Routing metrics: sgw_routing_decisions_total increments with the matching action and model.
  • PostgreSQL state: rows in budget_ledger and budget_reservations.

A typical operator query:

docker exec smart-supervisor-production-db-1 \
  psql -U sgwproduction -d sgwproduction -c \
  "SELECT available, hard_cap FROM budget_ledger;"

What happens if you exceed the hard cap

  1. The next request returns STOP_FOR_HUMAN (HTTP 403, detail includes cause=budget_hard_cap_exceeded).
  2. The reservation system is not affected; existing reservations settle normally.
  3. The operator can:
  4. raise the cap (requires a release),
  5. clear reservations manually (see docs-internal/operations/budget.md),
  6. restart the gateway (no-op for budget state — the DB is the source of truth).

See also