Skip to content

Routing behavior

This page explains, in plain terms, how the gateway picks a model for each request. The exact precedence is in Architecture → Routing pipeline. This page is the "what the user observes" companion.

The decision in 60 seconds

For every request, the gateway walks a fixed rule chain:

  1. Did the caller force a tier? smart-router-pro, X-Pro: true, etc. Yes → that tier wins, period.
  2. Is the content sensitive and the model free? Fail closed.
  3. Is the requested tier unhealthy? Substitute a healthy fallback or stop for a human.
  4. Is there an explicit "this one is hard" signal? Force Pro.
  5. Has the same error happened twice already? Force Pro.
  6. Was the last failure structural? Force Pro.
  7. Is the complexity score clearly low? Use Flash.
  8. Is the complexity score clearly high? Use Pro.
  9. Is the score in the gray zone? Ask the Scout (Nemotron) if the task is non-sensitive; otherwise fall back to Flash or Pro by midpoint.
  10. Otherwise? Use Flash.

Later rules never silently override earlier ones. So if step 2 says "stop", no amount of complexity score changes the outcome.

The Flash default

The default worker is deepseek/deepseek-v4-flash-0731. Roughly:

  • ordinary coding tasks,
  • tool loops,
  • tests,
  • small refactors,
  • ordinary reasoning.

Sending smart-router without any other context almost always lands here.

Pro escalation

Pro is forced by any of:

  • Explicit alias or header. smart-router-pro, X-Pro: true, metadata.routing = "pro".
  • High-reasoning directive. The instruction contains /pro, [route:pro], [reasoning:high], reasoning cao, high reasoning, deep reasoning, or think deeply (see frozen-v3/config/routing-policy.yaml::high_reasoning_markers).
  • Hard risk. The task is one of SECURITY, AUTH, PAYMENT, DESTRUCTIVE_MIGRATION. The risk level is CRITICAL. The blast radius is CRITICAL.
  • Volume. ≥ 20 files touched, ≥ 16 tool steps.
  • Repeated failure. Same error twice, or two flash attempts.
  • Structural failure. Last failure class was STRUCTURAL (e.g. SyntaxError, ImportError, SchemaError, ValidationError).
  • Score ≥ 65. The weighted complexity score lands in the "Pro" band.

The exact thresholds live in frozen-v3/config/routing-policy.yaml under hard_pro_rules and score.flash_max / score.pro_min.

Repeated normalized failure

The runtime extractor normalizes error text by stripping UUIDs, timestamps, line numbers, and hex addresses before hashing. Two errors with the same SHA-256 prefix count as one. So a Python ImportError that points to a different file each time still counts as the "same error" for routing purposes.

When that count reaches the same_error_count_gte threshold (default 2), the next request is escalated to Pro. Once Pro succeeds, the counter resets.

Structural failure

A structural failure is one that breaks parsing or interpretation:

  • SyntaxError
  • ImportError, ModuleNotFoundError
  • IndentationError
  • SchemaError, ValidationError
  • NameError, AttributeError

These never benefit from a cheap retry. The gateway escalates to Pro immediately, even on the first attempt.

The classification lives in src/smart_gateway/runtime_extract.py under STRUCTURAL_PATTERNS.

The Scout role

The Scout is nvidia/nemotron-3-ultra-550b-a55b:free. It is invoked only when:

  • the task is non-sensitive,
  • the score is in the gray zone (46–64 by default),
  • the policy enables Scout, and
  • the request did not already force a tier.

The Scout receives the original user instruction with a small "are we PRO or FLASH?" system prompt and replies with a single word. The gateway then routes the original request to Pro or Flash accordingly. The Scout never writes the answer; it only recommends.

The Scout's reply is not authoritative. If the gateway's later rules say Pro, Pro wins. So the Scout can recommend Flash for a structural failure, but the hard rule still escalates.

If the Scout fails (network, 5xx, refusal), the gateway falls back to a deterministic choice based on the score midpoint.

De-escalation

The model chosen for the previous task is recorded in routing events but is not inherited by the next subtask. The next request is reclassified from scratch unless the caller explicitly re-forces a tier.

This is a hard rule (frozen-v3/02-state-machine.md). Without it, a session that started hard would stay hard forever, which destroys the cost benefit of the gateway.

Model-health filtering

The gateway maintains a health registry per upstream model (src/smart_gateway/health.py). The states are HEALTHY, DEGRADED, UNHEALTHY, HALF_OPEN, DISABLED, STALE, QUARANTINE.

If the chosen model is UNHEALTHY, DISABLED, or QUARANTINE, the gateway substitutes:

  • Pro → glm-5.2 (cross-family fallback) or another healthy senior, then Flash as a last resort.
  • Flash → the free model.
  • Free → Flash.

If no healthy substitute exists, the gateway returns STOP_FOR_HUMAN instead of failing the request silently.

The full state-machine semantics are in frozen-v3/07-model-health-registry-spec.md.

Budget fail-closed behavior

The budget policy has two thresholds per scope:

Scope Soft Hard Soft behavior Hard behavior
Per goal $2.00 $5.00 Pro is disabled; Flash and Scout still allowed. STOP_FOR_HUMAN.
Per day $10.00 $25.00 Same. Same.

When the database is unavailable, the gateway enters SAFE_DEGRADED mode and refuses to make shared-budget decisions. A high-risk or Pro request that requires the shared hard budget "must fail closed or require human approval." See Architecture → Budget for the state diagram.

What the user sees

  • A request to smart-router on a non-sensitive, ordinary prompt almost always returns DeepSeek V4 Flash.
  • The same request marked sensitive: true may route identically — the privacy gate only fires when free models are involved.
  • A request that includes [reasoning:high] or /pro returns Pro, unless the budget soft cap is exhausted.
  • A request that fails twice with the same normalized error returns Pro on the third attempt.
  • A smart-router-free request on sensitive content returns HTTP 403 with the standard privacy error message.

You can preview any decision without making an upstream call using POST /v1/route/decision. See API reference → Route decision.

How to debug an unexpected decision

  1. Call POST /v1/route/decision with the same payload you would send to /v1/chat/completions and look at the cause field.
  2. Cross-reference the cause with the rule chain above.
  3. If you suspect a hard rule is mis-firing, check the evidence field — it lists which inputs matched which rule.
  4. If the cause is score_flash or score_pro, examine the evidence array; it includes the per-dimension complexity score.
  5. If the cause is all_models_unhealthy, see Troubleshooting → Providers.