Skip to content

Routing pipeline

The routing pipeline is a fixed-precedence rule chain implemented in src/smart_gateway/policy.py::decide. This page documents the exact precedence and the inputs each rule looks at.

Precedence (high to low)

flowchart TD
    S0["Request received"] --> S1{"1. Explicit route forced?<br/>(model alias / X-Pro / X-Scout)"}
    S1 -- "yes" --> S2["Apply that tier<br/>(subject to health & privacy)"]
    S1 -- "no" --> S3{"2. High-reasoning marker?<br/>(/pro, [reasoning:high],<br/>'think deeply', ...)"}
    S3 -- "yes" --> S4["Pro<br/>(unless budget soft cap)" ]
    S3 -- "no" --> S5{"3. Hard Pro rule?<br/>(AUTH, SECURITY, PAYMENT,<br/>DESTRUCTIVE_MIGRATION, CRITICAL risk,<br/>≥20 files, ≥16 tool steps,<br/>same error twice, structural failure)"}
    S5 -- "yes" --> S4
    S5 -- "no" --> S6{"4. Bounded free task?<br/>(SUMMARIZE, CLASSIFY, PUBLIC_REVIEW,<br/>non-sensitive)"}
    S6 -- "yes" --> S7["Nemotron (free)"]
    S6 -- "no" --> S8{"5. Score ≤ flash_max (45)?"}
    S8 -- "yes" --> S9["Flash"]
    S8 -- "no" --> S10{"6. Score ≥ pro_min (65)?"}
    S10 -- "yes" --> S4
    S10 -- "no" --> S11{"7. Scout enabled & non-sensitive?<br/>(gray zone 46–64)"}
    S11 -- "yes" --> S12["SCOUT → Nemotron recommends Pro or Flash"]
    S11 -- "no" --> S13{"8. Sensitive + gray zone"}
    S13 -- "yes" --> S14["Flash (≤ midpoint) or Pro (≥ midpoint)"]
    S13 -- "no" --> S9

Later rules never silently override earlier ones. If step 1 says "force Flash", step 4 (bounded free task) cannot turn the request into a Nemotron call.

Inputs

Task-level

Input Source Used by
task.explicit_route model alias, X-Pro, X-Scout step 1
task.reasoning metadata.reasoning, instruction markers step 2
task.instruction request body step 2 (markers)
task.kind metadata.kind step 3
task.risk_level metadata.risk_level step 3
task.blast_radius metadata.blast_radius step 3
task.sensitive metadata.sensitive, default true privacy (everywhere)
task.repo_context_present metadata.repo_context_present step 4
task.repo_free_opt_in metadata.repo_free_opt_in step 4
task.{reasoning_requirement, ambiguity, dependency_depth, verification_difficulty} request step 5 score
task.estimated_tool_depth request step 5 score

Runtime-level

Input Source Used by
runtime.attempts X-Attempts header / metadata.attempts informational
runtime.flash_attempts header / metadata step 3
runtime.same_error_count X-Same-Error-Count step 3
runtime.tests_failed header / metadata informational
runtime.files_touched header / metadata step 3
runtime.tool_depth header / metadata step 3
runtime.failure_class X-Failure-Class (STRUCTURAL triggers hard Pro) step 3

Health-level

Input Source Used by
model_health[model] in-memory registry inside route() after a tier is chosen

Budget-level

Input Source Used by
budget_spent_usd per-goal counter in DB inside route()

Step 1 — Explicit override

The earliest, strongest signal. Aliases (smart-router-pro, …) and headers (X-Pro, X-Scout) both reach this step. The chosen tier is then run through the privacy and health filters below.

Step 2 — High-reasoning marker

Markers from frozen-v3/config/routing-policy.yaml:

high_reasoning_markers:
  - "/pro"
  - "[route:pro]"
  - "[reasoning:high]"
  - "reasoning cao"
  - "high reasoning"
  - "deep reasoning"
  - "think deeply"

A case-insensitive substring match. Marker forced → Pro unless the budget soft cap is exceeded.

Step 3 — Hard Pro rule

Any of:

  • task.kind ∈ {SECURITY, AUTH, PAYMENT, DESTRUCTIVE_MIGRATION}
  • task.risk_level == CRITICAL
  • task.blast_radius == CRITICAL
  • runtime.same_error_count ≥ 2
  • runtime.flash_attempts ≥ 2
  • runtime.files_touched ≥ 20
  • runtime.tool_depth ≥ 16
  • runtime.failure_class == STRUCTURAL

Step 4 — Bounded free task

A small set of public tasks can run on Nemotron directly:

  • task.kind ∈ {SUMMARIZE, CLASSIFY, PUBLIC_REVIEW}
  • task.sensitive == false
  • If task.repo_context_present, then task.repo_free_opt_in == true

Steps 5 & 6 — Score bands

scorer.complexity_score produces a 0..100 weighted score:

Dimension Weight
reasoning_requirement 20
ambiguity 10
blast_radius 15
dependency_depth 10
tool_depth 10
verification_difficulty 10
runtime_failure 15
business_risk 10

Bands (configurable, defaults shown):

Score Decision
score ≤ 45 Flash
46 ≤ score ≤ 64 Gray zone
score ≥ 65 Pro

Step 7 — Scout

If Scout is enabled, the request is non-sensitive, and the score is in the gray zone, the gateway calls Nemotron with the user's prompt plus a small "are we PRO or FLASH?" system prompt and forwards the request to the model Nemotron recommends.

The Scout's recommendation is not authoritative; it cannot override any earlier rule.

Step 8 — Gray-zone tie-break

If the score is in the gray zone but Scout cannot run (sensitive or disabled), the gateway picks Flash below the midpoint and Pro above the midpoint. The midpoint is (flash_max + pro_min) / 2 = 55.

Privacy and health filters (inside every step)

After a model is chosen, two filters apply:

  1. Privacy. If the chosen tier is the free model and task.sensitive is true, the decision becomes STOP_FOR_HUMAN with cause="free_route_blocked_by_privacy" and sensitive_block_applied=true.
  2. Health. If the chosen tier is in {UNHEALTHY, QUARANTINE, DISABLED, DOWN}, the gateway substitutes a healthy fallback:
  3. Pro → glm-5.2 or another healthy senior, then Flash.
  4. Flash → Nemotron.
  5. Nemotron → Flash.

If no substitute is healthy, the decision becomes STOP_FOR_HUMAN with cause="all_models_unhealthy".

Outputs

The pipeline returns a RoutingDecision (src/smart_gateway/domain.py) with action, model, purpose, cause, score, confidence, hard_rule, sensitive_block_applied, evidence, and policy_version. This is the exact shape returned by /v1/route/decision.

Why this shape

  • Deterministic. No learning, no in-band model choice, no prompt-only router. Every step is a YAML rule.
  • Auditable. cause and evidence tell you exactly which rule fired.
  • Layered. Privacy is above health is above budget is above reasoning. A later rule cannot undo an earlier rule's safety guarantee.
  • Tunable in one file. Thresholds, markers, and the model registry all live in frozen-v3/config/. Changing a value requires editing the file, restarting the gateway, and observing the next request.