Streaming

The chat completions endpoint supports Server-Sent Events when "stream": true is set in the request body. The wire format is the standard OpenAI SSE shape; only the deepest internals differ.

Enabling streaming

Set stream to true in either the JSON body or the framework you are using:

curl -sN "$SMART_GATEWAY_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $SMART_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "smart-router-flash",
        "stream": true,
        "messages": [{"role": "user", "content": "Count to 5."}],
        "max_tokens": 64
      }'

The response is text/event-stream. Each event is one line of data: {…} followed by a blank line. The terminal event is data: [DONE].

Chunk shape

Each chunk is an OpenAI chat completion chunk:

{
  "id": "gen-…",
  "object": "chat.completion.chunk",
  "created": 1755000000,
  "model": "deepseek/deepseek-v4-flash-0731",
  "choices": [
    {
      "index": 0,
      "delta": {"role": "assistant", "content": "1"},
      "finish_reason": null
    }
  ]
}

The model field echoes the real OpenRouter model ID, not the alias.

Reasoning-only deltas

DeepSeek V4 models can produce "extended thinking" output. In that mode the upstream stream emits chunks where delta.reasoning is set but delta.content is null:

{
  "choices": [
    {
      "index": 0,
      "delta": {"reasoning": "Let me think about this carefully..."}
    }
  ]
}

Aider and some other OpenAI-compatible clients wait for a non-null content delta and treat reasoning-only chunks as "no reply yet". This produces an empty-looking final answer.

The gateway runs a small SSE normalizer on its way out (_stream_with_reasoning_normalization in src/smart_gateway/app.py):

• While no content delta has arrived, reasoning text accumulates.
• When the first content delta arrives, it is prefixed with the accumulated reasoning text.

The visible effect for clients is that the first content chunk contains the model's reasoning, then normal tokens continue. This keeps Aider 0.86.x working with --stream=true for normal coding tasks.

Known caveat

The streaming UX is acceptable for normal Aider usage (--no-stream is also supported as a workaround). For long reasoning passages the prefix can be large; if you need byte-perfect reasoning passthrough, disable the normalizer by using a non-DeepSeek model.

Timeouts

Streaming responses are subject to:

• Gateway → OpenRouter: 60 second per-request timeout, with up to 3 retries on 429/5xx (src/smart_gateway/clients/openrouter.py).
• nginx → gateway: proxy_read_timeout 300s (ops/nginx/smart-openrounter.bee1x.one.conf).

A typical streaming response completes in 1–5 seconds.

Disabling streaming

Set "stream": false (or omit the field). The response is then a single JSON object identical to the non-streaming OpenAI shape, with one addition: if the upstream payload puts reasoning into message.reasoning and leaves message.content null, the gateway copies the reasoning text into message.content so that the response is usable. See _normalize_extended_thinking_response in src/smart_gateway/app.py.

Canceling a stream

Close the connection from the client side. The gateway will cancel the upstream httpx request. The structured routing log records the request as completed (status="200") — the metrics layer does not distinguish "client closed early" from "client read everything" in v1.0.0.

Worked Aider example

.aider.model.settings.yml:

- name: smart-supervisor
  chat_model:
    api_base: https://smart-openrounter.bee1x.one/v1
    api_key: env:SMART_GATEWAY_API_KEY
    model: smart-router
    stream: true
  weak_model:
    api_base: https://smart-openrounter.bee1x.one/v1
    api_key: env:SMART_GATEWAY_API_KEY
    model: smart-router-flash

Then run:

export SMART_GATEWAY_API_KEY="<your-gateway-key>"
aider --model smart-supervisor path/to/file.py

If you see empty Aider responses, fall back to --no-stream and the behavior is identical to a non-streaming request.