API Transition: Streaming

The New API (https://llm-server.ai.t-systems.net/v1) continues to support Server-Sent Events (SSE). Most streaming behavior remains compatible, but token usage information is returned differently. For streaming basics, see Streaming.

What Changes?

Behavior	Legacy API	New API
Streaming protocol	SSE	SSE
Token usage	Separate final usage chunk	Included in the completion’s final chunk
Usage by default	Included without an explicit request	Included without an explicit request
Completion marker	`[DONE]`	`[DONE]`
Errors during streaming	Stream may terminate without an explicit error event	Structured errors may be included in the stream

{
  "choices": [],
  "usage": {
    "total_tokens": 150
  }
}

{
  "choices": [
    {
      "index": 0,
      "delta": {},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "total_tokens": 150
  }
}

Both APIs include token usage in successful streaming responses by default, even when stream_options.include_usage is not explicitly requested. The main difference is where this information appears in the stream.

What You Need to Check

Applications that process SSE responses directly should read token usage from whichever chunk contains the usage field, rather than relying on a separate chunk with an empty choices array.

Do not assume that streaming usage details will have an identical structure across all model providers.