Errors and reliability

Retry a failed call when the status is temporary. Error codes lists every status. Queued requests wait for rate-limit budget instead of returning 429 immediately.

Prerequisites

• An API key (get one here)
• The client from the quickstart

Retries

Minimal example

On HTTP 429, wait 2 ** attempt seconds and try again. Stop after 3 attempts. The model id is gpt-oss-120b.

Python

import time
from openai import OpenAI, RateLimitError

client = OpenAI()

def call_with_retry(messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model="gpt-oss-120b",
                messages=messages,
            )
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            wait = 2 ** attempt
            print(f"Rate limited. Retrying in {wait}s...")
            time.sleep(wait)

On 429 the function waits and tries again. On the third failure it re-raises the RateLimitError.

Parameters

max_retries is 3. The wait is 2 ** attempt seconds (1s, then 2s).

Limitations and troubleshooting

• 429 means the rate limit was exceeded. Wait and retry with exponential backoff, or use queued requests.
• 500 is a server-side error. Retry after a brief delay.
• 503 means the model is temporarily unavailable. Check model status and retry.
• 400, 401, 403, and 404 are request or credential errors. Retrying the same request will not succeed. See Error codes.

When a queued request is still over the limit after two minutes, the 429 carries a Retry-After header. See Queued requests.

Queued requests

The /queue endpoints work like the /v2 endpoints, with one difference: when you are over your rate limit, /v2 returns 429 Too Many Requests right away, while /queue waits up to two minutes for your limit to free up and then runs the request.

Caution

/queue is not asynchronous. There is no job ID and nothing to poll: you get the response on the same connection, as with /v2. It does not make requests faster or extend their timeouts.

How it works

• Within your limit: the request runs immediately.
• Over your limit: the request waits. Limits reset every minute, so it is checked again at the start of each minute.
• Still over your limit after two minutes: the wait is capped. The request stops waiting and you get 429 Too Many Requests with a Retry-After header telling you how many seconds to wait before retrying. The body says Queue wait limit of 120 seconds reached. The cap exists so that a starved project gets a clear answer instead of an opaque 504 Gateway Timeout from the network layer.

The cap is measured against the calendar minute, not from the moment your request arrives: the request is checked when it arrives and again at the next two minute resets, and gives up after the second one. So the actual wait is between one and two minutes, depending on when in the minute you sent the request.

Set your client timeout to at least two minutes plus the expected response time, otherwise your client may give up before the queue does. If you close the connection while waiting, the request is cancelled.

A 429 from /queue means you are sending more than your limit allows, not just a short burst. A single request with more input tokens than your per-minute token limit never fits and always times out after two minutes. Slow down, or ask for a higher limit.

When to use the queue API

• Use /queue for batch jobs that sometimes go over the limit, so short bursts wait instead of failing.
• Use /v2 for interactive apps, where an immediate 429 is better than waiting, and for long-running requests (use streaming).

Endpoint mapping

Every standard endpoint has a /queue equivalent:

Standard Endpoint	Queue Endpoint	Description
POST /v2/chat/completions	POST /queue/chat/completions	Chat completion
POST /v2/completions	POST /queue/completions	Text completion (/v2/completions is not available on this host at the moment; see API endpoints)
POST /v2/embeddings	POST /queue/embeddings	Embeddings
POST /v2/audio/transcriptions	POST /queue/audio/transcriptions	Audio transcription
POST /v2/audio/translations	POST /queue/audio/translations	Audio translation
POST /v2/images/generations	POST /queue/images/generations	Image generation
POST /v2/images/edits	POST /queue/images/edits	Image editing
GET /v2/models	GET /queue/models	List models

The request body for each queue endpoint is identical to its standard counterpart — just change the base path from /v2 to /queue.

Basic usage

Submit a chat completion to the queue:

curl

Terminal

curl -X POST "https://llm-server.llmhub.t-systems.net/queue/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a detailed analysis of renewable energy trends in Europe."}
    ],
    "max_tokens": 2000
  }'

Python

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

response = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a detailed analysis of renewable energy trends in Europe."},
    ],
    max_tokens=2000,
)

print(response.choices[0].message.content)

Node.js

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm-server.llmhub.t-systems.net/queue",
});

const response = await client.chat.completions.create({
  model: "gpt-oss-120b",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Write a detailed analysis of renewable energy trends in Europe." },
  ],
  max_tokens: 2000,
});

console.log(response.choices[0].message.content);

Batch processing example

Process multiple prompts efficiently using the queue:

Python

import asyncio

from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

prompts = [
    "Summarize the benefits of solar energy.",
    "Explain how wind turbines generate electricity.",
    "Describe the future of hydrogen fuel cells.",
]

async def process_prompt(prompt):
    response = await client.chat.completions.create(
        model="gpt-oss-120b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    return response.choices[0].message.content

async def main():
    tasks = [process_prompt(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    for prompt, result in zip(prompts, results):
        print(f"Prompt: {prompt[:50]}...")
        print(f"Result: {result[:100]}...\n")

asyncio.run(main())

Node.js

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm-server.llmhub.t-systems.net/queue",
});

const prompts = [
  "Summarize the benefits of solar energy.",
  "Explain how wind turbines generate electricity.",
  "Describe the future of hydrogen fuel cells.",
];

const results = await Promise.all(
  prompts.map((prompt) =>
    client.chat.completions.create({
      model: "gpt-oss-120b",
      messages: [{ role: "user", content: prompt }],
      max_tokens: 500,
    })
  )
);

results.forEach((result, i) => {
  console.log(`Prompt: ${prompts[i]}`);
  console.log(`Result: ${result.choices[0].message.content}\n`);
});

Queue endpoints for other APIs

Embeddings

Python

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

response = client.embeddings.create(
    model="text-embedding-bge-m3",
    input="The benefits of renewable energy in Europe",
)

print(f"Embedding dimension: {len(response.data[0].embedding)}")

Audio transcription

Python

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

with open("meeting_recording.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
    )

print(transcript.text)

Image generation

Python

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

result = client.images.generate(
    model="gpt-image-2",
    prompt="A futuristic data center powered by renewable energy",
)

print(result.data[0].b64_json[:50] + "...")

Next steps

Error codes Status codes, causes, and what to change.

Rate limits RPM and TPM for each plan, and the rate-limit headers.

API endpoints Every endpoint, including the queue variants.

Streaming Real-time token-by-token responses.