Asynchronous Requests (Queue API)

Senden Sie lang laufende API-Anfragen an eine Warteschlange zur asynchronen Verarbeitung. Die Queue API spiegelt die Standard-Endpunkte wider, verarbeitet Anfragen jedoch im Hintergrund, was sie ideal fuer Batch-Workloads und grosse Inferenz-Auftraege macht.

Was Sie lernen werden:

Wie Sie Anfragen an die Queue API senden
Wann Sie asynchrone vs. synchrone Endpunkte verwenden sollten
Wie Sie Ergebnisse abfragen und abrufen

Wann die Queue API verwenden

Verwenden Sie die /queue-Endpunkte wenn:

Lang laufende Inferenz — Grosse Modelle oder komplexe Prompts, die synchrone Timeout-Limits ueberschreiten koennten
Batch-Verarbeitung — Viele Anfragen senden, ohne auf jede einzelne warten zu muessen
Hintergrund-Auftraege — Fire-and-Forget-Workloads, bei denen Sie Ergebnisse spaeter verarbeiten
Rate-Limit-Management — Last ueber die Zeit verteilen statt in Spitzen

Fuer interaktive oder Echtzeit-Anwendungsfaelle (Chatbots, Streaming) verwenden Sie stattdessen die Standard-/v2-Endpunkte.

Endpunkt-Zuordnung

Jeder Standard-Endpunkt hat ein /queue-Aequivalent:

Standard-Endpunkt	Queue-Endpunkt	Beschreibung
`POST /v2/chat/completions`	`POST /queue/chat/completions`	Chat Completion
`POST /v2/completions`	`POST /queue/completions`	Text Completion
`POST /v2/embeddings`	`POST /queue/embeddings`	Embeddings
`POST /v2/audio/transcriptions`	`POST /queue/audio/transcriptions`	Audio Transcription
`POST /v2/audio/translations`	`POST /queue/audio/translations`	Audio Translation
`POST /v2/images/generations`	`POST /queue/images/generations`	Image Generation
`POST /v2/images/edits`	`POST /queue/images/edits`	Image Edit
`GET /v2/models`	`GET /queue/models`	Modelle auflisten

Der Request-Body fuer jeden Queue-Endpunkt ist identisch mit seinem Standard-Gegenstueck — aendern Sie einfach den Basispfad von /v2 zu /queue.

Grundlegende Verwendung

Senden Sie eine Chat Completion an die Warteschlange:

curl -X POST "https://llm-server.llmhub.t-systems.net/queue/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Llama-3.3-70B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a detailed analysis of renewable energy trends in Europe."}
    ],
    "max_tokens": 2000
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

response = client.chat.completions.create(
    model="Llama-3.3-70B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a detailed analysis of renewable energy trends in Europe."},
    ],
    max_tokens=2000,
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm-server.llmhub.t-systems.net/queue",
});

const response = await client.chat.completions.create({
  model: "Llama-3.3-70B-Instruct",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Write a detailed analysis of renewable energy trends in Europe." },
  ],
  max_tokens: 2000,
});

console.log(response.choices[0].message.content);

Beispiel fuer Batch-Verarbeitung

Verarbeiten Sie mehrere Prompts effizient ueber die Warteschlange:

Python
Node.js

import asyncio

from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

prompts = [
    "Summarize the benefits of solar energy.",
    "Explain how wind turbines generate electricity.",
    "Describe the future of hydrogen fuel cells.",
]

async def process_prompt(prompt):
    response = await client.chat.completions.create(
        model="Llama-3.3-70B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    return response.choices[0].message.content

async def main():
    tasks = [process_prompt(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    for prompt, result in zip(prompts, results):
        print(f"Prompt: {prompt[:50]}...")
        print(f"Result: {result[:100]}...\n")

asyncio.run(main())

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm-server.llmhub.t-systems.net/queue",
});

const prompts = [
  "Summarize the benefits of solar energy.",
  "Explain how wind turbines generate electricity.",
  "Describe the future of hydrogen fuel cells.",
];

const results = await Promise.all(
  prompts.map((prompt) =>
    client.chat.completions.create({
      model: "Llama-3.3-70B-Instruct",
      messages: [{ role: "user", content: prompt }],
      max_tokens: 500,
    })
  )
);

results.forEach((result, i) => {
  console.log(`Prompt: ${prompts[i]}`);
  console.log(`Result: ${result.choices[0].message.content}\n`);
});

Queue-Endpunkte fuer andere APIs

Embeddings

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

response = client.embeddings.create(
    model="text-embedding-bge-m3",
    input="The benefits of renewable energy in Europe",
)

print(f"Embedding dimension: {len(response.data[0].embedding)}")

Audio Transcription

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

with open("meeting_recording.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
    )

print(transcript.text)

Image Generation

from openai import OpenAI

client = OpenAI(
    base_url="https://llm-server.llmhub.t-systems.net/queue",
)

result = client.images.generate(
    model="gpt-image-1",
    prompt="A futuristic data center powered by renewable energy",
)

print(result.data[0].b64_json[:50] + "...")

Naechste Schritte

Chat Completions — Standard-synchrone Chat-API
Streaming — Echtzeit-Token-fuer-Token-Antworten
Rate Limits — TPM/RPM-Limits und Header verstehen
API-Endpunkte — Vollstaendige Endpunkt-Referenz