Prefix Caching

Prefix caching lets the model reuse the work it already did for the beginning of a prompt. When two requests start with the same tokens — a long system prompt, a document, a tool catalogue — the shared part does not have to be processed again, which lowers time-to-first-token and, for models that price cached tokens separately, the cost of the request.

What you’ll learn:

How prefix caching works and when it helps
How to enable it for T-Cloud-hosted open-source models with save_cache and pick a sharing scope with cache_salt
How to place cache_control breakpoints for Anthropic models
How to read cache usage from the response

How It Works

The model processes a prompt token by token and builds an internal key/value (KV) cache as it goes. That cache depends only on the tokens seen so far, so the entry for a given prefix is valid for any request that starts with exactly those tokens. Prefix caching stores those entries and reuses them on the next matching request.

Two consequences follow from this, and they drive every recommendation on this page:

Matching is exact and starts at token 0. A single changed character near the start of the prompt — a timestamp, a user name, a re-ordered JSON key — invalidates everything after it.
Only the prefix is reused. Everything from the first differing token onwards is processed normally.

So put the stable parts of your prompt first and the variable parts last:

✅ [ system prompt ][ document ][ few-shot examples ][ user question ]   ← cacheable prefix, variable tail
❌ [ "Current time: 14:32:07" ][ system prompt ][ document ]             ← prefix broken by the first token block

A coding assistant is the clearest case for this. Every turn resends the same coding instructions, the same tool definitions and the same source files, and only the developer’s latest instruction changes — so almost the entire prompt can come from cache:

✅ [ coding system prompt ][ tool definitions ][ repository files ][ conversation so far ][ new instruction ]
   └──────────────────── stable across the whole session ────────────────────┘ └─ grows ─┘ └─ changes ─┘

❌ [ "Session 8f3a · 2026-07-28 14:32" ][ coding system prompt ][ repository files ][ new instruction ]
   └─ a per-session header at position 0 makes every session start from a cold cache ─┘

Two details matter here:

Keep the file list in a fixed order. Reading a directory and sending the files in whatever order the filesystem returns produces a different prefix on every run. Sort by path.
Append, never rewrite. Adding the latest turn at the end keeps the previous prompt as a prefix. Re-summarising or re-ordering earlier turns invalidates the cache from the point of the change.

Prefix caching never changes the content of a response: the model sees the same tokens either way. It only affects latency and token accounting.

T-Cloud Hosted Open-Source Models

For T-Cloud-hosted open-source models (for example Llama-3.3-70B-Instruct, Qwen3-30B-A3B-FP8), prefix caching is opt-in per request. Two extra fields on the chat completion request control it:

Field	Type	Default	Purpose
`save_cache`	boolean	`false`	Master switch — must be explicitly `true` for the prompt prefix to be stored and reused
`cache_salt`	string	(empty)	Selects who may reuse the prefix. Only evaluated when `save_cache` is `true`

Both are extensions to the OpenAI schema, so with the OpenAI SDKs you pass them through extra_body (Python) or as extra request fields — see the examples below.

Cache Scope

save_cache decides whether the prefix is cacheable, cache_salt decides how widely it is shared. Requests only reuse each other’s prefix when they resolve to the same scope:

`save_cache`	`cache_salt`	Who can reuse the prefix
`false` or omitted	ignored	Nobody — prefix caching is off for this request
`true`	omitted / empty	Requests using the same API key
`true`	custom string	Requests in your project using the same salt — e.g. one chat session, one document, one batch job
`true`	`"org"`	Any request in the same project / organisation, across all its API keys
`true`	`"global"`	The model’s shared global pool — reuse across all tenants on that model

Wider scopes raise the hit rate but reduce prompt isolation, so the scope is always an explicit opt-in. Pick the narrowest one that gives you the reuse you need — and keep confidential prompts out of "global", whose pool is shared with other tenants.

Example

The system prompt and the document below are identical on every call, so only the question at the end has to be processed. cache_salt is set to a per-document value, which lets every question about that contract share one cache entry:

from openai import OpenAI

client = OpenAI()

SYSTEM_PROMPT = "You are a contract analyst. Answer only from the document."
DOCUMENT = open("contract.txt").read()  # long, stable, identical for every question

def ask(question: str):
    return client.chat.completions.create(
        model="Llama-3.3-70B-Instruct",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"{DOCUMENT}\n\nQuestion: {question}"},
            #                             ^ stable prefix     ^ variable tail
        ],
        temperature=0.1,
        extra_body={
            "save_cache": True,              # enable prefix caching
            "cache_salt": "contract-4711",   # shared by every question about this contract
        },
    )

for question in ["Who are the parties?", "When does it expire?", "What is the notice period?"]:
    response = ask(question)
    print(response.choices[0].message.content)
    print("cached tokens:", response.usage.prompt_tokens_details.cached_tokens)
    # First call warms the cache; the following calls reuse the document prefix.

curl -X POST "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Llama-3.3-70B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a contract analyst. Answer only from the document."},
      {"role": "user", "content": "<long document text>\n\nQuestion: What is the notice period?"}
    ],
    "temperature": 0.1,
    "save_cache": true,
    "cache_salt": "contract-4711"
  }'

import OpenAI from "openai";
import { readFileSync } from "node:fs";

const client = new OpenAI();

const SYSTEM_PROMPT = "You are a contract analyst. Answer only from the document.";
const DOCUMENT = readFileSync("contract.txt", "utf8"); // long, stable, identical for every question

const ask = (question) =>
  client.chat.completions.create({
    model: "Llama-3.3-70B-Instruct",
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      { role: "user", content: `${DOCUMENT}\n\nQuestion: ${question}` },
    ],
    temperature: 0.1,
    // @ts-expect-error — platform extensions, not part of the OpenAI schema
    save_cache: true,
    cache_salt: "contract-4711",
  });

for (const question of ["Who are the parties?", "When does it expire?"]) {
  const response = await ask(question);
  console.log(response.choices[0].message.content);
}

Choosing a Salt

Omit it when one API key does all the work — the default per-key scope already covers a single service.
Use a custom salt to group requests that legitimately share a prefix: a session ID, a document ID, a batch job name. Different salts are isolated from each other even for identical prompts.
Use "org" when several API keys of the same project hit the same prefix — for example a shared system prompt used by multiple services.
Use "global" only for content that is not sensitive, such as a public boilerplate prompt.
Set save_cache: false (or leave it out) for one-off or sensitive prompts you do not want reused at all.

Cache Control for Anthropic Models

Claude models do not cache implicitly. You mark the end of the cacheable region yourself by attaching cache_control to a content block; everything up to and including that block becomes the cached prefix.

This requires the structured content-block form of a message (a list of parts instead of a plain string):

Python
curl

from openai import OpenAI

client = OpenAI()

DOCUMENT = open("contract.txt").read()

response = client.chat.completions.create(
    model="claude-sonnet-4",
    messages=[
        {
            "role": "system",
            "content": [
                {"type": "text", "text": "You are a contract analyst. Answer only from the document."},
                {
                    "type": "text",
                    "text": DOCUMENT,
                    "cache_control": {"type": "ephemeral"},  # ← cache everything up to here
                },
            ],
        },
        {"role": "user", "content": "What is the notice period?"},  # variable tail, not cached
    ],
)

print(response.choices[0].message.content)

curl -X POST "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "messages": [
      {
        "role": "system",
        "content": [
          {"type": "text", "text": "You are a contract analyst. Answer only from the document."},
          {
            "type": "text",
            "text": "<long document text>",
            "cache_control": {"type": "ephemeral"}
          }
        ]
      },
      {"role": "user", "content": "What is the notice period?"}
    ]
  }'

Points to keep in mind:

The first request writes the cache, later ones read it. A cold call is slightly more expensive than an uncached one; the saving appears from the second matching request onwards.
Entries are short-lived. An ephemeral entry expires a few minutes after its last use, so caching pays off for bursts of related calls (an agent loop, a chat session, a batch over one document) rather than for requests spread over hours.
Short prefixes are not cached. Providers enforce a minimum cacheable prefix length — on the order of a thousand tokens — below which the breakpoint is ignored.
Place breakpoints on stable content only. A breakpoint after text that changes every call caches something that will never be read again.

Reading Cache Usage

The usage object of the response reports how much of the prompt came from cache. Check it before and after a change to confirm your prefix is actually being reused:

response = client.chat.completions.create(
    model="Llama-3.3-70B-Instruct",
    messages=messages,
    extra_body={"save_cache": True, "cache_salt": "contract-4711"},
)

usage = response.usage
cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0) if usage.prompt_tokens_details else 0

print(f"prompt tokens: {usage.prompt_tokens}")
print(f"cached tokens: {cached} ({cached / usage.prompt_tokens:.0%} of the prompt)")

Field	Meaning
`usage.prompt_tokens`	Total prompt tokens for the request
`usage.prompt_tokens_details.cached_tokens`	Prompt tokens served from cache
`usage.completion_tokens`	Generated tokens (never cached)

For Claude models the response may additionally carry provider-specific counters — cache_creation_input_tokens for tokens written to the cache and cache_read_input_tokens for tokens read from it.

If cached_tokens stays at 0 across repeated calls, work through this list:

save_cache is not true — the master switch is off, so nothing is stored (open-source models).
The two requests use different cache_salt values, which isolates them from each other.
A variable token sits near the start of the prompt and breaks the match.
The shared prefix is too short — aim for roughly a thousand tokens or more before expecting a hit.

Best Practices

Order by stability. System prompt → tool definitions → documents → few-shot examples → conversation → current question.
Set the scope once, in one place. Derive cache_salt from the thing the prefix belongs to (session, document, tenant) in a single helper, so every call site produces the same value.
Keep the prefix byte-identical. Build it from a constant, not from an f-string that interpolates a timestamp, request ID, or user name. If you need those, put them in the tail.
Serialise deterministically. json.dumps(..., sort_keys=True) for any structured data in the prefix; unordered dict output silently breaks matching.
Batch related calls together. Ten questions about one document, asked back to back, hit a warm cache. The same ten spread over a day mostly do not.
Do not restructure prompts blindly. Move content into the prefix only when the model output stays correct — cache efficiency is worth nothing if the answer degrades.

Limitations

Off by default. For open-source models nothing is cached unless save_cache is explicitly true.
Caching is best-effort. Entries are evicted under memory pressure and are not shared across model versions or deployments.
Exact token match only. Semantically identical but differently worded prefixes do not match, and a different cache_salt isolates them even when the tokens are identical.
Only the prompt is cached. Generated tokens are always computed fresh, so caching does not speed up long outputs.
Short prefixes are not cached. Below roughly a thousand tokens of shared prefix, expect no measurable reuse.

Next Steps

Chat Completions — Standard chat API usage
Streaming — Combine with streaming for the lowest perceived latency
Function Calling — Keep tool definitions in the cached prefix
Rate Limits — Throughput and quota behaviour