Token Budget Exhaustion in Production: Designing Fallback Chains When LLM Requests Fail Mid-Stream

You're streaming a response to a user. Twenty seconds in, the LLM hits its context window limit. The stream terminates mid-sentence. You have 200ms to decide: degrade gracefully, retry with a smaller model, reshape the request, or tell the user something went wrong.

This isn't a hypothetical. It happens regularly in production, and generic error handling won't save you. This post covers the operational reality of LLM failures and concrete patterns for building fallback chains that actually work under real constraints.

The Problem Space: Why LLM Failures Are Different

Traditional service failures are usually binary: the service responds or it doesn't. LLM failures are messier:

• Truncated responses: The model generates output but hits token limits before finishing. You have partial, often unusable data.
• Context window exhaustion: Your carefully crafted prompt + user input + retrieved context exceeds the model's limit. The request never completes.
• Rate limit cascades: Provider hits rate limits. You retry. You hit rate limits again. Your retry budget evaporates.
• Streaming interruptions: The connection closes after 15 tokens out of 500. Your client has incomplete output and no clear signal about whether to retry.
• Silent degradation: The model returns a response, but it's hallucinated or nonsensical because the context was truncated before reaching critical information.

The operational challenge: your user sees something fail, but the failure mode isn't always obvious. A truncated response looks like a complete response. A rate-limited retry looks like latency. A context-window error looks like a timeout.

Metric-Driven Fallback Design

Before implementing fallbacks, you need visibility into failure modes. These metrics matter:

Instrument these early. You'll need them to decide which fallback strategies are actually worth the complexity.

Pattern 1: The Cached Response Fallback

When to use: When you can tolerate stale or pre-computed answers. Common in Q&A, summarization, and classification tasks where the answer space is bounded.

How it works: Before calling the LLM, check if you have a cached response for this request (or a semantically similar one). If the primary LLM call fails, return the cached response instead of erroring.

class CachedLLMFallback:
    def __init__(self, llm_client, cache):
        self.llm = llm_client
        self.cache = cache
    
    def query(self, prompt, cache_key=None):
        cache_key = cache_key or hash(prompt)
        
        try:
            # Primary strategy: fresh LLM response
            response = self.llm.generate(prompt, timeout=5.0)
            self.cache.set(cache_key, response, ttl=3600)
            return response, "primary"
        except TokenLimitExceeded:
            # Fallback: cached response
            cached = self.cache.get(cache_key)
            if cached:
                logger.warning(f"Returning cached response for {cache_key}")
                return cached, "fallback_cache"
            else:
                raise
        except RateLimitError:
            # For rate limits, cached response is better than nothing
            cached = self.cache.get(cache_key)
            if cached:
                return cached, "fallback_cache_ratelimit"
            else:
                # No cache, will retry with exponential backoff
                raise

Trade-offs:

• ✅ Zero additional latency (cache lookup is sub-millisecond)
• ✅ Reduces load on LLM provider during outages
• ✅ Requires no additional model calls
• ❌ Serves stale data; users may see outdated answers
• ❌ Only works if you have cached responses (first-time users get nothing)
• ❌ Doesn't help with malformed requests or context-window issues

When it breaks: User asks a time-sensitive question ("What happened today?") and you return yesterday's cached answer. The cache is useless if your cache hit rate is low.

Pattern 2: Tiered Model Fallback

When to use: When you can trade response quality for reliability. You have a primary model (expensive, capable) and fallback models (cheaper, faster, less capable).

How it works: If the primary model fails, retry with a smaller model. The smaller model may have a smaller context window, so you also reshape the request.

class TieredModelFallback:
    def __init__(self, primary_model, fallback_models):
        self.primary = primary_model  # e.g., gpt-4, claude-opus
        self.fallbacks = fallback_models  # e.g., [gpt-3.5, claude-haiku]
    
    def query(self, prompt, context_budget=8000):
        models = [self.primary] + self.fallbacks
        
        for i, model in enumerate(models):
            try:
                # Adjust context budget for each tier
                adjusted_budget = context_budget // (2 ** i)
                truncated_prompt = self._truncate_to_budget(
                    prompt, 
                    adjusted_budget
                )
                
                response = model.generate(
                    truncated_prompt,
                    timeout=5.0,
                    max_tokens=adjusted_budget // 2
                )
                
                return response, f"tier_{i}"
            
            except (TokenLimitExceeded, RateLimitError, TimeoutError) as e:
                logger.warning(
                    f"Model {model.name} failed: {e}. Falling back to tier {i+1}."
                )
                if i == len(models) - 1:
                    raise  # All tiers exhausted
                continue
    
    def _truncate_to_budget(self, prompt, budget):
        # Aggressive truncation: remove context, keep instruction
        # This is lossy and will reduce quality
        tokens = self.tokenizer.encode(prompt)
        if len(tokens) > budget:
            # Keep first 20% (instruction), last 80% (context)
            instruction_tokens = tokens[:len(tokens) // 5]
            context_tokens = tokens[len(tokens) // 5:]
            context_tokens = context_tokens[-(budget - len(instruction_tokens)):]
            tokens = instruction_tokens + context_tokens
        
        return self.tokenizer.decode(tokens)

Trade-offs:

• ✅ High reliability: you have multiple fallback options
• ✅ Reduces provider lock-in: if one provider is down, use another
• ✅ Graceful quality degradation: users get an answer, but it might be worse
• ❌ Cost is unpredictable: fallback models are cheaper, but you might call all of them
• ❌ Quality variance is visible: users notice when responses get worse
• ❌ Request truncation is lossy: you lose context, responses become less accurate
• ❌ Latency adds up: each failed attempt consumes time from your latency budget

When it breaks: You truncate context so aggressively that the fallback model hallucinates. The user gets a confident-sounding wrong answer instead of an error.

Cost example:

• Primary (GPT-4): $0.03 per request
• Fallback 1 (GPT-3.5): $0.001 per request
• If 20% of requests hit the fallback, your cost is 0.8 * $0.03 + 0.2 * $0.001 = $0.0242 per request instead of $0.03. Not huge, but it compounds.

Pattern 3: Request Reshaping and Retry

When to use: When the failure is due to request structure, not model capability. You can simplify the request and retry.

How it works: If a request fails due to token limits, reshape it: remove examples, compress context, simplify the prompt. Retry with the reshaped request.

class ReshapingFallback:
    def query(self, prompt, examples=None, context=None, max_attempts=3):
        attempt = 0
        reshaping_strategies = [
            self._remove_examples,
            self._compress_context,
            self._simplify_prompt,
        ]
        
        while attempt < max_attempts:
            try:
                response = self.llm.generate(prompt, timeout=5.0)
                return response, f"attempt_{attempt}"
            
            except TokenLimitExceeded:
                if attempt >= len(reshaping_strategies):
                    raise  # No more strategies
                
                strategy = reshaping_strategies[attempt]
                logger.warning(
                    f"Token limit exceeded. Applying {strategy.__name__}."
                )
                prompt = strategy(prompt, examples, context)
                attempt += 1
                continue
    
    def _remove_examples(self, prompt, examples, context