Cost Attribution in Multi-Tenant GenAI Systems
Practical instrumentation strategies for measuring and allocating LLM costs in production across tenants, A/B tests, and feature flags without sacrificing latency.
You launch a feature that uses an LLM to generate summaries. Your pricing model assumes $0.002 per request. Three weeks later, your bill is 40% higher than forecast. You check the dashboard and find that one tenant's integration is retrying failed requests, another is sending 10x the context tokens you expected, and your A/B test variant is using a more expensive model. Now you're hunting through logs trying to figure out what went wrong.
This is the gap between benchmark pricing and production reality. It's where cost attribution becomes operational necessity rather than nice-to-have observability.
The Problem: Why Simple Per-Request Accounting Fails
Most teams start with a naive model:
cost = (prompt_tokens + completion_tokens) * price_per_token
This works until it doesn't. In production, you need to answer:
- Which tenant drove this cost spike?
- Which feature flag caused the model switch?
- Which variant in our A/B test is actually more expensive?
- Which retry loop is burning tokens silently?
- How much context bloat are we carrying per request?
Without instrumentation at the right layers, you're left guessing. And guessing wrong means either accepting margin erosion or making feature decisions based on incomplete data.
The challenge intensifies in multi-tenant systems where cost visibility directly impacts customer billing, SLA decisions, and feature rollout strategies.
Core Instrumentation Pattern: Cost Metadata on the Request Context
The foundation is attaching cost-relevant metadata to every LLM request early, then threading it through to the response.
type LLMRequestContext struct {
// Tenant and user identification
TenantID string
UserID string
// Feature and experiment tracking
FeatureFlag string // e.g., "summary_v2"
ExperimentID string // e.g., "model_comparison_exp_42"
ModelVariant string // e.g., "gpt-4-turbo" vs "gpt-3.5-turbo"
// Cost bucketing
CostBucket string // e.g., "high-latency", "fallback", "cached"
// Request characteristics
PromptLength int
ContextSize int
RetryCount int
}
type LLMResponse struct {
Content string
PromptTokens int
CompletionTokens int
// Cost attribution
Cost float64
CostContext *LLMRequestContext
CachingStrategy string // "hit", "miss", "bypass"
// Latency for cost-per-millisecond analysis
LatencyMS int
}This structure is lightweight—no serialization overhead, just pointers and strings. The key is creating it once at the request boundary and passing it through your LLM client library.
Bucketing Costs Across Billing Dimensions
Raw cost numbers are useless without dimensions. You need to slice costs by:
1. Tenant + Feature Flag
type CostBucket struct {
TenantID string
FeatureFlag string
ModelVariant string
TotalCost float64
RequestCount int
AvgCostPerReq float64
// Token waste indicators
PromptTokens int
CompletionTokens int
CacheHitRate float64
// Time window
Hour time.Time
}Every hour (or minute, depending on your scale), aggregate costs into buckets and ship to your observability backend:
func aggregateCosts(events []LLMResponse) map[string]*CostBucket {
buckets := make(map[string]*CostBucket)
for _, event := range events {
key := fmt.Sprintf("%s:%s:%s",
event.CostContext.TenantID,
event.CostContext.FeatureFlag,
event.CostContext.ModelVariant,
)
if _, exists := buckets[key]; !exists {
buckets[key] = &CostBucket{}
}
b := buckets[key]
b.TotalCost += event.Cost
b.RequestCount += 1
b.PromptTokens += event.PromptTokens
b.CompletionTokens += event.CompletionTokens
}
for _, b := range buckets {
b.AvgCostPerReq = b.TotalCost / float64(b.RequestCount)
}
return buckets
}2. Cost by Retry Count
Retries are hidden cost multipliers. Track them:
type RetryMetrics struct {
NoRetry float64 // baseline
OneRetry float64 // 1.5x-2x baseline
TwoRetries float64 // 2x-3x baseline
ThreeOrMore float64 // outlier detection
}If TwoRetries or ThreeOrMore spikes, you've found a problem—either rate limiting, transient failures, or a client bug causing unnecessary retries.
3. Cost by Context Window Usage
This is where margin dies quietly. Track the relationship between context size and cost:
type ContextCostAnalysis struct {
ContextPercentiles map[int]float64 // p50, p75, p95, p99 context sizes
CostPercentiles map[int]float64
ContextBloatRatio float64 // actual_avg_context / optimal_context
}If your p99 context is 2x your p50, you have bloat. Bloat comes from:
- Sending entire conversation history instead of summaries
- Including unnecessary metadata or formatting
- Loading full documents when snippets would suffice
A 30% context reduction can cut your LLM costs by 15-25% depending on your pricing model.
Detecting Cost Anomalies Without False Positives
Cost spikes happen. Not all of them are problems. You need signals that matter:
1. Cost-per-Request Deviation by Tenant
func detectTenantCostAnomaly(current, baseline *CostBucket) bool {
ratio := current.AvgCostPerReq / baseline.AvgCostPerReq
// Flag if >2x baseline AND request volume is significant
if ratio > 2.0 && current.RequestCount > 100 {
return true
}
// Flag if cost jumped but request count stayed flat
// (indicates token bloat, not volume increase)
if ratio > 1.5 &&
current.RequestCount > baseline.RequestCount * 0.9 &&
current.RequestCount < baseline.RequestCount * 1.1 {
return true
}
return false
}2. Cache Hit Rate Collapse
If you're using prompt caching or semantic caching, monitor hit rates by tenant:
type CacheMetrics struct {
TenantID string
HitRate float64 // should be stable
HitRateChange float64 // alert if > 0.15 (15% drop)
AvgCostPerCacheHit float64 // should be near-zero
}A sudden drop in cache hit rate often means:
- The tenant changed their workflow
- Your cache key strategy broke
- They're sending new context each request
3. Model Variant Cost Drift
If you're A/B testing models, track cost divergence:
type ModelComparison struct {
ModelA string
ModelB string
AvgCostA float64
AvgCostB float64
CostDifferential float64
ConfidenceInterval float64
// Flag for human review
IsSignificant bool
}Don't just look at average cost. Look at cost distribution—one model might be cheaper on average but have extreme outliers.
Integrating Cost Signals into Feature Decision-Making
Cost data should inform feature flags and routing decisions in real-time.
1. Cost-Aware Feature Rollout
type FeatureRolloutDecision struct {
FeatureFlag string
CostPerRequest float64
LatencyMS float64
ErrorRate float64
// Decision logic
CostBudgetPerReq float64
CanRollout bool
}
func shouldRollout(decision *FeatureRolloutDecision) bool {
// Cost check first
if decision.CostPerRequest > decision.CostBudgetPerReq {
return false
}
// Latency check
if decision.LatencyMS > 2000 {
return false
}
// Error rate check
if decision.ErrorRate > 0.01 {
return false
}
return true
}2. Dynamic Model Routing
Route requests to cheaper models when possible:
func routeToModel(ctx *LLMRequestContext, priority string