---
title: "Gemini — Conversation History Causes Quadratic Token Cost Growth"
aliases: [gemini-conversation-cost, gemini-token-scaling, gemini-accumulated-history, llm-conversation-cost]
tags: [gemini, cost-tracking, llm, ai, performance, budget]
sources:
  - "daily/2026-04-24.md"
created: 2026-04-24
updated: 2026-04-24
---

# Gemini — Conversation History Causes Quadratic Token Cost Growth

Gemini (and most LLM APIs) bill the full accumulated conversation history as prompt tokens on every turn. Each new message includes all previous messages in the request. This means cost grows quadratically with conversation length, not linearly — a 10-turn conversation costs roughly 5× more total than its message count suggests if estimated as individual turns.

## Key Points

- **Every Gemini API call sends the full history** — turn 10 includes the entire turns 1–9 as prompt context
- **Cost is quadratic**: total tokens ≈ `sum(messages_so_far_at_each_turn) = n * (n+1) / 2 * avg_turn_tokens`
- **Backfill/estimation scripts get this wrong** if they estimate each turn's input tokens as just the current message — actual prompt tokens are `base_template + all_prior_messages`
- **Gemini `usage_metadata.prompt_token_count`** from the live API response is always accurate — only estimation/backfill tools need special handling
- **200k token threshold** in Gemini pricing creates a tier split — conversations long enough to cross this boundary have different per-token prices mid-conversation

## Details

### Why Cost Grows Quadratically

```
Turn 1: sends [system_prompt + message_1]         → ~1500 tokens input
Turn 2: sends [system_prompt + message_1 + message_2] → ~2000 tokens input
Turn 3: sends [system_prompt + m1 + m2 + message_3]   → ~2500 tokens input
...
Turn N: sends [system_prompt + m1...m(N-1) + mN]      → ~(1500 + N*500) tokens input
```

**Total input tokens** = sum of all turns' input tokens ≈ `N² × avg_message_tokens / 2`

For a 20-turn conversation with ~500 tokens/turn:
- Naïve estimate: `20 × 500 = 10,000` input tokens
- Actual total: `~20 × 21 / 2 × 500 = 105,000` input tokens — **10× more than the naïve estimate**

### Live API vs Estimation

Gemini's live `usage_metadata` response always returns the real token count:

```python
response = await client.generate_content_async(messages)
# Always accurate — includes full history
input_tokens = response.usage_metadata.prompt_token_count
output_tokens = response.usage_metadata.candidates_token_count
```

The problem only arises in:
1. **Backfill scripts** that reconstruct historical usage from database logs — they often estimate `len(message_text) / 4` as the input token count for each turn
2. **Pre-call estimation** (preflight) for budget enforcement — must estimate accumulated context, not just the current message

### Correct Estimation Formula for Backfill

When reconstructing historical usage without the original API response:

```python
def estimate_turn_input_tokens(turn_index: int, avg_turn_tokens: int = 500) -> int:
    """
    Estimate input tokens for a conversation turn.
    Includes base template + all prior turns.
    """
    base_template_tokens = 1500  # system prompt + boilerplate
    return base_template_tokens + (turn_index * avg_turn_tokens)

# Total for a conversation:
total_input = sum(estimate_turn_input_tokens(i) for i in range(num_turns))
```

This still under-estimates (assumes constant avg_turn_tokens), but is far closer to reality than `len(text) / 4`.

### Implications for Cost Tracking and Budgeting

1. **Budget enforcement via preflight** must account for accumulated context:
   - Short early conversations are cheap to estimate
   - Long conversations (10+ turns) are expensive — budget may be exceeded faster than expected

2. **Per-user cost analytics** appear to show low costs early in the month but spike as conversations accumulate history

3. **RAG-based conversations** (inserting retrieved documents as context) amplify the effect — each retrieval adds thousands of tokens that accumulate across turns

4. **Mitigation strategies**:
   - Implement conversation summarization after N turns (replace full history with a summary)
   - Set a maximum conversation history window (e.g., last 10 turns only)
   - Track `conversation_turn_count` in the database and warn users when conversations grow expensive

### Gemini Pricing Tier (200k tokens)

Gemini has a pricing split at 200k input tokens per request. Conversations that grow beyond this threshold have the tokens above 200k billed at a higher rate. For cost tracking:

```python
GEMINI_PRICE_TIER_BOUNDARY = 200_000

def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    if input_tokens <= GEMINI_PRICE_TIER_BOUNDARY:
        input_cost = input_tokens * PRICE_PER_TOKEN_BELOW_200K[model]
    else:
        input_cost = (GEMINI_PRICE_TIER_BOUNDARY * PRICE_PER_TOKEN_BELOW_200K[model] + 
                      (input_tokens - GEMINI_PRICE_TIER_BOUNDARY) * PRICE_PER_TOKEN_ABOVE_200K[model])
    return input_cost + output_tokens * OUTPUT_PRICE[model]
```

## Related Concepts

- [[wiki/concepts/preflight-record-pattern]] — preflight/record pattern for AI cost tracking; preflight must estimate accumulated context
- [[wiki/concepts/litellm-pricing-source]] — LiteLLM as pricing source for Gemini models
- [[wiki/tech-patterns/python-ai-agents]] — Python AI agent patterns where conversation history management is relevant
- [[wiki/concepts/lazy-user-mirror]] — user mirror pattern in cost tracker; per-user cost shows the quadratic growth effect

## Sources

- [[daily/2026-04-24.md]] — Semblance usage backfill script was estimating input tokens as `len(output_text) / 4` for both input AND output — 60× underestimate because it ignored accumulated history; rewritten to use `base_template (~1500 tok) + cumulative conversation history per turn`; Gemini 200k pricing tier handled as a split calculation
