prompt-caching – skill Agent & Tự động hóa cho agent AI

Prompt Caching

Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and CAG (Cache Augmented Generation)

Capabilities

• prompt-cache
• response-cache
• kv-cache
• cag-patterns
• cache-invalidation

Prerequisites

• Knowledge: Caching fundamentals, LLM API usage, Hash functions
• Skills_recommended: context-window-management

Scope

• Does_not_cover: CDN caching, Database query caching, Static asset caching
• Boundaries: Focus is LLM-specific caching, Covers prompt and response caching

Ecosystem

Primary_tools

• Anthropic Prompt Caching - Native prompt caching in Claude API
• Redis - In-memory cache for responses (ioredis on servers; @upstash/redis over HTTP on serverless, see upstash-redis)
• OpenAI Caching - Automatic caching in OpenAI API

Patterns

Anthropic Prompt Caching

Use Claude's native prompt caching for repeated prefixes

When to use: Using Claude API with stable system prompts or context

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

// Cache the stable parts of your prompt

async function queryWithCaching(userQuery: string) {

const response = await client.messages.create({

model: "claude-sonnet-4-20250514",

max_tokens: 1024,

system: [

{

type: "text",

text: LONG_SYSTEM_PROMPT, // Your detailed instructions

cache_control: { type: "ephemeral" } // Cache this!

},

{

type: "text",

text: KNOWLEDGE_BASE, // Large static context

cache_control: { type: "ephemeral" }

}

],

messages: [

{ role: "user", content: userQuery } // Dynamic part

]

});

// Check cache usage

console.log(Cache read: ${response.usage.cache_read_input_tokens});

console.log(Cache write: ${response.usage.cache_creation_input_tokens});

return response;

}

// Measure actual cache-read/write costs and latency for the selected model/provider.

Response Caching

Cache full LLM responses for identical or similar queries

When to use: Same queries asked repeatedly

import { createHash } from 'crypto';

import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL);

// Serverless/edge alternative without a persistent connection:

// import { Redis } from '@upstash/redis'; const redis = Redis.fromEnv(); // then use redis.set(key, value, { ex: ttl })

class ResponseCache {

private ttl = 3600; // 1 hour default

// Exact match caching

async getCached(prompt: string): Promise<string | null> {

const key = this.hashPrompt(prompt);

return await redis.get(response:${key});

}

async setCached(prompt: string, response: string): Promise<void> {

const key = this.hashPrompt(prompt);

await redis.set(response:${key}, response, 'EX', this.ttl);

}

private hashPrompt(prompt: string): string {

return createHash('sha256').update(prompt).digest('hex');

}

// Semantic similarity caching

async getSemanticallySimilar(

prompt: string,

threshold: number = 0.95

): Promise<string | null> {

const embedding = await embed(prompt);

const similar = await this.vectorCache.search(embedding, 1);

if (similar.length && similar[0].similarity > threshold) {

return await redis.get(response:${similar[0].id});

}

return null;

}

// Temperature-aware caching

async getCachedWithParams(

prompt: string,

params: { temperature: number; model: string }

): Promise<string | null> {

// Only cache low-temperature responses

if (params.temperature > 0.5) return null;

const key = this.hashPrompt(

${prompt}|${params.model}|${params.temperature}

);

return await redis.get(response:${key});

}

}

Cache Augmented Generation (CAG)

Pre-cache documents in prompt instead of RAG retrieval

When to use: Document corpus is stable and fits in context

// CAG: Pre-compute document context, cache in prompt

// Better than RAG when:

// - Documents are stable

// - Total fits in context window

// - Latency is critical

class CAGSystem {

private cachedContext: string | null = null;

private lastUpdate: number = 0;

async buildCachedContext(documents: Document[]): Promise<void> {

// Pre-process and format documents

const formatted = documents.map(d =>

## ${d.title}\n${d.content}

).join('\n\n');

// Store with timestamp

this.cachedContext = formatted;

this.lastUpdate = Date.now();

}

async query(userQuery: string): Promise<string> {

// Use cached context directly in prompt

const response = await client.messages.create({

model: "claude-sonnet-4-20250514",

max_tokens: 1024,

system: [

{

type: "text",

text: "You are a helpful assistant with access to the following documentation.",

cache_control: { type: "ephemeral" }

},

{

type: "text",

text: this.cachedContext ?? "", // verify initialized context before querying

cache_control: { type: "ephemeral" }

}

],

messages: [{ role: "user", content: userQuery }]

});

return response.content.filter(block => block.type === "text").map(block => block.text).join("\n");

}

// Periodic refresh

async refreshIfNeeded(documents: Document[]): Promise<void> {

const stale = Date.now() - this.lastUpdate > 3600000; // 1 hour

if (stale) {

await this.buildCachedContext(documents);

}

}

}

// CAG vs RAG decision matrix:

// | Factor | CAG Better | RAG Better |

// |------------------|------------|------------|

// | Corpus size | < 100K tokens | > 100K tokens |

// | Update frequency | Low | High |

// | Latency needs | Critical | Flexible |

// | Query specificity| General | Specific |

Sharp Edges

Cache miss causes latency spike with additional overhead

Severity: HIGH

Situation: Slow response when cache miss, slower than no caching

Symptoms:

• Slow responses on cache miss
• Cache hit rate below 50%
• Higher latency than uncached

Why this breaks:

Cache check adds latency.

Cache write adds more latency.

Miss + overhead > no caching.

Recommended fix:

// Optimize for cache misses, not just hits

class OptimizedCache {

async queryWithCache(prompt: string): Promise<string> {

const cacheKey = this.hash(prompt);

// Non-blocking cache check

const cachedPromise = this.cache.get(cacheKey);

// Defer paid provider call until bounded cache lookup misses.

// Race: use cache if available before LLM returns

const cached = await Promise.race([

cachedPromise,

sleep(50).then(() => null) // also catch cache errors per availability policy // 50ms cache timeout

]);

if (cached) {

// Cancel LLM request if possible

return cached;

}

// Cache miss: continue with LLM

const response = await this.queryLLM(prompt);

// Async cache write (don't block response)

this.cache.set(cacheKey, response).catch(console.error);

return response;

}

}

// Alternative: Probabilistic caching

// Only cache if query matches known high-frequency patterns

class SelectiveCache {

private patterns: Map<string, number> = new Map();

shouldCache(prompt: string): boolean {

const pattern = this.extractPattern(prompt);

const frequency = this.patterns.get(pattern) || 0;

// Only cache high-frequency patterns

return frequency > 10;

}

recordQuery(prompt: string): void {

const pattern = this.extractPattern(prompt);

this.patterns.set(pattern, (this.patterns.get(pattern) || 0) + 1);

}

}

Cached responses become incorrect over time

Severity: HIGH

Situation: Users get outdated or wrong information from cache

Symptoms:

• Users report wrong information
• Answers don't match current data
• Complaints about outdated responses

Why this breaks:

Source data changed.

No cache invalidation.

Long TTLs for dynamic data.

Recommended fix:

// Implement proper cache invalidation

class InvalidatingCache {

// Version-based invalidation

private cacheVersion = 1;

getCacheKey(prompt: string): string {

return v${this.cacheVersion}:${this.hash(prompt)};

}

invalidateAll(): void {

this.cacheVersion++;

// Old keys automatically become orphaned

}

// Content-hash invalidation

async setWithContentHash(

key: string,

response: string,

sourceContent: str

… (nội dung đầy đủ có trong gói tải về)