Part 03

Context Engineering

the new "prompt engineering"

A dense, citation-backed reference for the seven techniques every production LLM system reaches for — context budgeting, summarization, caching, hierarchical memory, compression, tool-result truncation, and the "lost in the middle" failure mode. Each section ships in two voices: the engineer's version (with diagrams and papers) and a layman's version (plain English with analogies) at the bottom of each topic. প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

7 sections · 11 sources · three voices

Context window budgeting

A model has a fixed-size working memory. Every byte you allocate competes with every other byte. Context engineering is the discipline of deciding what lives where — system prompt, history, tool definitions, retrieved docs, prior outputs, the user's actual request.

The budget model

Treat the context window as a finite pie. Five claimants, each with a minimum allocation and a cost-per-token that varies:

A 200K-token window broken into system + tools + retrieval + history + current turn. Percentages are typical for an agentic RAG app, not universal.

এই ডায়াগ্রামে একটি 200K-token context window-কে SYSTEM (≈8%), TOOLS (≈7%), RETRIEVAL (≈40%), HISTORY (≈30%) এবং USER+OUTPUT (≈15%) অংশে ভাগ করে দেখানো হয়েছে। প্রতিটি অংশের প্রস্থ তার token বাজেটের সমান; RETRIEVAL সবচেয়ে বড়, তাই আগে সেটাই কমানো উচিত।

Allocation principles

→ System prompt: floor of ~1–2K tokens. Persona + schema + hard constraints. Larger prompts cost more per call AND dilute attention.[14]
→ Tool definitions: floor of ~1–3K tokens. Only include tools the current step might call; load specialised tools lazily.
→ Retrieval: budget = window − system − tools − current turn − reserved output. Cap retrieval chunks; prefer quality over quantity.
→ History: shrink aggressively. Use summarization (§2) or selective replay to keep it bounded.
→ Output reservation: always reserve enough tokens for the response. Forcing truncation produces silent failures.

python# A budget calculator
def budget(window: int, tools: list[dict], retrieved: str,
          system: str, history: str, max_out: int) -> dict:
    return {
        "system":      len(enc.encode(system)),
        "tools":       sum(len(enc.encode(json.dumps(t))) for t in tools),
        "retrieval":   len(enc.encode(retrieved)),
        "history":     len(enc.encode(history)),
        "max_output":  max_out,
        "total":       sum((len(enc.encode(x)) for x in [system, history, retrieved])) + max_out,
        "headroom":    window - "total",
    }

The "Lost in the Middle" effect. Liu et al. (2023) showed that models reliably recall information at the beginning and end of long contexts but perform close to random in the middle.[1] This is the single most important budget heuristic: put the things you most need retrieved in the start or the end of your prompt, not buried in the middle. (§7 expands this.)

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Fit a plan into an 8,000-token window

Why this matters

When you build a chat app that searches documents, the AI can read only a fixed amount of text at once. You must check that everything fits, or the answer gets cut off.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The AI will add up the numbers.

Prompt

A chat app has a context window of 8,000 tokens.
Fixed costs:
- System prompt: 1,500 tokens
- Tool definitions: 2,000 tokens
- Current user turn: 300 tokens
- Reserved for the answer: 1,000 tokens
The team wants to add:
- Chat history: 2,500 tokens
- Retrieved chunks (small pieces of documents found by search): 6 chunks of 700 tokens each

Task:
1. Show a table with every item and the total.
2. Say if the plan fits. Say how many tokens it is over or under.
3. If history stays at 2,500 tokens, how many 700-token chunks fit?
Show your arithmetic.

You should see: A table with a total of 11,500 tokens. The plan does not fit: it is 3,500 tokens over. Only 1 chunk fits.

Example answer

Item | Tokens
System prompt | 1,500
Tool definitions | 2,000
Current user turn | 300
Reserved for answer | 1,000
Chat history | 2,500
6 chunks x 700 | 4,200
Total | 11,500

The plan does not fit. It is 11,500 - 8,000 = 3,500 tokens over.
Without chunks the total is 7,300, so 700 tokens are free. Only 1 chunk of 700 tokens fits.

Your answer will be worded differently.

Step 2
Same chat
On paper, add the six numbers: 1,500 + 2,000 + 300 + 1,000 + 2,500 + 4,200. Compare your total with the total of the AI.
You should see: Your total is 11,500. If the AI gave a different total, the AI made a mistake.

Step 3

Same chat

Copy this prompt, paste it, and press Enter. The AI will make a smaller plan that fits.

Prompt

Now make a plan that fits in 8,000 tokens. Rules: keep the 1,000-token answer reserve, keep 3 chunks of 700 tokens each, and load only 1,000 tokens of tool definitions now (the rest load later, only when needed). Keep the system prompt (1,500) and the current user turn (300). Shrink the history by summarizing it. Give a new table with the largest history that fits, and how many tokens are left over.

You should see: A new table. The largest history is 2,100 tokens, and about 0 tokens are left over.

Example answer

Item | Tokens
System prompt | 1,500
Tool definitions (loaded now) | 1,000
Current user turn | 300
Reserved for answer | 1,000
3 chunks x 700 | 2,100
Summarized history | 2,100
Total | 8,000

The largest history that fits is 2,100 tokens. Tokens left over: 0.

Your answer will be worded differently.

Step 4
Same chat
Check the new plan on paper: 1,500 + 1,000 + 300 + 1,000 + 2,100. Then add the history size of the AI. The total must be 8,000 or less.
You should see: The first sum is 5,900. With a history of 2,100 the total is 8,000. If the AI chose another history size, the total must still be 8,000 or less.

What just happened

The context window is the most text the AI can read at one time, and the answer must fit inside it too. In the first plan, the fixed parts plus the documents were bigger than the window. To fix it, you had to cut several parts at once: a smaller history, fewer tools, and fewer chunks. Always keep room for the answer, or it gets cut off.

You are done when You have the total of the first plan (11,500) and a new plan that adds up to 8,000 or less.

Check yourselfWhy must you keep room for the answer when you plan what fits?Show answer

The window holds the input and the answer together. If you keep no room for the answer, the answer gets cut off, and nothing warns you.

Try a harder versionChange the window in the first prompt to 32,000 tokens and send it again. Does the plan fit now? Which part would you still limit first, and why?

Watch · 2 videos

Context Engineering for Agents - Lance Martin, LangChain
Latent Space
What is Context Engineering?
Standarity

Conversation summarization & rolling memory

A 50-turn chat is 50× the size of a 1-turn chat. Most of those tokens are noise — greetings, "can you clarify," half-finished thoughts. Summarization trades raw history for compressed state, keeping the model coherent without the linear cost.

When to summarize

Three triggers, in increasing aggressiveness:

Token-budget triggered: when history > N tokens (e.g. 4K), compress the oldest chunk into a summary block.
Turn-count triggered: every K turns, summarise the earliest K−1 into one block. Predictable, simpler.
Semantic triggered: summarise when the topic shifts (detected via an embedding similarity drop, or when a new user message is far from recent history).

The summarization pattern

Rolling memory: oldest turns are condensed by the model into a summary block, recent turns stay verbatim. Sliding window plus summarization.

এখানে rolling-window summarization দেখানো হয়েছে: সবচেয়ে পুরনো turn (T-7, T-6, T-5) মডেল নিজে একটি ছোট summary block-এ (200 tokens) সংকুচিত করে, আর T-4 থেকে T0 পর্যন্ত সাম্প্রতিক turn গুলো হুবহু থাকে। বাঁ থেকে ডানে পড়লে আগে ও পরের অবস্থা বোঝা যায়।

What to keep in the summary

A naive "summarise the conversation" produces mush. Use a structured schema:

# A summary schema that survives summarisation well
{
  "user_goal":         "<one sentence>",
  "decisions_made":    ["<list of conclusions reached>"],
  "open_questions":    ["<list of unanswered items>"],
  "key_facts_learned": ["<facts the user told us>"],
  "entities":          ["<names, ids, URLs the user mentioned>"],
  "user_preferences":  ["<style, format, tone preferences expressed>"],
  "tool_results_to_remember": ["<non-recreatable values>"]
}

Hierarchical summarisation — the technique Anthropic's prompt-engineering docs recommend: keep a rolling summary of the very oldest turns, a mid-resolution summary of the middle turns, and the recent turns verbatim. This preserves both long-arc continuity and recent detail.[14]

Information loss is real

Summarisation is lossy compression. Decisions, exact numbers, and quoted text are the first things to disappear. Mitigation:

Extract key facts into a separate structured slot that survives summarization.
Use the summary as index, not source of truth. When a detail matters, retrieve from the original log.
For high-stakes conversations, persist verbatim transcripts to a DB even after summarising into context.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Compare a vague and a structured summary

Why this matters

When you build a chatbot with long chats, you must shrink old messages into a summary. A bad summary can drop the budget or a promised call time, and the bot then gives wrong help.

Step 1

New chat

Copy this prompt, paste it, and press Enter. Copy the summary to a note.

Prompt

Summarise this conversation in 2 sentences.

User: I want a laptop for my son. He studies computer science in Chittagong.
Assistant: What is your budget?
User: Maximum 85,000 tk. I will pay by bKash, so the shop must accept it.
Assistant: Do you prefer a new or a used laptop?
User: New only. Used ones gave us problems before.
Assistant: Do you need a big screen?
User: No. He carries it every day. Lighter than 1.6 kg please.
Assistant: I suggest 16 GB RAM and a 512 GB SSD.
User: OK, 16 GB RAM. I am not sure about 512 GB. Ask me again later.
Assistant: Noted. I will check shops in Agrabad and send you a list.
User: Good. Call me on Friday after 5 pm. My number is 01700-000000.

You should see: A summary of 2 sentences. It usually keeps the main goal, but it often drops some exact details.

Example answer

A parent wants to buy a new laptop with 16 GB RAM for his son, who studies computer science in Chittagong. The assistant will check shops in Agrabad and send a list, and the parent asks for a call on Friday.

Your answer will be worded differently.

Step 2

New chat

Copy this prompt, paste it in a new chat, and press Enter. Copy the summary to a note.

Prompt

Summarise this conversation as JSON with exactly these keys: user_goal (one sentence), decisions_made (list), open_questions (list), key_facts_learned (list), entities (list of names, ids, numbers), user_preferences (list), tool_results_to_remember (list). Keep exact numbers. Do not add facts.

User: I want a laptop for my son. He studies computer science in Chittagong.
Assistant: What is your budget?
User: Maximum 85,000 tk. I will pay by bKash, so the shop must accept it.
Assistant: Do you prefer a new or a used laptop?
User: New only. Used ones gave us problems before.
Assistant: Do you need a big screen?
User: No. He carries it every day. Lighter than 1.6 kg please.
Assistant: I suggest 16 GB RAM and a 512 GB SSD.
User: OK, 16 GB RAM. I am not sure about 512 GB. Ask me again later.
Assistant: Noted. I will check shops in Agrabad and send you a list.
User: Good. Call me on Friday after 5 pm. My number is 01700-000000.

You should see: A JSON text with 7 keys. The numbers, such as 85,000 tk and 1.6 kg, are in the lists.

Example answer

{
  "user_goal": "Buy a laptop for his son, who studies computer science in Chittagong",
  "decisions_made": ["New laptop only", "16 GB RAM"],
  "open_questions": ["Whether to get a 512 GB SSD"],
  "key_facts_learned": ["Budget: maximum 85,000 tk", "Pays by bKash", "Lighter than 1.6 kg"],
  "entities": ["Agrabad", "01700-000000"],
  "user_preferences": ["New only", "Light laptop, carried every day", "Call on Friday after 5 pm"],
  "tool_results_to_remember": []
}

Your answer will be worded differently.

Step 3
Same chat
Count how many of these 6 facts each summary keeps: 85,000 tk, bKash, new only, under 1.6 kg, 16 GB RAM, call Friday after 5 pm.
You should see: The 2-sentence summary keeps fewer facts, often only 2 or 3. The JSON summary keeps most of them, often all 6.
Step 4
Same chat
Find the one question the user did not answer yet (the size of the SSD). Check if each summary shows it.
You should see: The JSON summary shows it in open_questions. The short summary often leaves it out.

What just happened

A summary is a lossy copy: the AI throws some details away to make the text short. A vague request like in 2 sentences loses exact numbers first. A fixed list of slots (the JSON keys) makes the AI keep a place for each kind of fact. The longer summary costs more tokens, but it loses less.

You are done when You have 2 summaries, and for each one you counted how many of the 6 facts it keeps.

Check yourselfWhy is a summary a bad source of truth for an exact number such as a budget?Show answer

A summary is a lossy copy. Exact numbers, decisions, and quotes disappear first. Use the summary as a guide, and keep key facts in a fixed slot or in the original chat log.

Try a harder versionAdd 4 more turns to the chat that change the budget to 90,000 tk. Make the JSON summary again. Does it show the new number and not the old one?

Watch · 3 videos

Context Management in Claude Code
Claude
Conversation Summary Memory (4.4)
Jeff Heaton
Memory in LangChain | Deep dive (python)
Eden Marco

Prompt caching strategies

Frontier providers let you cache a prefix of your prompt: identical tokens at the start of a request cost dramatically less and run faster. Used well, caching drops 80–90% of the input bill for apps with stable system prompts and long retrieval sections.

How caching works (Anthropic + OpenAI + Gemini)

Provider	Mechanism	Cost	TTL	Min cache size
Anthropic	Auto-cache of system + tools + first user message; explicit `cache_control` breakpoints for custom control	5–10× cheaper on cache hit (write pricing per tonne)	5 min (refreshes on hit)	1024 tokens (Claude 4 Sonnet), 4096 (Haiku)
OpenAI	Auto-cache on prompts >1024 tokens matching a recent prefix	50% off cached tokens	5–10 min (sliding window)	1024 tokens
Gemini	Explicit `cachedContent` resource	~75% off (free for system+prompt, regular per-token for output)	Configurable, default ~60 min	2048 tokens (Flash), 4096 (Pro)

Anthropic's cache breakpoints are the most flexible: you mark up to 4 ephemeral blocks within a prompt. Anything before the breakpoint is cached; anything after is rebuilt each call. This lets you cache system + tools + retrieved docs (system), and pass only the new turn's user message.[15]

The standard cache layout

Cache the prefix that doesn't change (system + tools + stable retrieval), pass only the new user message after the breakpoint.

ডায়াগ্রামে system prompt, tool schemas ও retrieved docs cached অংশ হিসেবে আর user msg প্রতিবার নতুন করে তৈরি হওয়া অংশ হিসেবে দেখানো হয়েছে। cache breakpoint-এর আগের সব কিছু আবার ব্যবহৃত হয়; তাই স্থির content আগে, পরিবর্তনশীল content শেষে রাখতে হয়।

What to cache vs what not to

Cache aggressively

System prompt (persona + rules)
Tool schemas (when tools are stable)
Large retrieved docs that are reused across turns
Few-shot examples (if they're static)
Long persona descriptions, style guides

Don't cache

The user's most recent message (changes every turn)
Retrieval that changes every call (e.g., today's news)
Token-by-token streaming prefixes (cache invalidates)
Anything where freshness matters per-call

The cost math

For an agent with a 60K-token system + tools prefix and a 4K-token user message:

→ No cache: 64K input tokens × price_per_m
→ With cache: 60K cached (≈10× cheaper) + 4K uncached
→ Effective savings: ~85% on input costs for the steady state

Cache invalidation is the gotcha. Even a single new character in the cached region busts the cache for everything after. Change a version number, fix a typo, even add a dynamic timestamp — cache miss. Cache control means also keeping the cached prefix byte-identical across calls. Hash your system prompt in CI; alert if it drifts.[16][17]

python# Anthropic prompt caching with explicit breakpoints
messages = [
    {"role": "system", "content": [
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ]},
    {"role": "user", "content": [
        {"type": "text", "text": tool_docs_str,
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": f"<retrieved>{retrieved}</retrieved>"},
        {"type": "text", "text": f"Question: {user_q}"},  # not cached
    ]},
]
# Inspect the response.usage to verify cache hit:
# response.usage.cache_read_input_tokens > 0  → cache hit

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

See which prompt layout costs less

Why this matters

When your app sends the same long system prompt in every call, the provider can cache it and charge less. If you put a changing line at the top, you lose that saving and pay full price.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The AI will work out the cost of two layouts.

Prompt

Prefix caching rule: the cache covers only the longest run of identical lines from the top of the prompt. The first line that differs, and every line after it, is a cache miss (full price). A cached token costs 0.1 units. An uncached token costs 1 unit.

Layout X. Request 1 and Request 2 have the same lines, except the lines marked CHANGES:
Line 1: Current time (15 tokens) - CHANGES each request
Line 2: System rules (1,800 tokens)
Line 3: Tool schemas (2,200 tokens)
Line 4: User question (100 tokens) - CHANGES each request

Layout Y. The same rule applies:
Line 1: System rules (1,800 tokens)
Line 2: Tool schemas (2,200 tokens)
Line 3: Current time (15 tokens) - CHANGES each request
Line 4: User question (100 tokens) - CHANGES each request

For Request 2 in each layout, give a table: cached tokens, uncached tokens, and cost in units. Then say which layout is cheaper and by about what percent.

You should see: Layout X: 0 cached, 4,115 uncached, cost 4,115 units. Layout Y: 4,000 cached, 115 uncached, cost 515 units. Layout Y is cheaper by about 87%.

Example answer

Layout X: cached 0, uncached 4,115 (15 + 1,800 + 2,200 + 100). Cost: 4,115 units.
Layout Y: cached 4,000 (1,800 + 2,200), uncached 115 (15 + 100). Cost: 4,000 x 0.1 + 115 = 400 + 115 = 515 units.

Layout Y is cheaper. It saves 4,115 - 515 = 3,600 units, which is about 87.5%.

Your answer will be worded differently.

Step 2

Same chat

Copy this prompt, paste it, and press Enter. See what one small typo fix does.

Prompt

Now a developer fixes one typo inside the System rules of Layout Y. The fix happens between Request 1 and Request 2, and it changes 1 character. What are the cached tokens, uncached tokens, and cost for Request 2 now? Explain in 2 sentences.

You should see: 0 cached tokens, 4,115 uncached tokens, and a cost of 4,115 units. The cache is lost from the changed line down.

Example answer

Cached tokens: 0. Uncached tokens: 4,115. Cost: 4,115 units.
The typo is in the System rules, which is the first line. So Request 2 no longer matches the cached start, and every line after it is a miss too.

Your answer will be worded differently.

Step 3
Same chat
Check the cost of Layout Y on paper: 4,000 x 0.1, then add 115 x 1. Compare with the AI cost.
You should see: 400 + 115 = 515 units.

What just happened

A prompt cache remembers the start of a prompt that the provider already read, so the same start costs less next time (here 0.1 instead of 1). The cache matches from the top and stops at the first line that changed. In Layout X the time was on line 1, so nothing matched. In Layout Y the changing lines were last, so 4,000 tokens matched.

You are done when You have the cost of Request 2 for Layout X (4,115), for Layout Y (515), and for Layout Y after the typo (4,115).

Check yourselfWhere should changing content go in a prompt that uses caching, and why?Show answer

Put the parts that stay the same first (system prompt, tools) and the parts that change last (time, user message). The cache keeps only an identical start, and a 1-character change breaks the cache for everything after it.

Try a harder versionAsk the AI to rewrite Layout Y for an app that also loads new news articles on every call. Where should the articles go, and should the provider cache them?

Watch · 3 videos

Build Hour: Prompt Caching
OpenAI
Prompt Caching: A Complete Technical Guide
Kartik Nighania
Cutting Claude API cost 80% in minutes using Anthropic Prompt Caching
Coev

Hierarchical memory

Humans don't have one type of memory — they have working memory, short-term, episodic, and procedural layers. AI systems that aspire to long-running tasks need the same stratification. Hierarchical memory = store different kinds of state at different timescales, summarised at each layer.

The four-layer model

Four-layer hierarchical memory. Each layer trades size for attention: working memory is small but always attended; external storage is unbounded but only retrieved on demand.

এই ডায়াগ্রামে Working memory থেকে Short-term, Long-term, Episodic ও External / Tools পর্যন্ত চার-পাঁচ স্তরের hierarchical memory দেখানো হয়েছে। নিচের অক্ষগুলো পড়লে বোঝা যায়, ওপরের স্তর ছোট, দ্রুত ও সবসময় attended; নিচের স্তর বড়, ধীর ও শুধু প্রয়োজনে retrieve করা হয়।

Layer	What lives there	Where it lives	Update cadence
Working memory	The current prompt: system + tools + retrieved + current request	API request payload	Every call (ephemeral)
Short-term	Recent turns of this session, recent tool results, recent decisions	Session store (DB row keyed by session_id)	Each turn, with rolling summarization
Long-term	User preferences, known facts about the user, project context	Vector store + structured DB keyed by user_id	Slow: when explicit facts are extracted
Episodic	Distilled "memories" of past sessions, lessons learned, what worked	Vector store keyed by user_id; written by a memory-update agent	End of session, or when triggered
External / tools	The world: documents, code, databases, calendars, anything retrievable	Your DBs, RAG indices, search APIs	Retrieved on demand

The MemGPT insight

Packer et al. (2023) proposed treating the LLM's context like virtual memory: the model itself decides when to evict information from working memory to external storage, and when to page it back in. The model uses tool calls (recall, archival_search) to manipulate its own memory.[2] This is the architectural blueprint for every serious long-running AI agent in 2024–2025.[14]

The memory-update pattern

python# An end-of-session memory consolidation step
def consolidate_memory(transcript: str, user_id: str):
    extracted = llm.extract_facts(transcript, schema={
        "user_preferences": [...],
        "project_facts": [...],
        "episodic_lessons": [...],
    })
    for fact in extracted.episodic_lessons:
        embedding = embed(fact.text)
        vector_store.upsert(user_id, embedding, fact.text)
    for pref in extracted.user_preferences:
        user_db.merge_pref(user_id, pref)   # structured update

Memory decay and contradiction. Long-term memory stores drift over time — a fact from a previous session may be contradicted by the user's current stance. Always tag memories with timestamps and a confidence score; surface potentially-stale memories for user confirmation, or weight them by recency at retrieval.[10]

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Sort 8 memories into memory layers

Why this matters

When you build a support agent that talks to the same user for weeks, it must know what to keep in the prompt, what to store, and what to fetch later. It must also notice when an old memory is no longer true.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The AI will sort 8 items into memory layers.

Prompt

An AI support agent has five memory places:
- Working memory: the current prompt
- Short-term: this chat session, kept in a session store
- Long-term: facts about the user, kept in a database
- Episodic: lessons from past sessions
- External: documents and databases, read only when needed

For each item below, choose one place, say where it is stored, and say when it is updated:
1. The current user question: "My router light is red."
2. The last 6 turns of this chat.
3. The user prefers answers in Bangla.
4. A 40-page router manual in PDF.
5. Lesson from last week: "Step-by-step answers with photos solved this user's problem fast."
6. Result of a tool call 2 turns ago: line status = up.
7. The user's name and city.
8. The company's price database.
Answer in a table. Mark any item that could fit two places.

You should see: A table with 8 rows. For example, the router manual goes to External, and the red-light question goes to Working memory. Some rows, such as the tool result, can fit two places.

Example answer

1. Current question: Working memory. In the prompt. Every call.
2. Last 6 turns: Short-term. Session store. Every turn.
3. Prefers Bangla: Long-term. User database. When the user says it.
4. 40-page manual: External. Document index. Read when needed.
5. Lesson from last week: Episodic. Memory store. End of session.
6. Tool result 2 turns ago: Short-term (could be Working). Session store. Each turn.
7. Name and city: Long-term. User database. When first learned.
8. Price database: External. Company database. Read when needed.

Your answer will be worded differently.

Step 2

New chat

Copy this prompt, paste it in a new chat, and press Enter. The AI will fix old memories.

Prompt

You are the memory manager of a cooking assistant. Today is 2026-06-01.
Stored memories:
M1 (saved 2025-01-10, confidence 0.9): The user is vegetarian.
M2 (saved 2025-02-02, confidence 0.9): The user lives in Sylhet.
New user message: "Plan a dinner for my friends tomorrow. I started eating fish last month, and I moved to Dhaka in April."

For M1 and M2, choose: keep, update, or ask the user to confirm. Give a reason. Then write the new memory records with a date and a confidence.

You should see: M2 is updated to Dhaka. M1 is updated (the user now eats fish) or the AI asks to confirm. The new records have a date and a confidence.

Example answer

M1: Update. The user now eats fish, so "vegetarian" is not fully true. The AI may ask if the user still avoids meat.
M2: Update. The user moved to Dhaka in April 2026, so Sylhet is old.

New records:
N1 (saved 2026-06-01, confidence 0.8): The user eats fish since May 2026, but may still avoid meat.
N2 (saved 2026-06-01, confidence 0.9): The user lives in Dhaka since April 2026.

Your answer will be worded differently.

Step 3
Same chat
Look at the new memory records. Check that each one has a date and a confidence number.
You should see: Each record has a date like 2026-06-01 and a number like 0.9. If one is missing, ask the AI to add it.

What just happened

Each kind of information has its own place. The prompt (working memory) is small, but the AI always reads it. Big things such as a manual stay outside and are fetched only when needed. A saved memory can become wrong, like M2 here. So each memory needs a timestamp (the date it was saved) and a confidence (a number for how sure you are).

You are done when You have a table of 8 sorted items and 2 new memory records, each with a date and a confidence.

Check yourselfWhy does a stored memory need a date and a confidence number?Show answer

A fact from an old session can become wrong when the user says something new. The date and the confidence let the system trust new facts more, or ask the user to confirm an old one.

Try a harder versionSend the second prompt again in a new chat, but change the new message to: I eat differently now. Does the AI ask a question, or does it guess?

Watch · 2 videos

The AI Operating System: Stateful Agents with Letta | Cameron Pfiffer, AI By the Bay25
FunctionalTV
MemGPT (Memory GPT) - Towards LLMs as Operating Systems
Shweta Lodha

Context compression & pruning

When summarization is too lossy and you have no time to spare, compression removes redundancy and pruning drops irrelevant tokens. The goal: same information density, fewer tokens.

Compression techniques

Technique	How it works	Compression ratio	Use case
LLM summarisation	Ask the model to summarise; structured schema for fact preservation	5–20×	Conversation history, long documents
LLMLingua	Small classifier prunes low-information words from prompts	5–20×	Latency-sensitive inference, prompt compression for cost
Selective context	Use self-information (perplexity) to drop uninformative phrases	5×	RAG context pruning
Embedding-based filtering	Score each chunk by relevance to the query, drop low-score chunks	Variable	RAG with many retrieved docs
Token-budget knapsack	Greedy: include chunks in relevance order until budget is filled	Variable	RAG with strict context caps
Deduplication	Near-duplicate detection (cosine > 0.95), keep one copy	1.5–3×	RAG over a corpus with overlapping content

LLMLingua — the canonical compression model

LLMLingua (Microsoft, 2023) trains a small classifier to predict which tokens are informative given the prompt; non-informative tokens are dropped. It can compress prompts 5–20× while preserving the LLM's ability to answer accurately.[3]

LLMLingua and similar tools identify low-information phrases ("the capital of Bangladesh") and drop them while preserving the meaning.

এখানে LLMLingua pipeline দেখানো হয়েছে: একটি classifier মূল prompt-এর কম-তথ্যপূর্ণ token চিহ্নিত করে, তারপর সংক্ষিপ্ত রূপ (প্রায় 40% tokens) তৈরি হয়। এই compression LLM call-এর আগেই হয়, ফলে input tokens ও latency কমে; তবে কিছু তথ্য হারাতে পারে।

Pruning: surgical removal of irrelevant context

Compression shrinks everything; pruning removes parts that aren't relevant to the current task:

Relevance pruning: score each retrieved chunk by similarity to the user's question; drop the bottom quartile.
Recency pruning: drop turns older than N if they've been incorporated into a summary.
Task-aware pruning: for tool calls, drop tool definitions not used by the current step.
Diversity pruning: when multiple retrieved docs cover the same fact, keep the highest-scoring one.

The combined approach

Retrieve K=20 candidate chunks (cheap, fast).
Re-rank with a cross-encoder (precise).
Prune to top 5 by relevance (drop redundancy).
Compress each chunk with LLMLingua if still over budget.
Inject into the prompt in a marked section (§7 — Lost in the Middle).

The "compression + pruning" combo. Compress everywhere, prune where it hurts least. Pruning preserves information density; compression reduces absolute size. They're complementary. The Anthropic context-engineering playbook recommends both, in that order, before any prompt goes to the model.[14]

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Prune chunks, then shorten a paragraph

Why this matters

When your RAG app finds 20 pieces of text for one question, many are repeats or off topic. If you send all of them, you pay for tokens that do not help, and the answer can get worse.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The AI will score and choose chunks (pieces of text).

Prompt

Question: How long does standard delivery take in Dhaka?

Retrieved chunks:
C1: Standard delivery in Dhaka takes 2 working days. Express delivery arrives the same day if you order before 12:00.
C2: Standard delivery inside Dhaka city takes 2 working days.
C3: Our office is closed on Fridays and public holidays.
C4: Delivery outside Dhaka takes 4 to 6 working days.
C5: The company was founded in 2014 and has 120 staff.
C6: Returns are accepted within 7 days if the box is sealed.

Task:
1. Give each chunk a relevance score from 0 to 10.
2. Find chunks that say the same thing. Keep only the best one of them.
3. The prompt has room for only 2 chunks. Choose the 2 best chunks that remain.
4. Answer the question using only the 2 chunks you chose.

You should see: A score for each chunk. C1 and C2 score high, and only one of them is kept. C5 and C6 score 0 or 1. The answer is 2 working days.

Example answer

Scores: C1 = 10, C2 = 9, C3 = 1, C4 = 3, C5 = 0, C6 = 0.
C2 repeats C1, so I drop C2.
Chosen chunks: C1 and C4.
Answer: Standard delivery in Dhaka takes 2 working days.

Your answer will be worded differently.

Step 2
Same chat
Check the AI work. Did it drop one of C1 and C2? Did it give C5 and C6 a low score?
You should see: Yes to both. Models may keep C1 or C2, and both choices are fine because they say the same thing.

Step 3

New chat

Copy this prompt, paste it in a new chat, and press Enter. The AI will make the paragraph shorter.

Prompt

Make the paragraph below less than half as long. Remove low-information words. Keep every number, every name, and every negation such as "not". Then answer the question using only your short text.

Paragraph: Thank you for your order. We would like to tell you that the Basic plan costs 500 tk per month, and the Pro plan costs 1,200 tk per month. You can cancel the Pro plan at any time, but we do not give a refund for a month that has already started. If you pay for a full year, you get 2 months free.

Question: How much does a full year of the Pro plan cost, and can I get a refund in the middle of a month?

You should see: A short text with all the numbers. Then the answer: 12,000 tk for a year of Pro, and no refund for a month that has started.

Example answer

Short text: Basic 500 tk/month. Pro 1,200 tk/month. Cancel Pro any time, no refund for a started month. Pay 1 year: 2 months free.

Answer: A full year of Pro costs 10 x 1,200 = 12,000 tk. No, you cannot get a refund in the middle of a month.

Your answer will be worded differently.

Step 4
Same chat
Check the short text: do 500, 1,200, "2 months free", and "no refund" stay? Also compute 10 months x 1,200 tk.
You should see: All of them stay. 10 x 1,200 = 12,000 tk.

What just happened

Pruning removes whole chunks that do not help, such as a repeat (C2) or an off-topic one (C5). Compression keeps all parts but cuts extra words, so the text is shorter. Both save tokens. Both must keep the numbers and words like "not", or the answer becomes wrong.

You are done when You have a list of kept and dropped chunks, and a short paragraph that still shows all the numbers and the "no refund" rule.

Check yourselfWhat is the difference between pruning and compression?Show answer

Pruning removes whole parts that are off topic or repeated. Compression keeps the parts but cuts extra words to make them shorter. You can use both together.

Try a harder versionIn the first prompt, change the question to: When is the office open? Does the AI now keep C3? What does this show about choosing chunks for each question?

Watch · 2 videos

Tool result truncation strategies

A single tool call can return 100K tokens — a long code file, a thousand-row database result, a 50-page web page. Left unchecked, one tool result can fill your entire context window. Truncation is the discipline of keeping tool results useful but bounded.

The truncation toolkit

Length-based

Hard cap: truncate to N tokens; append [...truncated...].
Head/tail: keep first K and last K tokens; drop the middle.
Top-N lines: for structured output, keep first and last N lines.

Structure-based

Keep schema, sample rows: for DB results, return schema + first 10 rows + count.
Keep signatures: for code, return function signatures + types; drop bodies.
Filter then return: ask the tool to return only what was asked.

Semantic

Re-rank: return all, but mark top-K by relevance.
Summarise on the fly: a small model summarises before returning.
Embed-then-page: store in vector DB, return pointer + summary.

Smart bounds

Reserve window: reserve tokens for tools in your budget (§1).
Iterate: page through results; don't return the entire corpus.
Just-in-time: store in object store, give reference, fetch on demand.

The "page through" pattern

The most production-friendly approach is to never return the whole result in one shot. Instead, the tool returns a small slice + a cursor, and the agent fetches more only if it needs to.

python# A paginated search tool — returns 5 results + a cursor
def search_docs(query: str, cursor: str = None) -> dict:
    results = vector_store.search(query, top_k=5, cursor=cursor)
    return {
        "results":      results.docs,            # < 2K tokens
        "next_cursor":  results.next_cursor,
        "total":        results.total_count,    # "5,234 more"
    }

# Agent can call search_docs again with the new cursor
# if it wants to see more — bounded cost, iterative exploration.

Where to truncate

Return the high-signal bits (summary, top slice, schema) and a pointer for the rest. Let the agent pull more if needed.

এই ডায়াগ্রামে একটি tool result-এর গঠন দেখানো হয়েছে: ওপরে summary header, মাঝে key slice (top 5 results), তারপর schema + counts এবং শেষে আরও ডেটার জন্য cursor। অর্থাৎ বেশি সিগন্যালযুক্ত অংশ আগে দিয়ে বাকিটা pointer দিয়ে পরে আনা যায়।

Don't truncate code blindly. Truncating a function body mid-line breaks parsing. Truncating a JSON result mid-object breaks the parser. Use structure-aware truncation: stop at function boundaries, object closing braces, line endings. Better yet, ask the tool to do the truncation at the source.[13]

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

See what a cut tool result hides

Why this matters

When your AI agent calls a tool, the result can be huge, so apps often cut it. If the cut falls in the wrong place, the agent can read a wrong number and give a customer a wrong answer.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The tool result in it was cut in the middle of a number.

Prompt

You are a support agent. You have a tool search_docs(query, cursor). You called search_docs('refund policy'). The tool result was cut by a hard cap, and this is all you received:

{"results": [{"id": "d1", "text": "Monthly plans can be refunded within 14 days of payment."}, {"id": "d2", "text": "Annual plans can be refunded within 3
[...truncated...]

User question: How many days do I have to ask for a refund on an annual plan? Answer from the tool result only. If you are not sure, say what you would do next.

You should see: A good AI says it is not sure, because the number stops at "3". It says it would fetch more. A weaker AI may guess.

Example answer

I am not sure. The tool result stops at "within 3", so the number may be 3, 30, or something else. Before I answer, I would call search_docs again or ask for the full text of d2.

Your answer will be worded differently.

Step 2

New chat

Copy this prompt, paste it in a new chat, and press Enter. This tool result is a small, complete page.

Prompt

You are a support agent. You have a tool search_docs(query, cursor). You called search_docs('refund policy'). The tool returned one page:

{"results": [{"id": "d1", "text": "Monthly plans can be refunded within 14 days of payment."}, {"id": "d2", "text": "Annual plans can be refunded within 30 days of payment."}, {"id": "d3", "text": "Refunds go back to the original payment method."}, {"id": "d4", "text": "Gift cards cannot be refunded."}, {"id": "d5", "text": "Refund requests are handled by the billing team."}], "next_cursor": "c_02", "total": 5234}

User question: How many days do I have to ask for a refund on an annual plan? Answer from the tool result only. Say if you need the next page.

You should see: A sure answer: 30 days. The AI says it does not need the next page.

Example answer

You have 30 days from the payment date to ask for a refund on an annual plan (document d2). This page already has the answer, so I do not need the next page.

Your answer will be worded differently.

Step 3
Same chat
Compare the two answers. In the first prompt, write down 3 numbers that could follow "within 3".
You should see: For example 3, 30, or 365. The first answer could not be sure. The second answer is sure: 30.

What just happened

A hard cap cuts the text at a fixed length, even in the middle of a number. Here it cut "30" to "3", so the AI could not be sure. A better tool returns a small page with a cursor (a bookmark for the next page) and a total count. The AI gets whole facts, and it asks for more only when it needs them.

You are done when You have one answer where the AI is not sure, and one answer where it says 30 days.

Check yourselfWhy is paging better than a hard cap?Show answer

A hard cap can cut a fact or a JSON object in the middle, which breaks the parser or changes the meaning. A small page with a cursor keeps results whole, and the agent fetches more only when it needs it.

Try a harder versionSend the second prompt again in a new chat, but change the question to: Which team handles a refund for a gift card? Does the AI ask for the next page? Is that the right choice?

Watch · 2 videos

Quit Tokenmaxxing: 4 Context Engineering Techniques for AI Agents
AWS Developers
Context Engineering for AI Agents with LangChain and Manus
LangChain

Lost in the Middle & mitigation

Liu et al. (2023) showed a robust phenomenon: language models attend strongly to information at the beginning and end of their input, and perform close to random for information buried in the middle. The longer the context, the worse the middle. This single finding reframes how you should structure prompts.

The evidence

Liu et al. inserted random "needle" facts at varying positions in a long context and measured retrieval accuracy. Result: U-shaped performance curve — recall is high at the start, collapses in the middle, recovers at the end. The effect is worse for longer contexts and weaker models, but persists even in frontier models with 100K+ context windows.[1]

The U-curve: retrieval accuracy vs. position. Endpoints are reliable; the middle is risky. Place critical information at start or end.

এখানে context-এর ভেতরে position অনুযায়ী retrieval accuracy-র একটি illustrative U-curve দেখানো হয়েছে। শুরু ও শেষে accuracy strong, কিন্তু মাঝের collapsed middle অংশে তা কমে যায়; তাই গুরুত্বপূর্ণ তথ্য মাঝের danger zone-এ রাখা উচিত নয়।

Why does it happen?

→ Attention is biased by position. Transformers (without relative position encodings) have weaker attention to tokens far from the query token at the end. Long-context extensions (RoPE, ALiBi, YaRN) help but don't fully fix it.
→ Training distribution. Most pre-training data has shorter contexts; models didn't learn to integrate information across long distances.
→ Positional primacy & recency. There's evidence models weight the first and last positions more heavily — the same bias that affects humans.

Mitigation tactics

1. Sandwich the critical content

Place the most important instructions and retrieved facts at the start of the system message and repeat them at the end of the user message. Hit both attention peaks.

2. Re-order retrieved docs

For multi-document RAG, place the most relevant doc at the start AND the end. Put mid-relevance docs in the middle (where they won't be relied on as heavily).

3. Trim aggressively

Shorter context > longer context for recall. The middle problem is exacerbated at 50K+ tokens. If you can fit in 8K, do.

4. Use structured delimiters

Wrap critical content in <important>…</important> tags. Strong models pay special attention to marked regions.

5. Re-query when uncertain

For RAG, design the system to: (1) read docs, (2) if answer not found, re-query with the partial answer as context. Two shorter queries beat one long one.

6. Explicit "answer here" markers

End the prompt with: "Based on the above, your answer:" — recent papers show that forcing the model to "look back" before answering helps retrieval from the middle.

The empirical reality check

Follow-up work (RULER, 2024; Summary-of-a-Haystack, 2024; BABILong, 2024) refined the picture:[6][5]

The U-curve flattens with better models and longer-context training, but never fully goes away.
Variable-length tasks (retrieval over changing input sizes) are harder than fixed-length.
Multi-hop reasoning across long contexts remains an open problem.
RAG + long-context is complementary, not redundant.

Practical rule: if a fact needs to be reliably recalled, put it in the first 10% or the last 10% of the context. If you can't, repeat it at both ends. If neither works, the model genuinely can't handle that fact — use a smaller context, or external retrieval, or a different architecture.[14]

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Test rules hidden in the middle

Why this matters

When you give an AI agent a long list of rules, it can miss the ones in the middle. A missed rule can mean a wrong tone, a banned promise, or a broken format in a real customer email.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The AI will write an email with 13 rules in the middle.

Prompt

You write short customer emails. Follow ALL rules.

Rules:
1. Write in English.
2. Use a polite tone.
3. Greet the customer by name.
4. Say sorry one time only.
5. Mention the order number.
6. Do not promise a delivery date.
7. Keep the email under 60 words.
8. Never use exclamation marks.
9. The last line of the email must be exactly: Ref: KX-482
10. Do not use emojis.
11. Do not mention other customers.
12. Use plain words.
13. Put the signature 'Support Team' before the Ref line.

Task: Write an email to Mina Akter. Her order #5521 is 2 days late. Offer a 10% discount on her next order.

You should see: A short email to Mina Akter. It should be under 60 words, have no exclamation marks, and end with Ref: KX-482.

Example answer

Hello Mina Akter,

I am sorry that your order #5521 is 2 days late. To thank you for your patience, we offer you 10% off your next order.

Support Team
Ref: KX-482

Your answer will be worded differently.

Step 2
Same chat
Check the email for 3 rules: under 60 words, no exclamation marks, and the last line is exactly Ref: KX-482. Count the words.
You should see: Most chats pass all 3 rules. A rule from the middle, such as the word count, is the one that fails most often.

Step 3

New chat

Copy this prompt, paste it in a new chat, and press Enter. The 3 key rules are now at the start and at the end.

Prompt

MOST IMPORTANT RULES: the email has under 60 words, has no exclamation marks, and its last line is exactly: Ref: KX-482

You write short customer emails. Follow ALL rules.

Rules:
1. Write in English.
2. Use a polite tone.
3. Greet the customer by name.
4. Say sorry one time only.
5. Mention the order number.
6. Do not promise a delivery date.
7. Keep the email under 60 words.
8. Never use exclamation marks.
9. The last line of the email must be exactly: Ref: KX-482
10. Do not use emojis.
11. Do not mention other customers.
12. Use plain words.
13. Put the signature 'Support Team' before the Ref line.

Task: Write an email to Mina Akter. Her order #5521 is 2 days late. Offer a 10% discount on her next order.

Before you answer, check again: under 60 words, no exclamation marks, last line exactly 'Ref: KX-482'.

You should see: Another short email with the same 3 rules followed. Most chats pass.

Example answer

Dear Mina Akter,

I am sorry that your order #5521 is 2 days late. As a thank you for waiting, we give you 10% off your next order.

Support Team
Ref: KX-482

Your answer will be worded differently.

Step 4
Same chat
Check this second email for the same 3 rules. Count the words.
You should see: All 3 rules pass. If the first email failed a rule, compare the two emails.
Step 5
Same chat
Optional: press the regenerate button (the redo button) 2 times in each chat, if it has one. Check the 3 rules each time and count the passes.
You should see: Most results pass all 3 rules. A failed rule is often an exclamation mark or too many words.

What just happened

Many AI models read the start and the end of a long prompt best, and the middle less well. This is called Lost in the Middle. The second prompt puts the key rules at the top and repeats them at the end, so the model sees them twice at the strong places. Your prompt here is short, so most models pass both tests. The problem grows when the prompt is long.

You are done when You have 2 emails, and for each one you checked 3 rules: under 60 words, no exclamation marks, last line Ref: KX-482.

Check yourselfWhere do you place a rule that the model must not miss, and why?Show answer

Place it at the start and repeat it at the end of the prompt. Models remember the start and the end of a long prompt best, and they are weaker in the middle.

Try a harder versionAdd 30 more filler rules to the middle of the first prompt and run it again. Does the model now miss more of the 3 rules? Does the second prompt still help?

Watch · 3 videos

Lost in the Middle: How Language Models use Long Context - Explained!
Weaviate vector database
Lost in the Middle: How Language Models Use Long Contexts Paper Reading
Arize AI
Context Rot: When Long Context Fails
Hamel Husain

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.

The budget model

Allocation principles

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

Step 4

What just happened

Watch · 2 videos

When to summarize

The summarization pattern

What to keep in the summary

Information loss is real

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

Step 4

What just happened

Watch · 3 videos

How caching works (Anthropic + OpenAI + Gemini)

The standard cache layout

What to cache vs what not to

Cache aggressively

Don't cache

The cost math

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

What just happened

Watch · 3 videos

The four-layer model

The MemGPT insight

The memory-update pattern

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

What just happened

Watch · 2 videos

Compression techniques

LLMLingua — the canonical compression model

Pruning: surgical removal of irrelevant context

The combined approach

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

Step 4

What just happened

Watch · 2 videos

The truncation toolkit

Length-based

Structure-based

Semantic

Smart bounds

The "page through" pattern

Where to truncate

Practice · 1 task

Why this matters

Step 1

Step 2

Step 3

What just happened

Watch · 2 videos

The evidence

Why does it happen?

Mitigation tactics

1. Sandwich the critical content

2. Re-order retrieved docs

3. Trim aggressively

4. Use structured delimiters

5. Re-query when uncertain

6. Explicit "answer here" markers