Skip to content
AI Engineering
বাংলা

Part 10

Cost and Performance Optimization

Model cascading, semantic / output / prompt caching, batch APIs, distillation, LLMLingua-style prompt compression, per-feature token budgets, and model gateways (LiteLLM, Portkey, OpenRouter) — a five-tier stack that can cut LLM spend by 5–50× without losing quality. Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

1 section · 30 sources · three voices

01

Cost & performance optimisation

LLM bills scale with usage. Three knobs let you cut cost by 5–50× without losing quality: cascade (cheap → expensive routing), cache (semantic, prefix, output), and compress (prompt compression, distillation).

Model cascading (cheap → expensive routing)

Route easy requests to a small, cheap model; only escalate to a large, expensive one when the small model's confidence is low (use logprobs, self-critique, or a separate classifier).[20] A well-tuned cascade often reaches frontier quality at 30–70% of the cost.

# Two-tier cascade
def answer(q):
    small = call("llama-3.1-8b", q, max_tokens=200)
    if small.confidence > 0.85 or is_simple(q):
        return small.text
    return call("claude-sonnet-4-5", q, system="Refine this draft: "+small.text)

Caching: semantic, output, prompt

Caching is the single most powerful cost lever. Three kinds:

  • Semantic cache: hash embeddings of the prompt; if a near-identical prompt was answered recently (cosine > 0.95), return the cached answer. Built into GPTCache, Redis with vector search, and Helicone.[21]
  • Output cache: for deterministic tasks (extraction, classification with temperature=0), a key-value cache keyed on the input.
  • Prompt cache: provider-level (Anthropic prompt_caching, OpenAI cached_prompt) reuses the KV of the system prompt across calls. 80–90% discount on cached prefix.

Batch APIs for non-real-time workloads

OpenAI's /v1/batches, Google Batch, Together batch — submit a JSONL file, get results within 24h, at half the cost. Use for: nightly evals, bulk labelling, backfills, anything that doesn't need an answer in the next 5 seconds.

Distillation (small model from big)

Take 50,000 (prompt, big_model_answer) pairs from production (with consent), fine-tune a small open model on them. DeepSeek-R1's "distilled Qwen" line showed that a well-distilled 7B beats an un-distilled 12B on reasoning benchmarks.[22] Distillation is the only technique that simultaneously cuts cost and latency and unlocks self-hosting.

Prompt compression (LLMLingua-style)

Tools like LLMLingua remove stopwords, paraphrase verbose passages, and drop low-information tokens — often 5–20× compression with <3% quality hit.[23] Particularly powerful for long RAG contexts where most chunks are filler.

Token budgeting per feature

Allocate a monthly token budget per feature, alert at 80%, throttle at 100%. This is the operational layer that prevents one runaway feature from blowing the company's bill.

Model gateways: LiteLLM, Portkey, OpenRouter

A model gateway sits between your app and every LLM provider. It gives you: one API for OpenAI + Anthropic + Together + your own vLLM; per-key rate limits; cost tracking; retries; fallbacks; semantic caching out of the box. The three major options are LiteLLM (Python proxy), Portkey (production-grade SaaS), and OpenRouter (consumer-grade aggregator).[24]

Cost optimisation stack — top-down, highest impact first Tier 1 — Eliminate the call entirely semantic cache · output cache · prompt cache · guardrail short-circuit typical savings: 30–80% of total cost zero quality impact if cache key is correct Tier 2 — Shrink the call prompt compression (LLMLingua) · short system prompts · trim tool output · max_tokens caps typical savings: 30–70% on remaining calls minor quality risk; measure before/after Tier 3 — Use a cheaper model cascade (8B → 70B) · distillation · batch API (50% off) · self-host open weights typical savings: 2–10× per remaining call requires quality testing per workload Tier 4 — Discount the call provider price matching · committed-use discounts · AWS / Azure credits typical savings: 10–40% at scale procurement play, not engineering Tier 5 — Stop runaway usage token budgets · per-user rate limits · loop detection · prompt-length caps typical savings: unpredictable but bounded prevents the $100k surprise on the 31st work top-down; Tiers 1–3 are engineering, Tiers 4–5 are procurement / ops. Each tier compounds with the ones above. gate each tier with an offline eval: measure quality before/after; refuse the optimisation if it breaks your golden dataset.
The cost-optimisation stack. Each tier compounds with the ones above; work top-down. Tiers 1–3 are engineering, Tier 4 is procurement, Tier 5 is ops.
খরচ-অপ্টিমাইজেশন স্ট্যাক। প্রতিটি স্তর উপরের স্তরগুলোর সাথে মিলিত হয়; উপর থেকে নিচে কাজ করুন। স্তর 1–3 ইসলেব-ই, স্তর 4 procurement, স্তর 5 ops।

Practice · 4 tasks

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Send easy requests to a cheap model

Why this matters

When you build a support chatbot, most questions are easy. If a big, costly model answers all of them, you pay too much. A router sends easy questions to a cheap model and hard ones to the big one.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. The AI acts as a router for 10 customer requests.

    Prompt
    You are a router for a customer-support system. Each request goes to a SMALL model (cost 1 credit) or a LARGE model (cost 20 credits). Use SMALL for easy requests. Use LARGE for requests that need long reasoning, code, or many steps.
For each request reply with: the number, SMALL or LARGE, and a reason of at most 6 words.

Requests:
1. What is your return window?
2. Where is my order #4471?
3. Translate "thank you" into Bangla.
4. Two of my orders came in one parcel. One item is damaged. I used a coupon that split the refund across both orders. How much should I get back?
5. How do I reset my password?
6. Summarise the legal risks of changing our refund policy for customers in three countries.
7. What is your phone number?
8. Write a Python script that matches our payment file to bank statements and flags the mismatches.
9. Is the blue shirt in stock in size M?
10. Explain why our conversion rate dropped 12% after the checkout redesign and propose three tests.

    You should see: A list of 10 lines. About 4 requests (the long, many-step ones) go to LARGE, and the rest go to SMALL.

    Example answer
    1. SMALL - simple policy fact
2. SMALL - order status lookup
3. SMALL - short translation
4. LARGE - multi-step refund maths
5. SMALL - standard how-to
6. LARGE - legal reasoning, three countries
7. SMALL - fixed contact fact
8. LARGE - needs code
9. SMALL - simple stock check
10. LARGE - analysis and test design

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Count the LARGE lines. Then calculate the cost: SMALL count x 1 + LARGE count x 20. Compare it with 10 x 20 = 200 credits.

    You should see: With 4 LARGE and 6 SMALL: 6 x 1 + 4 x 20 = 86 credits, against 200 credits if all go to LARGE. That saves 114 credits, which is 57%.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. Now a checker looks at a draft answer from the small model.

    Prompt
    A small model wrote a draft answer. Decide if the draft is good enough to send (KEEP) or if a large model must redo it (ESCALATE). Reply with KEEP or ESCALATE and one sentence of reason.

Question: A customer bought 3 shirts at 450 BDT each and 1 cap at 280 BDT. A 10% coupon applies to the whole order. Delivery costs 60 BDT and is not discounted. What is the total?

Draft answer: The shirts cost 1,350 BDT and the cap costs 280 BDT. The subtotal is 1,630 BDT. After the 10% coupon the price is 1,467 BDT. Delivery is 60 BDT, so the total is 1,527 BDT.

    You should see: KEEP with a reason. The draft is correct: 3 x 450 = 1,350, plus 280 = 1,630, minus 10% (163) = 1,467, plus 60 = 1,527.

    Example answer
    KEEP. The draft adds the items correctly (1,630 BDT), takes 10% off (1,467 BDT), and adds 60 BDT delivery after the discount, so 1,527 BDT is right.

    Your answer will be worded differently.

What just happened

A cascade means you try the cheap model first and use the expensive model only when needed. Here 6 of 10 requests used the cheap model, so the cost fell from 200 to 86 credits. The second prompt showed the other half: a check that decides if a cheap draft is good enough. A wrong KEEP sends a bad answer to a customer. A wrong ESCALATE wastes 20 credits.

You are done when You have a router list, a cost of about 86 credits against 200, and one KEEP or ESCALATE decision.

Check yourselfWhy does a cascade cost less than sending every request to the large model?Show answer

Most requests are easy, so the cheap model answers them at 1 credit each. Only the hard ones go to the large model at 20 credits, so the average cost per request falls.

Try a harder versionChange the draft to: The subtotal with delivery is 1,690 BDT. After the 10% coupon the total is 1,521 BDT. The delivery was discounted by mistake. Does the checker now say ESCALATE?

Task 2

Spot a wrong answer from a cache

Why this matters

When you build a chatbot, many users ask the same question in different words. A cache can reuse the old answer and save money. But if it reuses an answer for a question that only looks the same, the user gets a wrong answer.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. It shows 6 pairs of questions. The AI guesses if the old answer still fits the new question.

    Prompt
    A semantic cache stores answers to old questions. For a new question, it returns the stored answer if the new question looks very similar to the old one.
For each pair, give: a similarity score from 0 to 1 (your best guess), HIT or MISS (HIT if the score is above 0.9), and whether the old answer would be CORRECT or WRONG for the new question.
Reply with one short line per pair.

Pairs (old question | new question):
1. How do I reset my password? | How can I reset my password?
2. What is the refund status of order 1001? | What is the refund status of order 1002?
3. What is your return policy? | How long do I have to send an item back?
4. What is the weather in Dhaka today? | What is the weather in Dhaka tomorrow?
5. Please cancel my subscription. | Please do not cancel my subscription.
6. What time do you close at night? | When do you shut in the evening?

    You should see: 6 short lines. Pairs 2, 4, and 5 usually get a high score (HIT) and are marked WRONG. Scores differ between chats.

    Example answer
    1. 0.97 HIT CORRECT
2. 0.96 HIT WRONG (different order)
3. 0.78 MISS CORRECT
4. 0.94 HIT WRONG (today vs tomorrow)
5. 0.93 HIT WRONG (opposite meaning)
6. 0.80 MISS CORRECT

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Write down the pair numbers that are a HIT and also WRONG. Say in your own words what changed in each pair.

    You should see: Usually pairs 2, 4, and 5: a different order number, today vs tomorrow, and cancel vs do not cancel.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. The AI writes rules that stop a wrong cache hit.

    Prompt
    I run a semantic cache for a support chatbot. Give me 4 short rules that decide when a cached answer must NOT be used, even if the similarity score is very high. Give one example question pair for each rule. Keep each rule to one sentence.

    You should see: 4 short rules, for example: do not use the cache for different IDs, time words, negations (not), or personal data.

    Example answer
    1. Do not reuse an answer if the IDs or numbers differ. Example: order 1001 vs order 1002.
2. Do not reuse an answer if a time word differs. Example: today vs tomorrow.
3. Do not reuse an answer if one question has a negation. Example: cancel vs do not cancel.
4. Do not reuse an answer that holds personal data. Example: What is my balance?

    Your answer will be worded differently.

What just happened

A semantic cache compares the meaning of questions, not the exact words. Pairs 1, 3, and 6 change only the words, so the old answer is right, and reusing it saves money. Pairs 2, 4, and 5 change one small thing, so the old answer is wrong. Here the AI only guessed the scores. A real cache measures them with an embedding, which turns text into numbers.

You are done when You have a list of the HIT and WRONG pairs (usually 2, 4, and 5) and 4 rules that block them.

Check yourselfWhy is a high similarity score not enough to return a cached answer?Show answer

A small change, such as an order number, a day, or the word not, can change the right answer. The words stay almost the same, so the score stays high. Extra rules and a strict threshold lower the risk of a wrong hit.

Try a harder versionWrite two new pairs: one safe HIT with very different words, and one wrong HIT with almost the same words. Paste them into step 1 to test them.

Task 3

Shorten a long text and test it

Why this matters

When you build a chatbot with long documents, you pay for every word you send to the AI. If you cut the filler words, you pay less. But you must check that the short text still gives the same answers.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Write down the 3 answers. They are your baseline.

    Prompt
    Answer the three questions using only the context.

Context:
Thank you for choosing Padma Mart. We are very happy to tell you about our shipping policy. In general, orders that are placed before 2 PM on a working day are usually packed on the same day, and orders that are placed after 2 PM are usually packed on the next working day. Please note that we do not work on Fridays and public holidays. Delivery inside Dhaka normally takes one to two days, while delivery to other cities normally takes three to five days. Shipping inside Dhaka costs 60 BDT, and shipping to other cities costs 120 BDT. We are also glad to say that orders above 2,000 BDT get free shipping inside Dhaka only. If you are not at home, our courier will try to deliver your parcel two times, and after the second failed attempt the parcel returns to our warehouse. We hope you enjoy shopping with us!

Questions:
1. How much is shipping to Chattogram for a 1,500 BDT order?
2. When is a Thursday 3 PM order packed?
3. How many delivery attempts does the courier make?

    You should see: Three answers: shipping 120 BDT, packed on Saturday, and 2 delivery attempts.

    Example answer
    1. 120 BDT. Shipping to other cities costs 120 BDT, and free shipping is only inside Dhaka for orders above 2,000 BDT.
2. On Saturday. A Thursday order after 2 PM is packed on the next working day, and Friday is a non-working day.
3. 2 attempts.

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this prompt, paste it, and press Enter. Then copy the short text that the AI gives you.

    Prompt
    Compress the text below to about one third of its length. Remove filler words and greetings. Keep every number, price, time, and rule. Use short note-style phrases. Reply with the compressed text only.

Text:
Thank you for choosing Padma Mart. We are very happy to tell you about our shipping policy. In general, orders that are placed before 2 PM on a working day are usually packed on the same day, and orders that are placed after 2 PM are usually packed on the next working day. Please note that we do not work on Fridays and public holidays. Delivery inside Dhaka normally takes one to two days, while delivery to other cities normally takes three to five days. Shipping inside Dhaka costs 60 BDT, and shipping to other cities costs 120 BDT. We are also glad to say that orders above 2,000 BDT get free shipping inside Dhaka only. If you are not at home, our courier will try to deliver your parcel two times, and after the second failed attempt the parcel returns to our warehouse. We hope you enjoy shopping with us!

    You should see: A much shorter text in note style, about one third to one half of the original, with all the numbers still there.

    Example answer
    Padma Mart shipping. Order before 2 PM (working day): packed same day. After 2 PM: next working day. No work Fri, public holidays. Delivery: Dhaka 1-2 days, other cities 3-5 days. Shipping: Dhaka 60 BDT, other 120 BDT. Free shipping: Dhaka only, orders above 2,000 BDT. Courier tries 2 times, then parcel returns to warehouse.

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt and paste it. Replace the line PASTE YOUR COMPRESSED TEXT HERE with the short text from step 2. Press Enter.

    Prompt
    Answer the three questions using only the context.

Context:
PASTE YOUR COMPRESSED TEXT HERE

Questions:
1. How much is shipping to Chattogram for a 1,500 BDT order?
2. When is a Thursday 3 PM order packed?
3. How many delivery attempts does the courier make?

    You should see: Three answers again. They should match step 1: 120 BDT, Saturday, and 2 attempts.

    Example answer
    1. 120 BDT. Shipping to other cities costs 120 BDT, and free shipping is only inside Dhaka for orders above 2,000 BDT.
2. On Saturday. A Thursday order after 2 PM is packed on the next working day, and Friday is a non-working day.
3. 2 attempts.

    Your answer will be worded differently.

  4. Step 4
    Same chat

    Compare the 3 answers with step 1. Write down any answer that is different.

    You should see: Often all 3 answers are the same. If one is different, the short text lost a fact.

What just happened

You pay for the tokens (small pieces of words) that you send to the AI. Greetings and filler words use tokens but add no facts. The short text kept all the numbers and rules, so the AI gave the same answers for less cost. Always run this test after you compress. If a fact is lost, the AI gives a wrong answer.

You are done when You have two sets of 3 answers, and you know if they match.

Check yourselfWhy does compressing a prompt save money, and what must you check afterwards?Show answer

The cost depends on the number of tokens you send. Filler text uses tokens but adds no facts, so removing it saves money. Afterwards, check that the AI still gives the same answers, because a lost fact makes a wrong answer.

Try a harder versionAsk for compression to one sixth of the length. Run step 3 again. At what length does the first fact get lost?

Task 4

Find which feature will break its budget

Why this matters

When you build an AI product, every feature spends tokens, and tokens cost money. One feature that runs out of control can use up the whole month's bill. A budget per feature, with alerts, warns you early.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. The AI makes a budget table for 3 features.

    Prompt
    A company gives each LLM feature a monthly token budget. The rule: send an alert at 80% of the budget and throttle (slow down or stop) the feature at 100%.
Today is the end of day 12 of a 30-day month. Assume the daily use stays the same.

Feature | Monthly budget | Used so far (12 days)
Chat support | 40,000,000 tokens | 20,000,000 tokens
Document summary | 20,000,000 tokens | 4,000,000 tokens
Search rewrite | 6,000,000 tokens | 3,600,000 tokens

Make a table. For each feature show:
1) Tokens used per day.
2) Forecast usage at the end of the month, as a percent of the budget.
3) The day of the month when it reaches 80%.
4) The day of the month when it reaches 100%.
Write "never" if the feature stays below the limit.

    You should see: A table with 3 rows. The forecast for Search rewrite is the highest, at about 150%.

    Example answer
    Feature | Per day | Month end | 80% day | 100% day
Chat support | 1.67M | 125% | day 20 | day 24
Document summary | 0.33M | 50% | never | never
Search rewrite | 0.3M | 150% | day 16 | day 20

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Check one row by hand. For Search rewrite: 3,600,000 ÷ 12 = tokens per day. Then multiply by 30. Compare with the table.

    You should see: 300,000 tokens per day. 300,000 x 30 = 9,000,000 tokens. That is 150% of the 6,000,000 budget. It reaches 80% (4,800,000) on day 16, and 100% on day 20.

  3. Step 3
    Same chat

    Compare the other two rows with this key. If a number differs, ask the chat to calculate again. Key: Chat support 125%, day 20, day 24. Document summary 50%, never, never.

    You should see: The same numbers. Chat support uses 20,000,000 ÷ 12 = about 1.67M per day, so 1.67M x 30 = 50M, which is 125% of 40M.

  4. Step 4
    Same chat

    Copy this follow-up question, paste it, and press Enter.

    Prompt
    Which feature needs action first? Suggest two cheap ways to reduce its usage. Answer in 3 short lines.

    You should see: Search rewrite first, because its alert comes first (day 16), and two ideas such as a cache or a shorter prompt.

    Example answer
    Search rewrite needs action first. It reaches 80% on day 16.
1. Add a cache, so repeated questions do not call the AI.
2. Shorten the prompt, so each call uses fewer tokens.

    Your answer will be worded differently.

What just happened

A forecast is the use per day multiplied by the days in the month. The budget gives each feature its own limit. An alert at 80% gives you time to act. A throttle at 100% stops the bill from growing. Notice that the smallest budget (Search rewrite) is the first problem, not the feature that uses the most tokens.

You are done when You have a table with 3 forecasts, and you know that Search rewrite hits its 80% alert on day 16.

Check yourselfWhy set a budget per feature, and not one budget for the whole company?Show answer

With one budget per feature, you see which feature costs the money, and one runaway feature cannot use the whole bill. The 80% alert gives you time to act before the 100% throttle.

Try a harder versionSay a cache removes 40% of the calls for Search rewrite. Ask the chat for the new month-end forecast and the new alert day.

Watch · 8 videos

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.

  1. [1]Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
  2. [2]Wei et al., "Plan-and-Execute Agents" / "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — NeurIPS 2022
  3. [3]Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" — NeurIPS 2023
  4. [4]Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models"
  5. [5]Anthropic Engineering blog — "How we built our multi-agent research system" (2025)
  6. [6]LangGraph — official documentation (LangChain)
  7. [7]Temporal — "Durable Execution for LLM Agents"
  8. [8]Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" — 2023
  9. [9]Anthropic — "Introducing computer use, a new Claude 3.5 Sonnet capability" (Oct 2024)
  10. [10]Anthropic — Tool use / function calling documentation
  11. [11]OpenAI — Function calling guide
  12. [12]Anthropic — Model Context Protocol (MCP) specification
  13. [13]Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" — vLLM paper, SOSP 2023
  14. [14]Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" — 2023
  15. [15]llama.cpp — open-source GGUF-native inference (ggerganov)
  16. [16]Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" — MLSys 2024
  17. [17]Anthropic — Prompt caching documentation
  18. [18]Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — NeurIPS 2023
  19. [19]Es et al., "Ragas: Automated Evaluation of Information Retrieval Augmented Language Models" — 2023
  20. [20]Yue et al., "Large Language Model Cascades with Mixture of Thought Representations" — 2024
  21. [21]GPTCache — open-source semantic cache
  22. [22]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL" — 2025
  23. [23]Jiang et al., "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models" — EMNLP 2023
  24. [24]LiteLLM — unified interface for 100+ LLMs
  25. [25]OWASP — "Top 10 for LLM Applications" (LLM01 Prompt Injection, 2025)
  26. [26]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
  27. [27]NVIDIA NeMo Guardrails — open-source toolkit
  28. [28]Qwen2-VL Team, "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" — 2024
  29. [29]OpenAI — Realtime API documentation
  30. [30]OpenAI — Production best practices guide