Skip to content
AI Engineering
বাংলা

Part 01

LLM Fundamentals

from an engineer's lens

A dense, citation-backed reference for ten topics every ML engineer, applied researcher, and platform builder needs to internalise — tokens, families, sampling, structured outputs, tool use, multimodal, embeddings, reasoning, and the trade-offs that decide which model runs in production. Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies) at the bottom of each topic. প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

10 sections · 26 sources · three voices

01

Tokens, tokenization & context cost

Everything a model does — attention, generation, cost — is denominated in tokens, not characters or words. If your prompt shrinks from 800 to 200 tokens, you just cut cost by 75% and latency by a meaningful fraction. This is the single biggest engineering lever you have.

What is a token?

A token is one element in the model's vocabulary. Modern LLMs don't read characters; they read the output of a subword tokenizer — most commonly Byte-Pair Encoding (BPE), which iteratively merges the most frequent adjacent pairs in a corpus until a target vocabulary size (often 32k–200k) is reached.[15] For multilingual and code workloads, SentencePiece is the dominant alternative — it operates directly on raw Unicode bytes or whitespace, treating language as a configuration parameter.[16]

raw input unbelievable . _tokenization _ helps _compression . BPE merges (low→high freq) token IDs (vocab index) un believ able . _token ization _ helps _comp → 11 subword pieces (vs 33 raw chars). Frequent runs = single tokens; rare substrings split into 2–3. → The model never sees characters — only these integer IDs feed the embedding table.
BPE tokenization in practice: English byte-runs merge into common pieces; punctuation and whitespace often become standalone tokens.
এই diagram-এ দেখানো হয়েছে raw input টেক্সট কীভাবে BPE merges-এর মাধ্যমে subword pieces-এ ভাগ হয় এবং প্রতিটি piece একটি token ID পায়। উপর থেকে নিচে পড়ুন: মডেল কখনো characters দেখে না, শুধু এই integer IDs-ই embedding table-এ যায়।

Why this matters: the cost equation

APIs charge per token, not per character. A useful mnemonic: ~1 token ≈ 4 characters in English for most modern tokenizers, but this varies wildly by language — Chinese runs ~1 char ≈ 1.5 tokens, code ~1 token ≈ 3.5 chars, numbers are split digit-by-digit in early tokenizers but consolidated in newer ones.

# Counting tokens with tiktoken (OpenAI) or transformers
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Tokenization is the silent tax on every LLM call."
print(len(enc.encode(text)))   # → 11 tokens, not 8 words

# Concrete cost (input, gpt-4o-mini, Jul 2025 pricing)
tokens        = 11
price_per_m   = 0.15      # USD per 1M input tokens
cost_per_call = tokens / 1_000_000 * price_per_m   # → $0.00000165

Context windows — what they actually are

The context window is the maximum total tokens (input + output) the model can attend to in a single call. It has grown dramatically: GPT-3's 2K window in 2020 → 8K (GPT-3.5) → 128K (GPT-4o) → 1M+ (Gemini 1.5).[1][11]

GPT-3 (2020)
2,048 tokens
GPT-4 Turbo
128K tokens
Claude 4 Sonnet
200K (1M β)
Gemini 1.5 Pro
1M–2M tokens
Llama 3.1 405B
128K tokens
Qwen3
32K native, 1M w/ YaRN

Long context ≠ free performance. Even with 1M-token windows, retrieval-augmented generation, summarisation, or hierarchical attention beats dumping 800K tokens in. Needle-in-a-haystack benchmarks improve, but multi-hop reasoning across long contexts degrades sharply. Measure on your own workloads.

How tokens map to cost & latency

ComponentCost driverWhat dominates
Prefill (prompt processing)Tokens in × input $/MQuadratic attention compute; ~100 ms–2 s on modern APIs for 10K tokens
Decode (generation)Tokens out × output $/MSequential (one per step); output tokens are 3–5× more expensive than input on most APIs
CachingCache hits vs missesPrompt caching (Anthropic, OpenAI, Gemini) cuts repeated-prefix cost by ~80–90%
Long contextQuadratic attentionA 1M-token call isn't 125× a 8K call — closer to 5–8× due to attention scaling and chunked prefill

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Cut the prompt, keep the answer

Why this matters

When a support bot reads millions of tickets a month, every extra word in the prompt is paid for again on every call. A shorter prompt can save real money.

  1. Step 1
    New chat

    Copy this long prompt, paste it, and press Enter. Write down the answer.

    Prompt
    Hello! I hope you are having a wonderful day. I am working on a customer support system and I would really appreciate your kind help with a small task. Below you will find a customer ticket. Could you please read the ticket very carefully and then tell me which one of the following categories it belongs to? The categories are: billing, shipping, defect, account. Please choose only one category. When you answer, please do not write any explanation, any greeting, or any extra sentence. Just write the single category word. Thank you so much for your help!

Ticket: My order #48213 arrived yesterday and the screen is cracked. I want my money back.

    You should see: One word, most likely defect.

    Example answer
    defect

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this short prompt, paste it, and press Enter. Write down the answer.

    Prompt
    Classify the ticket as one of: billing, shipping, defect, account. Reply with the category only.

Ticket: My order #48213 arrived yesterday and the screen is cracked. I want my money back.

    You should see: The same word, defect, from a prompt that is more than 3 times shorter.

    Example answer
    defect

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. Check the numbers with a calculator.

    Prompt
    A support bot sends 800 input tokens per call. A shorter prompt sends 200 input tokens per call. The price is $0.50 per 1 million input tokens. The bot makes 2 million calls per month. Calculate the monthly input cost for each prompt, and the percentage saved. Show each step.

    You should see: $800 per month for the long prompt, $200 for the short one, and 75% saved.

    Example answer
    Long prompt: 800 tokens x 2,000,000 calls = 1,600,000,000 tokens (1,600 million). 1,600 x $0.50 = $800 per month.
Short prompt: 200 tokens x 2,000,000 calls = 400,000,000 tokens (400 million). 400 x $0.50 = $200 per month.
Saved: $800 - $200 = $600 per month.
Percentage saved: $600 / $800 = 75%.

    Your answer will be worded differently.

What just happened

The AI reads your text as tokens (small pieces of words). You pay for every token you send, on every call. The polite extra words in the long prompt did not change the answer, but they were extra tokens. With 2 million calls a month, 600 fewer tokens per call saves $600.

You are done when You have the same one-word answer from both prompts, and the numbers $800, $200, and 75%.

Check yourselfWhy does a shorter prompt save money when the answer stays the same?Show answer

APIs charge per token, and the model reads every input token on every call. A prompt with 75% fewer tokens costs 75% less for input, and it also has less to read, so it can answer faster.

Try a harder versionMake the short prompt even shorter without changing the answer. Then ask the AI: Does the same sentence use more tokens in Bangla than in English?

Watch · 3 videos

02

Model families & trade-offs

The frontier in 2025 is dominated by five families, each with a distinct architectural and commercial philosophy. Picking "the best model" is a misframe — the right question is which trade-off matches your workload's centre of gravity.

PRIVATE / PROPRIETARY OPEN-WEIGHT CHEAPER ↓ CAPABLE ↑ Anthropic Claude Opus / Sonnet / Haiku OpenAI GPT-4o / o1 / o3 / GPT-5 Google DeepMind Gemini 2.5 Pro / Flash Meta Llama 3.1 (8B–405B) Mistral AI Mistral / Mixtral (MoE) Alibaba Qwen3 (0.6B–235B-A22B) DeepSeek V3 / R1 (MoE + reasoning)
Frontier model families plotted by openness (x) and capability/cost tier (y). Position is qualitative, not benchmark-exact.
এই চার্টে বিভিন্ন frontier model family-কে PRIVATE / PROPRIETARY থেকে OPEN-WEIGHT (অনুভূমিক অক্ষ) এবং CHEAPER থেকে CAPABLE (উল্লম্ব অক্ষ) হিসেবে সাজানো হয়েছে। Anthropic, OpenAI, Google DeepMind, Meta, Mistral AI, Alibaba ও DeepSeek-এর অবস্থান গুণগত, benchmark-এর হুবহু মান নয়।

What each family is good at

FamilyFlagship model(s)Distinctive strengthBest fit for
OpenAIGPT-4o, o3, GPT-5Reasoning models, broad ecosystem, tooling maturity[13]General dev, rare-code agents, eval-driven pipelines
AnthropicClaude 4 Opus/SonnetLong context (200K), code agentic tasks, Constitutional AI safety[25]Rare-coding agents, legal/long-doc analysis
GoogleGemini 2.5 Pro/FlashMultimodal (native audio/video), 1–2M context[10][11][12]Long-context RAG, multimodal pipelines
MetaLlama 3.1 8B–405BOpen weights, permissive license, strong small models[2][3]Self-hosted infra, on-prem, fine-tuning
MistralMistral Large, Mistral SmallStrong open-weight small models, MoE efficiency[4][5]European data residency, edge deployment
AlibabaQwen3 (dense + MoE)Best-in-class small models, multilingual, Apache 2.0[8][9][28]Cheap self-host, Chinese/multilingual, agentic coding
DeepSeekDeepSeek-V3, R1Strong reasoning open-weight at low training cost, MLA architecture[6][7]Self-hosted reasoning, research, distillation

Architectural distinctions that matter

Dense transformers

Every parameter is used on every token. Predictable compute, simple serving. Examples: Llama 3.1 8B, Mistral 7B.[2][4]

Mixture-of-Experts (MoE)

Routes each token to a subset of "expert" FFNs. Total params large but active params per token small → cheaper inference, larger memory. Mixtral 8×7B, DeepSeek-V3 (671B/37B active), Qwen3-235B-A22B.[5][6][28]

Multi-Head Latent Attention (MLA)

DeepSeek's innovation: compresses the KV cache into a low-rank latent, slashing memory for long contexts.[6]

Sliding-window / hybrid

Mistral 7B pioneered sliding-window attention for cheap inference. Gemini 1.5 mixes local + global attention for million-token contexts.[4][11]

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Pick the right model for each job

Why this matters

When your team starts a new AI feature, you must choose a model. A wrong choice can cost thousands of dollars a month, or break a privacy rule.

  1. Step 1
    New chat

    Read the three jobs in the prompt and write down your own choice (X, Y, or Z) for each. Then copy the prompt, paste it, and press Enter.

    Prompt
    I must pick one model for each job. The three models are made up.

Model X: large, very strong at reasoning, slow, expensive per token, closed API.
Model Y: small, fast, cheap per token, good at simple tasks, closed API.
Model Z: medium, open weights, you run it on your own servers, you must pay for GPUs and operations.

Jobs:
1. A chat helper that sorts 5 million short support tickets per month into 4 categories.
2. A coding agent that fixes hard bugs across many files. Only 200 tasks per day.
3. A hospital in a country where patient data must never leave the building. Medium difficulty questions.

For each job, choose X, Y or Z. Give one sentence on the trade-off. Then say why "which model is the best" is the wrong question.

    You should see: A choice for each job with one sentence of reason. Most chats pick Y for job 1, X for job 2, and Z for job 3.

    Example answer
    1. Model Y. This is a simple job at a huge volume, so cheap and fast matters most.
2. Model X. Hard bugs across many files need the strongest reasoning, and 200 tasks a day is low volume, so the cost is fine.
3. Model Z. Patient data must stay in the building, so only a model on your own servers works.

"Which model is the best" is the wrong question because each model wins on a different trade-off: cost, quality, or data control.

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Compare the AI's choices with your own. Find one job where you chose differently, and decide who is right.

    You should see: Most jobs match. If one differs, the AI's reason shows a different trade-off, not always a wrong answer.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter.

    Prompt
    A Mixture-of-Experts model has 671B total parameters, but only 37B are active per token. A dense model has 70B parameters, and all of them are active on every token. Which model needs more GPU memory to load? Which model does less computing per token? Answer in 3 short sentences.

    You should see: MoE needs more memory (671B is stored), and MoE does less computing per token (37B is used).

    Example answer
    The Mixture-of-Experts model needs more GPU memory, because all 671B parameters must be loaded. The dense model needs less memory, only 70B. The Mixture-of-Experts model does less computing per token, because it uses only 37B parameters, while the dense model uses all 70B.

    Your answer will be worded differently.

What just happened

A model choice is a trade-off between quality, speed, cost, and data control. No model wins on all four, so "the best model" depends on the job. A MoE (mixture of experts) model is made of many expert parts, and each token uses only a few of them. All parts must sit in GPU memory, but only the active parts do work, so MoE needs more memory and less computing per token.

You are done when You have a model choice for each of the 3 jobs, and you know which of the two big models needs more memory.

Check yourselfWhy does a MoE model use more memory than a 70B dense model, but less computing per token?Show answer

All experts must be loaded in memory, so the total parameters (671B) decide the memory. Each token goes to only a few experts, so only the active parameters (37B) do the computing.

Try a harder versionChange job 1 to only 500 tickets per month. Ask the AI whether its choice changes, and why.

Watch · 3 videos

03

Closed vs open-weight models

"Open" splits into two camps that engineers often confuse: open-weight (you can download and run the model) and open-source (you also have training code, data, and licence to modify and redistribute). Most "open" LLMs are open-weight.

Closed / proprietary

Hosted behind APIs. Examples: GPT-4o/o3/GPT-5, Claude 4, Gemini 2.5. Advantages: highest capability ceiling, fastest hardware optimisation, integrated safety stack, no infra to run. Costs: vendor lock-in, no auditability of weights, data leaves your perimeter, price changes are unilateral.[14][13]

Open-weight

Weights released under a licence (often Apache 2.0, Llama Community License, or MIT). Examples: Llama 3.1 (8B–405B),[3] Mistral 7B & Mixtral 8×7B,[4][5] Qwen2.5/Qwen3,[8][9] DeepSeek-V3 / R1.[6][7] Advantages: self-host on your hardware, fine-tune on private data, audit weights, no per-token fees, no data egress. Costs: you operate the infra (GPUs, networking, monitoring), capability ceiling ~3–9 months behind the frontier, safety is your problem.

When open-weight wins: predictable cost at scale, data residency / compliance requirements, need to fine-tune on proprietary data, latency-sensitive serving where inference co-locates with state.

When closed wins: tasks where you're 3–9 months behind the SOTA matters (frontier coding agents, reasoning), small query volume makes self-hosting uneconomical, you need multimodal audio/video (Gemini).

Hybrid patterns becoming standard

  • Distillation: DeepSeek-R1's distilled Qwen/Llama variants prove you can train strong small open models from a closed teacher's outputs.[7]
  • Routing: route easy queries to a local 8B, hard ones to a frontier API. Saves cost without measurable quality loss on common tasks.
  • Open-weights with closed APIs: Llama 3.1 405B is open, and you can also call it via Together, Groq, Fireworks for managed serving — keeping the option to migrate.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Find when your own server pays off

Why this matters

When your team asks "should we run our own model?", the answer depends on how many tokens you use each month. A wrong guess can waste thousands of dollars.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Write down the break-even number (where both options cost the same).

    Prompt
    Compare two options for the same text task.

Option 1, closed API: $3 per 1 million tokens (input and output mixed). No other cost.
Option 2, open-weight model on our own server: $2,200 per month for the GPU server plus $1,500 per month for engineer time. It can serve up to 2 billion tokens per month. No per-token fee.

At how many tokens per month do both options cost the same? Show the formula and the steps. Which option is cheaper at 200 million tokens per month? Which at 1.8 billion?

    You should see: A break-even of about 1.23 billion tokens per month. The API is cheaper at 200 million, and the own server is cheaper at 1.8 billion.

    Example answer
    Own server cost = $2,200 + $1,500 = $3,700 per month, whatever the volume.
API cost = $3 per 1 million tokens.
Both cost the same when (tokens in millions) x $3 = $3,700, so tokens = 3,700 / 3 = 1,233.3 million, about 1.23 billion tokens per month.
At 200 million tokens: API = 200 x $3 = $600. Own server = $3,700. The API is cheaper.
At 1.8 billion tokens: API = 1,800 x $3 = $5,400. Own server = $3,700. The own server is cheaper.

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Check the maths yourself with a calculator: add $2,200 and $1,500, then divide by $3. Write down the result in millions of tokens.

    You should see: 3,700 / 3 = 1,233.3 million tokens. This matches the AI's number.

  3. Step 3
    New chat

    Read the four companies and write down your own choice for each. Then copy the prompt, paste it, and press Enter.

    Prompt
    For each company below, choose one: closed API, open-weight self-host, or hybrid (easy queries go to a small local model, hard queries go to a top API). Give one sentence of reason for each.

Company 1: A small start-up with 50,000 queries per month and no ML engineers.
Company 2: A bank that must keep all customer text inside its own data centre. It has a team that can run GPUs.
Company 3: A shopping app with 100 million queries per month. 80% are simple, and 20% need top quality.
Company 4: A team that must use the strongest coding agent available today.

    You should see: One choice and one reason per company. Most chats pick closed API for companies 1 and 4, open-weight for company 2, and hybrid for company 3.

    Example answer
    Company 1: Closed API. 50,000 queries is a small volume, and they have no engineers to run servers.
Company 2: Open-weight self-host. Customer text must stay in the data centre, and they can run GPUs.
Company 3: Hybrid. Send the 80% simple queries to a small local model and the 20% hard queries to a top API.
Company 4: Closed API. The strongest coding agents today are closed models.

    Your answer will be worded differently.

  4. Step 4
    Same chat

    Compare the AI's choices with yours. Pick one company where you disagree and type: Why not the other option?

    You should see: The AI gives a reason, often about volume, data rules, or staff. Decide if the reason is strong.

What just happened

A closed API charges per token, and there is no fixed cost. An open-weight model (a model whose files you can download and run yourself) has a fixed monthly cost for servers and people, whatever the volume. The break-even is the volume where both cost the same. Below it, the API is cheaper. Above it, your own server is cheaper.

You are done when You have a break-even number (about 1.23 billion tokens) and a choice for each of the 4 companies.

Check yourselfWhy can a closed API be cheaper for a small team, if open-weight models have no per-token fee?Show answer

With open weights you pay a fixed cost for servers and engineers, even at low volume. With few tokens, that fixed cost is higher than the API bill. Open-weight wins when the volume, the data rules, or the fine-tuning needs are high.

Try a harder versionRun the first prompt again, but with a server that costs $6,000 per month and serves up to 5 billion tokens. Keep the $1,500 for engineer time. Where is the new break-even, and is it inside the server limit?

Watch · 3 videos

04

Sampling parameters

A model outputs a probability distribution over every token in its vocabulary. Sampling is how you turn that distribution into text. The same logits can produce robotic lists, creative prose, or factual hallucinations depending on five knobs.

The knobs

logits → softmax → temperature → top-k → top-p → penalties → sample raw logits z softmax → probs temperature T→0: greedy / deterministic T=1.0: native distribution T→∞: uniform / noise truncation top_k keep only the k highest-prob tokens top_p keep smallest set whose sum ≥ k min_p drop tokens below p_min × top prob penalties freq −α · count(token) pres −α if token seen (any count)
Visualising how temperature flattens or sharpens the distribution, and where truncation steps (top_k, top_p, min_p) cut the tail.
এখানে দেখা যায় raw logits কীভাবে softmax, temperature, top_k, top_p, min_p ও repetition penalties পেরিয়ে শেষে sample হয়। বাঁ থেকে ডানে পড়ুন: T কম হলে বণ্টন sharp (greedy), T বেশি হলে flat (noise), আর truncation ধাপগুলো কম-সম্ভাবনার tail কেটে দেয়।
ParameterWhat it doesWhen to tune it
temperatureDivide logits by T before softmax. T<1 sharpens (more deterministic), T>1 flattens (more random).0 = code, JSON, classification, math.
0.7–1.0 = chat.
1.0–2.0 = creative writing, brainstorming.
top_kRestrict sampling to the k highest-prob tokens.k=1 = greedy. k≈40–100 typical. Avoid k<10 with high temperature.
top_p (nucleus)Keep the smallest set of tokens whose cumulative probability ≥ p.0.9–0.95 is the usual sweet spot. Often used with top_k.
min_pDrop tokens with prob < min_p × top_prob. Recent alternative to top_p.Scales better across model sizes than fixed top_p.
frequency_penaltySubtract α × (count of token) from logits each step.Discourages repetition. 0.1–1.0 typical.
presence_penaltySubtract α if token has appeared at all.Encourages topical diversity. 0–0.6 typical.
# OpenAI-compatible sampling params (works across providers)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    temperature=0.2,        # low for JSON extraction
    top_p=1.0,              # off when temperature low
    frequency_penalty=0.3,
    presence_penalty=0.0,
    max_tokens=512,
)

Don't combine top_k and top_p aggressively. Either pick top_k (fixed vocabulary cut) or top_p (probability mass cut). Using both with tight values double-strips the tail and can cause degeneration. Modern providers recommend temperature + top_p (or temperature + min_p), full stop.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

See sharp and flat word chances

Why this matters

When you build a chatbot, you must know when the AI is almost sure of its next word (a fact) and when many words fit (a story). This tells you when the answers will change from run to run.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Look at the top percentage.

    Prompt
    Estimate the next-word probabilities for this sentence start. List the 5 most likely next words, each with a percentage. The percentages must add up to about 100%. Reply with the list only.

The capital of France is

    You should see: One word, Paris, with almost all of the chance, for example 90% or more.

    Example answer
    Paris - 97%
Lyon - 1%
the - 1%
Marseille - 0.5%
a - 0.5%

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Copy this prompt, paste it, and press Enter. Compare this list with the first one.

    Prompt
    Estimate the next-word probabilities for this sentence start. List the 5 most likely next words, each with a percentage. The percentages must add up to about 100%. Reply with the list only.

Once upon a time, there was a little

    You should see: The chances are spread over several words, for example girl, boy, fox. No word has almost everything.

    Example answer
    girl - 35%
boy - 25%
fox - 15%
rabbit - 13%
princess - 12%

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. Write down the word.

    Prompt
    Complete this sentence with one word only. Reply with the word and nothing else.

Once upon a time, there was a little

    You should see: One word, for example girl.

    Example answer
    girl

    Your answer will be worded differently.

  4. Step 4
    New chat

    Do step 3 two more times, each time in a new chat. Write down all 3 words.

    You should see: Often 2 or 3 different words, for example girl, fox, boy.

What just happened

For each next word, the AI has a chance for many words. For "capital of France", one word takes almost all of the chance. For the story, many words have a similar chance. Sampling means picking the next word by these chances, so the story word changes from chat to chat. The percentages you saw are the AI's own guess, not its real inside numbers.

You are done when You have two percentage lists (one sharp, one flat) and 3 story words.

Check yourselfIf the AI always picked the single most likely word, what would change for the story sentence?Show answer

It would give the same word every time. This is called greedy decoding, and it is like setting temperature near 0. Sampling picks from all the words by chance, so a flat list gives different words, and a higher temperature makes the list flatter.

Try a harder versionAsk for the same list for the sentence start 2 + 2 equals. Is it sharper than the Paris list, or flatter than the story list? Why?

Watch · 3 videos

05

Streaming vs non-streaming

Generation is sequential — tokens come one at a time. Streaming returns them as they're produced via Server-Sent Events or chunked HTTP, so the UI can render the first word while the rest is still being decoded. Non-streaming waits for the full response.

time → non-streaming prefill (compute) waiting... TTFT = total time full text streaming prefill t1 t2 t3 t4 tokens stream back as decoded TTFT (time to first token) ≈ 100–500 ms · TPOT (time per output token) ≈ 30–80 ms Total wall time is identical; only the perceived latency differs.
Streaming shifts perceived latency from "all-or-nothing" to "incremental". Same compute, much better UX.
এই diagram-এ non-streaming ও streaming response-এর সময়রেখা তুলনা করা হয়েছে। Non-streaming-এ পুরো text একসাথে আসে, আর streaming-এ prefill-এর পর t1, t2, t3, t4 করে tokens ক্রমে আসে; মোট সময় একই, শুধু TTFT কমে বলে অনুভূত latency কম লাগে।

Trade-offs

Use streaming when

  • Any user-facing chat / completion UI
  • Long generations where the user needs feedback that the model is "thinking"
  • Tools like Cursor, Claude.ai, ChatGPT — every product is streaming by default now

Skip streaming when

  • You're feeding output back into a pipeline (function-call result, structured extractor)
  • You need the full response for JSON parsing / validation
  • Batch jobs with no UX
# Streaming with the OpenAI SDK
stream = client.chat.completions.create(
    model="gpt-4o", messages=messages,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

# Anthropic equivalent
with client.messages.stream(model=model, messages=msgs, max_tokens=1024) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Time the first word and the last

Why this matters

When you build a chat window, users leave if they stare at a blank screen. You need to know what streaming fixes, and when it breaks a step that needs the full answer.

  1. Step 1
    New chat

    Get a stopwatch (your phone is fine). Start it as you press Enter on this prompt. Write down the seconds until the first word, and until the last word.

    Prompt
    Write a story of about 400 words about a rickshaw driver in Dhaka who finds a lost umbrella during the monsoon.

    You should see: Most chats show the first word after 1 to 3 seconds. The whole story then needs many more seconds.

  2. Step 2
    New chat

    Start the stopwatch again. Copy this prompt, paste it, and press Enter. Write down the seconds until the answer is complete.

    Prompt
    Reply with only the word OK.

    You should see: The first word and the full answer appear almost together, in about 1 or 2 seconds.

    Example answer
    OK

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. After about 2 seconds, press the Stop button. Read the partial text. If there is no Stop button, read what is on screen at 2 seconds.

    Prompt
    Return only a JSON object, with no other text and no code fence. The object has these fields: id, name, email, city, plan, signup_date, last_login, tags (an array of 5 strings), and notes (a 2 sentence string). Use made-up values.

    You should see: Text that stops in the middle, with no closing } at the end. A program cannot read it as JSON yet.

    Example answer
    {
  "id": 1042,
  "name": "Amina Rahman",
  "email": "amina.rahman@example.com",
  "city": "Chattogram",
  "plan": "pro",
  "signup_date": "2024-0

    Your answer will be worded differently.

What just happened

Streaming means the AI sends the answer a few words at a time, so you see the first word early. It does not make the whole answer faster: the total time is the same. For a chat window this is great, because the user sees progress at once. For a program that needs the full JSON, a half answer is broken, so that step should wait for the complete answer.

You are done when You have a first-word time and a last-word time for the story, and you saw a half-written JSON that a program cannot use.

Check yourselfWhy is streaming good for a chat window, but often bad for a step in a data pipeline?Show answer

Streaming does not make the AI faster. It only shows the words as they arrive, so a person sees progress sooner. A pipeline step that must read or check JSON needs the complete answer, so it should wait for it.

Try a harder versionAsk for a 1,000 word story. Does the time to the first word change much, compared with the 400 word story? What does this tell you about how fast an app feels?

Watch · 3 videos

06

Structured outputs & JSON mode

Forcing free-form text into a typed schema unlocks LLMs as reliable components in larger pipelines. There are three progressively stricter approaches.

The three levels

strictness ↗ 1. JSON mode "respond with valid JSON" + simple regex / json.loads ↳ shape can drift; needs defensive parsing ~90% reliable 2. Schema-constrained decoding Pass JSON Schema; tokenizer masks invalid tokens at each step → guaranteed valid output OpenAI Structured Outputs, Anthropic tool-use schemas ~99% reliable 3. Typed parsing Parse the constrained output into typed objects in your code; Pydantic / Zod / Instructor / Outlines / Guidance libraries 100% by construction
JSON mode, schema-constrained decoding, and typed parsing — three guarantees, three reliability tiers.
এখানে structured output পাওয়ার তিনটি স্তর strictness বাড়ার ক্রমে দেখানো হয়েছে: JSON mode (~90% reliable), schema-constrained decoding (~99%, invalid tokens mask হয়) এবং typed parsing (Pydantic, Zod ইত্যাদি দিয়ে 100% by construction)। বাঁ থেকে ডানে গেলে নিশ্চয়তা বাড়ে।

Schema-constrained generation: how it actually works

The provider compiles the schema into a finite-state machine (or grammar) and, at each generation step, masks the logits of any token that would violate the schema. Output is guaranteed to be parseable by construction — no defensive try/except needed.[26]

# OpenAI Structured Outputs (Python)
from pydantic import BaseModel
from openai import OpenAI

class Person(BaseModel):
    name: str
    age: int
    role: str

client = OpenAI()
resp = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-09",
    messages=[{"role": "user", "content": "Extract: Ada Lovelace, 36, mathematician"}],
    response_format=Person,
)
person: Person = resp.choices[0].message.parsed
print(person.name, person.age)    # typed, valid by construction

Use structured outputs whenever you can. They turn LLM calls into well-typed function calls — no regex, no eval, no retry loops. The slight latency bump from FSM lookup is dwarfed by the saved post-validation.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Compare loose and strict JSON requests

Why this matters

When your code reads the AI's answer, one extra sentence or one wrong field name can crash the program. You must know how to ask for output that code can use.

  1. Step 1
    New chat

    Copy this loose prompt, paste it, and press Enter. Look at the answer: could a program read it as it is?

    Prompt
    Put the people from this note into JSON.

Note: Hi team, please welcome Rahim Uddin, who joined us last week as our new data engineer. He is thirty-four. Also, Sadia Akter (QA lead) starts on Monday. I do not have her age yet.

    You should see: Often an extra sentence before the JSON, a code fence, made-up field names, or the age as a word like thirty-four.

    Example answer
    Sure! Here is the JSON:

```json
{
  "people": [
    {"name": "Rahim Uddin", "job": "data engineer", "age": "thirty-four"},
    {"name": "Sadia Akter", "job": "QA lead", "age": "unknown"}
  ]
}
```

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this strict prompt, paste it, and press Enter. Look at the answer.

    Prompt
    Extract the people from the note. Reply with JSON only. No text before or after it. No markdown code fence.

The JSON must be an array. Each item must follow this schema:
{"name": string, "age": integer or null, "role": string}
Use a number for age, not words. Use null if the age is not in the note. Do not add any other fields.

Note: Hi team, please welcome Rahim Uddin, who joined us last week as our new data engineer. He is thirty-four. Also, Sadia Akter (QA lead) starts on Monday. I do not have her age yet.

    You should see: Only the JSON array. Rahim has age 34 as a number, and Sadia has age null.

    Example answer
    [
  {"name": "Rahim Uddin", "age": 34, "role": "data engineer"},
  {"name": "Sadia Akter", "age": null, "role": "QA lead"}
]

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Compare the two answers. For each one, check: only JSON, no extra sentence? Age is a number? Sadia's age is null?

    You should see: The strict answer passes most checks. The loose answer often fails one or more.

What just happened

A program needs an exact format. When you only say "JSON", the AI chooses the field names, the types, and the extra text. A schema (a description of the exact fields and their types) tells the AI what to return, and the answer becomes much cleaner. But asking does not guarantee it. Some AI providers can force the format, so a wrong answer is not even possible. This is called structured outputs.

You are done when You have two answers, and you checked that the strict one has age 34 as a number and age null for Sadia.

Check yourselfHow is forcing the format different from asking for JSON in a prompt?Show answer

A prompt only asks the AI to follow the format, so it can still fail. When the provider forces the format, it blocks every word that would break the schema, so the output always has the right shape.

Try a harder versionAdd a third person to the note, and give them two roles, for example designer and team lead. The schema allows one role. Ask the AI to follow the schema, and see how it handles this.

Watch · 3 videos

07

Function calling & tool use

Function calling lets the model emit a structured request to invoke a tool (a function, an API, a database query) instead of free-form prose. The model doesn't execute — it decides what to call and with what arguments, and your code runs the tool and feeds the result back. This is the basis of every LLM agent.

The loop

User prompt + tools schema LLM decides → emits JSON Your code parses, runs tool Tool result appended to context LLM final answer ↑ loop repeats if LLM calls another tool (Anthropic: max_iterations; OpenAI: 'auto' parallel)
Classic ReAct-style tool loop: model emits a tool-call JSON, your code runs it, the result is appended, repeat.
এই diagram-এ tool-calling loop দেখানো হয়েছে: User prompt ও tools schema থেকে LLM সিদ্ধান্ত নিয়ে JSON emit করে, আপনার code তা parse করে tool চালায়, আর Tool result context-এ যুক্ত হয়ে আবার LLM-এ যায়। LLM আরেকটি tool call করলে loop পুনরাবৃত্ত হয়, নইলে final answer আসে।

How it works under the hood

You supply a list of tools as JSON Schema (function name, parameters, description). The model is fine-tuned to emit a special tool_calls field instead of (or alongside) prose. Because the schema is constrained, the arguments are valid by construction. The leading paper is Toolformer (Meta, 2023),[18] with concurrent industry work on Gorilla (Berkeley, 2023) for massive API use,[17] and Anthropic's tool use docs.[27]

# Anthropic tool-use example
tools = [{
    "name": "get_weather",
    "description": "Get the current weather for a location.",
    "input_schema": {
        "type": "object",
        "properties": {"location": {"type": "string"}},
        "required": ["location"]
    }
}]

msg = client.messages.create(
    model="claude-sonnet-4-5", tools=tools,
    messages=[{"role": "user", "content": "Weather in Dhaka?"}],
)
if msg.stop_reason == "tool_use":
    tool = msg.content[1]                    # tool-use block
    result = get_weather(**tool.input)          # your code
    # feed result back, model produces final answer

Tool design matters as much as model choice. Tool names and descriptions are inserted verbatim into the prompt. "Retrieve flights" is ambiguous; search_flights(origin, destination, depart_date, cabin_class) with documented parameters in the description yields dramatically higher correct-call rates. Iterate on tool descriptions the same way you iterate on prompts.

Modern patterns

  • Parallel tool calls: OpenAI's tool_choice: "auto" can emit multiple independent calls per turn — the model decides what can run concurrently.
  • Chain-of-Abstraction: the model writes placeholders for tool results, then fills them in after parallel execution — saves wall time on multi-step agents.[19]
  • Computer use: Sonnet 4.5 and similar can drive a desktop via screenshot + mouse/keyboard primitives — the "ultimate tool" with the agent acting as the API.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Be the tool in a tool-calling loop

Why this matters

When you build an AI assistant that checks weather, orders, or bookings, the AI cannot do it alone. Your code must run the tool. Here you play that part by hand.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Look at the reply. It should be a tool call only, not an answer.

    Prompt
    You can use these tools. You cannot run them yourself. To use a tool, reply with ONLY a JSON object: {"tool": name, "arguments": {...}}. No other text.

Tools:
[
  {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}},
  {"name": "search_flights", "description": "Search flights between two airports.", "parameters": {"type": "object", "properties": {"origin": {"type": "string"}, "destination": {"type": "string"}, "depart_date": {"type": "string"}}, "required": ["origin", "destination", "depart_date"]}}
]

User question: Is it hot in Dhaka right now? Should I wear a jacket?

    You should see: Only a short JSON text like the example. It names the tool get_weather and the city Dhaka. It has no weather answer.

    Example answer
    {"tool": "get_weather", "arguments": {"location": "Dhaka", "unit": "celsius"}}

    Your answer will be worded differently.

  2. Step 2
    Same chat

    You are the tool now. Copy this fake result, paste it in the same chat, and press Enter.

    Prompt
    Tool result for your call:
{"location": "Dhaka", "temp_c": 38, "condition": "humid and sunny"}

Now answer the user question in two sentences, using only this result.

    You should see: A two-sentence answer that uses your fake 38 degrees, and says to leave the jacket at home.

    Example answer
    Yes, it is very hot in Dhaka right now: 38 degrees Celsius, humid and sunny. You do not need a jacket.

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Look at the first reply again. Write down the tool name the AI picked, and the city.

    You should see: The tool is get_weather, not search_flights, and the city is Dhaka.

What just happened

The AI did not look up the weather. It only wrote a tool call (a short JSON note that says which tool to run and with what inputs). In a real app, your code runs the tool and adds the result to the chat. You did that by hand, and the AI used your fake 38 degrees. The AI chose get_weather because the tool names and descriptions are part of its prompt.

You are done when You saw a tool call with no answer, and then a final answer that uses your fake 38 degrees.

Check yourselfIn function calling, who runs the tool: the AI or your code?Show answer

Your code runs the tool. The AI only decides which tool to call and with what inputs. Then it reads the result you add to the chat. This loop of call, run, add result, and repeat is the base of every AI agent.

Try a harder versionChange the get_weather description to just Get data, and run the first prompt in a new chat. Does the AI still pick the right tool? What does this tell you about tool descriptions?

Watch · 3 videos

08

Multimodal inputs (vision, audio, PDFs)

Modern frontier models ingest more than text: images, audio, video frames, and PDF pages become tokens the transformer can attend to. The unifying trick: any modality is converted into a sequence of embedding vectors and concatenated with the text sequence.

The encoder-per-modality architecture

different inputs → different encoders → unified token sequence image audio waveform PDF / doc page ViT / CLIP-style vision enc USM / Whisper-style audio enc OCR + page render + layout projection → token seq concat with text tokens LLM core joint attention
Each modality has a specialised encoder; a learned projection maps them into the LLM's token space, where a single transformer attends over all of them.
এখানে দেখা যায় image, audio waveform ও PDF / doc page প্রতিটি আলাদা encoder দিয়ে প্রক্রিয়া হয়। এরপর projection দিয়ে token sequence-এ রূপান্তরিত হয়ে text tokens-এর সাথে concat হয় এবং LLM core-এর joint attention সবগুলোকে একসাথে প্রক্রিয়া করে।

Capability comparison (2025)

ModelImageAudio in/outVideoPDFNotes
GPT-4o / GPT-5✓ native✓ (input via Whisper, output TTS)framesvia visionHigh-quality image reasoning[14]
Claude 4✓ nativeinput only (transcribe)framesvia visionBest on chart/diagram reading
Gemini 1.5 / 2.5✓ native✓ native in+out✓ native frames✓ nativeMost natively multimodal[10][11][12]
Qwen2.5-VL / Qwen3-VL✓ nativevia external ASRframesvia visionOpen-weight vision model
Llama 3.2 Vision✓ nativevia externalframesvia visionOpen-weight, smaller

Vision is not magic OCR. Frontier vision LLMs can count, read handwriting, reason about charts, and even interpret UI screenshots — but they still hallucinate on tiny text, fine numerics, and rotated pages. For high-precision OCR (invoices, IDs, scientific notation), use a dedicated OCR engine and feed the text to a text LLM.

PDFs are a special case

Frontier APIs accept PDFs directly (Claude, Gemini) and render each page as an image. The alternatives:
• Text-extract first: pymupdf, pdfplumber, unstructured.io — fast, cheap, but loses layout & visuals.
• Vision path: render pages → multimodal LLM — slower, expensive, but handles scanned docs, tables, charts.
• Hybrid: text-extract + page-image for tables/figures. This is what production systems ship.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Test an AI on sharp and tiny text

Why this matters

When you build an app that reads invoices, ID cards, or screenshots, you must know when the AI can read the image well, and when it only guesses.

  1. Step 1
    New chat

    Find the "Capability comparison" table in this section. Take a screenshot of only that table, at normal size. Use no private data. Your chat must allow image upload.

    You should see: A clear picture of the table on your computer. Nothing is sent yet.

  2. Step 2
    Same chat

    Attach the screenshot to the chat. Then copy this prompt, paste it, and press Enter.

    Prompt
    Look at the attached image. Copy the table into text, one row per line, with the exact words and symbols. After the copy, list every cell where you are not sure of the text.

    You should see: A text copy of the table: a header and 5 rows. Usually few or no mistakes, and a short list of unsure cells.

    Example answer
    Model | Image | Audio in/out | Video | PDF | Notes
GPT-4o / GPT-5 | native | input via Whisper, output TTS | frames | via vision | High-quality image reasoning
Claude 4 | native | input only (transcribe) | frames | via vision | Best on chart/diagram reading
Gemini 1.5 / 2.5 | native | native in+out | native frames | native | Most natively multimodal
...
Unsure cells: none.

    Your answer will be worded differently.

  3. Step 3
    New chat

    Zoom your browser out to about 30%. Take a new screenshot of the same table. Attach it, then paste the same prompt below and press Enter.

    Prompt
    Look at the attached image. Copy the table into text, one row per line, with the exact words and symbols. After the copy, list every cell where you are not sure of the text.

    You should see: More mistakes: missing rows, wrong words, or guessed cells. Many chats also list more unsure cells.

    Example answer
    Model | Image | Audio | Video | PDF | Notes
GPT-4o | native | Whisper | frames | vision | image reasoning
Claude 4 | native | input only | ? | via vision | chart reading
Gemini 2.5 | native | native | native | native | multimodal
...
Unsure cells: Claude 4 video, most of the Notes column, the last two rows.

    Your answer will be worded differently.

  4. Step 4
    Same chat

    Compare both copies with the real table. Count the wrong or missing cells in each copy. Write down both counts.

    You should see: A small count for the clear screenshot, and a bigger count for the tiny one. Counts differ between chats.

What just happened

The AI turns an image into many number lists (called embeddings) and reads them like text. In a tiny screenshot, the letters have too few pixels, so the number lists hold little detail, and the AI guesses. A guess often looks as sure as a correct reading. For exact text, such as invoices or IDs, use an OCR tool (software made only to read text from images).

You are done when You have two copies of the table, a count of mistakes for each, and the clear one has fewer mistakes.

Check yourselfWhy does the AI get worse when the text in the image is tiny?Show answer

The image becomes number lists that the AI reads like words. Tiny text has too little detail in those lists, so the AI guesses. For exact text such as invoices or IDs, a dedicated OCR tool is more reliable.

Try a harder versionTake a photo of a short handwritten note (or write one on paper). Send it with the same prompt and check which words the AI reads wrong.

Watch · 3 videos

09

Embedding models & reranker models

Two specialised model types that, together with an LLM, form the standard retrieval-augmented generation (RAG) stack. Embeddings compress text into a vector such that semantically similar items land near each other. Rerankers score (query, document) pairs with cross-attention for higher precision.

The two-stage retrieval pipeline

Corpus 10M docs bi-encoder embed(query), embed(doc) ANN search cosine / dot top-K ≈ 100 candidates cross-encoder reranker scores (q, d) jointly top-K ≈ 5–10 → LLM answer Stage 1: fast vector search over millions. Stage 2: expensive but accurate cross-attention over hundreds.
Two-stage retrieval: cheap dense retrieval narrows millions to a few hundred, then an expensive reranker picks the best few.
এই diagram-এ দুই ধাপের retrieval দেখানো হয়েছে: ১০M docs-এর corpus থেকে bi-encoder ও ANN search দ্রুত top-K ≈ 100 candidates আনে। তারপর cross-encoder reranker (q, d) জোড়া একসাথে score করে top-K ≈ 5–10 বেছে LLM answer-এ পাঠায়।

Embedding models (bi-encoders)

Encode each side (query and document) independently into a fixed-size vector. Similarity is a dot product or cosine — O(1) per pair, allowing billion-scale ANN search with FAISS, ScaNN, or HNSW. Modern examples: OpenAI text-embedding-3-large, Cohere embed-v3, BGE-M3, E5-Mistral-7B.[21]

# Modern embedding call (OpenAI v3)
from openai import OpenAI
client = OpenAI()
resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=["What is constitutional AI?", "harmlessness from AI feedback"],
    encoding_format="float",
)
vectors = [d.embedding for d in resp.data]   # 3072-d each
import numpy as np
sim = np.dot(vectors[0], vectors[1]) / (
    np.linalg.norm(vectors[0]) * np.linalg.norm(vectors[1]))

Reranker models (cross-encoders)

Rerankers concatenate query + document and run full transformer attention across them — much more accurate but O(n) per pair. Best used as the second stage on top-K=100 candidates.[22] Recent papers show LLM-based rerankers (even small open ones like Qwen2.5-7B) can match proprietary rerankers with listwise ranking prompts.

Bi-encoderCross-encoder reranker
SpeedEncode once, search billions in ms~50–200 ms per (q, d) pair
AccuracyGood recall; lower precisionBest in class precision
StorageStores vectors (KBs–MBs per doc)No storage; runs on text
UseStage 1: candidate generationStage 2: final ranking

The state-of-the-art 2025 stack: embed with E5-Mistral-7B-instruct or BGE-M3 for open weights, rerank with Qwen3-Reranker or Cohere Rerank v3, generate with your favourite LLM. Hybrid retrieval (sparse BM25 + dense) consistently beats either alone on BEIR.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Judge meaning, not matching words

Why this matters

When you build a search box or a support bot that finds answers in your documents, matching words is not enough. The AI must find the document that means the same thing as the question.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Look at the score for each pair.

    Prompt
    Rate how similar the meaning of each pair is, from 0 (unrelated) to 1 (same meaning). Reply with one line per pair: the pair number and the score only.

1. 'How do I reset my password?' / 'Steps to recover access to my account when I forgot my login secret'
2. 'The bank was flooded after heavy rain.' / 'I deposited money at the bank this morning.'
3. 'How do I reset my password?' / 'How do I reset my router?'
4. 'How do I reset my password?' / 'Best street food in Old Dhaka'

    You should see: Pair 1 gets a high score, even with few shared words. Pairs 2 and 3 get low scores, even with shared words. Pair 4 gets about 0.

    Example answer
    1. 0.92
2. 0.10
3. 0.30
4. 0.00

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this prompt, paste it, and press Enter. Look at which documents get the top scores.

    Prompt
    Query: Can I return a laptop after 40 days?

Documents:
D1: Laptops can be returned within 14 days of delivery if the box is unopened.
D2: Laptop deals: save 20% on laptops this week, with free delivery on every laptop.
D3: Standard items can be returned within 30 days. Electronics have a shorter window.
D4: How to pack a laptop for a return shipment: use the original box and two layers of tape.
D5: Every laptop has a 12 month warranty. Defective laptops can be repaired or replaced at any time in that period.

Score each document from 0 to 10 for how well it helps answer the query. Then list the documents from best to worst. Give one short reason for each.

    You should see: D1 and D3 get the top scores, because they answer the return question. D2 and D4 get low scores.

    Example answer
    D1: 9. Gives the return limit (14 days), so 40 days is too late.
D3: 8. Says electronics have a shorter window than 30 days.
D5: 4. Only helps if the laptop is defective.
D4: 3. About packing, not about the time limit.
D2: 0. An advert, not related to returns.
Best to worst: D1, D3, D5, D4, D2.

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Find the document that uses the word laptop the most (D2). Write down its score and say why it is low.

    You should see: D2 repeats laptop three times, but it scores 0 or close to 0. It is an advert that does not answer the question.

What just happened

Two texts can share many words and still mean different things, like bank of a river and bank for money. A search system first finds many possible documents with a fast method, using an embedding (a list of numbers that places similar meanings close together). Then a reranker (a second model that reads the question and one document together) picks the best ones. Your chat AI acted as both here, so its scores are rough guesses.

You are done when You have scores for 4 pairs and 5 documents, and you can say why D2 scored low.

Check yourselfWhy do search systems use an embedding step first, and a reranker second?Show answer

Embeddings are made once for each text, so the system can search millions of them very fast, but the match is less exact. A reranker reads the question and one document together, so it is more exact but slow. So it only re-scores the best few documents.

Try a harder versionAdd a document D6 that answers the question perfectly but uses different words: Items bought online may be sent back for a refund up to two weeks after arrival. Where does it rank?

Watch · 3 videos

10

Reasoning models & extended thinking

A new class of models trained to spend more compute at inference time on hard problems. Instead of answering immediately, they emit a chain of internal reasoning ("thinking tokens"), then produce a final answer. This is the most important capability shift since GPT-4.

Standard vs reasoning models

how the answer is produced Standard (GPT-4o, Claude, Gemini) user prompt answer one shot, direct Reasoning (o1/o3, DeepSeek-R1, Claude extended thinking) user prompt thinking tokens (hidden / returned) "Let's think step by step…" final answer
Reasoning models spend many more tokens on internal scratchpad, then answer.
এখানে Standard মডেল ও Reasoning মডেলের পার্থক্য দেখানো হয়েছে। Standard-এ user prompt থেকে সরাসরি one shot answer আসে; Reasoning মডেলে answer-এর আগে অনেক thinking tokens ("Let's think step by step…") খরচ হয়, তারপর final answer আসে।

The training trick: RL on reasoning chains

OpenAI's o1 paper showed that reinforcement learning on chain-of-thought traces dramatically improves math/physics/code reasoning — the model learns to think longer on hard problems.[13] DeepSeek-R1 replicated this with pure RL (no SFT), open-weights.[7] Anthropic's "extended thinking" exposes this as a togglable parameter.[28]

When extended thinking helps

TaskStandardReasoningNotes
Chat, summarisation, extraction✓ bestoverkillReasoning models waste tokens
Math, physics, formal proofspoor✓ bestLarge jumps from o1, R1
Hard multi-step coding agentsmixed✓ bestCursor / Devin ship reasoning
Planning & tool orchestrationmixed✓ bestCoT helps agent reliability
Simple Q&A, classification✓ bestoverkillUse small / fast model

Engineering trade-offs

Costs

Reasoning models can use 5–50× more tokens per query. A "thinking budget" cap is essential — set it to your latency/cost ceiling.

Latency

TTFT is often slower (longer prefill). Streaming the thinking tokens can help UX.

Distillation

DeepSeek-R1 distilled into Qwen 1.5B–32B gives small open models with reasoning capabilities.[7]

Process reward models

"Let's verify step by step" showed step-level reward models beat outcome-only supervision on math.[24]

# Anthropic extended thinking
resp = client.messages.create(
    model="claude-sonnet-4-5",
    thinking={"type": "enabled", "budget_tokens": 10000},
    messages=[{"role": "user", "content": "Prove sqrt(2) is irrational"}],
)
for block in resp.content:
    if block.type == "thinking":  print("…", block.thinking)
    if block.type == "text":      print(block.text)

Don't reach for reasoning models by reflex. They're 5–50× more expensive and slower. Use them when (1) the task is math/code/agentic, (2) standard models have plateaued on your benchmark, and (3) you can afford the latency. For most chat / extraction / RAG, a standard model + careful prompting wins.

The frontier as of Q3 2025

  • Reasoning models are the default for hard tasks: GPT-5, o3, DeepSeek-R1, Claude 4 with extended thinking.[13][7]
  • Open-weight models match closed on many benchmarks (Qwen3, Llama 3.1 405B, DeepSeek-V3).[9][3][6]
  • Multimodal is now table stakes: Gemini 2.5, GPT-4o, Claude 4, Qwen3-VL all native.[12]
  • Long context (1M+) is real but doesn't replace good retrieval.
  • Tool-use agents are the dominant deployment pattern.

Practice · 1 task

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Compare quick answers and step-by-step thinking

Why this matters

When you build an AI feature, thinking step by step can fix wrong answers on hard problems. But on easy questions it only adds cost and waiting time. You must know which is which.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. Write down the amount.

    Prompt
    A bakery bakes 8 trays with 25 buns each. 12% of all buns are burnt and thrown away. The rest are sold in full boxes of 6 buns at $4.50 per box. Buns that do not fill a box are given away free. How much money does the bakery earn? Reply with the final amount only. Do not show any working.

    You should see: A money amount only, with no working. Many modern chats get it right. A wrong amount is more likely when there is no working shown.

    Example answer
    $130.50

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this prompt, paste it, and press Enter. Compare the answer with step 1, and look at how long it is.

    Prompt
    A bakery bakes 8 trays with 25 buns each. 12% of all buns are burnt and thrown away. The rest are sold in full boxes of 6 buns at $4.50 per box. Buns that do not fill a box are given away free. How much money does the bakery earn? Think step by step. Then give the final line as: Answer: $amount

    You should see: A list of steps (200 buns, 24 burnt, 176 left, 29 full boxes), then Answer: $130.50. It is much longer than step 1.

    Example answer
    Step 1: Total buns = 8 x 25 = 200.
Step 2: Burnt buns = 12% of 200 = 24.
Step 3: Buns left = 200 - 24 = 176.
Step 4: Full boxes = 176 / 6 = 29 boxes (29 x 6 = 174), and 2 buns are left over and given away.
Step 5: Money = 29 x $4.50 = $130.50.
Answer: $130.50

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. Notice how much text comes back for such an easy question.

    Prompt
    Think step by step, then answer: What is the capital of Japan?

    You should see: Often several lines of steps for a one-word question, for example Tokyo. This is extra work that was not needed.

    Example answer
    Step 1: The question asks for the capital city of Japan.
Step 2: Japan's government and the Emperor are based in Tokyo.
Step 3: So the capital is Tokyo.
Answer: Tokyo

    Your answer will be worded differently.

  4. Step 4
    Same chat

    Now check the maths yourself with a calculator. Start with 8 x 25. Compare your final amount with the amounts in steps 1 and 2.

    You should see: Your answer is $130.50. If step 1 gave a different amount, the quick answer was wrong.

What just happened

In step 2 the AI wrote its thinking first, so each small step was easy and you could check it. This is the idea behind reasoning models (models that write many thinking words before the final answer). The thinking costs more words, so it costs more money and time. It helps on multi-step problems, but on "capital of Japan" it only wasted words.

You are done when You have two bakery answers and the correct amount, $130.50, from your calculator.

Check yourselfWhy do reasoning models use 5 to 50 times more tokens, and when is it worth it?Show answer

They write thinking tokens before the final answer, so they spend more compute on each question. This helps on hard maths, code, and planning. It adds cost and waiting time, so for simple chat or extraction a standard model is better.

Try a harder versionIf your chat has a Think or Reasoning mode, turn it on and send the step 1 prompt again in a new chat. Does the answer change? Does it take longer?

Watch · 3 videos

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.

  1. [1]Brown et al., "Language Models are Few-Shot Learners" — GPT-3 (NeurIPS 2020)
  2. [2]Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" — Meta AI, 2023
  3. [3]Dubey et al., "The Llama 3 Herd of Models" — Meta AI, 2024
  4. [4]Jiang et al., "Mistral 7B" — Mistral AI, 2023
  5. [5]Jiang et al., "Mixtral of Experts" — Mistral AI, 2024
  6. [6]DeepSeek-AI, "DeepSeek-V3 Technical Report" — 2024
  7. [7]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — 2025
  8. [8]Qwen Team, "Qwen2.5 Technical Report" — Alibaba, 2024
  9. [9]Qwen Team, "Qwen3 Technical Report" — Alibaba, 2025
  10. [10]Gemini Team, "Gemini: A Family of Highly Capable Multimodal Models" — Google DeepMind, 2023
  11. [11]Gemini Team, "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context" — 2024
  12. [12]Gemini Team, "Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities" — 2025
  13. [13]OpenAI, "OpenAI o1 System Card" — 2024
  14. [14]OpenAI, "GPT-4 Technical Report" — 2023
  15. [15]Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" — BPE paper, 2016
  16. [16]Kudo & Richardson, "SentencePiece: A simple and language independent subword tokenizer" — 2018
  17. [17]Patil et al., "Gorilla: Large Language Model Connected with Massive APIs" — Berkeley, 2023
  18. [18]Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" — Meta AI, 2023
  19. [19]Gao et al., "Efficient Tool Use with Chain-of-Abstraction Reasoning" — 2024
  20. [21]Wang et al., "Text Embeddings by Weakly-Supervised Contrastive Pre-training" (E5) — 2022
  21. [22]Santhanam et al., "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" — 2022
  22. [24]Lightman et al., "Let's Verify Step by Step" — process reward models, OpenAI, 2023
  23. [25]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
  24. [26]OpenAI — Structured Outputs documentation (JSON mode / schema-constrained decoding)
  25. [27]Anthropic — Tool use / function calling documentation
  26. [28]Qwen Team — Qwen3 blog announcement (May 2025)