LLM Fundamentals
from an engineer's lens
Learn how LLMs work in practice: tokens, model families, sampling, structured output, tool use, embeddings, reasoning, and how to choose the right model for production.
On this page
Section 1 of 10
Tokens, tokenization & context cost
Everything a model does — attention, generation, cost — is denominated in tokens, not characters or words. If your prompt shrinks from 800 to 200 tokens, you just cut cost by 75% and latency by a meaningful fraction. This is the single biggest engineering lever you have.
What is a token?
A token is one element in the model's vocabulary. Modern LLMs don't read characters; they read the output of a subword tokenizer — most commonly Byte-Pair Encoding (BPE), which iteratively merges the most frequent adjacent pairs in a corpus until a target vocabulary size (often 32k–200k) is reached.[15] For multilingual and code workloads, SentencePiece is the dominant alternative — it operates directly on raw Unicode bytes or whitespace, treating language as a configuration parameter.[16]
Why this matters: the cost equation
APIs charge per token, not per character. A useful mnemonic: ~1 token ≈ 4 characters in English for most modern tokenizers, but this varies wildly by language — Chinese runs ~1 char ≈ 1.5 tokens, code ~1 token ≈ 3.5 chars, numbers are split digit-by-digit in early tokenizers but consolidated in newer ones.
# Counting tokens with tiktoken (OpenAI) or transformers
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Tokenization is the silent tax on every LLM call."
print(len(enc.encode(text))) # → 11 tokens, not 8 words
# Concrete cost (input, gpt-4o-mini, Jul 2025 pricing)
tokens = 11
price_per_m = 0.15 # USD per 1M input tokens
cost_per_call = tokens / 1_000_000 * price_per_m # → $0.00000165Context windows — what they actually are
The context window is the maximum total tokens (input + output) the model can attend to in a single call. It has grown dramatically: GPT-3's 2K window in 2020 → 8K (GPT-3.5) → 128K (GPT-4o) → 1M+ (Gemini 1.5).[1][11]
Long context ≠ free performance. Even with 1M-token windows, retrieval-augmented generation, summarisation, or hierarchical attention beats dumping 800K tokens in. Needle-in-a-haystack benchmarks improve, but multi-hop reasoning across long contexts degrades sharply. Measure on your own workloads.
How tokens map to cost & latency
| Component | Cost driver | What dominates |
|---|---|---|
| Prefill (prompt processing) | Tokens in × input $/M | Quadratic attention compute; ~100 ms–2 s on modern APIs for 10K tokens |
| Decode (generation) | Tokens out × output $/M | Sequential (one per step); output tokens are 3–5× more expensive than input on most APIs |
| Caching | Cache hits vs misses | Prompt caching (Anthropic, OpenAI, Gemini) cuts repeated-prefix cost by ~80–90% |
| Long context | Quadratic attention | A 1M-token call isn't 125× a 8K call — closer to 5–8× due to attention scaling and chunked prefill |
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Cut the prompt, keep the answer
Why this matters
When a support bot reads millions of tickets a month, every extra word in the prompt is paid for again on every call. A shorter prompt can save real money.
Step 1
New chatCopy this long prompt, paste it, and press Enter. Write down the answer.
Prompt Hello! I hope you are having a wonderful day. I am working on a customer support system and I would really appreciate your kind help with a small task. Below you will find a customer ticket. Could you please read the ticket very carefully and then tell me which one of the following categories it belongs to? The categories are: billing, shipping, defect, account. Please choose only one category. When you answer, please do not write any explanation, any greeting, or any extra sentence. Just write the single category word. Thank you so much for your help! Ticket: My order #48213 arrived yesterday and the screen is cracked. I want my money back.
You should see: One word, most likely defect.
Example answer
defect
Your answer will be worded differently.
Step 2
New chatCopy this short prompt, paste it, and press Enter. Write down the answer.
Prompt Classify the ticket as one of: billing, shipping, defect, account. Reply with the category only. Ticket: My order #48213 arrived yesterday and the screen is cracked. I want my money back.
You should see: The same word, defect, from a prompt that is more than 3 times shorter.
Example answer
defect
Your answer will be worded differently.
Step 3
New chatCopy this prompt, paste it, and press Enter. Check the numbers with a calculator.
Prompt A support bot sends 800 input tokens per call. A shorter prompt sends 200 input tokens per call. The price is $0.50 per 1 million input tokens. The bot makes 2 million calls per month. Calculate the monthly input cost for each prompt, and the percentage saved. Show each step.
You should see: $800 per month for the long prompt, $200 for the short one, and 75% saved.
Example answer
Long prompt: 800 tokens x 2,000,000 calls = 1,600,000,000 tokens (1,600 million). 1,600 x $0.50 = $800 per month. Short prompt: 200 tokens x 2,000,000 calls = 400,000,000 tokens (400 million). 400 x $0.50 = $200 per month. Saved: $800 - $200 = $600 per month. Percentage saved: $600 / $800 = 75%.
Your answer will be worded differently.
What just happened
The AI reads your text as tokens (small pieces of words). You pay for every token you send, on every call. The polite extra words in the long prompt did not change the answer, but they were extra tokens. With 2 million calls a month, 600 fewer tokens per call saves $600.
You are done when You have the same one-word answer from both prompts, and the numbers $800, $200, and 75%.
Check yourselfWhy does a shorter prompt save money when the answer stays the same?Show answer
APIs charge per token, and the model reads every input token on every call. A prompt with 75% fewer tokens costs 75% less for input, and it also has less to read, so it can answer faster.
Try a harder version
Make the short prompt even shorter without changing the answer. Then ask the AI: Does the same sentence use more tokens in Bangla than in English?
Watch · 3 videos
Let's build the GPT Tokenizer Andrej Karpathy
Most devs don't understand how LLM tokens work Matt Pocock
Why AI Tokens are so Expensive - Computerphile Computerphile
Model families & trade-offs
The frontier in 2025 is dominated by five families, each with a distinct architectural and commercial philosophy. Picking "the best model" is a misframe — the right question is which trade-off matches your workload's centre of gravity.
What each family is good at
| Family | Flagship model(s) | Distinctive strength | Best fit for |
|---|---|---|---|
| OpenAI | GPT-4o, o3, GPT-5 | Reasoning models, broad ecosystem, tooling maturity[13] | General dev, rare-code agents, eval-driven pipelines |
| Anthropic | Claude 4 Opus/Sonnet | Long context (200K), code agentic tasks, Constitutional AI safety[25] | Rare-coding agents, legal/long-doc analysis |
| Gemini 2.5 Pro/Flash | Multimodal (native audio/video), 1–2M context[10][11][12] | Long-context RAG, multimodal pipelines | |
| Meta | Llama 3.1 8B–405B | Open weights, permissive license, strong small models[2][3] | Self-hosted infra, on-prem, fine-tuning |
| Mistral | Mistral Large, Mistral Small | Strong open-weight small models, MoE efficiency[4][5] | European data residency, edge deployment |
| Alibaba | Qwen3 (dense + MoE) | Best-in-class small models, multilingual, Apache 2.0[8][9][28] | Cheap self-host, Chinese/multilingual, agentic coding |
| DeepSeek | DeepSeek-V3, R1 | Strong reasoning open-weight at low training cost, MLA architecture[6][7] | Self-hosted reasoning, research, distillation |
Architectural distinctions that matter
Dense transformers
Every parameter is used on every token. Predictable compute, simple serving. Examples: Llama 3.1 8B, Mistral 7B.[2][4]
Mixture-of-Experts (MoE)
Routes each token to a subset of "expert" FFNs. Total params large but active params per token small → cheaper inference, larger memory. Mixtral 8×7B, DeepSeek-V3 (671B/37B active), Qwen3-235B-A22B.[5][6][28]
Multi-Head Latent Attention (MLA)
DeepSeek's innovation: compresses the KV cache into a low-rank latent, slashing memory for long contexts.[6]
Sliding-window / hybrid
Mistral 7B pioneered sliding-window attention for cheap inference. Gemini 1.5 mixes local + global attention for million-token contexts.[4][11]
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Pick the right model for each job
Why this matters
When your team starts a new AI feature, you must choose a model. A wrong choice can cost thousands of dollars a month, or break a privacy rule.
Step 1
New chatRead the three jobs in the prompt and write down your own choice (X, Y, or Z) for each. Then copy the prompt, paste it, and press Enter.
Prompt I must pick one model for each job. The three models are made up. Model X: large, very strong at reasoning, slow, expensive per token, closed API. Model Y: small, fast, cheap per token, good at simple tasks, closed API. Model Z: medium, open weights, you run it on your own servers, you must pay for GPUs and operations. Jobs: 1. A chat helper that sorts 5 million short support tickets per month into 4 categories. 2. A coding agent that fixes hard bugs across many files. Only 200 tasks per day. 3. A hospital in a country where patient data must never leave the building. Medium difficulty questions. For each job, choose X, Y or Z. Give one sentence on the trade-off. Then say why "which model is the best" is the wrong question.
You should see: A choice for each job with one sentence of reason. Most chats pick Y for job 1, X for job 2, and Z for job 3.
Example answer
1. Model Y. This is a simple job at a huge volume, so cheap and fast matters most. 2. Model X. Hard bugs across many files need the strongest reasoning, and 200 tasks a day is low volume, so the cost is fine. 3. Model Z. Patient data must stay in the building, so only a model on your own servers works. "Which model is the best" is the wrong question because each model wins on a different trade-off: cost, quality, or data control.
Your answer will be worded differently.
Step 2
Same chatCompare the AI's choices with your own. Find one job where you chose differently, and decide who is right.
You should see: Most jobs match. If one differs, the AI's reason shows a different trade-off, not always a wrong answer.
Step 3
New chatCopy this prompt, paste it, and press Enter.
Prompt A Mixture-of-Experts model has 671B total parameters, but only 37B are active per token. A dense model has 70B parameters, and all of them are active on every token. Which model needs more GPU memory to load? Which model does less computing per token? Answer in 3 short sentences.
You should see: MoE needs more memory (671B is stored), and MoE does less computing per token (37B is used).
Example answer
The Mixture-of-Experts model needs more GPU memory, because all 671B parameters must be loaded. The dense model needs less memory, only 70B. The Mixture-of-Experts model does less computing per token, because it uses only 37B parameters, while the dense model uses all 70B.
Your answer will be worded differently.
What just happened
A model choice is a trade-off between quality, speed, cost, and data control. No model wins on all four, so "the best model" depends on the job. A MoE (mixture of experts) model is made of many expert parts, and each token uses only a few of them. All parts must sit in GPU memory, but only the active parts do work, so MoE needs more memory and less computing per token.
You are done when You have a model choice for each of the 3 jobs, and you know which of the two big models needs more memory.
Check yourselfWhy does a MoE model use more memory than a 70B dense model, but less computing per token?Show answer
All experts must be loaded in memory, so the total parameters (671B) decide the memory. Each token goes to only a few experts, so only the active parameters (37B) do the computing.
Try a harder version
Change job 1 to only 500 tickets per month. Ask the AI whether its choice changes, and why.
Watch · 3 videos
How to Choose Large Language Models: A Developer’s Guide to LLMs IBM Technology
How to Choose the Right LLM for Your Project Marina Wyss - AI & Machine Learning
Small vs. Large AI Models: Trade-offs & Use Cases Explained IBM Technology
Closed vs open-weight models
"Open" splits into two camps that engineers often confuse: open-weight (you can download and run the model) and open-source (you also have training code, data, and licence to modify and redistribute). Most "open" LLMs are open-weight.
Closed / proprietary
Hosted behind APIs. Examples: GPT-4o/o3/GPT-5, Claude 4, Gemini 2.5. Advantages: highest capability ceiling, fastest hardware optimisation, integrated safety stack, no infra to run. Costs: vendor lock-in, no auditability of weights, data leaves your perimeter, price changes are unilateral.[14][13]
Open-weight
Weights released under a licence (often Apache 2.0, Llama Community License, or MIT). Examples: Llama 3.1 (8B–405B),[3] Mistral 7B & Mixtral 8×7B,[4][5] Qwen2.5/Qwen3,[8][9] DeepSeek-V3 / R1.[6][7] Advantages: self-host on your hardware, fine-tune on private data, audit weights, no per-token fees, no data egress. Costs: you operate the infra (GPUs, networking, monitoring), capability ceiling ~3–9 months behind the frontier, safety is your problem.
When open-weight wins: predictable cost at scale, data residency / compliance requirements, need to fine-tune on proprietary data, latency-sensitive serving where inference co-locates with state.
When closed wins: tasks where you're 3–9 months behind the SOTA matters (frontier coding agents, reasoning), small query volume makes self-hosting uneconomical, you need multimodal audio/video (Gemini).
Hybrid patterns becoming standard
- Distillation: DeepSeek-R1's distilled Qwen/Llama variants prove you can train strong small open models from a closed teacher's outputs.[7]
- Routing: route easy queries to a local 8B, hard ones to a frontier API. Saves cost without measurable quality loss on common tasks.
- Open-weights with closed APIs: Llama 3.1 405B is open, and you can also call it via Together, Groq, Fireworks for managed serving — keeping the option to migrate.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Find when your own server pays off
Why this matters
When your team asks "should we run our own model?", the answer depends on how many tokens you use each month. A wrong guess can waste thousands of dollars.
Step 1
New chatCopy this prompt, paste it, and press Enter. Write down the break-even number (where both options cost the same).
Prompt Compare two options for the same text task. Option 1, closed API: $3 per 1 million tokens (input and output mixed). No other cost. Option 2, open-weight model on our own server: $2,200 per month for the GPU server plus $1,500 per month for engineer time. It can serve up to 2 billion tokens per month. No per-token fee. At how many tokens per month do both options cost the same? Show the formula and the steps. Which option is cheaper at 200 million tokens per month? Which at 1.8 billion?
You should see: A break-even of about 1.23 billion tokens per month. The API is cheaper at 200 million, and the own server is cheaper at 1.8 billion.
Example answer
Own server cost = $2,200 + $1,500 = $3,700 per month, whatever the volume. API cost = $3 per 1 million tokens. Both cost the same when (tokens in millions) x $3 = $3,700, so tokens = 3,700 / 3 = 1,233.3 million, about 1.23 billion tokens per month. At 200 million tokens: API = 200 x $3 = $600. Own server = $3,700. The API is cheaper. At 1.8 billion tokens: API = 1,800 x $3 = $5,400. Own server = $3,700. The own server is cheaper.
Your answer will be worded differently.
Step 2
Same chatCheck the maths yourself with a calculator: add $2,200 and $1,500, then divide by $3. Write down the result in millions of tokens.
You should see: 3,700 / 3 = 1,233.3 million tokens. This matches the AI's number.
Step 3
New chatRead the four companies and write down your own choice for each. Then copy the prompt, paste it, and press Enter.
Prompt For each company below, choose one: closed API, open-weight self-host, or hybrid (easy queries go to a small local model, hard queries go to a top API). Give one sentence of reason for each. Company 1: A small start-up with 50,000 queries per month and no ML engineers. Company 2: A bank that must keep all customer text inside its own data centre. It has a team that can run GPUs. Company 3: A shopping app with 100 million queries per month. 80% are simple, and 20% need top quality. Company 4: A team that must use the strongest coding agent available today.
You should see: One choice and one reason per company. Most chats pick closed API for companies 1 and 4, open-weight for company 2, and hybrid for company 3.
Example answer
Company 1: Closed API. 50,000 queries is a small volume, and they have no engineers to run servers. Company 2: Open-weight self-host. Customer text must stay in the data centre, and they can run GPUs. Company 3: Hybrid. Send the 80% simple queries to a small local model and the 20% hard queries to a top API. Company 4: Closed API. The strongest coding agents today are closed models.
Your answer will be worded differently.
Step 4
Same chatCompare the AI's choices with yours. Pick one company where you disagree and type: Why not the other option?
You should see: The AI gives a reason, often about volume, data rules, or staff. Decide if the reason is strong.
What just happened
A closed API charges per token, and there is no fixed cost. An open-weight model (a model whose files you can download and run yourself) has a fixed monthly cost for servers and people, whatever the volume. The break-even is the volume where both cost the same. Below it, the API is cheaper. Above it, your own server is cheaper.
You are done when You have a break-even number (about 1.23 billion tokens) and a choice for each of the 4 companies.
Check yourselfWhy can a closed API be cheaper for a small team, if open-weight models have no per-token fee?Show answer
With open weights you pay a fixed cost for servers and engineers, even at low volume. With few tokens, that fixed cost is higher than the API bill. Open-weight wins when the volume, the data rules, or the fine-tuning needs are high.
Try a harder version
Run the first prompt again, but with a server that costs $6,000 per month and serves up to 5 billion tokens. Keep the $1,500 for engineer time. Where is the new break-even, and is it inside the server limit?
Watch · 3 videos
Sampling parameters
A model outputs a probability distribution over every token in its vocabulary. Sampling is how you turn that distribution into text. The same logits can produce robotic lists, creative prose, or factual hallucinations depending on five knobs.
The knobs
| Parameter | What it does | When to tune it |
|---|---|---|
temperature | Divide logits by T before softmax. T<1 sharpens (more deterministic), T>1 flattens (more random). | 0 = code, JSON, classification, math. 0.7–1.0 = chat. 1.0–2.0 = creative writing, brainstorming. |
top_k | Restrict sampling to the k highest-prob tokens. | k=1 = greedy. k≈40–100 typical. Avoid k<10 with high temperature. |
top_p (nucleus) | Keep the smallest set of tokens whose cumulative probability ≥ p. | 0.9–0.95 is the usual sweet spot. Often used with top_k. |
min_p | Drop tokens with prob < min_p × top_prob. Recent alternative to top_p. | Scales better across model sizes than fixed top_p. |
frequency_penalty | Subtract α × (count of token) from logits each step. | Discourages repetition. 0.1–1.0 typical. |
presence_penalty | Subtract α if token has appeared at all. | Encourages topical diversity. 0–0.6 typical. |
# OpenAI-compatible sampling params (works across providers)
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
temperature=0.2, # low for JSON extraction
top_p=1.0, # off when temperature low
frequency_penalty=0.3,
presence_penalty=0.0,
max_tokens=512,
)Don't combine top_k and top_p aggressively. Either pick top_k (fixed vocabulary cut) or top_p (probability mass cut). Using both with tight values double-strips the tail and can cause degeneration. Modern providers recommend temperature + top_p (or temperature + min_p), full stop.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
See sharp and flat word chances
Why this matters
When you build a chatbot, you must know when the AI is almost sure of its next word (a fact) and when many words fit (a story). This tells you when the answers will change from run to run.
Step 1
New chatCopy this prompt, paste it, and press Enter. Look at the top percentage.
Prompt Estimate the next-word probabilities for this sentence start. List the 5 most likely next words, each with a percentage. The percentages must add up to about 100%. Reply with the list only. The capital of France is
You should see: One word, Paris, with almost all of the chance, for example 90% or more.
Example answer
Paris - 97% Lyon - 1% the - 1% Marseille - 0.5% a - 0.5%
Your answer will be worded differently.
Step 2
Same chatCopy this prompt, paste it, and press Enter. Compare this list with the first one.
Prompt Estimate the next-word probabilities for this sentence start. List the 5 most likely next words, each with a percentage. The percentages must add up to about 100%. Reply with the list only. Once upon a time, there was a little
You should see: The chances are spread over several words, for example girl, boy, fox. No word has almost everything.
Example answer
girl - 35% boy - 25% fox - 15% rabbit - 13% princess - 12%
Your answer will be worded differently.
Step 3
New chatCopy this prompt, paste it, and press Enter. Write down the word.
Prompt Complete this sentence with one word only. Reply with the word and nothing else. Once upon a time, there was a little
You should see: One word, for example girl.
Example answer
girl
Your answer will be worded differently.
Step 4
New chatDo step 3 two more times, each time in a new chat. Write down all 3 words.
You should see: Often 2 or 3 different words, for example girl, fox, boy.
What just happened
For each next word, the AI has a chance for many words. For "capital of France", one word takes almost all of the chance. For the story, many words have a similar chance. Sampling means picking the next word by these chances, so the story word changes from chat to chat. The percentages you saw are the AI's own guess, not its real inside numbers.
You are done when You have two percentage lists (one sharp, one flat) and 3 story words.
Check yourselfIf the AI always picked the single most likely word, what would change for the story sentence?Show answer
It would give the same word every time. This is called greedy decoding, and it is like setting temperature near 0. Sampling picks from all the words by chance, so a flat list gives different words, and a higher temperature makes the list flatter.
Try a harder version
Ask for the same list for the sentence start 2 + 2 equals. Is it sharper than the Paris list, or flatter than the story list? Why?
Watch · 3 videos
Streaming vs non-streaming
Generation is sequential — tokens come one at a time. Streaming returns them as they're produced via Server-Sent Events or chunked HTTP, so the UI can render the first word while the rest is still being decoded. Non-streaming waits for the full response.
Trade-offs
Use streaming when
- Any user-facing chat / completion UI
- Long generations where the user needs feedback that the model is "thinking"
- Tools like Cursor, Claude.ai, ChatGPT — every product is streaming by default now
Skip streaming when
- You're feeding output back into a pipeline (function-call result, structured extractor)
- You need the full response for JSON parsing / validation
- Batch jobs with no UX
# Streaming with the OpenAI SDK
stream = client.chat.completions.create(
model="gpt-4o", messages=messages,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
# Anthropic equivalent
with client.messages.stream(model=model, messages=msgs, max_tokens=1024) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Time the first word and the last
Why this matters
When you build a chat window, users leave if they stare at a blank screen. You need to know what streaming fixes, and when it breaks a step that needs the full answer.
Step 1
New chatGet a stopwatch (your phone is fine). Start it as you press Enter on this prompt. Write down the seconds until the first word, and until the last word.
Prompt Write a story of about 400 words about a rickshaw driver in Dhaka who finds a lost umbrella during the monsoon.
You should see: Most chats show the first word after 1 to 3 seconds. The whole story then needs many more seconds.
Step 2
New chatStart the stopwatch again. Copy this prompt, paste it, and press Enter. Write down the seconds until the answer is complete.
Prompt Reply with only the word OK.
You should see: The first word and the full answer appear almost together, in about 1 or 2 seconds.
Example answer
OK
Your answer will be worded differently.
Step 3
New chatCopy this prompt, paste it, and press Enter. After about 2 seconds, press the Stop button. Read the partial text. If there is no Stop button, read what is on screen at 2 seconds.
Prompt Return only a JSON object, with no other text and no code fence. The object has these fields: id, name, email, city, plan, signup_date, last_login, tags (an array of 5 strings), and notes (a 2 sentence string). Use made-up values.
You should see: Text that stops in the middle, with no closing
}at the end. A program cannot read it as JSON yet.Example answer
{ "id": 1042, "name": "Amina Rahman", "email": "amina.rahman@example.com", "city": "Chattogram", "plan": "pro", "signup_date": "2024-0Your answer will be worded differently.
What just happened
Streaming means the AI sends the answer a few words at a time, so you see the first word early. It does not make the whole answer faster: the total time is the same. For a chat window this is great, because the user sees progress at once. For a program that needs the full JSON, a half answer is broken, so that step should wait for the complete answer.
You are done when You have a first-word time and a last-word time for the story, and you saw a half-written JSON that a program cannot use.
Check yourselfWhy is streaming good for a chat window, but often bad for a step in a data pipeline?Show answer
Streaming does not make the AI faster. It only shows the words as they arrive, so a person sees progress sooner. A pipeline step that must read or check JSON needs the complete answer, so it should wait for it.
Try a harder version
Ask for a 1,000 word story. Does the time to the first word change much, compared with the 400 word story? What does this tell you about how fast an app feels?
Watch · 3 videos
Streaming LLM Responses with FastAPI Irtiza Hafiz
SSE Deep Dive: How Server-Sent Events Stream LLM Responses #realtimecommunication Free World Coder
Server-Sent Events Crash Course Hussein Nasser
Structured outputs & JSON mode
Forcing free-form text into a typed schema unlocks LLMs as reliable components in larger pipelines. There are three progressively stricter approaches.
The three levels
Schema-constrained generation: how it actually works
The provider compiles the schema into a finite-state machine (or grammar) and, at each generation step, masks the logits of any token that would violate the schema. Output is guaranteed to be parseable by construction — no defensive try/except needed.[26]
# OpenAI Structured Outputs (Python)
from pydantic import BaseModel
from openai import OpenAI
class Person(BaseModel):
name: str
age: int
role: str
client = OpenAI()
resp = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-09",
messages=[{"role": "user", "content": "Extract: Ada Lovelace, 36, mathematician"}],
response_format=Person,
)
person: Person = resp.choices[0].message.parsed
print(person.name, person.age) # typed, valid by constructionUse structured outputs whenever you can. They turn LLM calls into well-typed function calls — no regex, no eval, no retry loops. The slight latency bump from FSM lookup is dwarfed by the saved post-validation.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Compare loose and strict JSON requests
Why this matters
When your code reads the AI's answer, one extra sentence or one wrong field name can crash the program. You must know how to ask for output that code can use.
Step 1
New chatCopy this loose prompt, paste it, and press Enter. Look at the answer: could a program read it as it is?
Prompt Put the people from this note into JSON. Note: Hi team, please welcome Rahim Uddin, who joined us last week as our new data engineer. He is thirty-four. Also, Sadia Akter (QA lead) starts on Monday. I do not have her age yet.
You should see: Often an extra sentence before the JSON, a code fence, made-up field names, or the age as a word like thirty-four.
Example answer
Sure! Here is the JSON: ```json { "people": [ {"name": "Rahim Uddin", "job": "data engineer", "age": "thirty-four"}, {"name": "Sadia Akter", "job": "QA lead", "age": "unknown"} ] } ```Your answer will be worded differently.
Step 2
New chatCopy this strict prompt, paste it, and press Enter. Look at the answer.
Prompt Extract the people from the note. Reply with JSON only. No text before or after it. No markdown code fence. The JSON must be an array. Each item must follow this schema: {"name": string, "age": integer or null, "role": string} Use a number for age, not words. Use null if the age is not in the note. Do not add any other fields. Note: Hi team, please welcome Rahim Uddin, who joined us last week as our new data engineer. He is thirty-four. Also, Sadia Akter (QA lead) starts on Monday. I do not have her age yet.You should see: Only the JSON array. Rahim has age 34 as a number, and Sadia has age
null.Example answer
[ {"name": "Rahim Uddin", "age": 34, "role": "data engineer"}, {"name": "Sadia Akter", "age": null, "role": "QA lead"} ]Your answer will be worded differently.
Step 3
Same chatCompare the two answers. For each one, check: only JSON, no extra sentence? Age is a number? Sadia's age is null?
You should see: The strict answer passes most checks. The loose answer often fails one or more.
What just happened
A program needs an exact format. When you only say "JSON", the AI chooses the field names, the types, and the extra text. A schema (a description of the exact fields and their types) tells the AI what to return, and the answer becomes much cleaner. But asking does not guarantee it. Some AI providers can force the format, so a wrong answer is not even possible. This is called structured outputs.
You are done when You have two answers, and you checked that the strict one has age 34 as a number and age null for Sadia.
Check yourselfHow is forcing the format different from asking for JSON in a prompt?Show answer
A prompt only asks the AI to follow the format, so it can still fail. When the provider forces the format, it blocks every word that would break the schema, so the output always has the right shape.
Try a harder version
Add a third person to the note, and give them two roles, for example designer and team lead. The schema allows one role. Ask the AI to follow the schema, and see how it handles this.
Watch · 3 videos
Function calling & tool use
Function calling lets the model emit a structured request to invoke a tool (a function, an API, a database query) instead of free-form prose. The model doesn't execute — it decides what to call and with what arguments, and your code runs the tool and feeds the result back. This is the basis of every LLM agent.
The loop
How it works under the hood
You supply a list of tools as JSON Schema (function name, parameters, description). The model is fine-tuned to emit a special tool_calls field instead of (or alongside) prose. Because the schema is constrained, the arguments are valid by construction. The leading paper is Toolformer (Meta, 2023),[18] with concurrent industry work on Gorilla (Berkeley, 2023) for massive API use,[17] and Anthropic's tool use docs.[27]
# Anthropic tool-use example
tools = [{
"name": "get_weather",
"description": "Get the current weather for a location.",
"input_schema": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"]
}
}]
msg = client.messages.create(
model="claude-sonnet-4-5", tools=tools,
messages=[{"role": "user", "content": "Weather in Dhaka?"}],
)
if msg.stop_reason == "tool_use":
tool = msg.content[1] # tool-use block
result = get_weather(**tool.input) # your code
# feed result back, model produces final answerTool design matters as much as model choice. Tool names and descriptions are inserted verbatim into the prompt. "Retrieve flights" is ambiguous; search_flights(origin, destination, depart_date, cabin_class) with documented parameters in the description yields dramatically higher correct-call rates. Iterate on tool descriptions the same way you iterate on prompts.
Modern patterns
- Parallel tool calls: OpenAI's
tool_choice: "auto"can emit multiple independent calls per turn — the model decides what can run concurrently. - Chain-of-Abstraction: the model writes placeholders for tool results, then fills them in after parallel execution — saves wall time on multi-step agents.[19]
- Computer use: Sonnet 4.5 and similar can drive a desktop via screenshot + mouse/keyboard primitives — the "ultimate tool" with the agent acting as the API.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Be the tool in a tool-calling loop
Why this matters
When you build an AI assistant that checks weather, orders, or bookings, the AI cannot do it alone. Your code must run the tool. Here you play that part by hand.
Step 1
New chatCopy this prompt, paste it, and press Enter. Look at the reply. It should be a tool call only, not an answer.
Prompt You can use these tools. You cannot run them yourself. To use a tool, reply with ONLY a JSON object: {"tool": name, "arguments": {...}}. No other text. Tools: [ {"name": "get_weather", "description": "Get the current weather for a city.", "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}, {"name": "search_flights", "description": "Search flights between two airports.", "parameters": {"type": "object", "properties": {"origin": {"type": "string"}, "destination": {"type": "string"}, "depart_date": {"type": "string"}}, "required": ["origin", "destination", "depart_date"]}} ] User question: Is it hot in Dhaka right now? Should I wear a jacket?You should see: Only a short JSON text like the example. It names the tool
get_weatherand the city Dhaka. It has no weather answer.Example answer
{"tool": "get_weather", "arguments": {"location": "Dhaka", "unit": "celsius"}}Your answer will be worded differently.
Step 2
Same chatYou are the tool now. Copy this fake result, paste it in the same chat, and press Enter.
Prompt Tool result for your call: {"location": "Dhaka", "temp_c": 38, "condition": "humid and sunny"} Now answer the user question in two sentences, using only this result.You should see: A two-sentence answer that uses your fake 38 degrees, and says to leave the jacket at home.
Example answer
Yes, it is very hot in Dhaka right now: 38 degrees Celsius, humid and sunny. You do not need a jacket.
Your answer will be worded differently.
Step 3
Same chatLook at the first reply again. Write down the tool name the AI picked, and the city.
You should see: The tool is
get_weather, notsearch_flights, and the city is Dhaka.
What just happened
The AI did not look up the weather. It only wrote a tool call (a short JSON note that says which tool to run and with what inputs). In a real app, your code runs the tool and adds the result to the chat. You did that by hand, and the AI used your fake 38 degrees. The AI chose get_weather because the tool names and descriptions are part of its prompt.
You are done when You saw a tool call with no answer, and then a final answer that uses your fake 38 degrees.
Check yourselfIn function calling, who runs the tool: the AI or your code?Show answer
Your code runs the tool. The AI only decides which tool to call and with what inputs. Then it reads the result you add to the chat. This loop of call, run, add result, and repeat is the base of every AI agent.
Try a harder version
Change the get_weather description to just Get data, and run the first prompt in a new chat. Does the AI still pick the right tool? What does this tell you about tool descriptions?
Watch · 3 videos
Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI AI Engineer
What is Tool Calling? Connecting LLMs to Your Data IBM Technology
How does OpenAI Function Calling work? Learn Data with Mark
Multimodal inputs (vision, audio, PDFs)
Modern frontier models ingest more than text: images, audio, video frames, and PDF pages become tokens the transformer can attend to. The unifying trick: any modality is converted into a sequence of embedding vectors and concatenated with the text sequence.
The encoder-per-modality architecture
Capability comparison (2025)
| Model | Image | Audio in/out | Video | Notes | |
|---|---|---|---|---|---|
| GPT-4o / GPT-5 | ✓ native | ✓ (input via Whisper, output TTS) | frames | via vision | High-quality image reasoning[14] |
| Claude 4 | ✓ native | input only (transcribe) | frames | via vision | Best on chart/diagram reading |
| Gemini 1.5 / 2.5 | ✓ native | ✓ native in+out | ✓ native frames | ✓ native | Most natively multimodal[10][11][12] |
| Qwen2.5-VL / Qwen3-VL | ✓ native | via external ASR | frames | via vision | Open-weight vision model |
| Llama 3.2 Vision | ✓ native | via external | frames | via vision | Open-weight, smaller |
Vision is not magic OCR. Frontier vision LLMs can count, read handwriting, reason about charts, and even interpret UI screenshots — but they still hallucinate on tiny text, fine numerics, and rotated pages. For high-precision OCR (invoices, IDs, scientific notation), use a dedicated OCR engine and feed the text to a text LLM.
PDFs are a special case
Frontier APIs accept PDFs directly (Claude, Gemini) and render each page as an image. The alternatives:
• Text-extract first: pymupdf, pdfplumber, unstructured.io — fast, cheap, but loses layout & visuals.
• Vision path: render pages → multimodal LLM — slower, expensive, but handles scanned docs, tables, charts.
• Hybrid: text-extract + page-image for tables/figures. This is what production systems ship.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Test an AI on sharp and tiny text
Why this matters
When you build an app that reads invoices, ID cards, or screenshots, you must know when the AI can read the image well, and when it only guesses.
Step 1
New chatFind the "Capability comparison" table in this section. Take a screenshot of only that table, at normal size. Use no private data. Your chat must allow image upload.
You should see: A clear picture of the table on your computer. Nothing is sent yet.
Step 2
Same chatAttach the screenshot to the chat. Then copy this prompt, paste it, and press Enter.
Prompt Look at the attached image. Copy the table into text, one row per line, with the exact words and symbols. After the copy, list every cell where you are not sure of the text.
You should see: A text copy of the table: a header and 5 rows. Usually few or no mistakes, and a short list of unsure cells.
Example answer
Model | Image | Audio in/out | Video | PDF | Notes GPT-4o / GPT-5 | native | input via Whisper, output TTS | frames | via vision | High-quality image reasoning Claude 4 | native | input only (transcribe) | frames | via vision | Best on chart/diagram reading Gemini 1.5 / 2.5 | native | native in+out | native frames | native | Most natively multimodal ... Unsure cells: none.
Your answer will be worded differently.
Step 3
New chatZoom your browser out to about 30%. Take a new screenshot of the same table. Attach it, then paste the same prompt below and press Enter.
Prompt Look at the attached image. Copy the table into text, one row per line, with the exact words and symbols. After the copy, list every cell where you are not sure of the text.
You should see: More mistakes: missing rows, wrong words, or guessed cells. Many chats also list more unsure cells.
Example answer
Model | Image | Audio | Video | PDF | Notes GPT-4o | native | Whisper | frames | vision | image reasoning Claude 4 | native | input only | ? | via vision | chart reading Gemini 2.5 | native | native | native | native | multimodal ... Unsure cells: Claude 4 video, most of the Notes column, the last two rows.
Your answer will be worded differently.
Step 4
Same chatCompare both copies with the real table. Count the wrong or missing cells in each copy. Write down both counts.
You should see: A small count for the clear screenshot, and a bigger count for the tiny one. Counts differ between chats.
What just happened
The AI turns an image into many number lists (called embeddings) and reads them like text. In a tiny screenshot, the letters have too few pixels, so the number lists hold little detail, and the AI guesses. A guess often looks as sure as a correct reading. For exact text, such as invoices or IDs, use an OCR tool (software made only to read text from images).
You are done when You have two copies of the table, a count of mistakes for each, and the clear one has fewer mistakes.
Check yourselfWhy does the AI get worse when the text in the image is tiny?Show answer
The image becomes number lists that the AI reads like words. Tiny text has too little detail in those lists, so the AI guesses. For exact text such as invoices or IDs, a dedicated OCR tool is more reliable.
Try a harder version
Take a photo of a short handwritten note (or write one on paper). Send it with the same prompt and check which words the AI reads wrong.
Watch · 3 videos
Embedding models & reranker models
Two specialised model types that, together with an LLM, form the standard retrieval-augmented generation (RAG) stack. Embeddings compress text into a vector such that semantically similar items land near each other. Rerankers score (query, document) pairs with cross-attention for higher precision.
The two-stage retrieval pipeline
Embedding models (bi-encoders)
Encode each side (query and document) independently into a fixed-size vector. Similarity is a dot product or cosine — O(1) per pair, allowing billion-scale ANN search with FAISS, ScaNN, or HNSW. Modern examples: OpenAI text-embedding-3-large, Cohere embed-v3, BGE-M3, E5-Mistral-7B.[21]
# Modern embedding call (OpenAI v3)
from openai import OpenAI
client = OpenAI()
resp = client.embeddings.create(
model="text-embedding-3-large",
input=["What is constitutional AI?", "harmlessness from AI feedback"],
encoding_format="float",
)
vectors = [d.embedding for d in resp.data] # 3072-d each
import numpy as np
sim = np.dot(vectors[0], vectors[1]) / (
np.linalg.norm(vectors[0]) * np.linalg.norm(vectors[1]))Reranker models (cross-encoders)
Rerankers concatenate query + document and run full transformer attention across them — much more accurate but O(n) per pair. Best used as the second stage on top-K=100 candidates.[22] Recent papers show LLM-based rerankers (even small open ones like Qwen2.5-7B) can match proprietary rerankers with listwise ranking prompts.
| Bi-encoder | Cross-encoder reranker | |
|---|---|---|
| Speed | Encode once, search billions in ms | ~50–200 ms per (q, d) pair |
| Accuracy | Good recall; lower precision | Best in class precision |
| Storage | Stores vectors (KBs–MBs per doc) | No storage; runs on text |
| Use | Stage 1: candidate generation | Stage 2: final ranking |
The state-of-the-art 2025 stack: embed with E5-Mistral-7B-instruct or BGE-M3 for open weights, rerank with Qwen3-Reranker or Cohere Rerank v3, generate with your favourite LLM. Hybrid retrieval (sparse BM25 + dense) consistently beats either alone on BEIR.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Judge meaning, not matching words
Why this matters
When you build a search box or a support bot that finds answers in your documents, matching words is not enough. The AI must find the document that means the same thing as the question.
Step 1
New chatCopy this prompt, paste it, and press Enter. Look at the score for each pair.
Prompt Rate how similar the meaning of each pair is, from 0 (unrelated) to 1 (same meaning). Reply with one line per pair: the pair number and the score only. 1. 'How do I reset my password?' / 'Steps to recover access to my account when I forgot my login secret' 2. 'The bank was flooded after heavy rain.' / 'I deposited money at the bank this morning.' 3. 'How do I reset my password?' / 'How do I reset my router?' 4. 'How do I reset my password?' / 'Best street food in Old Dhaka'
You should see: Pair 1 gets a high score, even with few shared words. Pairs 2 and 3 get low scores, even with shared words. Pair 4 gets about 0.
Example answer
1. 0.92 2. 0.10 3. 0.30 4. 0.00
Your answer will be worded differently.
Step 2
New chatCopy this prompt, paste it, and press Enter. Look at which documents get the top scores.
Prompt Query: Can I return a laptop after 40 days? Documents: D1: Laptops can be returned within 14 days of delivery if the box is unopened. D2: Laptop deals: save 20% on laptops this week, with free delivery on every laptop. D3: Standard items can be returned within 30 days. Electronics have a shorter window. D4: How to pack a laptop for a return shipment: use the original box and two layers of tape. D5: Every laptop has a 12 month warranty. Defective laptops can be repaired or replaced at any time in that period. Score each document from 0 to 10 for how well it helps answer the query. Then list the documents from best to worst. Give one short reason for each.
You should see: D1 and D3 get the top scores, because they answer the return question. D2 and D4 get low scores.
Example answer
D1: 9. Gives the return limit (14 days), so 40 days is too late. D3: 8. Says electronics have a shorter window than 30 days. D5: 4. Only helps if the laptop is defective. D4: 3. About packing, not about the time limit. D2: 0. An advert, not related to returns. Best to worst: D1, D3, D5, D4, D2.
Your answer will be worded differently.
Step 3
Same chatFind the document that uses the word laptop the most (D2). Write down its score and say why it is low.
You should see: D2 repeats laptop three times, but it scores 0 or close to 0. It is an advert that does not answer the question.
What just happened
Two texts can share many words and still mean different things, like bank of a river and bank for money. A search system first finds many possible documents with a fast method, using an embedding (a list of numbers that places similar meanings close together). Then a reranker (a second model that reads the question and one document together) picks the best ones. Your chat AI acted as both here, so its scores are rough guesses.
You are done when You have scores for 4 pairs and 5 documents, and you can say why D2 scored low.
Check yourselfWhy do search systems use an embedding step first, and a reranker second?Show answer
Embeddings are made once for each text, so the system can search millions of them very fast, but the match is less exact. A reranker reads the question and one document together, so it is more exact but slow. So it only re-scores the best few documents.
Try a harder version
Add a document D6 that answers the question perfectly but uses different words: Items bought online may be sent back for a refund up to two weeks after arrival. Where does it rank?
Watch · 3 videos
Reasoning models & extended thinking
A new class of models trained to spend more compute at inference time on hard problems. Instead of answering immediately, they emit a chain of internal reasoning ("thinking tokens"), then produce a final answer. This is the most important capability shift since GPT-4.
Standard vs reasoning models
The training trick: RL on reasoning chains
OpenAI's o1 paper showed that reinforcement learning on chain-of-thought traces dramatically improves math/physics/code reasoning — the model learns to think longer on hard problems.[13] DeepSeek-R1 replicated this with pure RL (no SFT), open-weights.[7] Anthropic's "extended thinking" exposes this as a togglable parameter.[28]
When extended thinking helps
| Task | Standard | Reasoning | Notes |
|---|---|---|---|
| Chat, summarisation, extraction | ✓ best | overkill | Reasoning models waste tokens |
| Math, physics, formal proofs | poor | ✓ best | Large jumps from o1, R1 |
| Hard multi-step coding agents | mixed | ✓ best | Cursor / Devin ship reasoning |
| Planning & tool orchestration | mixed | ✓ best | CoT helps agent reliability |
| Simple Q&A, classification | ✓ best | overkill | Use small / fast model |
Engineering trade-offs
Costs
Reasoning models can use 5–50× more tokens per query. A "thinking budget" cap is essential — set it to your latency/cost ceiling.
Latency
TTFT is often slower (longer prefill). Streaming the thinking tokens can help UX.
Distillation
DeepSeek-R1 distilled into Qwen 1.5B–32B gives small open models with reasoning capabilities.[7]
Process reward models
"Let's verify step by step" showed step-level reward models beat outcome-only supervision on math.[24]
# Anthropic extended thinking
resp = client.messages.create(
model="claude-sonnet-4-5",
thinking={"type": "enabled", "budget_tokens": 10000},
messages=[{"role": "user", "content": "Prove sqrt(2) is irrational"}],
)
for block in resp.content:
if block.type == "thinking": print("…", block.thinking)
if block.type == "text": print(block.text)Don't reach for reasoning models by reflex. They're 5–50× more expensive and slower. Use them when (1) the task is math/code/agentic, (2) standard models have plateaued on your benchmark, and (3) you can afford the latency. For most chat / extraction / RAG, a standard model + careful prompting wins.
The frontier as of Q3 2025
- Reasoning models are the default for hard tasks: GPT-5, o3, DeepSeek-R1, Claude 4 with extended thinking.[13][7]
- Open-weight models match closed on many benchmarks (Qwen3, Llama 3.1 405B, DeepSeek-V3).[9][3][6]
- Multimodal is now table stakes: Gemini 2.5, GPT-4o, Claude 4, Qwen3-VL all native.[12]
- Long context (1M+) is real but doesn't replace good retrieval.
- Tool-use agents are the dominant deployment pattern.
Practice · 1 task
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Compare quick answers and step-by-step thinking
Why this matters
When you build an AI feature, thinking step by step can fix wrong answers on hard problems. But on easy questions it only adds cost and waiting time. You must know which is which.
Step 1
New chatCopy this prompt, paste it, and press Enter. Write down the amount.
Prompt A bakery bakes 8 trays with 25 buns each. 12% of all buns are burnt and thrown away. The rest are sold in full boxes of 6 buns at $4.50 per box. Buns that do not fill a box are given away free. How much money does the bakery earn? Reply with the final amount only. Do not show any working.
You should see: A money amount only, with no working. Many modern chats get it right. A wrong amount is more likely when there is no working shown.
Example answer
$130.50
Your answer will be worded differently.
Step 2
New chatCopy this prompt, paste it, and press Enter. Compare the answer with step 1, and look at how long it is.
Prompt A bakery bakes 8 trays with 25 buns each. 12% of all buns are burnt and thrown away. The rest are sold in full boxes of 6 buns at $4.50 per box. Buns that do not fill a box are given away free. How much money does the bakery earn? Think step by step. Then give the final line as: Answer: $amount
You should see: A list of steps (200 buns, 24 burnt, 176 left, 29 full boxes), then Answer: $130.50. It is much longer than step 1.
Example answer
Step 1: Total buns = 8 x 25 = 200. Step 2: Burnt buns = 12% of 200 = 24. Step 3: Buns left = 200 - 24 = 176. Step 4: Full boxes = 176 / 6 = 29 boxes (29 x 6 = 174), and 2 buns are left over and given away. Step 5: Money = 29 x $4.50 = $130.50. Answer: $130.50
Your answer will be worded differently.
Step 3
New chatCopy this prompt, paste it, and press Enter. Notice how much text comes back for such an easy question.
Prompt Think step by step, then answer: What is the capital of Japan?
You should see: Often several lines of steps for a one-word question, for example Tokyo. This is extra work that was not needed.
Example answer
Step 1: The question asks for the capital city of Japan. Step 2: Japan's government and the Emperor are based in Tokyo. Step 3: So the capital is Tokyo. Answer: Tokyo
Your answer will be worded differently.
Step 4
Same chatNow check the maths yourself with a calculator. Start with 8 x 25. Compare your final amount with the amounts in steps 1 and 2.
You should see: Your answer is $130.50. If step 1 gave a different amount, the quick answer was wrong.
What just happened
In step 2 the AI wrote its thinking first, so each small step was easy and you could check it. This is the idea behind reasoning models (models that write many thinking words before the final answer). The thinking costs more words, so it costs more money and time. It helps on multi-step problems, but on "capital of Japan" it only wasted words.
You are done when You have two bakery answers and the correct amount, $130.50, from your calculator.
Check yourselfWhy do reasoning models use 5 to 50 times more tokens, and when is it worth it?Show answer
They write thinking tokens before the final answer, so they spend more compute on each question. This helps on hard maths, code, and planning. It adds cost and waiting time, so for simple chat or extraction a standard model is better.
Try a harder version
If your chat has a Think or Reasoning mode, turn it on and send the step 1 prompt again in a new chat. Does the answer change? Does it take longer?
Watch · 3 videos
How do thinking and reasoning models work? Google for Developers
What Are Large Reasoning Models (LRMs)? Smarter AI Beyond LLMs IBM Technology
Deep Dive into LLMs like ChatGPT Andrej Karpathy
Quiz
Check what you learned. 5 questions.
1. A prompt shrinks from 800 to 200 tokens. What does this Part say this does?
2. In function calling, the model does not run the tool itself. Your code runs the tool and feeds the result back.
3. Which temperature does this Part give for code, JSON, classification and math?
4. For which task does this Part say a reasoning model is the best choice?
5. This Part says that with a 1M-token window, it is better to put 800K tokens in the prompt than to use retrieval or summarisation.
Answer every question to check.
References
Sources
All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.
- [1]Brown et al., "Language Models are Few-Shot Learners" — GPT-3 (NeurIPS 2020)
- [2]Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" — Meta AI, 2023
- [3]Dubey et al., "The Llama 3 Herd of Models" — Meta AI, 2024
- [4]Jiang et al., "Mistral 7B" — Mistral AI, 2023
- [5]Jiang et al., "Mixtral of Experts" — Mistral AI, 2024
- [6]DeepSeek-AI, "DeepSeek-V3 Technical Report" — 2024
- [7]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — 2025
- [8]Qwen Team, "Qwen2.5 Technical Report" — Alibaba, 2024
- [9]Qwen Team, "Qwen3 Technical Report" — Alibaba, 2025
- [10]Gemini Team, "Gemini: A Family of Highly Capable Multimodal Models" — Google DeepMind, 2023
- [11]Gemini Team, "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context" — 2024
- [12]Gemini Team, "Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities" — 2025
- [13]OpenAI, "OpenAI o1 System Card" — 2024
- [14]OpenAI, "GPT-4 Technical Report" — 2023
- [15]Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" — BPE paper, 2016
- [16]Kudo & Richardson, "SentencePiece: A simple and language independent subword tokenizer" — 2018
- [17]Patil et al., "Gorilla: Large Language Model Connected with Massive APIs" — Berkeley, 2023
- [18]Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" — Meta AI, 2023
- [19]Gao et al., "Efficient Tool Use with Chain-of-Abstraction Reasoning" — 2024
- [21]Wang et al., "Text Embeddings by Weakly-Supervised Contrastive Pre-training" (E5) — 2022
- [22]Santhanam et al., "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" — 2022
- [24]Lightman et al., "Let's Verify Step by Step" — process reward models, OpenAI, 2023
- [25]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
- [26]OpenAI — Structured Outputs documentation (JSON mode / schema-constrained decoding)
- [27]Anthropic — Tool use / function calling documentation
- [28]Qwen Team — Qwen3 blog announcement (May 2025)