Interview questions

Everything a model does (attention, generation, cost) is denominated in tokens, not characters or words, and APIs charge per token. If a prompt shrinks from 800 to 200 tokens, you cut cost by 75% and also lower latency. Output tokens are 3–5× more expensive than input tokens on most APIs, and prompt caching cuts the cost of a repeated prefix by about 80–90%.

Temperature divides the logits by T before softmax. Use 0 for code, JSON, classification, and math, 0.7–1.0 for chat, and 1.0–2.0 for creative writing and brainstorming. Do not combine top_k and top_p aggressively: using both with tight values double-strips the tail and can cause degeneration. Modern providers recommend temperature + top_p (or temperature + min_p).

Reasoning models spend more compute at inference time: they emit thinking tokens, then the final answer. Use them when the task is math, code, or agentic work, when standard models have plateaued on your benchmark, and when you can afford the latency. They use 5–50× more tokens per query, so set a thinking budget cap at your latency and cost ceiling. For most chat, extraction, and RAG, a standard model with careful prompting wins.

Chain-of-thought (Wei et al., 2022) makes the model emit intermediate reasoning tokens before the answer, for example when you add "Let's think step by step". ReAct (Yao et al., 2022) interleaves reasoning with actions and observations, and it is the architectural pattern of every modern AI agent. Reflexion (Shinn et al., 2023) adds a loop in which the model reflects on what went wrong and writes that reflection into memory for the next attempt. A CoT trace can be 10–50× longer than a direct answer, so use it only for tasks like math, planning, debugging, and multi-hop reasoning.

Use three layers. First, constrain the model with JSON mode or structured outputs, which is the cheapest insurance. Second, validate the output at runtime with Pydantic, Zod, or JSON Schema, because even constrained output can have a wrong field or a hallucinated value. Third, recover gracefully: retry with a stricter prompt, fall back to defaults, or surface "I don't know" to the user. Never use eval() on model output.

Wrap every piece of untrusted input in clearly labeled tags, and tell the model in the system prompt that content inside the tags is data, not instructions. Repeat the critical rules at the end of the prompt, cap the size of injected content, constrain the output to a strict schema, and keep a tool allowlist. No prompt-layer defense is bulletproof, so combine them with system-layer defenses: a separate classifier that flags suspicious outputs, rate limits per user, human review for high-risk actions, and a deny-by-default tool allowlist.

Condense the oldest turns into a summary block written by the model, and keep the recent turns verbatim; the compression ratio is typically 5–20×. Trigger it by a token budget (for example when history is over 4K tokens), by turn count, or by a topic shift. Summarisation is lossy, and decisions, exact numbers, and quoted text disappear first, so use a structured summary schema and treat the summary as an index, not as the source of truth. For high-stakes conversations, persist the verbatim transcript to a DB.

Providers let you cache a prefix of your prompt, so identical tokens at the start of a request cost much less and run faster; this can drop 80–90% of the input bill for apps with stable system prompts and long retrieval sections. Put stable content first (system prompt, tool schemas, stable retrieval) and dynamic content last, such as the new user message. Anthropic supports up to 4 cache breakpoints. Even a single new character in the cached region busts the cache for everything after it, so keep the cached prefix byte-identical and hash your system prompt in CI.

Liu et al. (2023) showed that models recall information at the beginning and end of a long context well, but perform close to random for information in the middle, which gives a U-shaped curve. The effect persists even in frontier models with 100K+ context windows. To mitigate it, put critical content at the start and repeat it at the end, place the most relevant document at both the start and the end in RAG, and trim the context. As a rule, put any fact that must be recalled in the first 10% or the last 10% of the context.

Retrieval works on pieces of documents, so how you cut the text decides what the retriever can ever return: a chunk that is too coarse buries the answer in noise, and one that is too fine strips away the context. Fixed-size chunking shares chunk_overlap between neighbours so a fact on a boundary lands whole in at least one chunk, and the recursive splitter tries separators in order: paragraph, line, word, character. Late chunking (Günther et al., 2024) runs a long-context embedding model over the whole text and splits after the transformer and just before mean pooling, so every chunk vector carries the surrounding context. It needs no extra training, but it requires an embedding model whose context window can hold the document.

BM25 needs the query's exact words to appear, while dense vectors can blur rare identifiers, names, and codes, so hybrid search runs both and merges the results. RRF ignores raw scores and sums 1/(k + rank) over every ranking that contains the document, which sidesteps the problem that the two retrievers' scores are on different scales. A document that both lists rank well beats a document with only one strong rank. The paper fixed k = 60, and Elasticsearch and OpenSearch default to 60, but Qdrant's docs state a default of 2, so check the default of your engine.

First-stage retrieval is built to be fast over a huge corpus, so it scores each document without looking at the query and the document together. A cross-encoder reranker processes the pair jointly, so it is more accurate but slower, and it produces no reusable embedding. So use a two-stage funnel: a cheap first stage for recall over the whole corpus, then the reranker on only the N survivors, and the top-k go to the LLM. A reranker can only re-order what stage 1 returned, so if the right passage is not in the top-N, no reranker can recover it.

ReAct interleaves a Thought, an Action, and an Observation, so its traces are human-readable and you can see why the model picked an action. That is why it is the default for short, tool-heavy tasks and for debugging. Choose ReWOO for retrieval-heavy tasks with fan-out tool graphs, where the model does not need the result of one tool to plan the next. It plans the whole tool graph upfront, the worker runs everything in parallel, and the solver reads the evidence and answers. It uses 4–6× fewer tokens than ReAct on retrieval benchmarks, but its debug-ability is weak.

Use three layers. Retry transient failures with exponential backoff (1s, 2s, 4s, 8s) plus jitter, capped at 3–5 attempts; fall back to a different tool or model when the failure is not transient; and feed the failure back to the model to revise when it was a reasoning error. Always bound retries with a hard cap, a wall-clock timeout, and a circuit breaker. After 5 consecutive failures of the same kind the breaker opens and calls fail fast for 60 seconds, then it half-opens with one test call.

Use a risk ladder. Read-only actions are auto-approved, and local-write actions are auto-approved with a soft cap. External-write actions need approval per call, money actions need approval per call plus a daily cap, and destructive actions follow a two-person rule. The hard part is the UI, not the protocol, so budget at least 30% of agent project time for the human-facing surface.

Name the tool verb_noun, for example search_documents, because names sit in the model's context window. Write a description with one sentence on what it does, one on when to use it, and one on what it returns. Use explicit types and enums, avoid any, and return an error field instead of a 500. More than about 20 tools in one prompt measurably degrades selection accuracy, so split into sub-agents by domain or use a router.

MCP, the Model Context Protocol, is an open standard introduced by Anthropic in late 2024 and now adopted by OpenAI, Google, and the major IDE vendors. The model is the client, and the data or tool provider runs an MCP server. The server exposes three primitives: Tools (functions the model can call), Resources (read-only data the client can fetch on demand), and Prompts (named prompt templates). You write one MCP server, for example for Postgres, and every MCP-compatible client can use it.

Tool output is prompt content, because the model re-reads it on its next turn. A 10,000-row SQL result will blow your context window, so trim ruthlessly: return the top N and summarise the rest. Format for reading, not for storage, since Markdown tables beat JSON arrays for human-shaped reasoning. Add a "what this means" sentence when the result is non-obvious.

Start hosted: you can start in minutes, against weeks for self-hosting because of GPU procurement and ops. The rule of thumb is to move to self-hosting once your bill exceeds the cost of a half-time ML engineer plus two H100s. Hosted cost is $0.10–$15 per 1M tokens, and self-hosted cost is $0.04–$2 at scale. Self-hosting also gives you control of data privacy and full customisation, but you must scale for peak traffic yourself.

Quantization reduces the precision of model weights from 16-bit to 8- or 4-bit. It trades a small amount of quality for 2–4× less GPU memory and proportionally faster inference. GGUF is llama.cpp's single-file format with CPU+GPU hybrid support. AWQ gives 4-bit with near-FP16 quality and is popular for vLLM and SGLang.

Naive serving handles one request at a time per GPU. Continuous batching, used in vLLM, TGI, and SGLang, interleaves tokens from many requests so the GPU is never idle waiting for one slow request, and it typically delivers 5–20× higher throughput. Speculative decoding has a tiny draft model propose the next 5–10 tokens while the big model verifies them in parallel, for +2–3× throughput with no quality loss. vLLM adds a paged KV cache, allocated in pages like OS virtual memory.

An LLM trace is a tree of spans: the user request, the system prompt assembly, the LLM call, every tool invocation, and the result. Every request must record input tokens, output tokens, model, cost computed from the vendor's pricing, latency (TTFT and total), and cache hit. Tag each one with user_id, feature, and experiment. This data lets you answer questions like "is feature X losing money?" and "is the new model slower?"

Prompts are code: store them in git, version them, review them via PR, and deploy them via CI. Each prompt version gets a stable ID, and every trace carries that prompt ID, so you can instantly answer which prompt version users were on. For A/B tests, route traffic between two prompt versions or two models by user_id hash. Measure cost, latency, evals on the output, and user feedback, and stop when one variant wins on the metric you chose.

When a vendor swaps or updates a same-named model, your outputs can drift in ways your offline evals will not catch. Run a small fraction of production traffic (1–5%) through both the old and new model, compare evals, and alert on the delta; Arize Phoenix and Braintrust automate this. Capture thumbs-up and thumbs-down signals and feed the failures into your eval dataset. After three months a thousand real failures are worth more than ten thousand synthetic ones.

A golden dataset is a fixed set of (input, expected_output) pairs that the model runs against on every change. You build it once, grow it from real failures, and never let it decay. You run it in CI and block the release if accuracy drops below the threshold.

You use a strong LLM as a judge in one of two modes. Pairwise comparison shows the judge A and B and asks which is better, and it beats absolute scoring because relative judgements are easier than absolute ones. The other mode is rubric-based scoring with 4-6 criteria, each rated 0-5 with reasoning. Use a different, often stronger, model as the judge, rotate judge prompts, and periodically check the judge against human labels.

The Ragas framework standardised four RAG metrics: faithfulness, answer relevance, context precision, and context recall. Faithfulness asks whether the answer sticks to the retrieved chunks or hallucinates. Context recall asks whether all the chunks needed to answer were retrieved. For an agent, the right answer is not enough, so you also track trajectory match, tool-call accuracy, and step efficiency, besides task success rate.

You route easy requests to a small, cheap model and escalate to a large, expensive one only when the small model's confidence is low. You can measure confidence with logprobs, self-critique, or a separate classifier. A well-tuned cascade often reaches frontier quality at 30-70% of the cost.

Caching is the single most powerful cost lever, and there are three kinds. A semantic cache returns a cached answer when a near-identical prompt was answered recently (cosine > 0.95). An output cache is a key-value cache keyed on the input, for deterministic tasks such as extraction or classification with temperature=0. A prompt cache is provider-level: it reuses the KV of the system prompt across calls, with an 80-90% discount on the cached prefix.

You allocate a monthly token budget per feature, alert at 80%, and throttle at 100%, which is the operational layer that prevents one runaway feature from blowing the company's bill. A model gateway sits between your app and every LLM provider. It gives you one API for many providers, per-key rate limits, cost tracking, retries, and fallbacks, and the three major options are LiteLLM, Portkey, and OpenRouter.

Prompt injection is when user-controlled text hijacks the model's behaviour. In direct injection, the user types instructions that override the previous instruction. In indirect injection, the user plants malicious text in a document that the model will read. Indirect injection is the harder one because the user does not even need to interact with your system.

Jailbreaks are creative rewrites of forbidden prompts that slip past safety training, and they are a moving target because red-teams find new ones monthly. The defences are Constitutional AI fine-tuning, output classifiers, and robust system prompts. You also accept that 100% safety is unachievable.

You run a PII detector, such as Microsoft Presidio or spaCy, before you send user text to the model and before you render the model's reply. For compliance you often need both. Input redaction means you do not send PII to the vendor. Output redaction means you do not echo PII back to the wrong user.

A voice pipeline is STT → LLM → TTS. For STT the section lists Whisper, Deepgram, AssemblyAI, and ElevenLabs Scribe, and for TTS it lists ElevenLabs, Cartesia, OpenAI TTS, and PlayHT. Realtime APIs skip the chunky STT→LLM→TTS loop by streaming audio bidirectionally, and sub-500 ms turn-taking is now achievable. LiveKit and Vapi ship full agent frameworks on top of these.

Native multimodal models like GPT-4o, Claude 4, Gemini 2.5, and Qwen3-VL take images directly. An LLM beats Tesseract on messy photos, handwritten notes, and structured forms, and often no separate OCR step is needed. The same models do document AI (PDF understanding with layout awareness, for example Claude, Mistral OCR, and Qwen2-VL), read a chart and emit JSON, and answer visual Q&A.

Flux has open-weights availability and strong prompt fidelity. SDXL / SD3 are open and fine-tuneable with LoRA, Imagen 3 gives photorealism via Vertex, and DALL-E 3 / GPT-Image-1 are polished and instruction-following. For control, use ControlNet for structure (pose, depth, canny edges), LoRA for style or character consistency, inpainting for edits, and IP-Adapter for reference images. ComfyUI lets you wire these into reproducible pipelines.

Users expect words to appear as the model writes them, not after a 3-second pause. SSE is the simplest, works through any HTTP proxy, and is one-way (server to client). WebSockets are bidirectional and needed for voice and collaborative UIs, but have a heavier handshake. HTTP chunked transfer is a fallback for proxies that strip SSE headers.

Anything over about 10 seconds should not block an HTTP request. Push it to a queue (BullMQ, Celery, Inngest, Cloud Tasks, Temporal) and return a job_id, and the client polls or subscribes via SSE/WebSocket for updates. For failures, generate an idempotency key per request and reject duplicates, and store checkpoints of agent state so a resume replays only the failed steps. Use durable execution (Temporal/Restate) for anything longer than a minute.

Use fallback chains: if OpenAI returns 503, route to Anthropic, and if Anthropic returns 429, route to Together, with degraded prompts per provider. LiteLLM and Portkey do this out of the box, and a model gateway is essentially a fallback router plus cost tracking. Treat every prompt and every model as a feature flag, and roll out to 1%, then 10%, then 100%. Roll back instantly when evals regress.

Transformer Explainer, from the Polo Club of Data Science at Georgia Tech, runs a live GPT-2 (small) model inside your browser. It shows the stages tokenize, embed, attention, MLP, repeat, output, and sample, and it has sliders for temperature, top-k, and top-p. GPT-2 (small) has 124 million parameters and 12 blocks. It is from 2019, so it shows the shape of a transformer, not the quality of a modern model.

3Blue1Brown's Neural networks series is the best first stop if you want the idea before the code. Chapters 1 to 4 build one small network and teach how it learns, and chapters 5 to 7 apply the same ideas to the transformer in an LLM: transformers, attention step by step, and how an LLM may store facts. The videos teach ideas, not production code. They do not cover newer designs such as grouped-query attention or mixture of experts.

The LLM Inference Handbook is a free online guide to everything between your prompt and the answer. It separates four terms: inference, inference server or framework, serving, and inference optimization. It follows one request through tokenize, schedule, prefill, decode, and stream, and the two numbers to watch are TTFT (how long you wait for the first token) and TPOT (the gap between later tokens). Speed numbers depend on the model, the GPU, the batch size, and the prompt length, so measure your own system.