Part 07

Inference

API-based vs self-hosted inference; vLLM, TGI, SGLang, Ollama, llama.cpp; GGUF/AWQ/GPTQ/INT4-INT8 quantisation; KV cache, prefix caching, speculative decoding; continuous batching; TTFT vs TPS; GPU economics, edge inference, and the major hosted providers (Together, Fireworks, Groq, Cerebras, Replicate, AWS Bedrock). Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

1 section · 30 sources · three voices

Inference (servers, quantization, providers)

Every LLM call has two sides: the request you send and the inference that serves it. Most teams start by renting (Together, Fireworks, Groq, Cerebras, Replicate, AWS Bedrock). Once usage justifies it, you self-host with vLLM, TGI, SGLang, Ollama, or llama.cpp — and learn a different optimization vocabulary: KV cache, prefix caching, speculative decoding, continuous batching.

API-based vs self-hosted inference

Axis	Hosted API	Self-hosted
Time to first token	Minutes	Weeks (GPU procurement + ops)
Cost per 1M tokens	$0.10 – $15	$0.04 – $2 (at scale)
Data privacy	Vendor's terms	You control everything
Model selection	Vendor's catalogue	Anything you can fit on your GPU
Peak traffic	Vendor scales	You scale (autoscaler, queues)
Customisation	None / fine-tune API	Full (system prompts, prefixes, logprobs)

Rule of thumb: start hosted. Move to self-hosting once your bill exceeds the cost of a half-time ML engineer plus two H100s.

Inference servers

When you self-host, an inference server is the process that loads model weights into GPU memory and serves requests. The five that matter in 2025:

vLLM

The throughput champion. PagedAttention + continuous batching typically deliver 10–24× the throughput of naive Hugging Face.[13] Python API + OpenAI-compatible HTTP server.

TGI (text-generation-inference)

Hugging Face's Rust+Python server. Polished, supports the HF model hub natively, slightly lower raw TPS than TGI/vLLM.

SGLang

Optimised for structured-output and multi-call programs (RadixAttention for prefix caching across calls).[14]

Ollama

One-line CLI for local laptops and small servers. Best for dev and small-scale prod.

llama.cpp

CPU-first, GGUF-native, runs on phones and Raspberry Pis. Foundation for most edge inference.[15]

TensorRT-LLM / Triton

NVIDIA's compiled kernels for maximum H100 throughput. Most ops effort, best raw TPS.

Quantization: GGUF, AWQ, GPTQ, INT4/INT8

Quantization reduces the precision of model weights from 16-bit to 8- or 4-bit, trading a small amount of quality for 2–4× less GPU memory and proportionally faster inference. The common formats:

GGUF — llama.cpp's format. Single file, CPU+GPU hybrid, the de facto standard for open-weight local inference.[15]
AWQ — Activation-aware Weight Quantization. 4-bit with near-FP16 quality; popular for vLLM/SGLang.[16]
GPTQ — older, still common for GPU serving.
INT8 / FP8 — minimal-quality-loss quantisation on H100s; native hardware support.
INT4 / NF4 — bitsandbytes (QLora) for training; also used at inference time.

Quantisation trade-off (illustrative, not benchmark-exact). Going from FP16 to AWQ-4bit cuts memory 4× with small quality loss for most workloads.

কোয়ান্টাইজেশন ট্রেড-অফ (চিত্রলিখিত, benchmark-সঠিক নয়)। FP16 থেকে AWQ-4bit-এ গেলে বেশিরভাগ কাজে মানের সামান্য ক্ষতিতে মেমরি ৪× কমে।

KV cache, prefix caching, speculative decoding

Three ideas that show up in every modern inference server:

Paged KV cache: instead of one contiguous buffer per request, KV cache memory is allocated in pages (à la OS virtual memory). vLLM's PagedAttention removes the wasted-memory problem of naive serving.[13]
Prefix caching: if many requests share a long system prompt, the server hashes the prefix and reuses the cached KV. Anthropic's prompt_caching and OpenAI's cached_prompt save 80–90% of cost on the cached portion.[17]
Speculative decoding: a tiny draft model proposes the next 5–10 tokens; the big model verifies them in parallel. Net throughput +2–3× for free, no quality loss.

Continuous batching and throughput tuning

Naive serving processes one request at a time per GPU. Continuous batching (vLLM, TGI, SGLang) interleaves tokens from many requests so the GPU is never idle waiting for one slow request. This alone typically delivers 5–20× higher throughput.[13] Add max_num_seqs, max_model_len, and CUDA-graph capture for the next 2×.

Latency vs throughput (TTFT, TPS)

Metric	What it is	What moves it
TTFT (time to first token)	How long until the user sees the first word	Prompt length, prefill compute, network
TPS (tokens per second, per request)	Generation speed the user sees	Model size, batch size, KV-cache hits
Throughput (tokens/sec aggregate)	Total tokens the server emits	Continuous batching, GPU occupancy
p99 latency	Tail latency for worst 1%	Queue depth, KV-cache pressure

GPU economics, edge inference, on-device models

An H100 rents for $2–$4/hr on the major clouds, ~$30k to buy. At a 70B model, you serve ~30 concurrent requests before latency collapses — about $0.10 per 1k generated tokens fully loaded. Edge inference pushes smaller models onto the user's device (3B–8B on phones via MLX, llama.cpp, or vendor SDKs).[15] The trade-off: zero server cost, but quality and capability are limited.

Inference providers

Hosted inference in 2025 is a commodity market. Pick by which model you want at what price with what SLA:

Provider	Specialty	Models
Together	Open-weight catalogue, good default	Llama, Qwen, DeepSeek, Mistral
Fireworks	Fast, function-calling optimised	Most open + fine-tunes
Groq	Ultra-low latency LPU hardware	Llama, Mixtral, Whisper
Cerebras	Fast inference on wafer-scale chips	Llama, Qwen
Replicate	Pay-per-second, model marketplace	Anything on Hugging Face
AWS Bedrock / Azure AI Foundry	Enterprise compliance	Anthropic, Cohere, Llama, Mistral, etc.
OpenRouter	Aggregator — one API, many providers	All of the above

Practice · 4 tasks

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works

A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
Use made-up data only. Do not paste private information into a chat.

Task 1

Decide: hosted API or self-host

Why this matters

When your chatbot gets popular, someone asks: should we keep paying a hosted API, or run the model on our own GPUs? You must do the cost maths before you answer.

Step 1

New chat

Copy this prompt, paste it, and press Enter. It asks for the monthly cost of both options.

Prompt

Scenario (all numbers are invented):
A small company, ShopLink, runs a customer-support chatbot. It uses 2 billion tokens per month.

Option 1, hosted API: 0.50 USD per 1 million tokens.
Option 2, self-host: rent 2 GPUs at 3 USD per hour each, 730 hours per month, plus a half-time ML engineer at 3,000 USD per month. Ignore the small cost of electricity and storage.

Tasks:
1. Compute the monthly cost of each option. Show the arithmetic.
2. Compute the break-even: how many tokens per month make both options cost the same?
Reply with a short table, then one sentence on which option is cheaper now.

You should see: Hosted costs about 1,000 USD a month. Self-host costs about 7,380 USD. Break-even is about 14.8 billion tokens a month. At 2 billion, hosted is cheaper.

Example answer

Hosted: 2,000 million tokens × 0.50 USD = 1,000 USD per month.
Self-host: GPUs 2 × 3 × 730 = 4,380 USD, plus engineer 3,000 USD = 7,380 USD per month.
Break-even: 7,380 ÷ 0.50 = 14,760 million tokens, about 14.8 billion per month.
At 2 billion tokens, the hosted API is cheaper.

Your answer will be worded differently.

Step 2
Same chat
Check one number yourself with a calculator: 2 × 3 × 730. Compare it with the GPU cost in the answer.
You should see: 4,380 USD. It is the same as the GPU cost in the model's answer.

Step 3

Same chat

Copy this prompt, paste it in the same chat, and press Enter. The company grows and a privacy rule appears.

Prompt

Now the situation changes. ShopLink grows to 20 billion tokens per month. A new customer also says the chat text must never leave ShopLink's own servers. Use the same prices. Assume the same 2 GPUs can handle the load. What do you recommend now, and why? Give the monthly cost of both options and answer in under 120 words.

You should see: Hosted now costs about 10,000 USD. Self-host stays at 7,380 USD. Most models recommend self-hosting, for cost and for privacy.

Example answer

I recommend self-hosting.
Hosted: 20,000 million tokens × 0.50 USD = 10,000 USD per month.
Self-host: 7,380 USD per month, the same as before.
Self-hosting is cheaper now, and it keeps chat text on ShopLink's own servers, which the new customer requires.

Your answer will be worded differently.

What just happened

A hosted API costs more as your tokens grow (a token is a small piece of text, about a word). Self-hosting has a fixed cost: GPUs and an engineer, whatever the traffic. At 2 billion tokens the fixed cost is too big. At 20 billion tokens it is smaller than the API bill. Privacy rules can also force you to self-host.

You are done when You have two monthly costs for each situation, and you checked the GPU cost by hand.

Check yourselfWhy do most teams start with a hosted API and move to self-hosting later?Show answer

A hosted API is quick to start and the vendor handles the load. Self-hosting has fixed costs, so it pays off only when the token bill is large, or when privacy or full control matters.

Try a harder versionIn the first prompt, change one number, for example a hosted price of 5 USD per 1 million tokens. Predict first: does the break-even go up or down? Then check with the model.

Task 2

Calculate model memory at 16, 8, 4 bits

Why this matters

When you pick a model to run, the first question is: will it fit in my GPU or laptop memory? A simple calculation tells you in seconds, before you download a 140 GB file.

Step 1

New chat

Copy this prompt, paste it, and press Enter. The model makes a memory table for a 70 billion parameter model.

Prompt

A language model has 70 billion parameters. Use this rule: weight memory in GB = parameters in billions × bits per weight ÷ 8.

Make a table of weight memory at 16-bit, 8-bit, and 4-bit. Then say which of these fit on one GPU with 80 GB of memory. Finally, in one sentence, say what other memory the server needs at run time besides the weights.

You should see: 140 GB at 16-bit, 70 GB at 8-bit, 35 GB at 4-bit. Only 8-bit and 4-bit fit in 80 GB, and 8-bit leaves little room.

Example answer

Weight memory = 70 × bits ÷ 8
16-bit: 70 × 16 ÷ 8 = 140 GB
8-bit: 70 × 8 ÷ 8 = 70 GB
4-bit: 70 × 4 ÷ 8 = 35 GB
Fits on one 80 GB GPU: 8-bit (but little room is left) and 4-bit. 16-bit does not fit.
The server also needs memory for the KV cache, the model's short-term memory of the text so far.

Your answer will be worded differently.

Step 2

Same chat

Copy this prompt, paste it in the same chat, and press Enter. Now the target is a laptop with 8 GB free.

Prompt

I want to run an 8 billion parameter model on my laptop. The laptop has 16 GB of RAM, and about 8 GB of it is free for the model. Using the same rule, which precision (16-bit, 8-bit, or 4-bit) can I use? Give the memory for each. Then tell me one risk of using the smallest precision.

You should see: 8B model: 16 GB, 8 GB, 4 GB. Only 4-bit fits with room to spare. 8-bit uses all 8 GB, which is too tight.

Example answer

16-bit: 8 × 16 ÷ 8 = 16 GB. Too big.
8-bit: 8 × 8 ÷ 8 = 8 GB. It uses all your free memory, so it is too tight.
4-bit: 8 × 4 ÷ 8 = 4 GB. It fits.
Use 4-bit. One risk: the answers can get a little worse, for example more mistakes on hard questions.

Your answer will be worded differently.

What just happened

A model is a very large list of numbers called weights. At 16 bits each number takes 2 bytes. Quantization stores each number with fewer bits (8 or 4), so the model needs 2 to 4 times less memory. The cost is a small loss in quality. Weights are not all of the memory: the server also needs memory for the KV cache (its short-term memory of the text so far).

You are done when You have a memory table for the 70B model (140, 70, 35 GB) and one for the 8B model (16, 8, 4 GB).

Check yourselfWhat does quantization change, and what is the trade-off?Show answer

It stores the weights with fewer bits, for example 4 instead of 16, so the model needs about 2 to 4 times less memory and runs faster. The cost is a small loss in quality for most workloads.

Try a harder versionAsk the model for a test of 5 questions that compares a 16-bit and a 4-bit version of the same model. Which kinds of questions would show a quality drop first?

Task 3

Estimate savings from prefix caching

Why this matters

When you build a support bot, every request repeats the same long instructions. You pay for those tokens again and again. Caching can cut that bill a lot, if you write the prompt in the right order.

Step 1

New chat

First guess: will caching save more than half of the cost? Then copy this prompt, paste it, and press Enter.

Prompt

A support bot gets 10,000 requests per day (all numbers are invented). Every request starts with the same system prompt of 3,000 tokens. The user message adds 100 tokens. Input tokens cost 1 USD per 1 million. Cached input tokens cost 0.10 USD per 1 million (a 90% discount).

Compute the daily input cost (1) without prefix caching and (2) with prefix caching, where the system prompt is cached. Show the arithmetic. Then give the saving as a percentage.

You should see: Without caching: 31 USD a day. With caching: 4 USD a day. The saving is about 87%. Your guess was right if you said more than half.

Example answer

Without caching: 3,100 tokens × 10,000 = 31 million tokens × 1 USD = 31 USD per day.
With caching: cached part 3,000 × 10,000 = 30 million tokens × 0.10 USD = 3 USD. New part 100 × 10,000 = 1 million tokens × 1 USD = 1 USD. Total 4 USD.
Saving: 27 USD, about 87%.

Your answer will be worded differently.

Step 2

Same chat

Guess what goes wrong if the prompt starts with today's date and the user's name. Then copy this prompt, paste it in the same chat, and press Enter.

Prompt

The team now changes the system prompt. The first line becomes: "Today is {current date and time}. The user's name is {name}." The rest stays the same 3,000 tokens. Prefix caching stops working well. Explain in 3 short sentences why, and show how to change the prompt so the cache works again.

You should see: The date and name change on every request, so the start of the prompt is never the same. The cache cannot be reused. The fix: put the fixed text first and the changing text last.

Example answer

The date, time, and name change in every request. The cache only works when the start of the prompt is exactly the same, so the server must work from zero each time. Fix: put the 3,000 fixed tokens first, then add the date, time, and name at the end:
[fixed system prompt]
Today is {date}. The user's name is {name}.

Your answer will be worded differently.

What just happened

The server does heavy work on the start of your prompt and stores it. This stored work is the KV cache. With prefix caching, a new request that starts with the same text reuses it, and you pay about 90% less for that part. But the match must be exact from the first character. One changing line at the top breaks it.

You are done when You have a daily cost of 31 USD and 4 USD, and a new prompt with the fixed text first.

Check yourselfWhy must the shared part of the prompt come first?Show answer

The server reuses the cache only when the start of the prompt matches exactly. If something changes early in the prompt, the whole prompt looks new, so there is no cache hit.

Try a harder versionTry 1,000 requests per day instead of 10,000. How many USD do you save per day now? When is prefix caching not worth the effort?

Task 4

Match symptoms to inference metrics

Why this matters

When you run your own model server, users say "it is slow". But slow can mean four different things. You must name the right metric first, or you will pick the wrong fix.

Step 1

New chat

Copy this prompt, paste it, and press Enter. Before you read the answer, guess the metric for symptom 3.

Prompt

I run an LLM inference server. For each symptom, name the main metric that is bad and choose ONE best fix.

Metrics:
- TTFT: the time until the first word appears.
- TPS per request: how fast words stream for one user.
- Aggregate throughput: total tokens per second for all users together.
- p99 latency: how long the slowest 1% of requests take.

Fixes to choose from: prefix caching, continuous batching (the server mixes many requests on the GPU at once), speculative decoding (a small model drafts words and the big model checks them), a shorter prompt, a smaller or quantized model, or more GPU capacity.

Symptoms:
1. Users send a 20,000-token document. They wait 6 seconds before the first word appears. After that, text streams fast.
2. Even with only one user on the server, text appears at 8 tokens per second, which feels slow.
3. The GPU is idle about 60% of the time. The server finishes one request completely before it starts the next. Total tokens per second for all users is low.
4. Most requests are fast, but the slowest 1% take 40 seconds when the queue is long.

Reply with a table: symptom number, metric, fix, and a reason in 12 words or fewer.

You should see: A table with 4 rows. Roughly: 1 is TTFT, 2 is TPS per request, 3 is aggregate throughput, 4 is p99 latency. Some symptoms have more than one good fix.

Example answer

1 | TTFT | a shorter prompt | Long prompt makes the first word slow
2 | TPS per request | speculative decoding | Drafted words speed up one stream
3 | Aggregate throughput | continuous batching | Mixes requests so the GPU is never idle
4 | p99 latency | more GPU capacity | Long queue hurts the slowest requests

Your answer will be worded differently.

Step 2

Same chat

Copy this question, paste it in the same chat, and press Enter. Compare the answer with the table.

Prompt

Can a server have high aggregate throughput and still feel slow to one user? Answer in 2 short sentences.

You should see: Yes. Throughput counts all users together, but one user feels only their own wait and speed.

Example answer

Yes. Throughput counts tokens for all users together, so a busy server can score high. One user still feels their own wait for the first word and their own speed, and a long queue can make that wait long.

Your answer will be worded differently.

What just happened

"Slow" has four faces. TTFT (time to the first word) is about waiting. TPS per request is the speed of the stream for one user. Aggregate throughput is the total work of the server. p99 latency is the wait of the slowest 1% of users. Each one has a different fix.

You are done when You have a table with a metric and a fix for all 4 symptoms, and a clear answer on throughput versus one user's feeling.

Check yourselfSymptom 2 happens with only one user. Why can continuous batching not fix it?Show answer

Continuous batching helps when many users share the GPU, so it raises total throughput. With one user the GPU already works only for that user. To make that one stream faster, you need a smaller or quantized model, or speculative decoding.

Try a harder versionMake up a 5th symptom that has two possible fixes. Ask the model which fix to try first and why. Do you agree with the order?

Watch · 8 videos

Hosted LLM APIs vs. Self-Hosting: The Ultimate LLM Ops Decision Guide
Analytics Vidhya
What is vLLM? Efficient AI Inference for Large Language Models
IBM Technology
Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)
Maarten Grootendorst
KV Cache in LLMs Explained Visually | How LLMs Generate Tokens Faster
ExplainingAI
Accelerating LLM inference with speculative decoding: From Zero to Hero, By Eldar Kurtić
ARGO Vision
An Intermediate Guide to Inference Using vLLM
Red Hat Open
LLM’s INFERENCE: Cost vs. Latency vs. Throughput
César Soto Valero
TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google
AI Engineer

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.