Part 13
AI Application Architecture
Streaming over SSE / WebSockets / chunked HTTP, background jobs and queues for long agent runs, idempotency and resumability, multi-tenant prompt and key management, fallback chains across providers, feature flags for prompts and models, and API design patterns for AI features (cancel, retry, partial results). Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।
1 section · 30 sources · three voices
01
AI application architecture
An LLM call is just one HTTP request. Building a product around it requires the same plumbing any modern app needs, plus a few new primitives: streaming, resumability, multi-tenancy, fallbacks, feature flags, and an API surface designed for partial results.
Streaming over SSE / WebSockets / HTTP chunked
Users expect words to appear as the model writes them, not after a 3-second pause. Three transports:
- SSE (Server-Sent Events): the simplest; works through any HTTP proxy; one-way (server→client).
- WebSockets: bidirectional; needed for voice and collaborative UIs; heavier handshake.
- HTTP chunked transfer: a fallback for proxies that strip SSE headers.
# OpenAI streaming
for chunk in client.chat.completions.create(
model="gpt-4o", stream=True,
messages=messages):
token = chunk.choices[0].delta.content or ""
print(token, end="", file=sys.stdout)Background jobs and queues for long agent runs
Anything over ~10 seconds should not block an HTTP request. Push it to a queue (BullMQ, Celery, Inngest, Cloud Tasks, Temporal) and return a job_id. Client polls or subscribes via SSE/WebSocket for updates. This is the single biggest UX win for "the agent took 4 minutes" tasks.
Idempotency and resumability
LLM calls fail. Networks drop. Users refresh. Always:
- Generate an idempotency key per request; reject duplicates.
- Store checkpoints of agent state so a resume replays only the failed steps.
- Use durable execution (Temporal/Restate) for anything longer than a minute.
Multi-tenant prompt and key management
Per-tenant prompt overrides, per-tenant API keys, per-tenant rate limits, per-tenant tracing. The cleanest pattern: store all of these in a tenant_config row, fetch at the edge of every request, attach to the trace. Never hard-code a tenant's prompt in their codebase.
Fallback chains across providers
OpenAI 503s? Route to Anthropic. Anthropic 429s? Route to Together. With degraded prompts per provider ("you have no tool use; just answer"). LiteLLM and Portkey do this out of the box; a model gateway is essentially a fallback router plus cost tracking.
Feature flags for prompts and models
Treat every prompt and every model as a flag. Roll out "use GPT-5 for premium users only" to 1%, then 10%, then 100%. Roll back instantly when evals regress. LaunchDarkly, Statsig, PostHog, or a simple DB column — pick one and use it for prompts.
API design patterns for AI features
Five rules that distinguish an AI API from a normal REST API:[30]
- Cancel mid-flight: the client should be able to abort the request and stop the bill.
- Retry idempotently: same request + idempotency key → same response, no double billing.
- Return partial results: stream tokens, return intermediate tool states, never make the client wait until completion.
- Structured confidence: include a self-reported confidence, logprobs, or a "would I retry this?" field.
- Cost/latency budget in the request: let the client say "spend up to $0.05 and 8s"; the server respects it.
Practice · 4 tasks
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Time the first word of a streamed answer
Why this matters
When you build a chat feature, users leave if the screen stays empty for many seconds. Streaming shows the words as the model writes them, so the app feels fast.
Step 1
New chatCopy this prompt. Start a stopwatch when you press Enter. Write down the time of the first word and the time of the last word.
Prompt Write a story of about 400 words. A bus driver in Chattogram finds a lost bag on the last trip of the night. Start the story right away, with no introduction.
You should see: The first word appears fast, for example after 1 or 2 seconds. The last word comes much later, for example after 15 to 30 seconds.
Example answer
The last bus of the night rolled into the Chattogram depot at 11:40. Rafiq switched off the engine and, as he did every night, walked down the aisle to check the seats. Under the back row, he found a small black bag...
Your answer will be worded differently.
Step 2
Same chatImagine the chat showed nothing until the last word. Write how long the user would wait, and how that would feel.
You should see: The user would look at an empty screen for the full time you wrote down, for example 20 seconds. Most people would think the app is broken.
Step 3
New chatBefore you send this prompt, read the three features. Then paste it and press Enter.
Prompt Explain each of these in one simple sentence: SSE (Server-Sent Events), WebSockets, and HTTP chunked transfer. Then pick one for each feature and give one reason. Feature 1: An AI chat answer appears word by word. Data only goes from server to browser. The route is a normal HTTP proxy. Feature 2: A voice call where the user and the AI can talk at the same time. Feature 3: A fallback for an old company proxy that removes the SSE headers. Then list one cost of WebSockets compared to SSE.
You should see: SSE for feature 1, WebSockets for feature 2, and HTTP chunked for feature 3. WebSockets cost a heavier setup.
Example answer
SSE: the server pushes text to the browser over one normal HTTP connection. It is one-way. WebSockets: one open connection where both sides can send at any time. HTTP chunked: the server sends one response in many small pieces. Feature 1: SSE. Data only goes one way, and it works through a normal proxy. Feature 2: WebSockets. Both sides must send audio at the same time. Feature 3: HTTP chunked. It works when a proxy removes SSE headers. Cost of WebSockets: a heavier handshake (setup) and more work to run.
Your answer will be worded differently.
What just happened
A model writes tokens (small pieces of words) one by one. With streaming, the app shows each token at once. The total time does not change, but the user sees the first word after about a second and starts to read. SSE is the simplest way to stream, because the data only goes one way, from server to browser.
You are done when You have two times for the story (first word and last word), and a transport for each of 3 features.
Check yourselfThe total time to write the answer does not change with streaming. Why does the app still feel faster?Show answer
The user sees the first words after a short time and can start to read. Without streaming, the user would wait for the whole answer with an empty screen.
Task 2
Play a server that runs a long job
Why this matters
When you build an AI agent that works for 4 minutes, you cannot keep the web request open. Many servers cut a request that waits that long, and the user sees nothing.
Step 1
New chatCopy this prompt, paste it, and press Enter. The AI now plays an API server.
Prompt Act as the API server of a research agent. Reply only with an HTTP status line and a JSON body. No other text. Rules: - POST /jobs starts a job. Reply at once with 202 and a JSON body with job_id and status "queued". The first job_id is job_001. The job takes 245 seconds. - GET /jobs/{job_id} gives the status. Statuses are queued, running, done, and failed. - I tell you the elapsed seconds in each GET request. Use this table: 0 to 59 seconds: running, step "1 search sources" 60 to 179 seconds: running, step "2 read 12 sources" 180 to 244 seconds: running, step "3 write the report" 245 seconds or more: done, with a result field that has a short fake report title - When running, also show progress as a percent: elapsed divided by 245, rounded to a whole number. Say READY if you understand.You should see: The AI replies READY.
Step 2
Same chatCopy this request, paste it, and press Enter. This starts the job.
Prompt POST /jobs {"topic": "Mobile payment use in rural Bangladesh"}You should see: A fast reply with status 202 and a
job_id, for example job_001. It has no result yet.Example answer
HTTP/1.1 202 Accepted {"job_id": "job_001", "status": "queued"}Your answer will be worded differently.
Step 3
Same chatCopy this request, paste it, and press Enter. It asks for the status after 40 seconds.
Prompt GET /jobs/job_001 elapsed: 40 seconds
You should see: Status running, step 1, and progress of about 16%.
Example answer
HTTP/1.1 200 OK {"job_id": "job_001", "status": "running", "step": "1 search sources", "progress": "16%"}Your answer will be worded differently.
Step 4
Same chatCopy this request, paste it, and press Enter. It asks for the status after 120 seconds.
Prompt GET /jobs/job_001 elapsed: 120 seconds
You should see: Status running, step 2, and progress of about 49%.
Example answer
HTTP/1.1 200 OK {"job_id": "job_001", "status": "running", "step": "2 read 12 sources", "progress": "49%"}Your answer will be worded differently.
Step 5
Same chatCopy this request, paste it, and press Enter. It asks for the status after 245 seconds.
Prompt GET /jobs/job_001 elapsed: 245 seconds
You should see: Status done, and a result with a short report title.
Example answer
HTTP/1.1 200 OK {"job_id": "job_001", "status": "done", "result": "Mobile Payments in Rural Bangladesh: A Short Report"}Your answer will be worded differently.
What just happened
A long task should not hold the web request open. The server replies at once with a job_id, which is like a ticket number, and does the work in the background. The client then asks again and again with the job_id. This is called polling. Here the first reply was instant, but the job takes 245 seconds, about 4 minutes.
You are done when You have one fast reply with a job_id, and 3 polls that show running, running, and done.
Check yourselfWhy should a task of more than about 10 seconds return a job_id instead of the result?Show answer
A long task can make the connection time out, and the user sees nothing. With a job_id, the work runs in the background, and the client can check the progress or come back later.
Task 3
Stop double charges after a retry
Why this matters
When your app charges money for each AI run, a user who clicks Pay twice or refreshes the page must not pay twice. A failed long run must also not start from zero.
Step 1
New chatCopy this prompt, paste it, and press Enter. The AI now plays a billing API.
Prompt Act as an API server for report generation. Each new run costs 0.40 USD. Reply only with JSON for each request I send. Rules: - Every request has an idempotency_key and a body. - A new key makes a new run with a new run_id (run_1, run_2, ...). Reply with charged: 0.40 and replayed: false. - The same key with the same body returns the stored response again, with replayed: true and charged: 0. No new run. - The same key with a different body returns the error code 422 with a short message. Say READY if you understand.
You should see: The AI replies READY.
Step 2
Same chatCopy this prompt, paste it, and press Enter. Write down the total money charged after the three requests.
Prompt Handle these three requests in order. After the third, add a line with the total USD charged. 1. idempotency_key: k-1001, body: {"report": "monthly sales"} 2. idempotency_key: k-1001, body: {"report": "monthly sales"} (the user refreshed the page) 3. idempotency_key: k-1001, body: {"report": "yearly sales"}You should see: Three JSON replies. Request 1 charges 0.40. Request 2 is a replay with charge 0. Request 3 is an error 422. The total is 0.40 USD.
Example answer
{"run_id": "run_1", "charged": 0.40, "replayed": false} {"run_id": "run_1", "charged": 0, "replayed": true} {"error": 422, "message": "Key k-1001 was already used with a different body."} Total charged: 0.40 USDYour answer will be worded differently.
Step 3
Same chatCopy this prompt, paste it, and press Enter. Write down the cost with checkpoints and the cost without.
Prompt Now a long run. Run run_77 has 5 steps. Each step costs 0.10 USD, and a failed step costs nothing. Steps 1, 2, and 3 finished and were already charged. The server saved a checkpoint after each one. Step 4 failed with a network error. The client retries with the same idempotency_key. 1. Which steps do you run now, and what do you charge? 2. What would the retry cost if there were no checkpoints? 3. Name one thing the checkpoint must contain.
You should see: With checkpoints, only steps 4 and 5 run: 0.20 USD. Without checkpoints, all 5 steps run: 0.50 USD.
Example answer
1. Run only steps 4 and 5. Steps 1 to 3 are in the checkpoint. Charge: 2 × 0.10 = 0.20 USD. 2. With no checkpoints, all 5 steps run again: 5 × 0.10 = 0.50 USD. 3. The checkpoint must contain the saved results of steps 1 to 3, and the number of the last finished step.
Your answer will be worded differently.
What just happened
An idempotency key is a unique label that a client puts on a request. If the same request comes again, the server returns the saved answer and does not charge again. A checkpoint is a saved copy of the work done so far. After a failure, the server runs only the steps that are left: 2 steps (0.20 USD) instead of all 5 (0.50 USD).
You are done when You have a total of 0.40 USD for the three requests, and two costs for the retry: 0.20 USD and 0.50 USD.
Check yourselfA user refreshes the page, and the client sends the same request again. How does an idempotency key protect the user?Show answer
The server stores the response under the key. A second request with the same key gets the same response, with no new run and no new charge.
Task 4
Simulate a gateway that tries backup providers
Why this matters
When you build a product on an AI provider, that provider will sometimes fail. A gateway sends the request to a backup provider, so your users still get an answer.
Step 1
New chatCopy this prompt, paste it, and press Enter. It is the normal prompt, for a provider that has tools.
Prompt You can use one tool: get_weather(city: string), which returns live weather. User question: Do I need an umbrella in Sylhet today? Reply ONLY with the tool call as JSON, in the form {"tool": ..., "arguments": {...}}. Do not answer the question yet.You should see: Only a tool call as JSON, with no final answer. For example,
get_weatherwith the city Sylhet.Example answer
{"tool": "get_weather", "arguments": {"city": "Sylhet"}}Your answer will be worded differently.
Step 2
New chatCopy this prompt, paste it, and press Enter. It is the degraded prompt, for a provider with no tools.
Prompt You have no tools and no live data. Answer from general knowledge only. Say clearly that you cannot check today's weather, and give the most useful advice you can. User question: Do I need an umbrella in Sylhet today?
You should see: A plain answer that says the AI cannot check today's weather, and gives a useful tip.
Example answer
I cannot check today's weather, so I cannot be sure. Sylhet is one of the rainiest places in Bangladesh, especially in the monsoon, so carrying a small umbrella is a safe choice. Look at a weather app before you leave.
Your answer will be worded differently.
Step 3
New chatCopy this prompt, paste it, and press Enter. The AI now plays a gateway. Check each row of the table against the rules.
Prompt Act as an LLM gateway. The fallback chain is: Provider 1 (has tools), then Provider 2 (has tools), then Provider 3 (no tools, so use a degraded prompt that says: you have no tools, answer from general knowledge). Rules: on a 503 or 429 error, go to the next provider. On success, stop. Record the provider that answered, the number of attempts, and the cost. Costs per call: Provider 1 0.010 USD, Provider 2 0.006 USD, Provider 3 0.002 USD. Failed calls cost 0. Three requests came in: Request X: Provider 1 returns 200. Request Y: Provider 1 returns 503, Provider 2 returns 200. Request Z: Provider 1 returns 503, Provider 2 returns 429, Provider 3 returns 200. Give a table with: request, provider used, attempts, prompt used (normal or degraded), cost. Then add the total cost.
You should see: A table where X uses Provider 1, Y uses Provider 2, and Z uses Provider 3 with the degraded prompt. The total cost is 0.018 USD.
Example answer
| Request | Provider used | Attempts | Prompt | Cost (USD) | | X | Provider 1 | 1 | normal | 0.010 | | Y | Provider 2 | 2 | normal | 0.006 | | Z | Provider 3 | 3 | degraded | 0.002 | Total cost: 0.010 + 0.006 + 0.002 = 0.018 USD
Your answer will be worded differently.
What just happened
A provider can fail with an error such as 503 (the server is not ready) or 429 (too many requests). A provider can also lack a feature, such as tools. A gateway tries the next provider in a fixed order. For a provider with no tools, it uses a degraded prompt, which is simpler but still useful. Failed calls cost nothing here, so each request pays only for the provider that answered.
You are done when You have a tool call, an honest answer without tools, and a gateway table with a total of 0.018 USD.
Check yourselfWhy does a fallback chain use a different prompt for a provider that has no tool use?Show answer
A provider that cannot use tools would fail on the normal prompt. A degraded prompt, such as you have no tools, answer anyway, keeps the answer useful.
Watch · 6 videos
Streaming in React the Simple Way: Server-Sent Events (with FastAPI) Techno Pain
WebSockets vs Server-Sent Events (SSE) | Real-Time API Design AI TL;DR
Build an AI Agent with Temporal Temporal
Durable Agents: long running AI workflows in a flakey world | Samuel Colvin, AI By the Bay25 FunctionalTV
Why External APIs Break Idempotency CodeOpinion
Rate Limiting and Model Failover With Agentgateway solo.io
References
Sources
All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.
- [1]Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
- [2]Wei et al., "Plan-and-Execute Agents" / "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — NeurIPS 2022
- [3]Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" — NeurIPS 2023
- [4]Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models"
- [5]Anthropic Engineering blog — "How we built our multi-agent research system" (2025)
- [6]LangGraph — official documentation (LangChain)
- [7]Temporal — "Durable Execution for LLM Agents"
- [8]Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" — 2023
- [9]Anthropic — "Introducing computer use, a new Claude 3.5 Sonnet capability" (Oct 2024)
- [10]Anthropic — Tool use / function calling documentation
- [11]OpenAI — Function calling guide
- [12]Anthropic — Model Context Protocol (MCP) specification
- [13]Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" — vLLM paper, SOSP 2023
- [14]Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" — 2023
- [15]llama.cpp — open-source GGUF-native inference (ggerganov)
- [16]Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" — MLSys 2024
- [17]Anthropic — Prompt caching documentation
- [18]Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — NeurIPS 2023
- [19]Es et al., "Ragas: Automated Evaluation of Information Retrieval Augmented Language Models" — 2023
- [20]Yue et al., "Large Language Model Cascades with Mixture of Thought Representations" — 2024
- [21]GPTCache — open-source semantic cache
- [22]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL" — 2025
- [23]Jiang et al., "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models" — EMNLP 2023
- [24]LiteLLM — unified interface for 100+ LLMs
- [25]OWASP — "Top 10 for LLM Applications" (LLM01 Prompt Injection, 2025)
- [26]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
- [27]NVIDIA NeMo Guardrails — open-source toolkit
- [28]Qwen2-VL Team, "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" — 2024
- [29]OpenAI — Realtime API documentation
- [30]OpenAI — Production best practices guide