Skip to content

Part 13

AI Application Architecture

Streaming over SSE / WebSockets / chunked HTTP, background jobs and queues for long agent runs, idempotency and resumability, multi-tenant prompt and key management, fallback chains across providers, feature flags for prompts and models, and API design patterns for AI features (cancel, retry, partial results). Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

1 section · 30 sources · three voices

01

AI application architecture

An LLM call is just one HTTP request. Building a product around it requires the same plumbing any modern app needs, plus a few new primitives: streaming, resumability, multi-tenancy, fallbacks, feature flags, and an API surface designed for partial results.

Streaming over SSE / WebSockets / HTTP chunked

Users expect words to appear as the model writes them, not after a 3-second pause. Three transports:

  • SSE (Server-Sent Events): the simplest; works through any HTTP proxy; one-way (server→client).
  • WebSockets: bidirectional; needed for voice and collaborative UIs; heavier handshake.
  • HTTP chunked transfer: a fallback for proxies that strip SSE headers.
# OpenAI streaming
  for chunk in client.chat.completions.create(
        model="gpt-4o", stream=True,
        messages=messages):
    token = chunk.choices[0].delta.content or ""
    print(token, end="", file=sys.stdout)

Background jobs and queues for long agent runs

Anything over ~10 seconds should not block an HTTP request. Push it to a queue (BullMQ, Celery, Inngest, Cloud Tasks, Temporal) and return a job_id. Client polls or subscribes via SSE/WebSocket for updates. This is the single biggest UX win for "the agent took 4 minutes" tasks.

Idempotency and resumability

LLM calls fail. Networks drop. Users refresh. Always:

  • Generate an idempotency key per request; reject duplicates.
  • Store checkpoints of agent state so a resume replays only the failed steps.
  • Use durable execution (Temporal/Restate) for anything longer than a minute.

Multi-tenant prompt and key management

Per-tenant prompt overrides, per-tenant API keys, per-tenant rate limits, per-tenant tracing. The cleanest pattern: store all of these in a tenant_config row, fetch at the edge of every request, attach to the trace. Never hard-code a tenant's prompt in their codebase.

Fallback chains across providers

OpenAI 503s? Route to Anthropic. Anthropic 429s? Route to Together. With degraded prompts per provider ("you have no tool use; just answer"). LiteLLM and Portkey do this out of the box; a model gateway is essentially a fallback router plus cost tracking.

Feature flags for prompts and models

Treat every prompt and every model as a flag. Roll out "use GPT-5 for premium users only" to 1%, then 10%, then 100%. Roll back instantly when evals regress. LaunchDarkly, Statsig, PostHog, or a simple DB column — pick one and use it for prompts.

API design patterns for AI features

Five rules that distinguish an AI API from a normal REST API:[30]

  • Cancel mid-flight: the client should be able to abort the request and stop the bill.
  • Retry idempotently: same request + idempotency key → same response, no double billing.
  • Return partial results: stream tokens, return intermediate tool states, never make the client wait until completion.
  • Structured confidence: include a self-reported confidence, logprobs, or a "would I retry this?" field.
  • Cost/latency budget in the request: let the client say "spend up to $0.05 and 8s"; the server respects it.

Practice · 4 tasks

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Time the first word of a streamed answer

Why this matters

When you build a chat feature, users leave if the screen stays empty for many seconds. Streaming shows the words as the model writes them, so the app feels fast.

  1. Step 1
    New chat

    Copy this prompt. Start a stopwatch when you press Enter. Write down the time of the first word and the time of the last word.

    Prompt
    Write a story of about 400 words. A bus driver in Chattogram finds a lost bag on the last trip of the night. Start the story right away, with no introduction.

    You should see: The first word appears fast, for example after 1 or 2 seconds. The last word comes much later, for example after 15 to 30 seconds.

    Example answer
    The last bus of the night rolled into the Chattogram depot at 11:40. Rafiq switched off the engine and, as he did every night, walked down the aisle to check the seats. Under the back row, he found a small black bag...

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Imagine the chat showed nothing until the last word. Write how long the user would wait, and how that would feel.

    You should see: The user would look at an empty screen for the full time you wrote down, for example 20 seconds. Most people would think the app is broken.

  3. Step 3
    New chat

    Before you send this prompt, read the three features. Then paste it and press Enter.

    Prompt
    Explain each of these in one simple sentence: SSE (Server-Sent Events), WebSockets, and HTTP chunked transfer.
    Then pick one for each feature and give one reason.
    Feature 1: An AI chat answer appears word by word. Data only goes from server to browser. The route is a normal HTTP proxy.
    Feature 2: A voice call where the user and the AI can talk at the same time.
    Feature 3: A fallback for an old company proxy that removes the SSE headers.
    Then list one cost of WebSockets compared to SSE.

    You should see: SSE for feature 1, WebSockets for feature 2, and HTTP chunked for feature 3. WebSockets cost a heavier setup.

    Example answer
    SSE: the server pushes text to the browser over one normal HTTP connection. It is one-way.
    WebSockets: one open connection where both sides can send at any time.
    HTTP chunked: the server sends one response in many small pieces.
    Feature 1: SSE. Data only goes one way, and it works through a normal proxy.
    Feature 2: WebSockets. Both sides must send audio at the same time.
    Feature 3: HTTP chunked. It works when a proxy removes SSE headers.
    Cost of WebSockets: a heavier handshake (setup) and more work to run.

    Your answer will be worded differently.

What just happened

A model writes tokens (small pieces of words) one by one. With streaming, the app shows each token at once. The total time does not change, but the user sees the first word after about a second and starts to read. SSE is the simplest way to stream, because the data only goes one way, from server to browser.

You are done when You have two times for the story (first word and last word), and a transport for each of 3 features.

Check yourselfThe total time to write the answer does not change with streaming. Why does the app still feel faster?Show answer

The user sees the first words after a short time and can start to read. Without streaming, the user would wait for the whole answer with an empty screen.

Try a harder versionAsk for a 1,000-word story and measure again. Does the time of the first word change much? What does that tell you?

Task 2

Play a server that runs a long job

Why this matters

When you build an AI agent that works for 4 minutes, you cannot keep the web request open. Many servers cut a request that waits that long, and the user sees nothing.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. The AI now plays an API server.

    Prompt
    Act as the API server of a research agent. Reply only with an HTTP status line and a JSON body. No other text.
    Rules:
    - POST /jobs starts a job. Reply at once with 202 and a JSON body with job_id and status "queued". The first job_id is job_001. The job takes 245 seconds.
    - GET /jobs/{job_id} gives the status. Statuses are queued, running, done, and failed.
    - I tell you the elapsed seconds in each GET request. Use this table:
      0 to 59 seconds: running, step "1 search sources"
      60 to 179 seconds: running, step "2 read 12 sources"
      180 to 244 seconds: running, step "3 write the report"
      245 seconds or more: done, with a result field that has a short fake report title
    - When running, also show progress as a percent: elapsed divided by 245, rounded to a whole number.
    Say READY if you understand.

    You should see: The AI replies READY.

  2. Step 2
    Same chat

    Copy this request, paste it, and press Enter. This starts the job.

    Prompt
    POST /jobs
    {"topic": "Mobile payment use in rural Bangladesh"}

    You should see: A fast reply with status 202 and a job_id, for example job_001. It has no result yet.

    Example answer
    HTTP/1.1 202 Accepted
    {"job_id": "job_001", "status": "queued"}

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Copy this request, paste it, and press Enter. It asks for the status after 40 seconds.

    Prompt
    GET /jobs/job_001
    elapsed: 40 seconds

    You should see: Status running, step 1, and progress of about 16%.

    Example answer
    HTTP/1.1 200 OK
    {"job_id": "job_001", "status": "running", "step": "1 search sources", "progress": "16%"}

    Your answer will be worded differently.

  4. Step 4
    Same chat

    Copy this request, paste it, and press Enter. It asks for the status after 120 seconds.

    Prompt
    GET /jobs/job_001
    elapsed: 120 seconds

    You should see: Status running, step 2, and progress of about 49%.

    Example answer
    HTTP/1.1 200 OK
    {"job_id": "job_001", "status": "running", "step": "2 read 12 sources", "progress": "49%"}

    Your answer will be worded differently.

  5. Step 5
    Same chat

    Copy this request, paste it, and press Enter. It asks for the status after 245 seconds.

    Prompt
    GET /jobs/job_001
    elapsed: 245 seconds

    You should see: Status done, and a result with a short report title.

    Example answer
    HTTP/1.1 200 OK
    {"job_id": "job_001", "status": "done", "result": "Mobile Payments in Rural Bangladesh: A Short Report"}

    Your answer will be worded differently.

What just happened

A long task should not hold the web request open. The server replies at once with a job_id, which is like a ticket number, and does the work in the background. The client then asks again and again with the job_id. This is called polling. Here the first reply was instant, but the job takes 245 seconds, about 4 minutes.

You are done when You have one fast reply with a job_id, and 3 polls that show running, running, and done.

Check yourselfWhy should a task of more than about 10 seconds return a job_id instead of the result?Show answer

A long task can make the connection time out, and the user sees nothing. With a job_id, the work runs in the background, and the client can check the progress or come back later.

Try a harder versionAdd a rule: after 150 seconds, step 2 fails. Poll at 160 seconds. What status and message should the server return, and should the client retry?

Task 3

Stop double charges after a retry

Why this matters

When your app charges money for each AI run, a user who clicks Pay twice or refreshes the page must not pay twice. A failed long run must also not start from zero.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. The AI now plays a billing API.

    Prompt
    Act as an API server for report generation. Each new run costs 0.40 USD. Reply only with JSON for each request I send.
    Rules:
    - Every request has an idempotency_key and a body.
    - A new key makes a new run with a new run_id (run_1, run_2, ...). Reply with charged: 0.40 and replayed: false.
    - The same key with the same body returns the stored response again, with replayed: true and charged: 0. No new run.
    - The same key with a different body returns the error code 422 with a short message.
    Say READY if you understand.

    You should see: The AI replies READY.

  2. Step 2
    Same chat

    Copy this prompt, paste it, and press Enter. Write down the total money charged after the three requests.

    Prompt
    Handle these three requests in order. After the third, add a line with the total USD charged.
    1. idempotency_key: k-1001, body: {"report": "monthly sales"}
    2. idempotency_key: k-1001, body: {"report": "monthly sales"} (the user refreshed the page)
    3. idempotency_key: k-1001, body: {"report": "yearly sales"}

    You should see: Three JSON replies. Request 1 charges 0.40. Request 2 is a replay with charge 0. Request 3 is an error 422. The total is 0.40 USD.

    Example answer
    {"run_id": "run_1", "charged": 0.40, "replayed": false}
    {"run_id": "run_1", "charged": 0, "replayed": true}
    {"error": 422, "message": "Key k-1001 was already used with a different body."}
    Total charged: 0.40 USD

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Copy this prompt, paste it, and press Enter. Write down the cost with checkpoints and the cost without.

    Prompt
    Now a long run. Run run_77 has 5 steps. Each step costs 0.10 USD, and a failed step costs nothing. Steps 1, 2, and 3 finished and were already charged. The server saved a checkpoint after each one. Step 4 failed with a network error. The client retries with the same idempotency_key.
    1. Which steps do you run now, and what do you charge?
    2. What would the retry cost if there were no checkpoints?
    3. Name one thing the checkpoint must contain.

    You should see: With checkpoints, only steps 4 and 5 run: 0.20 USD. Without checkpoints, all 5 steps run: 0.50 USD.

    Example answer
    1. Run only steps 4 and 5. Steps 1 to 3 are in the checkpoint. Charge: 2 × 0.10 = 0.20 USD.
    2. With no checkpoints, all 5 steps run again: 5 × 0.10 = 0.50 USD.
    3. The checkpoint must contain the saved results of steps 1 to 3, and the number of the last finished step.

    Your answer will be worded differently.

What just happened

An idempotency key is a unique label that a client puts on a request. If the same request comes again, the server returns the saved answer and does not charge again. A checkpoint is a saved copy of the work done so far. After a failure, the server runs only the steps that are left: 2 steps (0.20 USD) instead of all 5 (0.50 USD).

You are done when You have a total of 0.40 USD for the three requests, and two costs for the retry: 0.20 USD and 0.50 USD.

Check yourselfA user refreshes the page, and the client sends the same request again. How does an idempotency key protect the user?Show answer

The server stores the response under the key. A second request with the same key gets the same response, with no new run and no new charge.

Try a harder versionAsk the AI what should happen if two requests with the same key arrive at the same moment, before the first one finishes.

Task 4

Simulate a gateway that tries backup providers

Why this matters

When you build a product on an AI provider, that provider will sometimes fail. A gateway sends the request to a backup provider, so your users still get an answer.

  1. Step 1
    New chat

    Copy this prompt, paste it, and press Enter. It is the normal prompt, for a provider that has tools.

    Prompt
    You can use one tool: get_weather(city: string), which returns live weather.
    User question: Do I need an umbrella in Sylhet today?
    Reply ONLY with the tool call as JSON, in the form {"tool": ..., "arguments": {...}}. Do not answer the question yet.

    You should see: Only a tool call as JSON, with no final answer. For example, get_weather with the city Sylhet.

    Example answer
    {"tool": "get_weather", "arguments": {"city": "Sylhet"}}

    Your answer will be worded differently.

  2. Step 2
    New chat

    Copy this prompt, paste it, and press Enter. It is the degraded prompt, for a provider with no tools.

    Prompt
    You have no tools and no live data. Answer from general knowledge only. Say clearly that you cannot check today's weather, and give the most useful advice you can.
    User question: Do I need an umbrella in Sylhet today?

    You should see: A plain answer that says the AI cannot check today's weather, and gives a useful tip.

    Example answer
    I cannot check today's weather, so I cannot be sure. Sylhet is one of the rainiest places in Bangladesh, especially in the monsoon, so carrying a small umbrella is a safe choice. Look at a weather app before you leave.

    Your answer will be worded differently.

  3. Step 3
    New chat

    Copy this prompt, paste it, and press Enter. The AI now plays a gateway. Check each row of the table against the rules.

    Prompt
    Act as an LLM gateway. The fallback chain is: Provider 1 (has tools), then Provider 2 (has tools), then Provider 3 (no tools, so use a degraded prompt that says: you have no tools, answer from general knowledge).
    Rules: on a 503 or 429 error, go to the next provider. On success, stop. Record the provider that answered, the number of attempts, and the cost.
    Costs per call: Provider 1 0.010 USD, Provider 2 0.006 USD, Provider 3 0.002 USD. Failed calls cost 0.
    Three requests came in:
    Request X: Provider 1 returns 200.
    Request Y: Provider 1 returns 503, Provider 2 returns 200.
    Request Z: Provider 1 returns 503, Provider 2 returns 429, Provider 3 returns 200.
    Give a table with: request, provider used, attempts, prompt used (normal or degraded), cost. Then add the total cost.

    You should see: A table where X uses Provider 1, Y uses Provider 2, and Z uses Provider 3 with the degraded prompt. The total cost is 0.018 USD.

    Example answer
    | Request | Provider used | Attempts | Prompt | Cost (USD) |
    | X | Provider 1 | 1 | normal | 0.010 |
    | Y | Provider 2 | 2 | normal | 0.006 |
    | Z | Provider 3 | 3 | degraded | 0.002 |
    Total cost: 0.010 + 0.006 + 0.002 = 0.018 USD

    Your answer will be worded differently.

What just happened

A provider can fail with an error such as 503 (the server is not ready) or 429 (too many requests). A provider can also lack a feature, such as tools. A gateway tries the next provider in a fixed order. For a provider with no tools, it uses a degraded prompt, which is simpler but still useful. Failed calls cost nothing here, so each request pays only for the provider that answered.

You are done when You have a tool call, an honest answer without tools, and a gateway table with a total of 0.018 USD.

Check yourselfWhy does a fallback chain use a different prompt for a provider that has no tool use?Show answer

A provider that cannot use tools would fail on the normal prompt. A degraded prompt, such as you have no tools, answer anyway, keeps the answer useful.

Try a harder versionAdd a rule: if the user is on a free plan, skip Provider 1. Then ask how a feature flag could turn this rule on for 10 percent of users first.

Watch · 6 videos

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.

  1. [1]Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
  2. [2]Wei et al., "Plan-and-Execute Agents" / "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — NeurIPS 2022
  3. [3]Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" — NeurIPS 2023
  4. [4]Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models"
  5. [5]Anthropic Engineering blog — "How we built our multi-agent research system" (2025)
  6. [6]LangGraph — official documentation (LangChain)
  7. [7]Temporal — "Durable Execution for LLM Agents"
  8. [8]Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" — 2023
  9. [9]Anthropic — "Introducing computer use, a new Claude 3.5 Sonnet capability" (Oct 2024)
  10. [10]Anthropic — Tool use / function calling documentation
  11. [11]OpenAI — Function calling guide
  12. [12]Anthropic — Model Context Protocol (MCP) specification
  13. [13]Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" — vLLM paper, SOSP 2023
  14. [14]Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" — 2023
  15. [15]llama.cpp — open-source GGUF-native inference (ggerganov)
  16. [16]Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" — MLSys 2024
  17. [17]Anthropic — Prompt caching documentation
  18. [18]Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — NeurIPS 2023
  19. [19]Es et al., "Ragas: Automated Evaluation of Information Retrieval Augmented Language Models" — 2023
  20. [20]Yue et al., "Large Language Model Cascades with Mixture of Thought Representations" — 2024
  21. [21]GPTCache — open-source semantic cache
  22. [22]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL" — 2025
  23. [23]Jiang et al., "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models" — EMNLP 2023
  24. [24]LiteLLM — unified interface for 100+ LLMs
  25. [25]OWASP — "Top 10 for LLM Applications" (LLM01 Prompt Injection, 2025)
  26. [26]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
  27. [27]NVIDIA NeMo Guardrails — open-source toolkit
  28. [28]Qwen2-VL Team, "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" — 2024
  29. [29]OpenAI — Realtime API documentation
  30. [30]OpenAI — Production best practices guide