Skip to content

Part 12

Multimodal Engineering

Vision (OCR-via-LLM, document AI), voice (STT/TTS, realtime APIs from OpenAI / LiveKit / Vapi), image generation (Flux, SDXL, Imagen, DALL-E; ControlNet, LoRA, ComfyUI), and video generation (Sora, Veo, Runway, Kling). Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।

1 section · 30 sources · three voices

01

Multimodal engineering (vision, voice, video)

Text was the first modality. In 2025, every production AI app combines at least two of vision, voice, image generation, and video generation. Multimodal engineering is the discipline of choosing models, latency budgets, and pipelines for each.

Vision: image understanding, OCR-via-LLM, document AI

Native multimodal models (GPT-4o, Claude 4, Gemini 2.5, Qwen3-VL) take images directly. The killer use cases:

  • OCR-via-LLM: an LLM beats Tesseract on messy photos, handwritten notes, and structured forms. Often no separate OCR step needed.[28]
  • Document AI: PDF understanding with layout awareness (Claude, Mistral OCR, Qwen2-VL).
  • Chart/diagram extraction: ask the model to read a chart and emit JSON.
  • Visual Q&A: "what's wrong with this screenshot?"

Voice agents: STT, TTS, realtime APIs

A voice pipeline is STT → LLM → TTS. Each has its own SOTA:

ComponentOptions
STT (speech → text)Whisper, Deepgram, AssemblyAI, ElevenLabs Scribe
TTS (text → speech)ElevenLabs, Cartesia, OpenAI TTS, PlayHT
RealtimeOpenAI Realtime API, LiveKit Agents, Vapi, Retell, Daily

Realtime APIs skip the chunky STT→LLM→TTS loop by streaming audio bidirectionally. Sub-500 ms turn-taking is now achievable. LiveKit and Vapi ship full agent frameworks on top of these.[29]

Image generation

ModelStyle
Flux (Black Forest Labs)Open-weights availability, strong prompt fidelity
SDXL / SD3 (Stability)Open, fine-tuneable with LoRA
Imagen 3 (Google)Photorealism, available via Vertex
DALL-E 3 / GPT-Image-1 (OpenAI)Polished, instruction-following
Ideogram, Recraft, ReveTypography and design-oriented

Workflows: ControlNet for structure (pose, depth, canny edges), LoRA for style or character consistency, inpainting for edits, IP-Adapter for reference images. ComfyUI lets you wire these into reproducible pipelines.

Video generation

ModelLength / strength
Sora (OpenAI)Up to 20 s, strong physics
Veo 3 (Google)Native audio + 4K, prompt-adherent
Runway Gen-4Reference-image continuity for character
Kling 2.0 (Kuaishou)Long-form, motion quality
Pika, Luma, HailuoSmaller, cheaper, faster

ComfyUI, Replicate, Fal for media pipelines

  • ComfyUI — node-based UI for Stable Diffusion / Flux workflows. The "Figma for diffusion." Production teams version-control the JSON graphs.
  • Replicate — pay-per-second API for any open-source model. The fastest way to deploy something you found on Hugging Face.
  • Fal — fast hosted inference for diffusion, with streaming partial results.

Practice · 4 tasks

Open any free AI chat: ChatGPT · Claude · Gemini

How practice works
  • A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
  • Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
  • To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
  • AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
  • Use made-up data only. Do not paste private information into a chat.

Task 1

Turn a messy photo into JSON

Why this matters

When you build an app that reads receipts or forms, the photos are often blurry. You must know that the AI can read them, but can also guess a character wrong.

  1. Step 1
    New chat

    Take a photo of a receipt, a sign, a menu, or a handwritten note. Attach it, paste this prompt, and press Enter.

    Prompt
    Look at the attached image.
    Step 1: Write all the text you can read, in order.
    Step 2: Return JSON with these keys:
    - item_type (for example receipt, sign, menu, note)
    - title (the main heading, or null)
    - dates (list)
    - numbers (list of every number you can read, as written)
    - unclear (list of parts you could not read well)
    Do not guess. If a part is hard to read, put it in unclear and do not put it in the other keys.

    You should see: The AI first writes the text it can read. Then it gives a JSON block with the keys item_type, title, dates, numbers, and unclear.

  2. Step 2
    New chat

    No image upload in your chat? Do this step instead of step 1. Copy this prompt, paste it, and press Enter.

    Prompt
    Below is the text that came from a blurry photo of a shop receipt. Some characters are wrong.
    
    ---
    RAHIM GR0CERY STOR3
    12 Moy 2O25
    Rice 5kg ....... 4SO
    Oil 1L ......... 18O
    Eggs (12) ...... ??5
    Total .......... 8l5
    ---
    
    Return JSON with these keys:
    - item_type
    - title
    - dates (list)
    - numbers (list of every number, as written)
    - unclear (list of parts you could not read well)
    Do not guess. If a part is hard to read, put it in unclear and do not put it in the other keys.

    You should see: A JSON block. item_type is receipt. The unclear list names the broken parts, for example ??5.

    Example answer
    {
      "item_type": "receipt",
      "title": "RAHIM GR0CERY STOR3",
      "dates": ["12 Moy 2O25"],
      "numbers": ["5kg", "4SO", "1L", "18O", "12", "8l5"],
      "unclear": [
        "Shop name: 0 and 3 may be O and E",
        "Date: Moy may be May, 2O25 may be 2025",
        "Rice price 4SO may be 450",
        "Oil price 18O may be 180",
        "Eggs price ??5 has missing digits",
        "Total 8l5 may be 815"
      ]
    }

    Your answer will be worded differently.

  3. Step 3
    Same chat

    Check each JSON value against your photo, or against the receipt text. Write down every value that is wrong.

    You should see: Most values are right. One or two may be wrong, or the AI may fix a broken character and not tell you.

  4. Step 4
    Same chat

    Read the unclear list. Write down if it names the parts that were hard to read.

    You should see: A good answer puts the hard parts in unclear, and not in the other keys. For the typed receipt, that includes the egg price ??5.

What just happened

The AI can read text in a picture without a separate tool. People call this OCR (turning a picture of text into text). When you ask for JSON, a program can use the answer at once. But the AI can still guess a bad character. The unclear list makes it say where it is unsure, so you check only those parts.

You are done when You have one JSON answer, and you checked every value and the unclear list.

Check yourselfWhy add an unclear list and the rule do not guess to the prompt?Show answer

The AI can read a blurry character wrong and sound sure. The unclear list makes it report doubt, so you or your program can check those parts by hand.

Try a harder versionIn the receipt, the total is 815, rice is 450, and oil is 180. Ask the AI for the egg price. Check the maths: 815 − 450 − 180 = 185. Should you trust that guess?

Task 2

Compare a vague and a precise question

Why this matters

When you build a feature that checks screenshots, for example for a QA team, a vague question gives a polite answer that helps nobody. You need to know how to ask for real problems.

  1. Step 1
    New chat

    Take a screenshot of any app or web page. Attach it, paste this question, and press Enter. Save the answer.

    Prompt
    What do you think of this image?

    You should see: A general, polite answer about the look of the screen. It does not point to a clear problem.

    Example answer
    It looks like a simple checkout page. The layout is clean and the green button is easy to see. Some text could be easier to read, and the form could give clearer feedback. Overall it is a good start, but it needs a few small improvements.

    Your answer will be worded differently.

  2. Step 2
    New chat

    Attach the same screenshot. Paste this prompt and press Enter.

    Prompt
    This is a screenshot of an app screen.
    1. Say in one sentence what the screen is for.
    2. List up to 3 things that look wrong or confusing, such as cut-off text, low contrast, or unclear buttons. For each one, say where it is on the screen.
    3. List what you cannot judge from this image.
    Be specific. Do not praise the design.

    You should see: A one-sentence purpose, up to 3 problems that say where they are on the screen, and a list of what the AI cannot judge.

  3. Step 3
    New chat

    No image upload in your chat? Do this step instead of steps 1 and 2. Paste this text and press Enter. Save the answer.

    Prompt
    Here is a description of a screenshot, because I cannot upload it.
    The screen is a checkout page for a phone shop. At the top, a title says "Pay" in light grey on a white background. In the middle, a field named "Phone numbr" has a red border but no message. At the bottom, two buttons sit next to each other: a big green button with the text "OK" and a small grey button with the text "Cancel". The price "BDT 12,4" is cut off at the right edge.
    
    What do you think of this screen?

    You should see: A general, polite answer, much like the example. It does not point to the cut-off price or the missing message.

    Example answer
    It looks like a simple checkout page. The layout is clean and the green button is easy to see. Some text could be easier to read, and the form could give clearer feedback. Overall it is a good start, but it needs a few small improvements.

    Your answer will be worded differently.

  4. Step 4
    New chat

    Still no image upload? Open a new chat, paste this prompt, and press Enter.

    Prompt
    Here is a description of a screenshot, because I cannot upload it.
    The screen is a checkout page for a phone shop. At the top, a title says "Pay" in light grey on a white background. In the middle, a field named "Phone numbr" has a red border but no message. At the bottom, two buttons sit next to each other: a big green button with the text "OK" and a small grey button with the text "Cancel". The price "BDT 12,4" is cut off at the right edge.
    
    1. Say in one sentence what the screen is for.
    2. List up to 3 things that look wrong or confusing. For each one, say where it is on the screen.
    3. List what you cannot judge from this description.
    Be specific. Do not praise the design.

    You should see: A one-sentence purpose, up to 3 problems with their place on the screen, and what the AI cannot judge. For example, the cut-off price at the right edge.

    Example answer
    1. A checkout page where the user pays for a phone.
    2. Problems:
    - Top: the title "Pay" is light grey on white, so it is hard to read.
    - Middle: the "Phone numbr" field has a red border but no message, and its label has a typo.
    - Right edge: the price "BDT 12,4" is cut off, so the user cannot see the full price.
    3. I cannot judge: what happens when a button is tapped, how the page looks on other screens, or the real colours.

    Your answer will be worded differently.

  5. Step 5
    Same chat

    Put the two answers side by side. Write which one you could use to fix the screen, and why.

    You should see: The second answer. It gives problems you can find on the screen and fix one by one.

What just happened

A question about a picture is still a prompt. A vague question lets the AI describe the picture in a general way. A precise question gives it a job: find problems, say where they are, and say what it cannot judge. The place of each problem lets you check the answer on the screenshot.

You are done when You have two answers for the same screen, one vague and one precise, and you know which one is more useful.

Check yourselfThe image is the same. Why does the precise prompt give a more useful answer?Show answer

The AI needs a task, a format, and a limit on what to report. A vague question leaves it free to describe the image. A precise question makes it find problems and say where they are.

Try a harder versionAdd this to the precise prompt: Return the problems as JSON with the keys problem, location, and severity. Does your program now get an answer it can read?

Task 3

Add up the delays of a voice agent

Why this matters

When you build a phone voice agent, a caller who waits in silence thinks the call is broken. You must find which stage adds the most delay.

  1. Step 1
    New chat

    Read the six delays in this prompt. Write your guess of the total. Then paste the prompt and press Enter.

    Prompt
    I build a voice agent for a bank help line. The design is speech-to-text (STT), then an LLM, then text-to-speech (TTS). After the caller stops talking, the delays are:
    - Wait to be sure the caller finished speaking: 600 ms
    - STT finishes the final text: 400 ms
    - LLM sends its first token: 700 ms
    - The system waits for the LLM to finish a full sentence before it sends text to TTS: 300 ms
    - TTS makes the first audio: 350 ms
    - Network to the caller: 150 ms
    
    1. Add up the total delay before the caller hears the first sound.
    2. Name the stage that is the biggest and the stage that is the easiest to cut.
    3. Suggest two changes. For each, say how many ms it might save and what could go wrong.
    Use a small table.

    You should see: A small table and the total 2,500 ms. The biggest stage is the LLM first token (700 ms). Check that your guess was close.

    Example answer
    1. Total: 600 + 400 + 700 + 300 + 350 + 150 = 2,500 ms (2.5 s).
    2. Biggest: the LLM first token (700 ms). Easiest to cut: the wait for a full sentence (300 ms).
    3.
    | Change | Saves | What could go wrong |
    | Stream LLM text to TTS in small pieces | about 300 ms | The voice may sound choppy |
    | Shorten the end-of-speech wait to 400 ms | about 200 ms | The agent may cut in when the caller pauses |

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Copy this prompt, paste it, and press Enter. Write down how many stages disappear.

    Prompt
    Now my team thinks about a realtime audio API. Audio streams in both directions, and the model starts to speak after about 450 ms in total.
    1. Which stages of my old design disappear?
    2. Give me two things I might lose, for example control, logging, or the choice of voice.
    3. Give me one rule to decide between the two designs for a bank help line.
    Answer in 8 lines or fewer.

    You should see: About 3 stages disappear: the separate STT, the wait for a full sentence, and the separate TTS. The AI also names 2 things you may lose, for example control or logging.

    Example answer
    1. The separate STT stage, the wait for a full sentence, and the separate TTS stage disappear. One model hears and speaks.
    2. You lose control (you cannot check the text between stages). Logging is harder, and the voice choice is smaller.
    3. If every call needs a checked and saved transcript, use the pipeline. If a natural, fast talk matters most, use realtime.

    Your answer will be worded differently.

  3. Step 3
    Same chat

    From the step 1 answer, pick the change that costs the least work. Write it down, with one reason.

    You should see: For example: stream the LLM text to TTS, because you only change how the stages connect.

What just happened

A voice agent has three stages. STT turns speech into text, the LLM writes the reply, and TTS turns the reply into speech. Each stage waits for the one before it, so the delays add up: 2,500 ms here. A realtime API streams audio both ways, so most of the waits go away (about 450 ms, which is 2,050 ms less). The price is less control over each stage.

You are done when You have the total delay (2,500 ms), the biggest stage, and one change you would make first.

Check yourselfWhy is an STT, LLM, and TTS pipeline slower than a realtime audio API?Show answer

In a pipeline, each stage waits for the one before it to finish, so the delays add up. A realtime API streams audio in both directions, so the waits mostly go away.

Try a harder versionChange one number in the first prompt, for example make the LLM first token 1,500 ms. Send it in a new chat and ask which change now saves the most time.

Task 4

Pick the right image tool for four jobs

Why this matters

When you build an image feature, for example posters for a shop, the wrong tool wastes days. One tool keeps a pose. Another keeps a style. You must know which one fits the job.

  1. Step 1
    New chat

    Read the four jobs in this prompt. On paper, write your own tool for each one. Then paste the prompt and press Enter.

    Prompt
    I use a diffusion image pipeline. These tools exist: ControlNet (structure such as pose, depth, or edges), LoRA (a small trained add-on for a style or a character), inpainting (edit one masked area), and IP-Adapter (use a reference image as a guide).
    
    For each job, pick the best tool, give one reason, and say what input I must have.
    Job 1: Make 20 posters with the same cartoon girl in the same drawing style.
    Job 2: Keep the exact pose of a person in my photo, but turn the photo into a painting.
    Job 3: Change only the sky in a product photo and leave the product as it is.
    Job 4: Make a new picture that has the colours and mood of one reference image, with no training.
    If two tools could work, say so.

    You should see: One tool for each job. Most chats pick ControlNet for job 2, inpainting for job 3, and IP-Adapter for job 4. Job 1 gets LoRA, and maybe IP-Adapter too.

    Example answer
    Job 1: LoRA. It teaches the model the girl and the style, so all posters match. Input: example images of the girl to train on. IP-Adapter could also work.
    Job 2: ControlNet. It keeps the pose. Input: your photo (a pose map from it) and a text like "oil painting".
    Job 3: Inpainting. It edits only the masked area. Input: the photo, a mask over the sky, and a text for the new sky.
    Job 4: IP-Adapter. It uses the reference as a guide. Input: the reference image and a text.

    Your answer will be worded differently.

  2. Step 2
    Same chat

    Compare the answer with your paper. Write down each job where you and the AI picked a different tool.

    You should see: Most of your picks match the AI. For a different pick, read the reason of the AI and decide who is right.

  3. Step 3
    Same chat

    Copy this prompt, paste it, and press Enter. Check that each node uses the output of the node before it.

    Prompt
    Take Job 2 and write the pipeline as a numbered list of nodes, as in ComfyUI. For each node, write its name, its input, and its output. Then give two reasons why a team should keep the pipeline as a saved graph file in version control.

    You should see: A numbered list of 5 to 7 nodes, from loading the photo to saving the new image. Each node has a name, an input, and an output.

    Example answer
    1. Load Image: in = photo file, out = image.
    2. Pose Detector: in = image, out = pose map.
    3. Text Prompt: in = "an oil painting of a person", out = text guide.
    4. ControlNet Apply: in = text guide + pose map, out = guided input.
    5. Sampler: in = guided input, out = new image.
    6. Save Image: in = new image, out = file.
    Why keep the graph file: the team can review changes like code, and can go back to an older version.

    Your answer will be worded differently.

What just happened

Each tool solves a different problem. ControlNet keeps the structure of one image, such as a pose. A LoRA is a small add-on, trained to teach the model a style or a character. Inpainting edits only a masked area, and IP-Adapter uses a reference image as a guide with no training. In ComfyUI you connect these as nodes, and the graph is a file that a team can keep like code.

You are done when You have a tool for each of the 4 jobs, and a node list for job 2.

Check yourselfWhat is the difference between ControlNet and LoRA?Show answer

ControlNet controls the structure of one image, such as a pose or edges, from an input you give each time. A LoRA is a small trained add-on that teaches the model a style or a character, so it can repeat them in many images.

Try a harder versionAdd job 5: put a person from a photo on a new background and keep the face the same. Ask which tools you need, and in what order.

Watch · 6 videos

References

Sources

All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.

  1. [1]Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
  2. [2]Wei et al., "Plan-and-Execute Agents" / "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — NeurIPS 2022
  3. [3]Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" — NeurIPS 2023
  4. [4]Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models"
  5. [5]Anthropic Engineering blog — "How we built our multi-agent research system" (2025)
  6. [6]LangGraph — official documentation (LangChain)
  7. [7]Temporal — "Durable Execution for LLM Agents"
  8. [8]Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" — 2023
  9. [9]Anthropic — "Introducing computer use, a new Claude 3.5 Sonnet capability" (Oct 2024)
  10. [10]Anthropic — Tool use / function calling documentation
  11. [11]OpenAI — Function calling guide
  12. [12]Anthropic — Model Context Protocol (MCP) specification
  13. [13]Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" — vLLM paper, SOSP 2023
  14. [14]Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" — 2023
  15. [15]llama.cpp — open-source GGUF-native inference (ggerganov)
  16. [16]Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" — MLSys 2024
  17. [17]Anthropic — Prompt caching documentation
  18. [18]Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — NeurIPS 2023
  19. [19]Es et al., "Ragas: Automated Evaluation of Information Retrieval Augmented Language Models" — 2023
  20. [20]Yue et al., "Large Language Model Cascades with Mixture of Thought Representations" — 2024
  21. [21]GPTCache — open-source semantic cache
  22. [22]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL" — 2025
  23. [23]Jiang et al., "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models" — EMNLP 2023
  24. [24]LiteLLM — unified interface for 100+ LLMs
  25. [25]OWASP — "Top 10 for LLM Applications" (LLM01 Prompt Injection, 2025)
  26. [26]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
  27. [27]NVIDIA NeMo Guardrails — open-source toolkit
  28. [28]Qwen2-VL Team, "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" — 2024
  29. [29]OpenAI — Realtime API documentation
  30. [30]OpenAI — Production best practices guide