Part 12
Multimodal Engineering
Vision (OCR-via-LLM, document AI), voice (STT/TTS, realtime APIs from OpenAI / LiveKit / Vapi), image generation (Flux, SDXL, Imagen, DALL-E; ControlNet, LoRA, ComfyUI), and video generation (Sora, Veo, Runway, Kling). Each section ships in two voices: the engineer's version (technical, with code & diagrams) and a layman's version (plain English with analogies). প্রতিটি টপিকের শেষে বাংলা ব্যাখ্যাও আছে।
1 section · 30 sources · three voices
01
Multimodal engineering (vision, voice, video)
Text was the first modality. In 2025, every production AI app combines at least two of vision, voice, image generation, and video generation. Multimodal engineering is the discipline of choosing models, latency budgets, and pipelines for each.
Vision: image understanding, OCR-via-LLM, document AI
Native multimodal models (GPT-4o, Claude 4, Gemini 2.5, Qwen3-VL) take images directly. The killer use cases:
- OCR-via-LLM: an LLM beats Tesseract on messy photos, handwritten notes, and structured forms. Often no separate OCR step needed.[28]
- Document AI: PDF understanding with layout awareness (Claude, Mistral OCR, Qwen2-VL).
- Chart/diagram extraction: ask the model to read a chart and emit JSON.
- Visual Q&A: "what's wrong with this screenshot?"
Voice agents: STT, TTS, realtime APIs
A voice pipeline is STT → LLM → TTS. Each has its own SOTA:
| Component | Options |
|---|---|
| STT (speech → text) | Whisper, Deepgram, AssemblyAI, ElevenLabs Scribe |
| TTS (text → speech) | ElevenLabs, Cartesia, OpenAI TTS, PlayHT |
| Realtime | OpenAI Realtime API, LiveKit Agents, Vapi, Retell, Daily |
Realtime APIs skip the chunky STT→LLM→TTS loop by streaming audio bidirectionally. Sub-500 ms turn-taking is now achievable. LiveKit and Vapi ship full agent frameworks on top of these.[29]
Image generation
| Model | Style |
|---|---|
| Flux (Black Forest Labs) | Open-weights availability, strong prompt fidelity |
| SDXL / SD3 (Stability) | Open, fine-tuneable with LoRA |
| Imagen 3 (Google) | Photorealism, available via Vertex |
| DALL-E 3 / GPT-Image-1 (OpenAI) | Polished, instruction-following |
| Ideogram, Recraft, Reve | Typography and design-oriented |
Workflows: ControlNet for structure (pose, depth, canny edges), LoRA for style or character consistency, inpainting for edits, IP-Adapter for reference images. ComfyUI lets you wire these into reproducible pipelines.
Video generation
| Model | Length / strength |
|---|---|
| Sora (OpenAI) | Up to 20 s, strong physics |
| Veo 3 (Google) | Native audio + 4K, prompt-adherent |
| Runway Gen-4 | Reference-image continuity for character |
| Kling 2.0 (Kuaishou) | Long-form, motion quality |
| Pika, Luma, Hailuo | Smaller, cheaper, faster |
ComfyUI, Replicate, Fal for media pipelines
- ComfyUI — node-based UI for Stable Diffusion / Flux workflows. The "Figma for diffusion." Production teams version-control the JSON graphs.
- Replicate — pay-per-second API for any open-source model. The fastest way to deploy something you found on Hugging Face.
- Fal — fast hosted inference for diffusion, with streaming partial results.
Practice · 4 tasks
Open any free AI chat: ChatGPT · Claude · Gemini
How practice works
- A new chat is a fresh conversation with no history. In ChatGPT, Claude, or Gemini, click New chat. It is usually at the top left.
- Same chat means: keep typing in the chat that is already open. The AI remembers what you sent before.
- To use a prompt: click Copy, click in the message box of the chat, paste it (Ctrl+V, or press and hold on a phone), and press Enter.
- AI answers are a little different every time. Your answer does not need to match the example word for word. Look for the same idea.
- Use made-up data only. Do not paste private information into a chat.
Task 1
Turn a messy photo into JSON
Why this matters
When you build an app that reads receipts or forms, the photos are often blurry. You must know that the AI can read them, but can also guess a character wrong.
Step 1
New chatTake a photo of a receipt, a sign, a menu, or a handwritten note. Attach it, paste this prompt, and press Enter.
Prompt Look at the attached image. Step 1: Write all the text you can read, in order. Step 2: Return JSON with these keys: - item_type (for example receipt, sign, menu, note) - title (the main heading, or null) - dates (list) - numbers (list of every number you can read, as written) - unclear (list of parts you could not read well) Do not guess. If a part is hard to read, put it in unclear and do not put it in the other keys.
You should see: The AI first writes the text it can read. Then it gives a JSON block with the keys
item_type,title,dates,numbers, andunclear.Step 2
New chatNo image upload in your chat? Do this step instead of step 1. Copy this prompt, paste it, and press Enter.
Prompt Below is the text that came from a blurry photo of a shop receipt. Some characters are wrong. --- RAHIM GR0CERY STOR3 12 Moy 2O25 Rice 5kg ....... 4SO Oil 1L ......... 18O Eggs (12) ...... ??5 Total .......... 8l5 --- Return JSON with these keys: - item_type - title - dates (list) - numbers (list of every number, as written) - unclear (list of parts you could not read well) Do not guess. If a part is hard to read, put it in unclear and do not put it in the other keys.
You should see: A JSON block.
item_typeis receipt. Theunclearlist names the broken parts, for example ??5.Example answer
{ "item_type": "receipt", "title": "RAHIM GR0CERY STOR3", "dates": ["12 Moy 2O25"], "numbers": ["5kg", "4SO", "1L", "18O", "12", "8l5"], "unclear": [ "Shop name: 0 and 3 may be O and E", "Date: Moy may be May, 2O25 may be 2025", "Rice price 4SO may be 450", "Oil price 18O may be 180", "Eggs price ??5 has missing digits", "Total 8l5 may be 815" ] }Your answer will be worded differently.
Step 3
Same chatCheck each JSON value against your photo, or against the receipt text. Write down every value that is wrong.
You should see: Most values are right. One or two may be wrong, or the AI may fix a broken character and not tell you.
Step 4
Same chatRead the
unclearlist. Write down if it names the parts that were hard to read.You should see: A good answer puts the hard parts in
unclear, and not in the other keys. For the typed receipt, that includes the egg price ??5.
What just happened
The AI can read text in a picture without a separate tool. People call this OCR (turning a picture of text into text). When you ask for JSON, a program can use the answer at once. But the AI can still guess a bad character. The unclear list makes it say where it is unsure, so you check only those parts.
You are done when You have one JSON answer, and you checked every value and the unclear list.
Check yourselfWhy add an unclear list and the rule do not guess to the prompt?Show answer
The AI can read a blurry character wrong and sound sure. The unclear list makes it report doubt, so you or your program can check those parts by hand.
Task 2
Compare a vague and a precise question
Why this matters
When you build a feature that checks screenshots, for example for a QA team, a vague question gives a polite answer that helps nobody. You need to know how to ask for real problems.
Step 1
New chatTake a screenshot of any app or web page. Attach it, paste this question, and press Enter. Save the answer.
Prompt What do you think of this image?
You should see: A general, polite answer about the look of the screen. It does not point to a clear problem.
Example answer
It looks like a simple checkout page. The layout is clean and the green button is easy to see. Some text could be easier to read, and the form could give clearer feedback. Overall it is a good start, but it needs a few small improvements.
Your answer will be worded differently.
Step 2
New chatAttach the same screenshot. Paste this prompt and press Enter.
Prompt This is a screenshot of an app screen. 1. Say in one sentence what the screen is for. 2. List up to 3 things that look wrong or confusing, such as cut-off text, low contrast, or unclear buttons. For each one, say where it is on the screen. 3. List what you cannot judge from this image. Be specific. Do not praise the design.
You should see: A one-sentence purpose, up to 3 problems that say where they are on the screen, and a list of what the AI cannot judge.
Step 3
New chatNo image upload in your chat? Do this step instead of steps 1 and 2. Paste this text and press Enter. Save the answer.
Prompt Here is a description of a screenshot, because I cannot upload it. The screen is a checkout page for a phone shop. At the top, a title says "Pay" in light grey on a white background. In the middle, a field named "Phone numbr" has a red border but no message. At the bottom, two buttons sit next to each other: a big green button with the text "OK" and a small grey button with the text "Cancel". The price "BDT 12,4" is cut off at the right edge. What do you think of this screen?
You should see: A general, polite answer, much like the example. It does not point to the cut-off price or the missing message.
Example answer
It looks like a simple checkout page. The layout is clean and the green button is easy to see. Some text could be easier to read, and the form could give clearer feedback. Overall it is a good start, but it needs a few small improvements.
Your answer will be worded differently.
Step 4
New chatStill no image upload? Open a new chat, paste this prompt, and press Enter.
Prompt Here is a description of a screenshot, because I cannot upload it. The screen is a checkout page for a phone shop. At the top, a title says "Pay" in light grey on a white background. In the middle, a field named "Phone numbr" has a red border but no message. At the bottom, two buttons sit next to each other: a big green button with the text "OK" and a small grey button with the text "Cancel". The price "BDT 12,4" is cut off at the right edge. 1. Say in one sentence what the screen is for. 2. List up to 3 things that look wrong or confusing. For each one, say where it is on the screen. 3. List what you cannot judge from this description. Be specific. Do not praise the design.
You should see: A one-sentence purpose, up to 3 problems with their place on the screen, and what the AI cannot judge. For example, the cut-off price at the right edge.
Example answer
1. A checkout page where the user pays for a phone. 2. Problems: - Top: the title "Pay" is light grey on white, so it is hard to read. - Middle: the "Phone numbr" field has a red border but no message, and its label has a typo. - Right edge: the price "BDT 12,4" is cut off, so the user cannot see the full price. 3. I cannot judge: what happens when a button is tapped, how the page looks on other screens, or the real colours.
Your answer will be worded differently.
Step 5
Same chatPut the two answers side by side. Write which one you could use to fix the screen, and why.
You should see: The second answer. It gives problems you can find on the screen and fix one by one.
What just happened
A question about a picture is still a prompt. A vague question lets the AI describe the picture in a general way. A precise question gives it a job: find problems, say where they are, and say what it cannot judge. The place of each problem lets you check the answer on the screenshot.
You are done when You have two answers for the same screen, one vague and one precise, and you know which one is more useful.
Check yourselfThe image is the same. Why does the precise prompt give a more useful answer?Show answer
The AI needs a task, a format, and a limit on what to report. A vague question leaves it free to describe the image. A precise question makes it find problems and say where they are.
Task 3
Add up the delays of a voice agent
Why this matters
When you build a phone voice agent, a caller who waits in silence thinks the call is broken. You must find which stage adds the most delay.
Step 1
New chatRead the six delays in this prompt. Write your guess of the total. Then paste the prompt and press Enter.
Prompt I build a voice agent for a bank help line. The design is speech-to-text (STT), then an LLM, then text-to-speech (TTS). After the caller stops talking, the delays are: - Wait to be sure the caller finished speaking: 600 ms - STT finishes the final text: 400 ms - LLM sends its first token: 700 ms - The system waits for the LLM to finish a full sentence before it sends text to TTS: 300 ms - TTS makes the first audio: 350 ms - Network to the caller: 150 ms 1. Add up the total delay before the caller hears the first sound. 2. Name the stage that is the biggest and the stage that is the easiest to cut. 3. Suggest two changes. For each, say how many ms it might save and what could go wrong. Use a small table.
You should see: A small table and the total 2,500 ms. The biggest stage is the LLM first token (700 ms). Check that your guess was close.
Example answer
1. Total: 600 + 400 + 700 + 300 + 350 + 150 = 2,500 ms (2.5 s). 2. Biggest: the LLM first token (700 ms). Easiest to cut: the wait for a full sentence (300 ms). 3. | Change | Saves | What could go wrong | | Stream LLM text to TTS in small pieces | about 300 ms | The voice may sound choppy | | Shorten the end-of-speech wait to 400 ms | about 200 ms | The agent may cut in when the caller pauses |
Your answer will be worded differently.
Step 2
Same chatCopy this prompt, paste it, and press Enter. Write down how many stages disappear.
Prompt Now my team thinks about a realtime audio API. Audio streams in both directions, and the model starts to speak after about 450 ms in total. 1. Which stages of my old design disappear? 2. Give me two things I might lose, for example control, logging, or the choice of voice. 3. Give me one rule to decide between the two designs for a bank help line. Answer in 8 lines or fewer.
You should see: About 3 stages disappear: the separate STT, the wait for a full sentence, and the separate TTS. The AI also names 2 things you may lose, for example control or logging.
Example answer
1. The separate STT stage, the wait for a full sentence, and the separate TTS stage disappear. One model hears and speaks. 2. You lose control (you cannot check the text between stages). Logging is harder, and the voice choice is smaller. 3. If every call needs a checked and saved transcript, use the pipeline. If a natural, fast talk matters most, use realtime.
Your answer will be worded differently.
Step 3
Same chatFrom the step 1 answer, pick the change that costs the least work. Write it down, with one reason.
You should see: For example: stream the LLM text to TTS, because you only change how the stages connect.
What just happened
A voice agent has three stages. STT turns speech into text, the LLM writes the reply, and TTS turns the reply into speech. Each stage waits for the one before it, so the delays add up: 2,500 ms here. A realtime API streams audio both ways, so most of the waits go away (about 450 ms, which is 2,050 ms less). The price is less control over each stage.
You are done when You have the total delay (2,500 ms), the biggest stage, and one change you would make first.
Check yourselfWhy is an STT, LLM, and TTS pipeline slower than a realtime audio API?Show answer
In a pipeline, each stage waits for the one before it to finish, so the delays add up. A realtime API streams audio in both directions, so the waits mostly go away.
Task 4
Pick the right image tool for four jobs
Why this matters
When you build an image feature, for example posters for a shop, the wrong tool wastes days. One tool keeps a pose. Another keeps a style. You must know which one fits the job.
Step 1
New chatRead the four jobs in this prompt. On paper, write your own tool for each one. Then paste the prompt and press Enter.
Prompt I use a diffusion image pipeline. These tools exist: ControlNet (structure such as pose, depth, or edges), LoRA (a small trained add-on for a style or a character), inpainting (edit one masked area), and IP-Adapter (use a reference image as a guide). For each job, pick the best tool, give one reason, and say what input I must have. Job 1: Make 20 posters with the same cartoon girl in the same drawing style. Job 2: Keep the exact pose of a person in my photo, but turn the photo into a painting. Job 3: Change only the sky in a product photo and leave the product as it is. Job 4: Make a new picture that has the colours and mood of one reference image, with no training. If two tools could work, say so.
You should see: One tool for each job. Most chats pick ControlNet for job 2, inpainting for job 3, and IP-Adapter for job 4. Job 1 gets LoRA, and maybe IP-Adapter too.
Example answer
Job 1: LoRA. It teaches the model the girl and the style, so all posters match. Input: example images of the girl to train on. IP-Adapter could also work. Job 2: ControlNet. It keeps the pose. Input: your photo (a pose map from it) and a text like "oil painting". Job 3: Inpainting. It edits only the masked area. Input: the photo, a mask over the sky, and a text for the new sky. Job 4: IP-Adapter. It uses the reference as a guide. Input: the reference image and a text.
Your answer will be worded differently.
Step 2
Same chatCompare the answer with your paper. Write down each job where you and the AI picked a different tool.
You should see: Most of your picks match the AI. For a different pick, read the reason of the AI and decide who is right.
Step 3
Same chatCopy this prompt, paste it, and press Enter. Check that each node uses the output of the node before it.
Prompt Take Job 2 and write the pipeline as a numbered list of nodes, as in ComfyUI. For each node, write its name, its input, and its output. Then give two reasons why a team should keep the pipeline as a saved graph file in version control.
You should see: A numbered list of 5 to 7 nodes, from loading the photo to saving the new image. Each node has a name, an input, and an output.
Example answer
1. Load Image: in = photo file, out = image. 2. Pose Detector: in = image, out = pose map. 3. Text Prompt: in = "an oil painting of a person", out = text guide. 4. ControlNet Apply: in = text guide + pose map, out = guided input. 5. Sampler: in = guided input, out = new image. 6. Save Image: in = new image, out = file. Why keep the graph file: the team can review changes like code, and can go back to an older version.
Your answer will be worded differently.
What just happened
Each tool solves a different problem. ControlNet keeps the structure of one image, such as a pose. A LoRA is a small add-on, trained to teach the model a style or a character. Inpainting edits only a masked area, and IP-Adapter uses a reference image as a guide with no training. In ComfyUI you connect these as nodes, and the graph is a file that a team can keep like code.
You are done when You have a tool for each of the 4 jobs, and a node list for job 2.
Check yourselfWhat is the difference between ControlNet and LoRA?Show answer
ControlNet controls the structure of one image, such as a pose or edges, from an input you give each time. A LoRA is a small trained add-on that teaches the model a style or a character, so it can repeat them in many images.
Watch · 6 videos
Qwen2-VL: The Best Open Source Vision Model for OCR & VQA AI Anytime
AI Dev 25 | Justin Uberti: Introduction to the OpenAI Realtime API DeepLearningAI
Build a Real-Time Voice Agent with LiveKit + Speechmatics (Step-by-Step) Speechmatics
How AI Image Generators Work (Stable Diffusion / Dall-E) - Computerphile Computerphile
How does OpenAI's Sora work? Jia-Bin Huang
ComfyUI Tutorial: How to Build and Experiment with AI pipelines – No Coding Needed Super Data Science
References
Sources
All citations resolve to primary papers (arXiv), vendor docs (OpenAI, Anthropic, Google), or official team blogs. IDs were verified by fetching each page and extracting the citation metadata.
- [1]Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (ICLR 2023)
- [2]Wei et al., "Plan-and-Execute Agents" / "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — NeurIPS 2022
- [3]Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" — NeurIPS 2023
- [4]Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models"
- [5]Anthropic Engineering blog — "How we built our multi-agent research system" (2025)
- [6]LangGraph — official documentation (LangChain)
- [7]Temporal — "Durable Execution for LLM Agents"
- [8]Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" — 2023
- [9]Anthropic — "Introducing computer use, a new Claude 3.5 Sonnet capability" (Oct 2024)
- [10]Anthropic — Tool use / function calling documentation
- [11]OpenAI — Function calling guide
- [12]Anthropic — Model Context Protocol (MCP) specification
- [13]Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" — vLLM paper, SOSP 2023
- [14]Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs" — 2023
- [15]llama.cpp — open-source GGUF-native inference (ggerganov)
- [16]Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" — MLSys 2024
- [17]Anthropic — Prompt caching documentation
- [18]Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — NeurIPS 2023
- [19]Es et al., "Ragas: Automated Evaluation of Information Retrieval Augmented Language Models" — 2023
- [20]Yue et al., "Large Language Model Cascades with Mixture of Thought Representations" — 2024
- [21]GPTCache — open-source semantic cache
- [22]Guo et al., "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL" — 2025
- [23]Jiang et al., "LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models" — EMNLP 2023
- [24]LiteLLM — unified interface for 100+ LLMs
- [25]OWASP — "Top 10 for LLM Applications" (LLM01 Prompt Injection, 2025)
- [26]Bai et al., "Constitutional AI: Harmlessness from AI Feedback" — Anthropic, 2022
- [27]NVIDIA NeMo Guardrails — open-source toolkit
- [28]Qwen2-VL Team, "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution" — 2024
- [29]OpenAI — Realtime API documentation
- [30]OpenAI — Production best practices guide