LLMOps and Observability

Here is the flat log of one agent run (invented data). A span is one step of the run, with a start and an end.

10:02:11.000 REQUEST_START POST /chat user=u_204 prompt_id=support-v7
10:02:11.020 SPAN_START build_system_prompt
10:02:11.040 SPAN_END   build_system_prompt
10:02:11.040 SPAN_START llm_call_1 model=model-x
10:02:12.440 SPAN_END   llm_call_1 in_tokens=1200 out_tokens=60 decision="call tool search_orders"
10:02:12.440 SPAN_START tool_search_orders args={"customer":"u_204"}
10:02:16.940 SPAN_END   tool_search_orders status=timeout
10:02:16.940 SPAN_START llm_call_2 model=model-x
10:02:18.140 SPAN_END   llm_call_2 in_tokens=1350 out_tokens=80
10:02:18.150 REQUEST_END status=200 answer="Sorry, I could not check your order right now."

Tasks:
1. Draw this run as a trace: a tree of spans, with indentation. Show the duration of each span in seconds.
2. Name the slowest span and say what share of the total time it used.
3. Name the span where the real problem started.

The request returned status 200. A classic DevOps dashboard shows green for it. Explain in 3 short sentences what a plain 200 check misses for this run. Then list 4 fields that the trace should carry so that I can find this failure later.

Here are logged LLM requests (invented data). Prices per 1 million tokens: small model = 0.20 USD input and 0.80 USD output. Large model = 3.00 USD input and 15.00 USD output.

id | feature       | user | model | in_tokens | out_tokens | latency_s
1  | autocomplete  | u1   | small | 300       | 20         | 0.4
2  | autocomplete  | u2   | small | 280       | 25         | 0.5
3  | autocomplete  | u1   | small | 310       | 18         | 0.4
4  | report_writer | u3   | large | 12000     | 1500       | 14.2
5  | report_writer | u4   | large | 9000      | 1200       | 11.8
6  | report_writer | u3   | large | 15000     | 2000       | 18.5

Revenue: each autocomplete request earns 0.001 USD. Each report_writer request earns 0.04 USD.

Tasks:
1. Compute the cost of each request in USD. Show the formula once, then give a table.
2. Sum cost and revenue per feature.
3. Say which feature loses money, and by how much per request on average.

Now suppose we move report_writer to the small model. Use the same token counts for requests 4, 5, and 6. What is the new average cost per request? What must we measure before we make this change in production? Answer in under 100 words.

We test two versions of our support-bot prompt in production (invented data). Traffic is split by a hash of user_id. Version support-v7 is the current prompt. Version support-v8 is a new, longer prompt.

After day 1 (afternoon):
- v7: 40 thumbs-up out of 60 rated answers
- v8: 45 thumbs-up out of 60 rated answers

After day 7:
- v7: 290 thumbs-up out of 420 rated answers, average cost 0.8 cents per answer, average latency 2.1 s
- v8: 292 thumbs-up out of 420 rated answers, average cost 1.1 cents per answer, average latency 3.3 s

Tasks:
1. Compute the thumbs-up rate for each version at day 1 and at day 7.
2. Should we stop and ship v8 after day 1? Why or why not?
3. What should we do at day 7?
4. Which metrics should we have chosen before the test started?
Be short and direct.

We split traffic by a hash of user_id (a fixed rule that sends each user to the same version every time). Why do we do that, and not pick at random for every message? Also, why should every trace carry the prompt version ID, such as support-v8? Answer in 4 short sentences.

Our app asks a model to extract JSON like {"name": ..., "amount_bdt": ...} from a message. The vendor updated the model behind the same model name. We ran 5 production inputs through the old and new versions (invented data).

1. Input: "Rahim paid 1,200 taka"
   OLD: {"name":"Rahim","amount_bdt":1200}
   NEW: {"name":"Rahim","amount_bdt":1200}
2. Input: "Sumi sent 5k BDT"
   OLD: {"name":"Sumi","amount_bdt":5000}
   NEW: {"name":"Sumi","amount_bdt":"5k"}
3. Input: "Tanvir paid 300 taka and Mina paid 450 taka"
   OLD: [{"name":"Tanvir","amount_bdt":300},{"name":"Mina","amount_bdt":450}]
   NEW: {"name":"Tanvir","amount_bdt":300}
4. Input: "Lina paid 75 taka"
   OLD: {"name":"Lina","amount_bdt":75}
   NEW: Sure! Here is the JSON: {"name":"Lina","amount_bdt":75}
5. Input: "Arif paid 2,000 taka"
   OLD: {"name":"Arif","amount_bdt":2000}
   NEW: {"name":"Arif","amount_bdt":2000}

Tasks:
1. For each case say SAME or CHANGED, and name the type of change.
2. For each change, write one automatic check (a rule a program can test).
3. Say what share of live traffic we should send through both models, and what we should alert on.

A user clicked thumbs-down on this answer.
Input: "Karim paid 1.5 lakh taka" (1 lakh = 100,000)
Output: {"name":"Karim","amount_bdt":1.5}

Turn this failure into one test case for our eval dataset. Give the input, the expected output, and the rule that checks it. Then say in 2 sentences why a thousand real failures like this are more useful than synthetic tests.