Finding the slow service in a trace

Computing · Computer Systems · ages 23-24

Name ______________________ Date ____________

What does a distributed trace connect?
- All requests merged into one system-wide latency average
- Instrumented operations for one request
- Only the longest operation recorded for each request
A request has serial, non-overlapping spans A = 10 ms, B = 90 ms and C = 20 ms. There is no other time. What does this trace establish?
- B explains every slow request.
- B accounts for 90 ms on this measured serial path.
- All spans ran together because they belong to one distributed trace.
The mean latency alone always reveals how slow the slowest requests are.
Circle one: True False
Three branches start simultaneously and reply after 20, 30 and 400 ms. The request waits for every reply and has no other work or overhead. How long does it take?
- 450 ms: sum of waits
- 30 ms: middle reply
- 400 ms: final reply
Of 100 requests, 98 take 10 ms and 2 take 500 ms. Use nearest-rank p99: sort the latencies and select rank ceil(0.99 × 100), counting from 1. What is p99?
- 500 ms at rank 99
- 10 ms, the sample minimum
- 255 ms, the mean of the two distinct values
A root span lasts 150 ms: 30 ms of its own processing and 120 ms waiting for a child call. The child span lasts 120 ms. Which next investigation is supported?
- Inspect the child's work and waits within the root's duration.
- Assign all 150 ms to root processing.
- Add the two span totals and investigate a supposed 270 ms request duration.
Four branches always finish together: all take 10 ms on 99% of runs and 200 ms on 1%. They start together, and wait-all adds no other time. How often does the fan-out take 200 ms?
- 4% of runs
- 1% of runs
- 100% of runs
A sample's p99 is 50 ms and its maximum is 2000 ms. A teammate says p99 proves no request exceeded 100 ms. Which response fits the measurements?
- Accept p99 as proof that every request stayed below 100 ms.
- Discard the maximum because p99 replaces individual observations.
- Use the maximum and count requests above 100 ms.

Answer key

For grown-ups. Fold this page away before handing over the rest.

Finding the slow service in a trace W1-mt_NT5tmumdfA-s1

Instrumented operations for one request · A trace links recorded operations belonging to one request. It can cross service boundaries, but uninstrumented work may be missing.
B accounts for 90 ms on this measured serial path. · B accounts for the largest measured interval on this serial path. Duration alone does not explain why it took that time or describe other requests.
False · The mean reports an average. Many fast requests can keep it relatively low while a smaller group takes much longer.
400 ms: final reply · All waits start together. With no other time to add, the request finishes when the slowest required branch replies at 400 ms.
500 ms at rank 99 · The selected rank is 99. Ranks 1 through 98 are 10 ms, while ranks 99 and 100 are 500 ms. P99 is therefore 500 ms.
Inspect the child's work and waits within the root's duration. · The root includes the 120 ms child wait. Inspecting the child can help explain that delay. Adding 150 and 120 would count the child interval twice.
1% of runs · The branch delays coincide. Their maximum is 200 ms on the same 1% of runs, not on four separate sets of runs.
Use the maximum and count requests above 100 ms. · The maximum already shows that at least one request exceeded 100 ms. Counting requests above that limit measures how often it happened; p99 alone cannot do that.