Observability and Tail Latency Across Services
Distributed traces connect instrumented operations for one request. Span relationships help locate measured delays without proving their cause. Means and percentiles summarize different aspects of latency. Simultaneous fan-out that waits for every reply waits for its slowest branch, plus any other work.
What a learner can do afterwards
- Interpret serial and nested spans without double-counting included time or claiming an unproven cause
- Compare mean latency with nearest-rank p99 and explain why a percentile can miss rarer delays
- Calculate simultaneous wait-all fan-out duration under stated assumptions and explain why related branch delays affect tail amplification
1 · Read
A distributed trace connects instrumented operations for one request, possibly across services. Each span records an operation's start and end. Missing instrumentation can leave gaps. Serial spans do not overlap: a span from 10 to 50 ms lasts 40 ms. Compare durations to locate measured delays. With overlapping work, follow the waits that delay completion. Duration alone does not prove a cause.
These spans run one after another. Their durations are 10, 40 and 20 ms.
A parent span lasts 80 ms and includes a 60 ms child call. Their durations do not add to 140 ms: the child interval is already inside the parent interval. A long parent can include time spent waiting for a child. Inspect the child's work and waits before assigning all the parent's time to its own processing. The timing alone may not explain why the child was slow.
The mean is total latency divided by request count. For nearest-rank p99, sort n observed latencies and select rank ceil(0.99 × n), counting from 1. Here ceil means round upward to an integer. At least 99% of observations are at or below that value. P99 is a cutoff, not the average of the slowest 1%. Other percentile conventions can give different results.
Of 100 requests, 95 take 12 ms and 5 take 180 ms. The mean is (95 × 12 + 5 × 180) / 100 = 20.4 ms. Nearest-rank p99 selects sorted rank 99, which is 180 ms. The mean describes the average but does not reveal every slow request. Compare summaries with the distribution and the time period measured.
Bar heights count requests, not milliseconds.
If branches start together and a request waits for every reply, its waiting interval is the largest branch duration. Add setup or later processing separately. Serial calls instead add their durations. These rules need the stated timing assumptions; the term fan-out alone does not specify start times, overhead or whether every reply is required.
P99 is not a maximum. With 100 requests, rank 99 can be fast even if rank 100 is very slow. To check whether any request exceeded a limit, inspect the maximum. To measure how often it happened, count requests above that limit. A low p99 alone cannot prove that no request was slow.
More branches can raise the chance that at least one is slow, but the increase depends on how branch delays are related. If all branches always finish together, their maximum equals each branch's duration. They share the same slow runs, so adding their slow-run percentages would count those runs repeatedly. Adding branches does not always increase the frequency of slow requests.
Read span relationships before assigning causes. Compare means, defined percentiles and rare-delay counts. Simultaneous wait-all fan-out waits for the largest branch duration, plus other work.
Original lesson, questions, worked solutions and flashcards about request traces and latency, written for this dataset.
2 · Watch
به انگلیسیTake it off screen
Where it sits
Optional connections
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.