Observability and Tail Latency Across Services
Once a request touches a dozen services, an average tells nobody anything. Traces follow one request across process boundaries, percentiles expose the slow tail, and a request that fans out inherits the worst of everything it waited on.
What a learner can do afterwards
- Read a distributed trace and say which span holds the latency
- Explain why the mean hides a problem that the ninety-ninth percentile shows
- Say why a request fanning out to many services is slower than any one of them typically is
1 · Read
One request can pass through a dozen services. A distributed trace follows that single request across every process boundary, and each span records what one service did with it. The span holding the most time is where your latency lives.
An average over all requests tells you almost nothing, because fast requests drown out the slow ones. The ninety-ninth percentile shows the slow tail instead: the latency that only the worst one percent of requests feel. When the mean looks fine but the tail is bad, a few users suffer while the average smiles.
Picture a request that fans out to several services and waits for every reply before it can answer. Each service is usually quick, but one of them is slow this time. The whole request inherits the slowest wait, so the fan-out is slower than any one service typically is.
When a request is slow, line the spans up and blame the longest one. Never trust the average to reveal the problem; check the tail first.
Follow one request with a trace, judge slowness by the tail percentile, and remember that waiting on many services means waiting on the slowest.
2 · Watch
Take it off screen
Where it sits
Learn first
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.