Tail latency is a distribution, not a number

A dashboard shows p99 at 40 ms. Users complain about two-second stalls. Both are true, and the gap between them is the difference between a summary and a distribution.

How p99 lies

Percentiles are computed over a window. If traffic is bursty, the window is dominated by the quiet periods, and the burst that caused the stall is a handful of samples that sit above p99 and never show up. Averaging percentiles across instances makes it worse: the average of ten p99s is not the p99 of the combined traffic and is usually lower.

What I keep instead

A small fixed histogram with log-spaced buckets, one per endpoint, reset on scrape. Sixteen buckets from 1 ms to 32 s cover everything I have ever cared about, and the whole type is an array of counters.

type Hist struct{ b [16]uint64 }

func (h *Hist) Observe(d time.Duration) {
    ms := d.Milliseconds()
    i := bits.Len64(uint64(ms)) // log2 bucket
    if i > 15 { i = 15 }
    atomic.AddUint64(&h.b[i], 1)
}

Histograms merge by adding buckets, so aggregating across instances is correct. Any percentile can be read off after the fact. And the top bucket answers the question the p99 could not: how many requests took longer than a second, as a count, not a rate.

The habit

Whenever a metric is an average or a percentile, ask what distribution it summarised and whether you could get the distribution instead. Usually you can, and usually it is cheaper than the summary.