"Our latency is 40 microseconds" is not a statement about a system. It is a statement about one measurement of one segment of one path, on a day somebody chose to quote. It cannot be acted on, and it usually cannot be reproduced.
Measure the path, not the box
The number that matters is the interval between the event that should trigger a decision and the moment the resulting order leaves your network. That path crosses the NIC, the feed handler, the book builder, the strategy, the risk checks, the order gateway, and the wire out. Each of those is a segment with its own budget, and the total is what the venue sees.
Teams that only instrument the strategy will spend months shaving microseconds off code that accounts for a tenth of the path, while the risk check quietly takes a third of it. Instrument every hop with a common timebase before optimising anything.
Which percentile should I care about?
Not the mean, and usually not the median. The mean is dominated by the common case, which is already fine; what costs money is the tail, because the tail correlates with exactly the moments when the market is moving and your decision was worth something.
Track the 99th and the 99.9th, and track the maximum separately. A system with a 40-microsecond median and a 4-millisecond 99.9th is a system that is slow precisely when it matters, and the average will never tell you that.
What causes the tail
Tail latency is rarely about algorithmic complexity. It comes from garbage collection, from page faults on memory that was not pre-touched, from a lock contended once every few thousand operations, from a log write that hit a synchronous flush, from a socket buffer filling, or from the operating system scheduling your thread off a core at the wrong moment.
The fixes are correspondingly unglamorous: pre-allocate, pin threads to isolated cores, keep the hot path free of allocation and of anything that can block, and make logging asynchronous with a bounded queue that drops rather than stalls. None of this is clever. All of it is the difference between a stated latency and an actual one.
Budget, then enforce
Write the budget down: feed handler 3µs, book build 5µs, strategy 8µs, risk 4µs, gateway 6µs. Then make the pipeline fail when a segment exceeds its allocation on the reference replay. A latency budget that is not enforced by CI decays within a quarter, because every individual regression is small enough to wave through.
The budget also makes the trade-off explicit when someone asks for a new pre-trade check. The question stops being "can we add this?" and becomes "which segment gives up two microseconds?" — which is a conversation the desk and the engineers can actually have.