Ask a team whether they can replay a trading day and get the same answer twice. The confident yes is rare, and it is almost always the answer from a team that has recently done the work to make it true. Everyone else discovers, on first attempt, that the replay diverges somewhere after lunch and nobody can say why.
That divergence is not a curiosity. It is the reason the team cannot confidently answer "would this change have done anything different yesterday?" — which is the only pre-deployment question that really matters.
Where determinism leaks
The leaks are mundane and there are only about six of them. Wall-clock reads inside logic that should be driven by event time. Map or set iteration where the order is unspecified. Unseeded randomness, usually in a jitter or tie-break. Concurrency, where two feeds are merged by arrival rather than by timestamp. Floating-point accumulation that depends on the order of additions. And silent dependence on external state — a config file, a reference-data lookup, a cache that was warm the first time.
Every one of these is easy to fix in isolation and hard to find retrospectively in a large system. The practical approach is to fix them at the boundary: make it structurally impossible for logic to read the clock, generate randomness, or iterate an unordered collection, rather than trying to audit every site where it might.
Event time is the only clock
The single highest-value rule is that business logic never calls the system clock. Time arrives as a field on an event, and the only component allowed to read the wall clock is the one at the edge that stamps incoming messages. In replay, that edge is swapped for one that reads timestamps from the capture. Everything downstream is unchanged and cannot tell the difference — which is exactly the point.
This also fixes a subtler class of production bug, where a system behaves correctly under normal load and incorrectly when it falls behind, because logic that reads "now" starts disagreeing with the events it is processing.
Capture what you actually received
A replay is only as good as the capture, and the common mistake is capturing the normalised, cleaned-up view rather than the raw bytes as they arrived. When a bug turns out to be in the normaliser, a capture of the normaliser’s output cannot reproduce it. Capture at the wire, keep the arrival timestamps, and normalise during replay exactly as you do live.
Storage is cheap relative to a day spent unable to reproduce an incident. Retain full-fidelity captures for the recent window, downsample beyond it, and treat the capture pipeline as production infrastructure rather than a debugging convenience.
How do I know determinism is holding?
Make it a test, not a discipline. A nightly job replays a fixed reference day twice and compares the full decision stream — every order, cancel and amend, with timestamps. Any difference fails the build. Because the check is cheap and automatic, drift gets caught by the change that introduced it rather than six weeks later during an incident review.
Once that job is green and stays green, everything else becomes tractable: strategy changes can be diffed against a known baseline, refactors can be proven behaviour-preserving, and "it works on my machine" stops being a category of argument.