Most teams that say they "have CI" mean they have a build server. Something runs on push, a badge turns green, and once a fortnight somebody investigates why it went red. That is a start, but it is not what continuous integration is for — and in a trading business the gap between those two things is expensive.
The point of CI is not automation for its own sake. It is to shorten the distance between a change being made and that change being proven safe. In a trading system, "safe" has a specific meaning: the system still prices the same way, still sizes the same way, still refuses the same orders it refused yesterday, and still recovers correctly when a venue drops the connection.
Why does trading make continuous integration harder?
Three properties make a trading stack unusually awkward to test continuously. It is stateful — position, exposure and working orders carry across the boundary of any single request. It is time-dependent — the same input at 09:29:59 and 09:30:01 is two different events. And it is externally coupled — the venue, the market data feed and the clearing path are all systems you do not control and cannot spin up in a container.
The usual response is to give up on the hard parts and test only the pure functions: the indicator maths, the position-sizing formula, the message parsers. Those tests are worth having, and they catch approximately none of the failures that actually take a desk offline. The failures that matter live in sequencing, in reconnection, in partial fills, in the half-second where two components disagree about what the position is.
What a real pipeline covers
A pipeline worth the name has four layers, and each one buys something different. Unit tests cover the maths and run in seconds. Component tests run a single service against a stubbed venue and cover protocol handling, reconnect logic and order-state transitions. Replay tests run the whole stack against captured market data and assert that yesterday’s decisions reproduce exactly. Shadow runs put the new build alongside production on live data with orders suppressed, and diff the two.
The first two layers tell you the code is correct. The second two tell you the system is unchanged — which is the question you actually care about before a deploy. Most incidents we are called in to unpick were changes that were individually correct and collectively surprising.
How long should a pipeline take?
Unit and component layers should finish inside ten minutes, because that is roughly the limit of a developer’s attention before they context-switch and the feedback loop breaks. Replay and shadow layers can take hours; they run on merge and overnight, not on every push. Splitting the pipeline this way is the single highest-leverage change most teams can make, and it costs nothing but a config file.
If your full suite takes fifty minutes and runs on every push, people will start pushing less, batching changes, and merging on faith when the release window is closing. The pipeline has then made the system less safe than no pipeline at all, because it has taught the team to route around it.
The artefact rule
Build once, promote the same artefact. If the binary that ran in staging is not byte-for-byte the binary that runs in production, everything the pipeline proved in staging is an anecdote. This sounds obvious and is violated constantly, usually by a "quick rebuild with the prod config baked in" at the last step.
Configuration is the same principle applied to data: it should be versioned, reviewed and deployed exactly like code, because a fat-fingered risk limit is indistinguishable from a bug in the risk engine. Most of the config-related outages we have seen would have been caught by the same review process the team already applied faithfully to source files.
What good looks like
A team with working CI can answer, without a meeting: what is in production right now, what changed since yesterday, what proved it safe, and how long would it take to put it back. If any of those takes more than a minute to answer, the pipeline is documentation rather than infrastructure.
That is the standard we hold work to, and it is why CI is the first thing we look at on any engagement. Everything else — latency, throughput, new venues, new strategies — gets easier once changing the system is boring.