Separate Protocol Logic from I/O for Deterministic Tests • Somtochi Onyekwere • Bug Bash EU 2026
Somtochi Onyekwere of Fly.io explains how separating protocol logic from I/O lets distributed protocols run under a deterministic, virtual-time simulation, with injected randomness and failures, to test a 2,000-node gossip broadcast before it reaches production.
Why distributed protocols are hard to test
Distributed systems carry a lot of hidden state. Consensus and broadcast protocols depend on messages that get delayed, reordered, or lost, on timers that fire at unpredictable moments, and on nodes that go down mid-operation. Integration tests make this worse. Simulating a 2,000-node cluster means standing up thousands of containers, which is expensive, slow, and rarely reproducible, because each run plays out differently.
Separating protocol logic from I/O
At Fly.io, the team was building Corrosion, their SQLite replication system, and wanted to change its broadcast protocol. The protocol as written in papers is simple, but real implementations tangle that logic with thread spawning, socket writes, timers, and sleeps. Sleeps in particular introduce flakiness into tests.
The approach was to make the protocol a state machine. It receives events, such as a message arriving, a peer joining or leaving, or a timer firing, and returns a list of effects it wants carried out. The protocol never opens a socket, reads a clock, or schedules a timer itself. Effects are defined by an interface, a trait in Rust, that the caller implements: send a message to a peer, send to a number of peers, deliver a payload to the application, notify the application, or schedule a timer. When the timer fires, the caller passes that event back into the protocol. Randomness is injected the same way. In production it comes from the OS, while in tests it comes from a seed, so every run can be reproduced exactly.
The example used is Plumtree, a gossip protocol. Eager peers receive the full payload and forward it onward, while lazy peers receive only the message ID and can request the payload if they have not seen it. The specific protocol matters less than the boundary it illustrates. In Corrosion, the glue code is small. Production batches outgoing messages in short windows for efficiency, but another user of the library could make a different choice without touching the protocol.
Deterministic simulation in tests
Because the effects are abstracted, the test runtime can differ from production. Instead of threads and sockets, the test keeps a binary heap of events ordered by time. A single loop pops the earliest event, applies it to the target node’s protocol state, and enqueues whatever effects result. A map from node ID to protocol state holds all 2,000 nodes, and the whole simulation runs on one thread with no concurrency to manage.
Time is virtual. Rather than waiting for a timer, the loop jumps straight to the next scheduled event, so an entire cluster can run through its protocol in very little real time. Failures are modeled as events too. A node can be taken down by dropping any messages sent to it and notifying every other node that it is gone, then brought back at a chosen moment. Since everything flows through one ordered queue and one seed, any failure the simulation finds can be replayed exactly.
The team also used the harness to evaluate their modifications to Plumtree. The original paper selects eager peers from a partial membership protocol, whereas Corrosion keeps full membership, with every node knowing about every other node. This raised a design question: should eager peer selection stay random, or should it favor nearby nodes, and how would that change delivery times? Because virtual time is precise, they could measure the latency from when a message was sent to when every node in the cluster had received it, and compare the variants before anything reached production.
Trade-offs and limits
This approach does not remove complexity. It makes the boundary between logic and effects explicit, which is what makes the logic easy to test. Testing the protocol in isolation does not establish that it works in the running system, though. Integration code still has to be tested. A common failure is a caller that blocks on a send while incoming messages go unprocessed, which breaks the protocol even though its logic is correct. Library users also take on some complexity, since they must supply the environment, including time and randomness, rather than relying on defaults. The author’s view is that this cost is worth paying for deterministic, reproducible testing of distributed protocols. The pattern is known as sans I/O, and it appears in networking libraries that already keep protocol logic free of I/O.