Distributed systems / reliability
Raft KV Store
A strongly consistent key-value store built in Rust, designed to remain understandable while the network, clocks, and machines misbehave.
The problem
Keep a replicated key-value store consistent when machines restart and network messages arrive late or disappear.
My contribution
Designed and implemented the replication protocol, durable storage, and deterministic failure-testing tools.
The outcome
The placeholder validation run recorded no linearizability violations across 240 hours of adversarial histories.
The full story
Failure is the normal case
A distributed system is not a faster single machine. Messages arrive late, nodes restart halfway through a write, and a leader can lose contact with the majority without knowing it. The design begins from those conditions rather than treating them as exceptional branches.
The core safety rule is simple to state:
The implementation is organised so that this invariant remains visible in the code. Protocol transitions are explicit, persistent state is small, and each mutation can be replayed in the simulator.
Testing the history, not the endpoint
An apparently correct final value says very little. The Jepsen workload records every invocation and response, then checks whether the whole history admits a legal linearizable ordering. Network partitions, process pauses, restarts, and clock skew are applied while clients continue to read and write.
The result is not merely a passing test suite. It is a body of evidence explaining which failures the system tolerates and where its guarantees end.
Timeline
Progress,
without revisionism.
A chronological record of milestones, course corrections, and the next concrete step.
- 01complete
Protocol study
Compared Raft, Paxos, and Viewstamped Replication; chose explicit understandability.
- 02complete
Elections under partition
Deterministic simulation covered split votes and delayed messages.
- 03complete
Durable replication
AppendEntries, commit tracking, snapshots, and crash recovery converged.
- 04complete
Jepsen validation
No linearizability violations across 240 hours of adversarial histories.
- 05complete
From protocol to practice
Published the engineering notes behind durable replication and deterministic failure testing: the decisions that mattered once the cluster met unreliable machines.
Writing · Read the deep dive Building a distributed KV store from scratch