Distributed systems / reliability

Raft KV Store

A strongly consistent key-value store built in Rust, designed to remain understandable while the network, clocks, and machines misbehave.

RustRaftJepsengRPC
IP / DISTRIBUTED
Statuscompleted
Year2025
ConsistencyLinearizable
Fault modelCrash-stop
Test duration240 hours

The problem

Keep a replicated key-value store consistent when machines restart and network messages arrive late or disappear.

My contribution

Designed and implemented the replication protocol, durable storage, and deterministic failure-testing tools.

The outcome

The placeholder validation run recorded no linearizability violations across 240 hours of adversarial histories.

The full story

Failure is the normal case

A distributed system is not a faster single machine. Messages arrive late, nodes restart halfway through a write, and a leader can lose contact with the majority without knowing it. The design begins from those conditions rather than treating them as exceptional branches.

The core safety rule is simple to state:

committed_entry  ⟹  majority_replicationcommitted\_entry \implies majority\_replication

The implementation is organised so that this invariant remains visible in the code. Protocol transitions are explicit, persistent state is small, and each mutation can be replayed in the simulator.

Testing the history, not the endpoint

An apparently correct final value says very little. The Jepsen workload records every invocation and response, then checks whether the whole history admits a legal linearizable ordering. Network partitions, process pauses, restarts, and clock skew are applied while clients continue to read and write.

The result is not merely a passing test suite. It is a body of evidence explaining which failures the system tolerates and where its guarantees end.

Timeline

Progress,
without revisionism.

A chronological record of milestones, course corrections, and the next concrete step.

  1. 01

    Protocol study

    Compared Raft, Paxos, and Viewstamped Replication; chose explicit understandability.

    complete
  2. 02

    Elections under partition

    Deterministic simulation covered split votes and delayed messages.

    complete
  3. 03

    Durable replication

    AppendEntries, commit tracking, snapshots, and crash recovery converged.

    complete
  4. 04

    Jepsen validation

    No linearizability violations across 240 hours of adversarial histories.

    complete
  5. 05

    From protocol to practice

    Published the engineering notes behind durable replication and deterministic failure testing: the decisions that mattered once the cluster met unreliable machines.

    Building a distributed KV store from scratch
    complete