Distributed systems / reliability

Raft KV Store

A strongly consistent key-value store built in Rust, designed to remain understandable while the network, clocks, and machines misbehave.

RustRaftJepsengRPC
IP / DISTRIBUTED
Statuscompleted
Year2025
ConsistencyLinearizable
Fault modelCrash-stop
Test duration240 hours
ResearchPrototypeValidationRelease

System anatomy

How the project
fits together.

This map is driven directly by the project data in Keystatic. Every layer groups the decisions, components, and current state of the work.

01

Protocol

Establish a single replicated history despite partial failure.

complete

Leader election

Randomised timeouts and term fencing prevent competing leaders.

complete

Log replication

Quorum commits preserve Raft's leader-completeness invariant.

02

Storage

Make the replicated state durable without stalling the protocol.

complete

Write-ahead log

Segmented append-only records recover safely after torn writes.

complete

Snapshots

Incremental compaction bounds replay time and disk growth.

03

Verification

Attack the assumptions before production has the opportunity.

complete

Model checking

A reduced protocol model explores elections and partitions.

complete

Jepsen suite

Nemeses inject packet loss, pauses, restarts, and clock skew.

04

Operations

Expose enough evidence to operate the cluster with confidence.

complete

Observability

Term, commit lag, quorum health, and storage latency are first-class metrics.

complete

Rolling upgrade

Version negotiation allows one node at a time to be replaced.

The full story

Failure is the normal case

A distributed system is not a faster single machine. Messages arrive late, nodes restart halfway through a write, and a leader can lose contact with the majority without knowing it. The design begins from those conditions rather than treating them as exceptional branches.

The core safety rule is simple to state:

committed_entry    majority_replicationcommitted\_entry \implies majority\_replication

The implementation is organised so that this invariant remains visible in the code. Protocol transitions are explicit, persistent state is small, and each mutation can be replayed in the simulator.

Testing the history, not the endpoint

An apparently correct final value says very little. The Jepsen workload records every invocation and response, then checks whether the whole history admits a legal linearizable ordering. Network partitions, process pauses, restarts, and clock skew are applied while clients continue to read and write.

The result is not merely a passing test suite. It is a body of evidence explaining which failures the system tolerates and where its guarantees end.

Timeline

Progress,
without revisionism.

A chronological record of milestones, course corrections, and the next concrete step.

  1. 01

    Protocol study

    Compared Raft, Paxos, and Viewstamped Replication; chose explicit understandability.

    complete
  2. 02

    Elections under partition

    Deterministic simulation covered split votes and delayed messages.

    complete
  3. 03

    Durable replication

    AppendEntries, commit tracking, snapshots, and crash recovery converged.

    complete
  4. 04

    Jepsen validation

    No linearizability violations across 240 hours of adversarial histories.

    complete