完整资料库

6.5840 General Information

6.5840 - Spring 2026

General Information

Structure //
Grading policy //
Staff //
Office Hours //
Acknowledgements //

Structure

6.5840 is a 12-unit core graduate subject with
lectures, labs, an optional project, a mid-term exam, and a final exam.

Class meets TR1-2:30 in person in 54-100.
Most class meetings will be part lecture
  and part paper discussion.  You should read the paper before
  coming to class, and be prepared to discuss it.
  The schedule indicates the paper to read
  for each meeting.

We will post a question about each paper 24 hours before the beginning
of class on the schedule (see the Question
link for each paper).  Your answer need only be long enough to
demonstrate that you understand the paper; a paragraph or two will
usually be enough.  We won't give feedback, but we will glance at your
answers to make sure they make sense, and they will contribute to your
grade.  If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

6.5840 will have a midterm exam during the ordinary lecture time,
  and a final exam during finals week.
  You must attend both exams.
  There will be no make-up or alternate conflict times for the exams.
  If you take 6.5840, please do not register for any other
  class with a conflicting lecture time.

There are programming labs due every week or two throughout the
term. The labs will help you understand more deeply some of the ideas
discussed in 6.5840; a more general goal is for you to gain experience
programming and debugging distributed systems. During the semester we
will ask you to attend five check-off meetings, for randomly chosen
labs, in which we will ask you questions about how your lab code
works.

Towards the end of the term you can choose between doing a final
project based on your own ideas, or doing Lab 5.  If you
want to do a project, you must form a team of two or
three people, the project must be closely related to 6.5840 topics,
and we must approve it in advance. You'll hand in a short project proposal, and,
if we approve, you'll design and build a system; at the end of the
term you'll hand in a summary of your results (which we'll post)
and your code, and do a short
presentation and demo in class.

To do well in 6.5840, you should already be familiar with computer
systems to the level of 6.1910 (6.004) and at least one of 6.1800 (6.033)
or 6.1810, and you should be good at debugging, implementing, and designing software,
perhaps as a result of taking programming-intensive courses such as 6.1810
and 6.1100 (6.035).

Grading

Final course grades will be based on:

    40% labs (programming assignments), including optional project
    20% mid-term exam
    20% final exam
    15% lab check-offs
    5% paper question answers

Each lab submission is weighted proportionally to the number
of weeks that you have to complete the assignment, excluding the first
week of classes, midterm week, and spring break week.

To help you cope with unexpected emergencies, you can hand in your Lab
1, 2, 3, 4, and 5A
solutions late, but the total amount of lateness summed over all the
lab deadlines must not exceed 72 hours. You can divide up your 72 hours
among the labs however you like; you don't have to ask or tell us.
You can only use late hours for Labs 1, 2, 3, 4, and 5A; you cannot
use late hours for Lab 5B-D or for any aspect of the project.

If you hand a lab in late, and your total late time (including
the late time for that assignment) exceeds 72 hours, and you hand it
in by the last day of classes, then we'll give it half the credit we
would have given if you had handed it in on time. Please send us
e-mail if you want us to grade an assignment that's more than 72 hours
late. We will not accept any work after the last day of classes,
regardless of late hours. If you don't hand in an assignment by the
last day of classes, we'll give the assignment zero credit.

If you want an exception to
these rules, please ask S3 to send us an excuse note.

Collaboration policy

Please do the course labs individually: you are welcome to discuss the
labs with others, but please do not look at (or hand in) anyone else's
solution. If you are tempted to use an AI assistant to write code for
you, consider that you'll then learn correspondingly less from the
labs. Regardless, we will expect you to understand and be able to
explain all of the code that you hand in, and to be able to reason
about lab-related questions on the exams.

Please do not publish your code or make
it available to current or future 6.5840 students.
github.com repositories are public by default, so please
don't put your code there unless you make the repository private. You
may find it convenient to use
MIT's GitHub,
but be sure to create a private repository.

You may discuss the paper questions with other
students, but you may not look at other students' answers.  You must
write your answers yourself.

Staff

Please use 6824-staff@lists.csail.mit.edu
to send questions or comments about the course to the staff.

    Lecturer

    Frans Kaashoek
    32-G992
    kaashoek at csail.mit.edu

    Robert Morris
    32-G972
    rtm at csail.mit.edu



    Teaching assistants

Baltasar Dinis

Ayana Alemayehu

Upamanyu Sharma

Yun-Sheng Chang

Danny Villanueva

Brian Shi

Nour Massri

Beshr Islam Bouli


Office hours



      Day
      Time
      Location
      TA





      [官方原文中的待定标记]







Appointments with staff outside of the listed office hours can be setup via
email or private Piazza post.

Acknowledgements



Robert Morris, Frans Kaashoek, and Nickolai Zeldovich developed
much of the 6.5840 course material.
The course was called 6.824 before 2023.

Questions or comments regarding 6.5840?  Send e-mail to
6824-staff@lists.csail.mit.edu.

Top //
6.5840 home //

6.5840 Home Page: Spring 2026

6.5840: Distributed Systems

Spring 2026

· Information
· Schedule
· Submissions
· Labs:
1
2
3
4
5
· Questions
· Past Exams
· 2025 Web Site
· 2024 Web Site
· 2023 Web Site
· 2022 Web Site
· 2021 Web Site
· 2020 Web Site
· 2018 Web Site
· 2017 Web Site
· 2016 Web Site
· 2015 Web Site
· 2014 Web Site
· 2013 Web Site
· 2012 Web Site
· 2011 Web Site
· 2010 Web Site
· 2009 Web Site
· 2007 Web Site
· 2006 Web Site
· 2005 Web Site
· 2004 Web Site
· 2002 Web Site
· 2001 Web Site
· 2000 Web  Site



Announcements:

Jan 25:
Please use Piazza
to read announcements and ask and answer questions about labs, lectures, and papers.

What is 6.5840 about?

6.5840 is a core 12-unit graduate subject
with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.
It will present abstractions and implementation
techniques for engineering distributed systems. Major topics include
fault tolerance, replication, and consistency. Much of the class
consists of studying and discussing case studies of distributed
systems.

 Prerequisites: 6.1910 (6.004) and one of 6.1800 (6.033) or 6.1810, or equivalent.
Substantial programming experience will be helpful for the
lab assignments.

Questions or comments regarding 6.5840?  Send e-mail to the TA at 6824-staff@lists.csail.mit.edu

6.5840 Collaboration policy

Collaboration Policy

You must write all the code you hand in for 6.5840, except for code
that we give you as part of assignments. You are not allowed to
look at anyone else's solution, and you are not allowed to look at
solutions from previous 6.5840 or 6.824 years. You may discuss the assignments with
other students, but you may not look at or copy each others' code. The
reason for this rule is that we believe you will learn the most by
designing and implementing your lab solution yourself.

Please do not publish your code or make
it available to current or future 6.5840 students.
github.com repositories are public by default, so please
don't put your code there unless you make the repository private. You
may find it convenient to use
MIT's GitHub,
but be sure to create a private repository.

6.5840 Go

Go

You'll implement all the labs in
Go. The Go web site contains lots
of tutorial information. You should use Go 1.22 or any later version.

You can check your Go version by running go version.

We recommend that you work on the labs on your own machine, so you can
use the tools, text editors, etc. that you are already familiar with. Many
editors have plug-ins for Go, e.g.
the
Go extension for
VS Code. Some commercial IDEs like
GoLand have
free educational licenses.
We are happy to provide support over Piazza and in office hours to help
you set up Go.
The labs probably won't work on Athena.

macOS

You can use Homebrew to install Go. After
installing Homebrew, run brew install go.

Linux

Depending on your Linux distribution, you might be able to get an up-to-date
version of Go from the package repository, e.g. by running apt install
golang. Otherwise, you can manually install a binary from Go's website.
First, make sure that you're running a 64-bit kernel (uname -a should
mention "x86_64 GNU/Linux"), and then run:

$ wget -qO- https://go.dev/dl/go1.23.5.linux-amd64.tar.gz | sudo tar xz -C /usr/local

You'll need to make sure /usr/local/go/bin is on your PATH.
You can do this by adding export PATH=$PATH:/usr/local/go/bin to your
shell's init file ( commonly this is one of .bashrc,  .bash_profile or .zshrc)

Windows

The labs are believed to work under Microsoft's WSL2 (Windows Subsystem for
Linux, version 2).

 To use WSL 2, first make sure you have
the Windows
Subsystem for Linux installed. Then
add Ubuntu
24.04 from the Microsoft Store. Afterwards you should be able to
launch Ubuntu Linux. Then you can follow
the directions for Linux (above).

Make sure that you are running version 2 of WSL.
WSL 1 does not work with the labs.
To check,
run  wsl -l -v  in a Windows terminal to confirm that WSL
2 and the correct Ubuntu version are installed.

Lab guidance


Lab guidance

Hardness of assignments


Each lab task is tagged to indicate
roughly how long we expect the task to take:

    Easy: A few hours.

    Moderate: ~ 6 hours (per week).

    Hard: More than 6 hours (per week). If
    you start late, your solution is unlikely to pass all tests.



Most of the labs require only a modest amount of code
(perhaps a
few hundred lines per lab part), but can be conceptually difficult
and may require a good deal of thought and debugging.
Some of the tests are difficult to pass.

Don't start a lab the night before it is due; it's more
efficient to do the labs in several sessions spread over multiple
days. Tracking down bugs in distributed systems is difficult,
because of concurrency, crashes, and an unreliable network.

Tips


Do the Online Go tutorial and
  consult
  Effective Go.
    See Editors to
    set up your editor for Go.

The lab Makefiles are set up to use
Go's race detector.
Fix any races it reports.

Advice on locking in labs.

Advice on structuring your Raft lab.

This Diagram of Raft interactions may
help you understand code flow
between different parts of the system.

Learn about Go's Printfformat strings:
Go format strings.

 To learn more about git, look at the
Pro Git book or the
git user's manual.

Debugging

Efficient debugging takes experience. It
helps to be systematic: form a hypothesis about a possible cause of the
problem; collect evidence that might be relevant; think about the
information you've gathered; repeat as needed. For extended debugging
sessions it helps to keep notes, both to accumulate evidence and to
remind yourself why you've discarded specific earlier hypotheses.

The most effective debugging technique is often to add print
statements to your code, run the test that is failing and collect the
print output in a file, and then look through the output file to
identify the point at which things start to go wrong. You may need to
iterate, adding more print statements as you learn more about what is
going wrong.

Concurrency among different peers and among the threads in a single
peer can cause actions to be interleaved in unexpected ways. For
example, it's quite possible for a Raft peer to be elected leader
while the previous leader still thinks it is the leader, or for a
leader to send an RPC but receive the reply after it has lost
leadership. Adding print statements may help you spot such situations.

Feel free to examine the test code (mr/mt_test.go,
raft1/raft_test.go, &c) to understand what the tests are
exploring. You can add print statements to the tests to help you
understand what they are doing and why they are failing, but be sure
you code passes with the original test code before submitting.

The Raft paper's Figure 2 must be followed fairly exactly. It is easy
to miss a condition that Figure 2 says must be checked, or a state
change that it says must be made. If you have a bug, re-check that all
of your code adheres closely to Figure 2.

As you're writing code (i.e., before you have a bug), it may be worth
adding explicit checks for conditions that the code assumes to be
true, perhaps using Go's
panic. Such checks may
help detect situations where later code unwittingly violates the
assumptions.

The TAs are happy to help you think about your code during office
hours, but you're likely to get the most mileage out of limited office
hour time if you've already dug as deep as you can into the situation.

6.5840 Lab 4: Fault-tolerant Key/Value Service

6.5840 - Spring 2026

6.5840 Lab 4: Fault-tolerant Key/Value Service

  Collaboration policy //
  Submit lab //
  Setup Go //
  Guidance //
  Piazza

Introduction

In this lab you will build a fault-tolerant key/value storage
service using your Raft library from
Lab 3.
To clients, the service looks similar to the server of
Lab 2.
However, instead of a single server, the service consists
of a set of servers that use Raft to help them maintain
identical databases.
Your key/value service should continue to
process client requests as long as a majority of the servers
are alive and can communicate, in spite of other failures or
network partitions.

After Lab 4, you will have implemented all parts (Clerk, Service, and Raft) shown in the diagram of Raft interactions.

  Clients will interact with your key/value service through a Clerk,
  as in Lab 2. A Clerk implements the Put and Get
  methods with the same semantics as Lab 2: Puts are at-most-once
  and the Puts/Gets must form a linearizable history.


Providing linearizability is relatively easy for a single server. It is
harder if the service is replicated, since all servers must choose the same
execution order for concurrent requests, must avoid replying to
clients using state that isn't up to date, and must recover their
state after a failure in a way that preserves all acknowledged client
updates.

  This lab has three parts. In part A, you will implement a
  replicated-state machine package, rsm, using your raft
  implementation; rsm is agnostic of the requests that it
  replicates.  In part B, you will implement a replicated key/value
  service using rsm, but without using snapshots.  In part C,
  you will use your snapshot implementation from Lab 3D, which will
  allow Raft to discard old log entries. Please submit each part by
  the respective deadline.

You should review the
extended Raft paper,
in particular Section 7 (but not 8). For a wider
perspective, have a look at Chubby, Paxos Made Live,
Spanner, Zookeeper, Harp, Viewstamped Replication, and
Bolosky et al.

Start early.

Getting Started

We supply you with skeleton code and tests in src/kvraft1.
The skeleton code uses the skeleton package src/kvraft1/rsm to
replicate a server.  A server must implement
the StateMachine interface defined in rsm to
replicate itself using rsm. Most of your work will be
implementing rsm to provide server-agnostic replication.  You
will also need to modify kvraft1/client.go
and kvraft1/server.go to implement the server-specific parts.
This split allows you to re-use rsm in the next lab.  You may
be able to re-use some of your Lab 2 code (e.g., re-using the server
code by copying or importing the "src/kvsrv1" package) but it is not a
requirement.

To get up and running, execute the following commands.
Don't forget the git pull to get the latest software.

$ cd ~/6.5840
$ git pull
..

Part A: replicated state machine (RSM)

$ cd src
$ make rsm1
=== RUN   TestBasic4A
Test RSM basic (reliable network)...
    rsm_test.go:28: expected 0 instead of 0

In the common situation of a client/server service using Raft for
replication, the service interacts with Raft in two ways: the service
leader submits client operations by calling raft.Start(), and all
service replicas receive committed operations via Raft's applyCh,
which they execute. On the leader, these two activities interact. At
any given time, some server goroutines are handling client requests,
have called raft.Start(), and each is waiting for its operation to
commit and to find out what the result of executing the operation is.
And as committed operations appear on the applyCh, each needs to be
executed by the service, and the results need to be handed to the goroutine that
called raft.Start() so that it can return the result to the client.

The rsm package encapsulates the above interaction. It sits
as a layer between the service (e.g. a key/value database) and Raft.
In rsm/rsm.go you will need to implement a "reader" goroutine
that reads the applyCh, and a rsm.Submit() function that calls
raft.Start() for a client operation and then waits for the reader
goroutine to hand it the result of executing that operation.

The service that is using rsm appears to the rsm
reader goroutine as a StateMachine object providing
a DoOp() method. The reader goroutine should hand each
committed operation to DoOp(); DoOp()'s return value
should be given to the corresponding rsm.Submit() call for
it to return.
DoOp()'s argument and return value have type
any; the actual values should have the same types as the
argument and return values that the service passes
to rsm.Submit(), respectively.

The service should pass each client operation
to rsm.Submit(). To help the reader goroutine match applyCh
messages with waiting calls
to rsm.Submit(), Submit() should wrap each client
operation in an Op structure along with a unique identifier.
Submit() should then wait until the operation has committed
and been executed, and return the result of execution (the
value returned by DoOp()).
If raft.Start() indicates that the current peer is not the
Raft leader, Submit() should return
an rpc.ErrWrongLeader error. Submit() should detect
and handle
the situation in which leadership changed just after it
called raft.Start(), causing the operation to be lost (never
committed).

For Part A, the rsm tester acts as the service, submitting
operations that it interprets as increments on a state consisting of a
single integer. In Part B you'll use rsm as part of a key/value service that
implements StateMachine (and DoOp()),
and calls rsm.Submit().

If all goes well, the sequence of events for a client request is:

     The client sends a request to the service leader.
     The service leader calls rsm.Submit() with the request.
     rsm.Submit() calls raft.Start() with
      the request, and then waits.
     Raft commits the request and sends it on all peers' applyChs.
     The rsm reader goroutine on each peer reads the
         request from the applyCh and passes it to the service's
         DoOp().
     On the leader, the rsm reader goroutine hands
         the DoOp() return value to the Submit()
         goroutine that originally submitted the request, and
         Submit() returns that value.


  Implement rsm.go: the Submit() method and
  a reader goroutine.
  You have completed this task if
  you pass the rsm 4A tests:

  $ cd src
  $ make RUN="-run 4A" rsm1
go build -race -o main/rsm1d main/rsm1d.go
cd kvraft1/rsm; go test -v -race -run 4A
=== RUN   TestBasic4A
Test RSM basic (reliable network)...
  ... Passed --  time  4.2s #peers 3 #RPCs    50 #Ops   10
--- PASS: TestBasic4A (4.57s)
=== RUN   TestConcurrent4A
Test concurrent submit (reliable network)...
  ... Passed --  time  1.0s #peers 3 #RPCs    28 #Ops   50
--- PASS: TestConcurrent4A (1.39s)
=== RUN   TestLeaderFailure4A
Test Leader Failure (reliable network)...
  ... Passed --  time  2.9s #peers 3 #RPCs    32 #Ops    2
--- PASS: TestLeaderFailure4A (3.29s)
=== RUN   TestLeaderPartition4A
Test Leader Partition (reliable network)...
2026/03/11 10:43:46 partition leader 0
  ... Passed --  time  3.6s #peers 3 #RPCs    61 #Ops    2
--- PASS: TestLeaderPartition4A (4.04s)
=== RUN   TestRestartReplay4A
Test Restart (reliable network)...
  ... Passed --  time 28.4s #peers 3 #RPCs   467 #Ops  101
--- PASS: TestRestartReplay4A (28.79s)
=== RUN   TestShutdown4A
Test Shutdown (reliable network)...
  ... Passed --  time 10.0s #peers 3 #RPCs     0 #Ops    0
--- PASS: TestShutdown4A (10.38s)
=== RUN   TestRestartSubmit4A
Test Restart and submit (reliable network)...
  ... Passed --  time 39.8s #peers 3 #RPCs   463 #Ops  102
--- PASS: TestRestartSubmit4A (40.21s)
PASS
ok      6.5840/kvraft1/rsm  93.691s

You should not need to add any fields to the Raft ApplyMsg,
or to Raft RPCs such as AppendEntries, but you are allowed to do so.

 Your solution needs to handle an rsm leader that has
called Start() for a request submitted with Submit() but
  loses its leadership before the request is committed to the log.
  One way to do this is for
the
rsm to detect that it has lost leadership,
by noticing that Raft's term has changed or
a different request has
appeared at the index returned by Start(),
and return rpc.ErrWrongLeader from Submit().
If the ex-leader is partitioned by
itself, it won't know about new leaders; but
any client in the same partition won't be able
to talk to a new leader either, so it's OK in
this case for the server to wait
indefinitely until the partition heals.

Part B: Key/value service without snapshots

$ cd src
$ make RUN="-run 4B" kvraft1
go build -race -o main/kvraft1d main/kvraft1d.go
cd kvraft1 && go test -v -race -run 4B
=== RUN   TestBasic4B
Test: one client (4B basic) (reliable network)...
Fatal: Wrong error

Now you will use the rsm package to replicate a
  key/value server. Each of the servers ("kvservers") will have an
  associated rsm/Raft peer.  Clerks send Put()
  and Get() RPCs to the kvserver whose associated Raft is the
  leader. The kvserver code submits the Put/Get operation to rsm,
  which replicates it using Raft and invokes your
  server's DoOp at each peer, which should apply the
  operations to the peer's key/value database; the intent is for the
  servers to maintain identical replicas of the key/value database.

A Clerk sometimes doesn't
know which kvserver is the Raft leader. If the Clerk sends an
RPC to the wrong kvserver, or if it cannot reach the kvserver,
the Clerk should re-try by sending to a different kvserver.
If the key/value service commits the operation to its Raft log
(and hence applies the operation to the key/value state machine), the
leader reports the result to the Clerk by responding to its
RPC. If the operation failed to commit (for example, if the leader was
replaced), the server reports an error, and the Clerk retries with a
different server.

Your first task is to implement a solution that works when there are no dropped
messages, and no failed servers.

Feel free to copy your client code from Lab 2 (kvsrv1/client.go)
into kvraft1/client.go. You will need to add logic for deciding which
kvserver to send each RPC to.

You'll also need to implement Put() and Get() RPC
handlers in server.go. These handlers should submit the
request to Raft using rsm.Submit(). As the rsm
package reads commands from
applyCh, it should invoke the DoOp method, which you
will have to implement in server.go.

You have completed this task when you
reliably pass the first test in the
test suite, with make RUN="-run TestBasic4B" kvraft1.

A kvserver should not complete a Get() RPC if it is not part
of a majority (so that it does not serve stale data). A simple
solution is to enter every Get() (as well as
each Put()) in the Raft log using Submit(). You don't
have to implement the optimization for read-only operations that is
described in Section 8.

It's best to add locking from the start because the
need to avoid deadlocks sometimes affects overall code design. The
tester runs your code with the race detector by default.

Now you should modify your solution to continue in the face of network
and server failures.
One problem you'll face is that a
Clerk may have to send an RPC multiple times until it finds a
kvserver that replies positively. If a leader fails just after
committing an entry to the Raft log, the Clerk may not
receive a reply, and thus may
re-send the request to another leader.
Each call to
Clerk.Put() should
result in just a single execution for a particular version number.

  Add code to handle failures.  Your Clerk can use a similar
  retry plan as in lab 2, including returning ErrMaybe if a
  response to a retried Put RPC is lost.
  You are done when your code reliably passes all the 4B tests, with
  make RUN="-run 4B" kvraft1.

  Recall that the rsm leader may lose its leadership and return
    rpc.ErrWrongLeader from Submit().  In this case you
    should arrange for the Clerk to re-send the request to other
    servers until it finds the new leader.

You will probably have to modify your
Clerk to remember which server turned out to
be the leader for the last RPC, and send the
next RPC to that server first. This will avoid
wasting time searching for the leader on every
RPC, which may help you pass some of the tests
quickly enough.

Your code should now pass the Lab 4B tests, like this:

$ cd src
$ make RUN="-run 4B" kvraft1
go build -race -o main/kvraft1d main/kvraft1d.go
cd kvraft1 && go test -v -race -run 4B
=== RUN   TestBasic4B
Test: one client (4B basic) (reliable network)...
  ... Passed --  time  3.5s #peers 5 #RPCs   395 #Ops  122
--- PASS: TestBasic4B (4.11s)
=== RUN   TestSpeed4B
Test: one client (4B speed) (reliable network)...
  ... Passed --  time 33.4s #peers 3 #RPCs  3291 #Ops 1002
--- PASS: TestSpeed4B (33.80s)
=== RUN   TestConcurrent4B
Test: many clients (4B many clients) (reliable network)...
  ... Passed --  time  4.1s #peers 5 #RPCs   953 #Ops  558
--- PASS: TestConcurrent4B (4.69s)
=== RUN   TestUnreliable4B
Test: many clients (4B many clients) (unreliable network)...
  ... Passed --  time  4.6s #peers 5 #RPCs   685 #Ops  210
--- PASS: TestUnreliable4B (5.22s)
=== RUN   TestOnePartition4B
Test: one client (4B progress in majority) (unreliable network)...
  ... Passed --  time  4.9s #peers 5 #RPCs   231 #Ops    4
Test: no progress in minority (4B) (unreliable network)...
  ... Passed --  time  1.8s #peers 5 #RPCs   110 #Ops    7
Test: completion after heal (4B) (unreliable network)...
  ... Passed --  time  1.1s #peers 5 #RPCs    43 #Ops    4
--- PASS: TestOnePartition4B (8.36s)
=== RUN   TestManyPartitionsOneClient4B
Test: partitions, one client (4B partitions, one client) (reliable network)...
  ... Passed --  time  9.4s #peers 5 #RPCs   520 #Ops  114
--- PASS: TestManyPartitionsOneClient4B (10.08s)
=== RUN   TestManyPartitionsManyClients4B
Test: partitions, many clients (4B partitions, many clients (4B)) (reliable network)...
  ... Passed --  time 16.1s #peers 5 #RPCs  1271 #Ops  558
--- PASS: TestManyPartitionsManyClients4B (16.68s)
=== RUN   TestPersistOneClient4B
Test: restarts, one client (4B restarts, one client 4B ) (reliable network)...
  ... Passed --  time  8.4s #peers 5 #RPCs   311 #Ops   62
--- PASS: TestPersistOneClient4B (9.01s)
=== RUN   TestPersistConcurrent4B
Test: restarts, many clients (4B restarts, many clients) (reliable network)...
  ... Passed --  time  8.5s #peers 5 #RPCs   994 #Ops  350
--- PASS: TestPersistConcurrent4B (9.11s)
=== RUN   TestPersistConcurrentUnreliable4B
Test: restarts, many clients (4B restarts, many clients ) (unreliable network)...
  ... Passed --  time 10.3s #peers 5 #RPCs   672 #Ops  114
--- PASS: TestPersistConcurrentUnreliable4B (10.89s)
=== RUN   TestPersistPartition4B
Test: restarts, partitions, many clients (4B restarts, partitions, many clients) (reliable network)...
  ... Passed --  time 14.3s #peers 5 #RPCs   804 #Ops   94
--- PASS: TestPersistPartition4B (14.95s)
=== RUN   TestPersistPartitionUnreliable4B
Test: restarts, partitions, many clients (4B restarts, partitions, many clients) (unreliable network)...
  ... Passed --  time 22.0s #peers 5 #RPCs  1229 #Ops  102
--- PASS: TestPersistPartitionUnreliable4B (22.64s)
=== RUN   TestPersistPartitionUnreliableLinearizable4B
Test: restarts, partitions, random keys, many clients (4B restarts, partitions, random keys, many clients) (unreliable network)...
  ... Passed --  time 24.1s #peers 7 #RPCs  4464 #Ops  444
--- PASS: TestPersistPartitionUnreliableLinearizable4B (24.94s)
PASS
ok      6.5840/kvraft1  175.518s

The numbers after each Passed are real time in seconds,
number of peers, number of RPCs sent (including client RPCs), and
number of key/value operations executed (Clerk Get/Put
calls).

Part C: Key/value service with snapshots

As things stand now, your key/value server doesn't call your Raft
library's Snapshot() method, so a rebooting server has to
replay the complete persisted Raft log in order to restore its state.
Now you'll modify kvserver and rsm to cooperate with Raft to save log space
and reduce restart time, using Raft's Snapshot() from Lab 3D.

The tester passes maxraftstate to your
StartKVServer(), which passes it
to rsm. maxraftstate indicates the maximum allowed
size of your persistent Raft state in bytes (including the log, but
not including snapshots). You should compare maxraftstate
to rf.PersistBytes().  Whenever your rsm
detects that the Raft state size is approaching this threshold, it
should save a snapshot by calling
Raft's Snapshot.  rsm can create this snapshot
by calling the Snapshot method of the StateMachine
interface to obtain a snapshot of the kvserver.
If maxraftstate is -1, you do not have to snapshot.
The maxraftstate limit applies to the GOB-encoded
bytes your Raft passes as the first argument to
persister.Save().

You can find the source for the persister object in
tester1/persister.go.

Modify your rsm so that it detects when the persisted Raft state
grows too large, and then hands a snapshot to Raft.  When
a rsm server restarts, it should read the snapshot
with persister.ReadSnapshot() and, if the snapshot's
length is greater than zero, pass the snapshot to the
StateMachine's Restore() method.
You complete this task if you
pass TestSnapshot4C in rsm.

$ cd src
$ make RUN="-run 4C" kvraft1
go build -race -o main/kvraft1d main/kvraft1d.go
cd kvraft1 && go test -v -race -run 4C
=== RUN   TestSnapshotRPC4C
Test: snapshots, one client (4C SnapshotsRPC) (reliable network)...
Test: InstallSnapshot RPC (4C) (reliable network)...
signal: killed
FAIL    6.5840/kvraft1  61.186s

Think about when rsm should snapshot its state and what
should be included in the snapshot beyond just the server state. Raft
stores each snapshot in the persister object using
Save(),
along with corresponding Raft state.
You can read the
latest stored snapshot using ReadSnapshot().
Capitalize all fields of structures stored in the snapshot.

  Implement the kvraft1/server.go Snapshot() and Restore()
  methods, which rsm calls.
  Modify rsm to handle applyCh messages that contain
  snapshots.

You may have bugs in your Raft and rsm library that this task
  exposes.  If you make changes to your Raft implementation make sure
  it continues to pass all of the Lab 3 tests.

A reasonable amount of time to take for the Lab 4 tests is 400 seconds
of real time and 700 seconds of CPU time.

Your code should pass the 4C tests (as in the example here) as well
as the 4A+B tests (and your Raft must continue to pass the Lab 3 tests).

$ make RUN="-run 4C" kvraft1
go build -race -o main/kvraft1d main/kvraft1d.go
cd kvraft1 && go test -v -race -run 4C
=== RUN   TestSnapshotRPC4C
Test: snapshots, one client (4C SnapshotsRPC) (reliable network)...
Test: InstallSnapshot RPC (4C) (reliable network)...
  ... Passed --  time  4.8s #peers 3 #RPCs   248 #Ops   72
--- PASS: TestSnapshotRPC4C (5.18s)
=== RUN   TestSnapshotSize4C
Test: snapshots, one client (4C snapshot size is reasonable) (reliable network)...
  ... Passed --  time 21.0s #peers 3 #RPCs  2569 #Ops 1200
--- PASS: TestSnapshotSize4C (21.42s)
=== RUN   TestSpeed4C
Test: snapshots, one client (4C speed) (reliable network)...
  ... Passed --  time 24.9s #peers 3 #RPCs  3208 #Ops 1002
--- PASS: TestSpeed4C (25.32s)
=== RUN   TestSnapshotRecover4C
Test: restarts, snapshots, one client (4C restarts, snapshots, one client) (reliable network)...
  ... Passed --  time  8.2s #peers 5 #RPCs   273 #Ops   50
--- PASS: TestSnapshotRecover4C (8.78s)
=== RUN   TestSnapshotRecoverManyClients4C
Test: restarts, snapshots, many clients (4C restarts, snapshots, many clients ) (reliable network)...
info: linearizability check timed out, assuming history is ok
info: linearizability check timed out, assuming history is ok
info: linearizability check timed out, assuming history is ok
  ... Passed --  time 12.5s #peers 5 #RPCs  3525 #Ops 1670
--- PASS: TestSnapshotRecoverManyClients4C (13.15s)
=== RUN   TestSnapshotUnreliable4C
Test: snapshots, many clients (4C unreliable net, snapshots, many clients) (unreliable network)...
  ... Passed --  time  5.5s #peers 5 #RPCs   773 #Ops  230
--- PASS: TestSnapshotUnreliable4C (6.16s)
=== RUN   TestSnapshotUnreliableRecover4C
Test: restarts, snapshots, many clients (4C unreliable net, restarts, snapshots, many clients) (unreliable network)...
  ... Passed --  time 10.7s #peers 5 #RPCs   804 #Ops   78
--- PASS: TestSnapshotUnreliableRecover4C (11.28s)
=== RUN   TestSnapshotUnreliableRecoverConcurrentPartition4C
Test: restarts, partitions, snapshots, many clients (4C unreliable net, restarts, partitions, snapshots, many clients) (unreliable network)...
  ... Passed --  time 17.4s #peers 5 #RPCs   894 #Ops   94
--- PASS: TestSnapshotUnreliableRecoverConcurrentPartition4C (17.97s)
=== RUN   TestSnapshotUnreliableRecoverConcurrentPartitionLinearizable4C
Test: restarts, partitions, snapshots, random keys, many clients (4C unreliable net, restarts, partitions, snapshots, random keys, many clients) (unreliable network)...
  ... Passed --  time 19.6s #peers 7 #RPCs  2957 #Ops  368
--- PASS: TestSnapshotUnreliableRecoverConcurrentPartitionLinearizable4C (20.45s)
PASS
ok      6.5840/kvraft1  130.724s

6.5840 Lab 2: Key/Value Server

6.5840 - Spring 2026

6.5840 Lab 2: Key/Value Server

  Collaboration policy //
  Submit lab //
  Setup Go //
  Guidance //
  Piazza

Introduction

In this lab you will build a key/value server for a single machine
that ensures that each Put operation is executed at-most-once
despite network failures and that the operations
are linearizable. You will use this KV server to implement a
lock. Later labs will replicate a server like this one to handle
server crashes.

KV server

Each client interacts with the key/value server using a Clerk,
a set of library routines which sends RPCs to the server.
Clients can send two different RPCs to the server:
Put(key, value, version) and Get(key). The server
maintains an in-memory map that records for each key a (value,
version) tuple.  Keys and values are strings. The version number
records the number of times the key has been written.

Put(key, value, version) installs or replaces the value for a
particular key in the map only if the Put's version number
matches the server's version number for the key.  If the version
numbers match, the server also increments the version number of the
key.  If the version numbers don't match, the server should
return rpc.ErrVersion. A client can create a new key by invoking Put with
version number 0 (and the resulting version stored by the server
will be 1).  If the version number of the Put is larger than 0
and the key doesn't exist, the server should return rpc.ErrNoKey.

Get(key) fetches the current value for the key and its associated
version.  If the key doesn't exist at the server, the server should
return rpc.ErrNoKey.

Maintaining a version number for each key will be useful for
implementing locks using Put and ensuring
at-most-once semantics for Put's when the network is
unreliable and the client retransmits.

When you've finished this lab and passed all the tests, you'll
have a linearizable key/value service from the point
of view of clients calling
Clerk.Get and Clerk.Put.
That is,
if client operations aren't concurrent, each
client Clerk.Get and Clerk.Put will observe the
modifications to the state implied by the preceding sequence of
operations. For concurrent operations, the return values and final state will be
the same as if the operations had executed one at a time in some
order. Operations are concurrent if they overlap in time: for example, if
client X calls Clerk.Put(), and client Y
calls Clerk.Put(), and then client X's call returns. An operation
must observe the effects of all operations that have completed before the
operation starts.  See the FAQ
on linearizability for
more background.

Linearizability is convenient for applications because it's the
behavior you'd see from a single server that processes requests one at
a time. For example, if one client gets a successful response from the
server for an update request, subsequently launched reads from other
clients are guaranteed to see the effects of that update. Providing
linearizability is relatively easy for a single server.

Getting Started

We supply you with skeleton code and tests in src/kvsrv1.
kvsrv1/client.go implements a Clerk that clients use
to manage RPC interactions with the server; the Clerk provides
Put and Get methods.
kvsrv1/server.go contains the server code,
including the Put and Get handlers that
implement the server side of RPC requests.
You will need to modify client.go and server.go.
The RPC requests, replies, and
error values are defined in
the kvsrv1/rpc package in the file kvsrv1/rpc/rpc.go,
which you should look at, though
you don't have to modify rpc.go.

To get up and running, execute the following commands.
Don't forget the git pull to get the latest software.

$ cd ~/6.5840
$ git pull
...
$ cd src
$ make kvsrv1
go build -race -o main/kvsrv1d main/kvsrv1d.go
cd kvsrv1 && go test -v -race
=== RUN   TestReliablePut
One client and reliable Put (reliable network)...
    kvsrv_test.go:25: Put err ErrNoKey
--- FAIL: TestReliablePut (0.31s)
...
$

Key/value server with reliable network

Your first task is to implement a solution that works when there are
no dropped messages.
You'll need to add RPC-sending code to the Clerk Put/Get
methods in client.go, and implement
Put and Get RPC handlers in
server.go.

You have completed this task when you
pass the Reliable tests in the
test suite:

$ cd src
$ make RUN="-run Reliable" kvsrv1
go build -race -o main/kvsrv1d main/kvsrv1d.go
cd kvsrv1 && go test -v -race -run Reliable
=== RUN   TestReliablePut
One client and reliable Put (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     5 #Ops    5
--- PASS: TestReliablePut (0.12s)
=== RUN   TestPutConcurrentReliable
Test: many clients racing to put values to the same key (reliable network)...
  ... Passed --  time  6.3s #peers 1 #RPCs 11025 #Ops 22050
--- PASS: TestPutConcurrentReliable (6.36s)
=== RUN   TestMemPutManyClientsReliable
Test: memory use many put clients (reliable network)...
  ... Passed --  time 29.0s #peers 1 #RPCs 50000 #Ops 50000
--- PASS: TestMemPutManyClientsReliable (52.91s)
PASS
ok      6.5840/kvsrv1   60.732s
$

The numbers after each Passed are real time in seconds,
the constant 1,
the number of RPCs sent (including client RPCs), and the number of key/value
operations executed (Clerk Get and Put calls).

Implementing a lock using key/value clerk

In many distributed applications, clients running on different
  machines use a key/value server to coordinate their activities.  For
  example, ZooKeeper and Etcd allow clients to coordinate using a
  distributed lock, in analogy with how threads in a Go program can
  coordinate with locks (i.e., sync.Mutex).  Zookeeper and Etcd
  implement such a lock with conditional put.

Your task is to implement locks,
using your key/value server to store whatever per-lock
state your design needs.
There can be multiple independent locks, each with its own
name, passed as an argument to MakeLock.
A lock supports two
methods: Acquire and Release. The specification is that
only one client can successfully acquire a given lock at a time;
other clients
must wait until the first client has released the lock
using Release.

We supply you with skeleton code and tests
in src/kvsrv1/lock/. You will need to
modify src/kvsrv1/lock/lock.go.
Your Acquire and Release should
store each lock's state in your key/value server,
by calling lk.ck.Put() and lk.ck.Get().

If a client crashes while holding a lock,
the lock will never be released.
In a design more sophisticated than this lab,
the client would attach a
lease
to a lock.  When the lease expires, the lock server would release the lock on
behalf of the client.  In this lab clients don't crash and you
can ignore this problem.

Implement Acquire and Release.
You have completed this exercise when your
code passes these tests:

$ cd src
$ make RUN="-run Reliable" lock1
go build -race -o main/kvsrv1d main/kvsrv1d.go
cd kvsrv1/lock; go test -v -race -run Reliable
=== RUN   TestReliableBasic
Test: a single Acquire and Release (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     4 #Ops    4
--- PASS: TestReliableBasic (0.13s)
=== RUN   TestReliableNested
Test: one client, two locks (reliable network)...
  ... Passed --  time  0.1s #peers 1 #RPCs    17 #Ops   17
--- PASS: TestReliableNested (0.17s)
=== RUN   TestOneClientReliable
Test: 1 lock clients (reliable network)...
  ... Passed --  time  2.0s #peers 1 #RPCs   477 #Ops  477
--- PASS: TestOneClientReliable (2.14s)
=== RUN   TestManyClientsReliable
Test: 10 lock clients (reliable network)...
  ... Passed --  time  2.2s #peers 1 #RPCs  5704 #Ops 5704
--- PASS: TestManyClientsReliable (2.36s)
PASS
ok      6.5840/kvsrv1/lock      5.817s
$

If you haven't implemented the lock yet, the first two
tests will succeed.

This exercise requires little code but a bit more
  independent thought than the previous exercise.



     You will need a unique identifier for each lock client;
      call kvtest.RandValue(8) to generate a random string.

Key/value server with dropped messages

The main challenge in this exercise is that the network may re-order,
delay, or discard RPC requests and/or replies.  To recover from
discarded requests/replies, the Clerk must keep re-trying each RPC
until it receives a reply from the server.

If the network discards an RPC request message, then the
client re-sending
the request will solve the problem: the server will receive and
execute just the re-sent request.

However, the network might instead discard an RPC reply message.
The client does not know which message was discarded; the client only
observes that it received no reply.
If it was the reply that was discarded, and the client re-sends
the RPC request, then the server will receive two copies of
the request.
That's OK for a Get,
since Get doesn't modify the server state.
It is safe to resend a Put RPC with the
same version number, since the server executes Put
conditionally on the version number; if the server received and
executed a
Put RPC, it will respond to a re-transmitted
copy of that RPC with rpc.ErrVersion rather than
executing the Put a second time.

A tricky case is if the server replies with
  an rpc.ErrVersion in a response to an RPC that the Clerk
  retried.  In this case, the Clerk cannot know if the
  Clerk's Put was executed by the server or not: the first
  RPC might have been executed by the server but the network may have
  discarded the successful response from the server, so that the server
  sent rpc.ErrVersion only for the retransmitted RPC.
  Or, it might be that another
  Clerk updated the key before the Clerk's first RPC arrived at the server,
  so that the server executed neither of the Clerk's RPCs and replied
  rpc.ErrVersion to both.
  Therefore, if a Clerk receives rpc.ErrVersion for a
  retransmitted Put RPC,
  Clerk.Put must return rpc.ErrMaybe to the
  application instead of rpc.ErrVersion since the request may
  have been executed.  It is then up to the application to handle this
  case.
 If the server responds to an initial (not retransmitted)
 Put RPC with rpc.ErrVersion, then the Clerk
should return rpc.ErrVersion to the application, since the RPC
was definitely not executed by the server.

It would be more convenient for application developers if Put's were
  exactly-once (i.e., no rpc.ErrMaybe errors) but that is
  difficult to guarantee without maintaining state at the server for
  each Clerk.
  In the last exercise of this lab, you will
  implement a lock using your Clerk to explore how to
  program with at-most-once Clerk.Put.

Now you should modify your kvsrv1/client.go
to continue in the face of dropped
RPC requests and replies.
A return value of true from
the client's ck.clnt.Call()
indicates that the client
received an RPC reply from the
server; a return value of false indicates that it
did not receive a reply (more precisely, Call() waits
for a reply message for a timeout interval, and returns false
if no reply arrives within that time).
Your
Clerk should keep re-sending an RPC until it receives
a reply.
Keep in mind the discussion of rpc.ErrMaybe above.
Your solution shouldn't require
any changes to the server.

 Add code to Clerk to retry if doesn't receive a reply.
Your have completed this task if your code passes all the tests
for kvsrv1:

$ make kvsrv1
go build -race -o main/kvsrv1d main/kvsrv1d.go
cd kvsrv1 && go test -v -race
=== RUN   TestReliablePut
One client and reliable Put (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     5 #Ops    5
--- PASS: TestReliablePut (0.12s)
=== RUN   TestPutConcurrentReliable
Test: many clients racing to put values to the same key (reliable network)...
  ... Passed --  time  6.4s #peers 1 #RPCs 11021 #Ops 22042
--- PASS: TestPutConcurrentReliable (6.52s)
=== RUN   TestMemPutManyClientsReliable
Test: memory use many put clients (reliable network)...
  ... Passed --  time 28.8s #peers 1 #RPCs 50000 #Ops 50000
--- PASS: TestMemPutManyClientsReliable (52.44s)
=== RUN   TestUnreliableNet
One client (unreliable network)...
  ... Passed --  time  4.0s #peers 1 #RPCs   268 #Ops  422
--- PASS: TestUnreliableNet (4.13s)
PASS
ok      6.5840/kvsrv1   64.442s
$


   Before the client retries, it should wait a little bit; you can
    use go's time package and call time.Sleep(100 *
    time.Millisecond)


Implementing a lock using key/value clerk and unreliable
network


Modify your lock implementation to
work correctly with your modified key/value client
when the network is not reliable.
You
have completed this exercise when your code passes all the
lock1 tests:

$ make lock1
go build -race -o main/kvsrv1d main/kvsrv1d.go
cd kvsrv1/lock; go test -v -race
=== RUN   TestReliableBasic
Test: a single Acquire and Release (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     4 #Ops    4
--- PASS: TestReliableBasic (0.13s)
=== RUN   TestReliableNested
Test: one client, two locks (reliable network)...
  ... Passed --  time  0.1s #peers 1 #RPCs    17 #Ops   17
--- PASS: TestReliableNested (0.17s)
=== RUN   TestOneClientReliable
Test: 1 lock clients (reliable network)...
  ... Passed --  time  2.0s #peers 1 #RPCs   471 #Ops  471
--- PASS: TestOneClientReliable (2.13s)
=== RUN   TestManyClientsReliable
Test: 10 lock clients (reliable network)...
  ... Passed --  time  2.2s #peers 1 #RPCs  5658 #Ops 5658
--- PASS: TestManyClientsReliable (2.35s)
=== RUN   TestOneClientUnreliable
Test: 1 lock clients (unreliable network)...
  ... Passed --  time  2.1s #peers 1 #RPCs    66 #Ops   57
--- PASS: TestOneClientUnreliable (2.18s)
=== RUN   TestManyClientsUnreliable
Test: 10 lock clients (unreliable network)...
  ... Passed --  time  4.1s #peers 1 #RPCs   778 #Ops  617
--- PASS: TestManyClientsUnreliable (4.23s)
PASS
ok      6.5840/kvsrv1/lock      12.227s
$

6.5840 Lab 1: MapReduce

6.5840 - Spring 2026

6.5840 Lab 1: MapReduce

  Collaboration policy //
  Submit lab //
  Setup Go //
  Guidance //
  Piazza

Introduction

In this lab you'll build a MapReduce system.
You'll implement a worker process that calls application Map and Reduce
functions and handles reading and writing files,
and a coordinator process that hands out tasks to
workers and copes with failed workers.
You'll be building something similar to the
MapReduce paper.
(Note: this lab uses "coordinator" instead of the paper's "master".)

Getting started

You need to setup Go to do the labs.

Fetch the initial lab software with
git (a version control system).
To learn more about git, look at the
Pro Git book or the
git user's manual.

$ git clone git://g.csail.mit.edu/6.5840-golabs-2026 6.5840
$ cd 6.5840
$ ls
Makefile src
$

We supply you with a simple sequential mapreduce implementation in
src/main/mrsequential.go. It runs the maps and reduces one at a
time, in a single process. We also
provide you with a couple of MapReduce applications: word-count
in mrapps/wc.go, and a text indexer
in mrapps/indexer.go. You can run
word count sequentially as follows:

$ cd ~/6.5840
$ cd src/main
$ go build -buildmode=plugin ../mrapps/wc.go
$ rm mr-out*
$ go run mrsequential.go wc.so pg*.txt
$ sort mr-out-0
A 509
ABOUT 2
ACT 8
ACTRESS 1
...

(You might
need to set LC_COLLATE=C environment variable for
sort to produce the above output: LC_COLLATE=C sort mr-out-0)

mrsequential.go leaves its output in the file mr-out-0.
The input is from the text files named pg-xxx.txt.

Feel free to borrow code from mrsequential.go.
You should also have a look at mrapps/wc.go to see what
MapReduce application code looks like.

For this lab and all the others, we might issue updates to the code we
provide you. To ensure that you can fetch those updates and easily
merge them using git pull, it's best to leave the code we
provide in the original files. You can add to the code we provide as
directed in the lab write-ups; just don't move it. It's OK to put your
own new functions in new files.

Your Job

Your job is to implement a distributed MapReduce, consisting of
two programs, the coordinator and the worker. There will be
just one coordinator process, and one or more worker processes executing in
parallel. In a real system the workers would run on a bunch of
different machines, but for this lab you'll run them all on a single machine.
The workers will talk to the coordinator via RPC. Each worker process will,
in a loop, ask
the coordinator for a task, read the task's input from one or more files,
execute the task, write the task's output to one
or more files, and again ask the coordinator for a
new task. The coordinator should notice if a worker hasn't completed
its task in a reasonable amount of time (for this lab, use ten
seconds), and give the same task to a different worker.

We have given you a little code to start you off. The "main" routines for
the coordinator and worker are in main/mrcoordinator.go and main/mrworker.go;
don't change these files. You should put your implementation in mr/coordinator.go,
mr/worker.go, and mr/rpc.go.

Here's how to run your code on the word-count MapReduce
application. First, build the word-count plugin:

$ cd main
$ go build -buildmode=plugin ../mrapps/wc.go

In one window, run the coordinator:

$ rm mr-out*
$ go run mrcoordinator.go sock123 pg-*.txt

The sock123 argument specifies a socket on which the
coordinator receives RPCs from workers.
The pg-*.txt arguments to mrcoordinator.go are
the input files; each file corresponds to one "split", and is the
input to one Map task.

In one or more other windows, run some workers:

$ go run mrworker.go wc.so sock123

When the workers and coordinator have finished, look at the output
in mr-out-*. When you've completed the lab, the
sorted union of the output files should match the sequential
output, like this:

$ cat mr-out-* | sort | more
A 509
ABOUT 2
ACT 8
ACTRESS 1
...

We supply you with all the tests that we'll use to grade your submitted
lab.  The source code for the tests are in mr/mr_test.go.
You can run the tests in the src directory:

$ cd src
$ make mr
...

The tests check that the wc and indexer MapReduce
applications produce the correct output when given
the pg-xxx.txt files as input. The tests also check that your
implementation runs the Map and Reduce tasks in parallel, and that
your implementation recovers from workers that crash while running
tasks.

If you run the tests now, they will hang in the first test:

$ cd ~/6.5840/src
$ make mr
...
cd mr; go test -v -race
=== RUN   TestWc
...

You can change ret := false to true in the Done function in mr/coordinator.go
so that the coordinator exits immediately. Then:

$ make mr
...
=== RUN   TestWc
2026/01/22 14:56:24 reduce created no mr-out-X output files!
exit status 1
FAIL    6.5840/mr       4.516s
make: *** [Makefile:44: mr] Error 1
$

The tests expect to see output in files named mr-out-X, one
for each reduce task. The empty implementations of mr/coordinator.go
and mr/worker.go don't produce those files (or do much of
anything else), so the test fails.

When you've finished, the test output should look like this:

$ make mr
...
=== RUN   TestWc
--- PASS: TestWc (8.64s)
=== RUN   TestIndexer
--- PASS: TestIndexer (5.90s)
=== RUN   TestMapParallel
--- PASS: TestMapParallel (7.05s)
=== RUN   TestReduceParallel
--- PASS: TestReduceParallel (8.05s)
=== RUN   TestJobCount
--- PASS: TestJobCount (10.04s)
=== RUN   TestEarlyExit
--- PASS: TestEarlyExit (6.05s)
=== RUN   TestCrashWorker
2026/01/22 14:58:14 *re*-starting map ../../main/pg-tom_sawyer.txt 0
2026/01/22 14:58:14 *re*-starting map ../../main/pg-metamorphosis.txt 2
2026/01/22 14:58:39 *re*-starting map ../../main/pg-metamorphosis.txt 2
2026/01/22 14:58:40 map 2 already done
2026/01/22 14:58:45 *re*-starting reduce 0
--- PASS: TestCrashWorker (40.18s)
PASS
ok      6.5840/mr       86.932s
$

Depending on your strategy for terminating worker processes, you may see errors like:

2026/02/11 16:21:32 dialing:dial unix /var/tmp/5840-mr-501: connect: connection refused

It is fine to see a handful of these messages per test; they arise when the worker is unable to contact the coordinator RPC server after
the coordinator has exited.

A few rules:

The map phase should divide the intermediate keys into buckets for
nReduce reduce tasks,
where nReduce is the number of reduce tasks -- the argument that
main/mrcoordinator.go passes to MakeCoordinator().
Each mapper should create nReduce intermediate files for
consumption by the reduce tasks.

 The worker implementation should put the output of the X'th
reduce task in the file mr-out-X.

 A mr-out-X file should contain one line per Reduce
function output. The line should be generated with the Go "%v %v"
format, called with the key and value. Have a look in main/mrsequential.go
for the line commented "this is the correct format".
The tests will fail if your implementation deviates too much from this format.

 You can modify mr/worker.go, mr/coordinator.go, and mr/rpc.go.
You can temporarily modify other files for testing, but make sure your code works
with the original versions; we'll test with the original versions.

 The worker should put intermediate Map output in files in the current
directory, where your worker can later read them as input to Reduce tasks.

 main/mrcoordinator.go expects mr/coordinator.go to implement a
Done() method that returns true when the MapReduce job is completely finished;
at that point, mrcoordinator.go will exit.

 When the job is completely finished, the worker processes should exit.
A simple way to implement this is to use the return value from call():
if the worker fails to contact the coordinator, it can assume that the coordinator has exited
because the job is done, so the worker can terminate too. Depending on your
design, you might also find it helpful to have a "please exit" pseudo-task
that the coordinator can give to workers.

Hints

The Guidance page has some
  tips on developing and debugging.

 One way to get started is to modify mr/worker.go's
Worker() to send an RPC to the coordinator asking for a task. Then
modify the coordinator to respond with the file name of an as-yet-unstarted
map task. Then modify the worker to read that file and call the
application Map function, as in mrsequential.go.

 The application Map and Reduce functions are loaded at run-time
using the Go plugin package, from files whose names end in .so.

 If you change anything in the mr/ directory, you will
probably have to re-build any MapReduce plugins you use, with
  something like go build -buildmode=plugin ../mrapps/wc.go.
  make mr builds the plugins for you.  You can run an
  individual test using make RUN="-run Wc" mr`, which passes
  "-run Wc" to go go test, and selects any test
  from mr/mr_test.go matching Wc.

 This lab relies on the workers sharing a file system.
That's straightforward when all workers run on the same machine, but would require a global
filesystem like GFS if the workers ran on different machines.

 A reasonable naming convention for intermediate files is mr-X-Y,
where X is the Map task number, and Y is the reduce task number.

 The worker's map task code will need a way to store intermediate
key/value pairs in files in a way that can be correctly read back
during reduce tasks. One possibility is to use Go's encoding/json package. To
write key/value pairs in JSON format to an open file:

  enc := json.NewEncoder(file)
  for _, kv := ... {
    err := enc.Encode(&kv)

and to read such a file back:

  dec := json.NewDecoder(file)
  for {
    var kv KeyValue
    if err := dec.Decode(&kv); err != nil {
      break
    }
    kva = append(kva, kv)
  }

 The map part of your worker can use the ihash(key) function
(in worker.go) to pick the reduce task for a given key.

 You can steal some code from mrsequential.go for reading
Map input files, for sorting intermedate key/value pairs between the
Map and Reduce, and for storing Reduce output in files.

 The coordinator, as an RPC server, will be concurrent; don't forget
to lock shared data.

 Workers will sometimes need to wait, e.g. reduces can't start
until the last map has finished. One possibility is for workers to
periodically ask the coordinator for work, sleeping
with time.Sleep() between each request. Another possibility
is for the relevant RPC handler in the coordinator to have a loop that
waits, either with time.Sleep() or sync.Cond. Go
runs the handler for each RPC in its own thread, so the fact that one
handler is waiting needn't prevent the coordinator from processing other
RPCs.

 The coordinator can't reliably distinguish between crashed workers,
workers that are alive but have stalled for some reason,
and workers that are executing but too slowly to be useful.
The best you can do is have the coordinator wait for
some amount of time, and then give up and re-issue the task to
a different worker. For this lab, have the coordinator wait for
ten seconds; after that the coordinator should assume the worker has
died (of course, it might not have).

 If you choose to implement Backup Tasks (Section 3.6), note that we test that your code doesn't
    schedule extraneous tasks when workers execute tasks without crashing.  Backup tasks should only
    be scheduled after some relatively long period of time (e.g., 10s).

 To test crash recovery, you can use the mrapps/crash.go
application plugin. It randomly exits in the Map and Reduce functions.

 To ensure that nobody observes partially written files in the presence of
    crashes, the MapReduce paper mentions the trick of using a temporary file
    and atomically renaming it once it is completely written. You can use
    ioutil.TempFile (or os.CreateTemp if you are running
    Go 1.17 or later) to create a temporary file and os.Rename
    to atomically rename it.

Go RPC sends only struct fields whose names start with capital letters.
  Sub-structures must also have capitalized field names.

 When calling the RPC call() function, the
reply struct should contain all default values. RPC calls
should look like this:

  reply := SomeType{}
  call(..., &reply)

without setting any fields of reply before the call. If you
pass reply structures that have non-default fields, the RPC
system may silently return incorrect values.

No-credit challenge exercises

Implement your own MapReduce application (see examples in mrapps/*), e.g., Distributed
Grep (Section 2.3 of the MapReduce paper).

Get your MapReduce coordinator and workers to run on separate machines, as they would in practice.
You will need to set up your RPCs to communicate over TCP/IP instead of Unix sockets (see
the commented out line in Coordinator.server()), and read/write files using a shared file
system. For example, you can ssh into multiple
Athena cluster
machines at MIT, which use
AFS
to share files; or you could rent a couple AWS instances and use
S3 for storage.

6.5840 Lab 3: Raft

6.5840 - Spring 2026

6.5840 Lab 3: Raft

  Collaboration policy //
  Submit lab //
  Setup Go //
  Guidance //
  Piazza

Introduction

This is the first in a series of labs in which you'll build a
fault-tolerant key/value storage system. In this
lab you'll implement Raft, a replicated state machine protocol.
In the next lab you'll build a key/value service on top of
Raft. Then you will “shard” your service over
multiple replicated state machines for higher performance.

A replicated service achieves fault
tolerance by storing complete copies of its state (i.e., data)
on multiple replica servers.
Replication allows
the service to continue operating even if some of
its servers experience failures (crashes or a broken or flaky
network). The challenge is that failures may cause the
replicas to hold differing copies of the data.

Raft organizes client requests into a sequence, called
the log, and ensures that all the replica servers see the same log.
Each replica executes client requests
in log order, applying them to its local copy of the service's state.
Since all the live replicas
see the same log contents, they all execute the same requests
in the same order, and thus continue to have identical service
state. If a server fails but later recovers, Raft takes care of
bringing its log up to date. Raft will continue to operate as
long as at least a majority of the servers are alive and can
talk to each other. If there is no such majority, Raft will
make no progress, but will pick up where it left off as soon as
a majority can communicate again.

In this lab you'll implement Raft as a Go object type
with associated methods, meant to be used as a module in a
larger service. A set of Raft instances talk to each other with
RPC to maintain replicated logs. Your Raft interface will
support an indefinite sequence of numbered commands, also
called log entries. The entries are numbered with index
numbers. The log entry with a given index will eventually
be committed. At that point, your Raft should send the log
entry to the larger service for it to execute.

You should follow the design in the
extended Raft paper,
with particular attention to Figure 2.
You'll implement most of what's in the paper, including saving
persistent state and reading it after a node fails and
then restarts. You will not implement cluster
membership changes (Section 6).

This lab is due in four parts. You must submit each part on the
corresponding due date.

Getting Started

If you have done Lab 1, you already have a copy of the lab
source code.
If not,
you can find directions for obtaining the source via git
in the Lab 1 instructions.

We supply you with skeleton code src/raft1/raft.go. We also
supply a set of tests, which you should use to drive your
implementation efforts, and which we'll use to grade your submitted
lab. The tests are in src/raft1/raft_test.go.

When we grade your submissions, we will run the tests without the -race flag.
However, you should test with -race.

To get up and running, execute the following commands.
Don't forget the git pull to get the latest software.

$ cd ~/6.5840
$ git pull
...
$ cd src
$ make raft1
go build -race -o main/raft1d main/raft1d.go
cd raft1 && go test -v -race
=== RUN   TestInitialElection3A
Test (3A): initial election (reliable network)...
Fatal: expected one leader, got none
        /Users/rtm/824-process-raft/src/raft1/test.go:151
        /Users/rtm/824-process-raft/src/raft1/raft_test.go:36
info: wrote visualization to /var/folders/x_/vk0xmxwn1sj91m89wsn5b1yh0000gr/T/porcupine-2242138501.html
--- FAIL: TestInitialElection3A (5.51s)
...
$

The code

Implement Raft by adding code to
raft1/raft.go. In that file you'll find
skeleton code, plus examples of how to send and receive
RPCs.

Your implementation must support the following interface, which
the tester and (eventually) your key/value server will use.
You'll find more details in comments in raft.go
and in raftapi/raftapi.go.

// create a new Raft server instance:
rf := Make(peers, me, persister, applyCh)

// start agreement on a new log entry:
rf.Start(command interface{}) (index, term, isleader)

// ask a Raft for its current term, and whether it thinks it is leader
rf.GetState() (term, isLeader)

// each time a new entry is committed to the log, each Raft peer
// should send an ApplyMsg to the service (or tester).
type ApplyMsg

A service calls Make(peers,me,…) to create a
Raft peer. The peers argument is an array of network identifiers
of the Raft peers (including this one), for use with RPC. The
me argument is the index of this peer in the peers
array. Start(command) asks Raft to start the processing
to append the command to the replicated log. Start()
should return immediately, without waiting for the log appends
to complete. The service expects your implementation to send an
ApplyMsg for each newly committed log entry to the
applyCh channel argument to Make().

raft.go contains example code that sends an RPC
(sendRequestVote()) and that handles an incoming RPC
(RequestVote()).
Your Raft peers should exchange RPCs using the labrpc Go
package (source in src/labrpc).
The tester can tell labrpc to delay RPCs,
re-order them, and discard them to simulate various network failures.
While you can temporarily modify labrpc,
make sure your Raft works with the original labrpc,
since that's what we'll use to test and grade your lab.
Your Raft instances must interact only through RPC; for example,
they are not allowed to communicate using shared Go variables
or files.

Subsequent labs build on this lab, so it is important to give
yourself enough time to write solid code.

Part 3A: leader election

Implement Raft leader election and heartbeats (AppendEntries RPCs with no
log entries). The goal for Part 3A is for a
single leader to be elected, for the leader to remain the leader
if there are no failures, and for a new leader to take over if the
old leader fails or if packets to/from the old leader are lost.
Run make RUN="-run 3A" raft1 in the src
directory to test your 3A code.

Follow the paper's Figure 2. At this point you care about sending
and receiving RequestVote RPCs, the Rules for Servers that relate to
elections, and the State related to leader election,

Add the Figure 2 state for leader election
to the Raft struct in raft.go.

Fill in the RequestVoteArgs and
RequestVoteReply structs. Modify
Make() to create a background goroutine that will kick off leader
 election periodically by sending out RequestVote RPCs when it hasn't
 heard from another peer for a while.
 Implement
 the RequestVote() RPC handler so that servers will vote for one
 another.

To implement heartbeats, define an
AppendEntries RPC struct (though you may not
need all the arguments yet), and have the leader send
them out periodically. Write an
AppendEntries RPC handler method.

The tester requires that the leader send heartbeat RPCs no more than
ten times per second.

The tester requires your Raft to elect a new leader within five
seconds of the failure of the old leader (if a majority of peers can
still communicate).

The paper's Section 5.2 mentions election timeouts in the range of 150
to 300 milliseconds. Such a range only makes sense if the leader
sends heartbeats considerably more often than once per 150
milliseconds (e.g., once per 10 milliseconds). Because the tester limits you tens of heartbeats per
second, you will have to use an election timeout larger
than the paper's 150 to 300 milliseconds, but not too large, because then you
may fail to elect a leader within five seconds.

You may find Go's
rand
useful.

You'll need to write code that takes actions periodically or
after delays in time. The easiest way to do this is to create
a goroutine with a loop that calls
time.Sleep();
see the ticker() goroutine that Make()
creates for this purpose.
Don't use Go's time.Timer or time.Ticker, which
are difficult to use correctly.

If your code has trouble passing the tests,
read the paper's Figure 2 again; the full logic for leader
election is spread over multiple parts of the figure.

Don't forget to implement GetState().

Go RPC sends only struct fields whose names start with capital letters.
  Sub-structures must also have capitalized field names (e.g. fields of log records
  in an array). The labgob package will warn you about this;
  don't ignore the warnings.

The most challenging part of this lab may be the debugging. Refer to
the Guidance page for debugging tips.

If you fail a test, the tester produces a file that visualizes a timeline with
events marked along it, including network partitions, crashed servers, and
checks performed. Here's an example of the
visualization. Further, you can add your own annotations by writing, for
example,
tester.Annotate("Server 0", "short description", "details").

Be sure you pass the 3A tests before submitting Part 3A, so that
you see something like this:

$ make RUN="-run 3A" raft1
go build -race -o main/raft1d main/raft1d.go
cd raft1 && go test -v -race -run 3A
=== RUN   TestInitialElection3A
Test (3A): initial election (reliable network)...
  ... Passed --  time  3.5s #peers 3 #RPCs    32 #Ops    0
--- PASS: TestInitialElection3A (3.84s)
=== RUN   TestReElection3A
Test (3A): election after network failure (reliable network)...
  ... Passed --  time  6.2s #peers 3 #RPCs    68 #Ops    0
--- PASS: TestReElection3A (6.54s)
=== RUN   TestManyElections3A
Test (3A): multiple elections (reliable network)...
  ... Passed --  time  9.8s #peers 7 #RPCs   684 #Ops    0
--- PASS: TestManyElections3A (10.68s)
PASS
ok      6.5840/raft1    22.095s
$

Each "Passed" line contains five numbers; these are the time that the
test took in seconds, the number of Raft peers, the
number of RPCs sent during the test, the total number of bytes in the
RPC messages, and the number of log entries
that Raft reports were committed. Your numbers will differ from those
shown here. You can ignore the numbers if you like, but they may help
you sanity-check the number of RPCs that your implementation sends.
For all of labs 3, 4, and 5, the grading script will fail your
solution if it takes more than 600 seconds for all of the tests,
or if any individual test takes more than 120
seconds.

When we grade your submissions, we will run the tests without
the -race
flag. However, you should make sure that your code consistently
passes the tests with the -race flag.

Part 3B: log

Implement the leader and follower code to append new log entries,
so that make RUN="-run 3B" raft1 passes all tests.

Run git pull to get the latest lab software.

The Raft paper views the log as 1-indexed, but we suggest that you implement
it as 0-indexed, starting
with a dummy entry at index=0 that has term 0. That allows the very
first AppendEntries RPC to contain 0 as PrevLogIndex, and be a valid index into
the log.

Your first goal should be to pass TestBasicAgree3B().
Start by implementing Start(), then write the code
to send and receive new log entries via AppendEntries RPCs,
following Figure 2. Send each newly committed entry
on applyCh on each peer.

You will need to implement the election
restriction (section 5.4.1 in the paper).

Your code may have loops that repeatedly check for certain events.
Don't have these loops
execute continuously without pausing, since that
will slow your implementation enough that it fails tests.
Use Go's
condition variables,
or insert a
time.Sleep(10 * time.Millisecond) in each loop iteration.

Do yourself a favor for future labs and write (or re-write) code
that's clean and clear.

If you fail a test, look at
  raft_test.go and trace the test code from there to
  understand what's being tested.

The tests for upcoming labs may fail your code if it runs too slowly.
You can check how much real time and CPU time your solution uses with
the time command. Here's typical output:

$ make RUN="-run 3B" raft1
go build -race -o main/raft1d main/raft1d.go
cd raft1 && go test -v -race -run 3B
=== RUN   TestBasicAgree3B
Test (3B): basic agreement (reliable network)...
  ... Passed --  time  1.6s #peers 3 #RPCs    18 #Ops    3
--- PASS: TestBasicAgree3B (1.96s)
=== RUN   TestRPCBytes3B
Test (3B): RPC byte count (reliable network)...
  ... Passed --  time  3.3s #peers 3 #RPCs    50 #Ops   11
--- PASS: TestRPCBytes3B (3.71s)
=== RUN   TestFollowerFailure3B
Test (3B): test progressive failure of followers (reliable network)...
  ... Passed --  time  5.4s #peers 3 #RPCs    58 #Ops    3
--- PASS: TestFollowerFailure3B (5.77s)
=== RUN   TestLeaderFailure3B
Test (3B): test failure of leaders (reliable network)...
  ... Passed --  time  6.5s #peers 3 #RPCs   110 #Ops    3
--- PASS: TestLeaderFailure3B (6.89s)
=== RUN   TestFailAgree3B
Test (3B): agreement after follower reconnects (reliable network)...
  ... Passed --  time  6.0s #peers 3 #RPCs    61 #Ops    7
--- PASS: TestFailAgree3B (6.37s)
=== RUN   TestFailNoAgree3B
Test (3B): no agreement if too many followers disconnect (reliable network)...
  ... Passed --  time  4.0s #peers 5 #RPCs   107 #Ops    2
--- PASS: TestFailNoAgree3B (4.55s)
=== RUN   TestConcurrentStarts3B
Test (3B): concurrent Start()s (reliable network)...
  ... Passed --  time  1.4s #peers 3 #RPCs    12 #Ops    0
--- PASS: TestConcurrentStarts3B (1.75s)
=== RUN   TestRejoin3B
Test (3B): rejoin of partitioned leader (reliable network)...
  ... Passed --  time  7.8s #peers 3 #RPCs   120 #Ops    4
--- PASS: TestRejoin3B (8.15s)
=== RUN   TestBackup3B
Test (3B): leader backs up quickly over incorrect follower logs (reliable network)...
  ... Passed --  time 27.7s #peers 5 #RPCs  1370 #Ops  102
--- PASS: TestBackup3B (28.27s)
=== RUN   TestCount3B
Test (3B): RPC counts aren't too high (reliable network)...
  ... Passed --  time  2.7s #peers 3 #RPCs    32 #Ops    0
--- PASS: TestCount3B (3.05s)
PASS
ok      6.5840/raft1    71.716s
$

The "ok 6.5840/raft 71.716s" means that Go measured the time taken for the 3B
tests to be 71.716 seconds of real (wall-clock) time.
If your solution uses much more than a few minutes of real time
for the 3B tests, you may run
into trouble later on. Look for time spent sleeping or waiting for RPC
timeouts, loops that run without sleeping or waiting for conditions or
channel messages, or large numbers of RPCs sent.

Part 3C: persistence

If a Raft-based server reboots it should resume service
where it left off. This requires
that Raft keep persistent state that survives a reboot. The
paper's Figure 2 mentions which state should be persistent.

A real implementation would write
Raft's persistent state to disk each time it changed, and would read the
state from
disk when restarting after a reboot. Your implementation won't use
the disk; instead, it will save and restore persistent state
from a Persister object (see tester1/persister.go).
Whoever calls Raft.Make() supplies a Persister
that initially holds Raft's most recently persisted state (if
any). Raft should initialize its state from that
Persister, and should use it to save its persistent
state each time the state changes. Use the Persister's
ReadRaftState() and Save() methods.

Complete the functions
persist()
and
readPersist() in raft.go
by adding code to save and restore persistent state. You will need to encode
(or "serialize") the state as an array of bytes in order to pass it to
the Persister. Use the labgob encoder;
see the comments in persist() and readPersist().
labgob is like Go's gob encoder but
prints error messages if
you try to encode structures with lower-case field names.
For now, pass nil as the second argument to persister.Save().
Insert calls to persist() at the points where
your implementation changes persistent state.
Once you've done this,
and if the rest of your implementation is correct,
you should pass all of the 3C tests.

You will probably need the optimization that backs up
nextIndex by more than one entry
at a time. Look at the extended Raft paper starting at
the bottom of page 7 and top of page 8 (marked by a gray line).
The paper is vague about the details; you will need to fill in the gaps.
One possibility is to have a rejection message include:

    XTerm:  term in the conflicting entry (if any)
    XIndex: index of first entry with that term (if any)
    XLen:   log length

Then the leader's logic can be something like:

  Case 1: leader doesn't have XTerm:
    nextIndex = XIndex
  Case 2: leader has XTerm:
    nextIndex = (index of leader's last entry for XTerm) + 1
  Case 3: follower's log is too short:
    nextIndex = XLen

A few other hints:

Run git pull to get the latest lab software.

The 3C tests are more demanding than those for 3A or 3B, and failures
may be caused by problems in your code for 3A or 3B.

Your code should pass all the 3C tests (as shown below), as well as
the 3A and 3B tests.

$ make RUN="-run 3C" raft1
go build -race -o main/raft1d main/raft1d.go
cd raft1 && go test -v -race -run 3C
=== RUN   TestPersist13C
Test (3C): basic persistence (reliable network)...
  ... Passed --  time  7.6s #peers 3 #RPCs    58 #Ops    6
--- PASS: TestPersist13C (7.99s)
=== RUN   TestPersist23C
Test (3C): more persistence (reliable network)...
  ... Passed --  time 21.6s #peers 5 #RPCs   287 #Ops   16
--- PASS: TestPersist23C (22.17s)
=== RUN   TestPersist33C
Test (3C): partitioned leader and one follower crash, leader restarts (reliable network)...
  ... Passed --  time  3.8s #peers 3 #RPCs    30 #Ops    4
--- PASS: TestPersist33C (4.11s)
=== RUN   TestFigure83C
Test (3C): Figure 8 (reliable network)...
  ... Passed --  time 48.5s #peers 5 #RPCs   499 #Ops    2
--- PASS: TestFigure83C (49.08s)
=== RUN   TestUnreliableAgree3C
Test (3C): unreliable agreement (unreliable network)...
  ... Passed --  time  5.1s #peers 5 #RPCs   288 #Ops  246
--- PASS: TestUnreliableAgree3C (5.68s)
=== RUN   TestFigure8Unreliable3C
Test (3C): Figure 8 (unreliable) (unreliable network)...
  ... Passed --  time 53.6s #peers 5 #RPCs  3200 #Ops    2
--- PASS: TestFigure8Unreliable3C (54.19s)
=== RUN   TestReliableChurn3C
Test (3C): churn (reliable network)...
  ... Passed --  time 18.2s #peers 5 #RPCs  1701 #Ops    1
--- PASS: TestReliableChurn3C (18.80s)
=== RUN   TestUnreliableChurn3C
Test (3C): unreliable churn (unreliable network)...
  ... Passed --  time 17.3s #peers 5 #RPCs  1253 #Ops    1
--- PASS: TestUnreliableChurn3C (17.92s)
PASS
ok      6.5840/raft1    180.983s
$

It is a good idea to run the tests multiple times before submitting.

Part 3D: log compaction

As things stand now, a rebooting server replays the
complete Raft log in order to restore its state. However, it's not
practical for a long-running service to remember the complete Raft log
forever. Instead, you'll modify Raft to cooperate with services that
persistently store a "snapshot" of their state from time to time, at
which point Raft discards log entries that precede the snapshot. The
result is a smaller amount of persistent data and faster restart.
However, it's now possible for a follower to fall so far behind that
the leader has discarded the log entries it needs to catch up; the
leader must then send a snapshot plus the log starting at the time of
the snapshot. Section 7 of the
extended Raft paper
outlines the scheme; you will have to design the details.

Your Raft must provide the following function that the service
can call with a serialized snapshot of its state:

Snapshot(index int, snapshot []byte)

In Lab 3D, the tester calls Snapshot() periodically. In Lab 4, you will
write a key/value server that calls Snapshot(); the snapshot
will contain the complete table of key/value pairs.
The service layer calls Snapshot() on every peer (not
just on the leader).

The index argument indicates the highest log entry that's
reflected in the snapshot. Raft should discard its log entries before
that point. You'll need to revise your Raft code to operate while
storing only the tail of the log.

You'll need to implement the InstallSnapshot RPC discussed in
the paper that allows a Raft leader to tell a lagging Raft peer to
replace its state with a snapshot. You will likely need to think
through how InstallSnapshot should interact with the state and rules
in Figure 2.

When a follower's Raft code receives an InstallSnapshot RPC, it can
use the applyCh to send the snapshot to the service in
an ApplyMsg. The ApplyMsg struct definition
in raftapi/raftapi.go already
contains the fields you will need (and which the tester expects). Take
care that these snapshots only advance the service's state, and don't
cause it to move backwards.

If a server crashes, it must restart from persisted data. Your Raft
should persist both Raft state and the corresponding snapshot.
Use the second argument to
persister.Save() to save the snapshot.
If there's no snapshot, pass nil as the second
argument.

When a server restarts, the application layer reads the persisted
snapshot and restores its saved application state. After a restart,
the application layer expects the first message on applyCh to either
contain a snapshot with a SnapshotIndex higher than that of the
initial restored snapshot, or an ordinary command with CommandIndex
immediately following the index of the initial restored snapshot.

Implement Snapshot() and the InstallSnapshot RPC, as well as the
changes to Raft to support these (e.g, operation with a
trimmed log).  Your solution is complete when it passes the 3D tests
(and all the previous Lab 3 tests).

 git pull to make sure you have the latest software.

 A good place to start is to modify your code to so that it is
able to store just the part of the log
starting at some index X. Initially you can set X to zero and
run the 3B/3C tests.
Then make Snapshot(index) discard the log before index,
and set X equal to index. If all goes well you should
now pass the first 3D test.

A common reason for failing the first 3D test is that followers take too long to
catch up to the leader.

Next: have the leader send an InstallSnapshot RPC if it doesn't
have the log entries required to bring a follower up to date.

Send the entire snapshot in a single InstallSnapshot RPC.
Don't implement Figure 13's offset mechanism for
splitting up the snapshot.

 Raft must discard old log entries in a way that allows the Go garbage collector to free and re-use the
memory; this requires that there be no reachable references (pointers)
to the discarded log entries.

When a Raft peer is re-started, the persister passed to Make()
will contain a snapshot of application state as well as Raft's saved
state. Raft must include a non-nil snapshot with every call
to persister.Save() (if the log has been trimmed), which
means that it's a good idea for
Make() to call persister.ReadSnapshot() and save the
result.

A reasonable amount of time to consume for the full set of
Lab 3 tests (3A+3B+3C+3D) without -race is 6 minutes of real time and one
minute of CPU time. When running with -race, it is about 10 minutes of real
time and two minutes of CPU time.

Your code should pass all the 3D tests (as shown below), as well as the 3A, 3B, and 3C tests.

$ make RUN="-run 3D" raft1
go build -race -o main/raft1d main/raft1d.go
cd raft1 && go test -v -race -run 3D
=== RUN   TestSnapshotBasic3D
Test (3D): snapshots basic (reliable network)...
  ... Passed --  time  8.4s #peers 3 #RPCs   279 #Ops   31
--- PASS: TestSnapshotBasic3D (8.74s)
=== RUN   TestSnapshotInstall3D
Test (3D): install snapshots (disconnect) (reliable network)...
  ... Passed --  time 59.6s #peers 3 #RPCs   919 #Ops   91
--- PASS: TestSnapshotInstall3D (59.99s)
=== RUN   TestSnapshotInstallUnreliable3D
Test (3D): install snapshots (disconnect) (unreliable network)...
  ... Passed --  time 82.1s #peers 3 #RPCs  1083 #Ops   91
--- PASS: TestSnapshotInstallUnreliable3D (82.49s)
=== RUN   TestSnapshotInstallCrash3D
Test (3D): install snapshots (crash) (reliable network)...
  ... Passed --  time 53.6s #peers 3 #RPCs   685 #Ops   91
--- PASS: TestSnapshotInstallCrash3D (53.99s)
=== RUN   TestSnapshotInstallUnCrash3D
Test (3D): install snapshots (crash) (unreliable network)...
  ... Passed --  time 66.2s #peers 3 #RPCs   717 #Ops   91
--- PASS: TestSnapshotInstallUnCrash3D (66.60s)
=== RUN   TestSnapshotAllCrash3D
Test (3D): crash and restart all servers (unreliable network)...
  ... Passed --  time 20.4s #peers 3 #RPCs   244 #Ops   45
--- PASS: TestSnapshotAllCrash3D (20.79s)
=== RUN   TestSnapshotInit3D
Test (3D): snapshot initialization after crash (unreliable network)...
  ... Passed --  time  7.4s #peers 3 #RPCs    79 #Ops   14
--- PASS: TestSnapshotInit3D (7.77s)
PASS
ok      6.5840/raft1    301.406s
$

6.5840 Lab 5: Sharded Key/Value Service



            6.5840 - Spring 2026

            6.5840 Lab 5: Sharded Key/Value Service


  Collaboration policy //
  Submit lab //
  Setup Go //
  Guidance //
  Piazza





                Introduction


                  You can either do a final project based
                  on your own ideas, or this lab.


        In this lab you'll build a key/value storage system
        that "shards," or partitions, the keys over a set of
                Raft-replicated key/value server groups
        (shardgrps).  A shard is a subset of the
        key/value pairs; for example, all the keys starting
        with "a" might be one shard, all the keys starting
        with "b" another, etc. The reason for sharding is
        performance. Each shardgrp
        handles puts and gets for just a few of
        the shards, and the shardgrps operate in parallel;
        thus total system throughput (puts and gets per unit
        time) increases in proportion to the number of
        shardgrps.





        The sharded key/value service has the components shown
        above. Shardgrps (shown with blue squares) store
        shards with keys: shardgrp 1 holds a shard storing key
        "a", and shardgrp 2 holds a shard storing key
        "b". Clients of the sharded key/value service interact
        with the service through a clerk (shown with a green
        circle), which implements Get
        and Put methods.  To find the shardgrp for a
        key passed to Put/Get, the clerk
        gets the configuration from the kvsrv (shown with a
        black square), which you implemented in Lab 2. The
        configuration (not shown) describes the mapping from
        shards to shardgrps (e.g., shard 1 is served by
        shardgrp 3).

                An administrator (i.e., the tester) uses another
                client, the controller (shown with a purple circle),
                to add/remove shardgrps from the cluster and
                update which shardgrp should serve a shard. The
                controller has one main
                method: ChangeConfigTo, which takes as
                argument a new configuration and changes the system
                from the current configuration to the new
                configuration; this involves moving shards to new
                shardgrps that are joining the system and moving
                shards away from shardgrps that are leaving the system.  To
                do so the controller 1) makes RPCs
                (FreezeShard, InstallShard,
                and DeleteShard) to shardgrps, and 2) updates the
                configuration stored in kvsrv.

         The reason for the controller is that a sharded
        storage system must be able to shift shards among
        shardgrps. One reason is that some shardgrps may
        become more loaded than others, so that shards need to
        be moved to balance the load. Another reason is that
        shardgrps may join and leave the system: new shardgrps
        may be added to increase capacity, or existing
        shardgrps may be taken offline for repair or
        retirement.

         The main challenges in this lab will be ensuring
        linearizability of
                Get/Put operations while handling 1)
                  changes in the assignment of shards to shardgrps,
                and 2) recovering from a controller that fails or is partitioned
                during ChangeConfigTo.


                ChangeConfigTo moves shards from one
        shardgrp to another. A risk is that some clients might
        use the old shardgrp while other clients use the new
        shardgrp, which could break linearizability.  You will
        need to ensure that at most one shardgrp is serving
        requests for each shard at any one time.

                If ChangeConfigTo fails while
                reconfiguring, some shards may be inaccessible if they
                have started but not completed moving from one
                shardgrp to another.  To make forward progress, the
                tester starts a new controller, and your job is to
                ensure that the new one completes the reconfiguration
                that the previous controller started.




        This lab uses "configuration" to refer to the
        assignment of shards to shardgrps. This is not the
        same as Raft cluster membership changes. You don't
        have to implement Raft cluster membership changes.

                 A shardgrp server is a member of only
                  a single shardgrp. The set of servers in a
                  given shardgrp will never change.


        Only RPC may be used for interaction among clients and servers.
        For example, different instances of your server are not allowed
        to share Go variables or files.


    In Part A, you will implement a working shardctrler,
    which will store and retrieve configurations in a kvsrv.
    You will also implement the shardgrp, replicated
    with your Raft rsm package, and a corresponding
    shardgrp clerk. The shardctrler talks
    to the shardgrp clerks to move shards between
    different groups.


    In Part B, you will modify your shardctrler to handle
    failures and partitions during config changes. In Part C,
    you will extend your shardctrler to allow for concurrent
    controllers without interfering with each other. Finally, in
    Part D, you will have the opportunity to extend your solution
    in any way you like.


        This lab's sharded key/value service follows the same
        general design as Flat Datacenter Storage, BigTable,
        Spanner, FAWN, Apache HBase, Rosebud, Spinnaker, and
        many others. These systems differ in many details from
        this lab, though, and are also typically more
        sophisticated and capable. For example, the lab
        doesn't evolve the sets of peers in each Raft group;
        its data and query models are simple; and so on.


          Lab 5 will use your kvsrv from Lab 2, and your
          rsm and Raft from Lab 4. Your Lab 5
      and Lab 4 must use the same rsm and Raft
      implementations.


      You may use late hours for Part A, but you may not use
      late hours for Parts B-D.

        Getting Started



        Do a git pull to get the latest lab software.



        We supply you with tests and skeleton code
        in src/shardkv1:

          client.go for the shardkv clerk
                   shardcfg package for computing shard
            configurations
        shardgrp package: for the shardgrp
        clerk and server.
        shardctrler
          package, which contains shardctrler.go with
                  methods for the controller to change a configuration
          (ChangeConfigTo) and to get a
          configuration (Query)


        To get up and running, execute the following commands:

$ cd ~/6.5840
$ git pull
...
$ cd src
$ make RUN="-run 5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5A
=== RUN  TestInitQuery5A
Test (5A): Init and Query ... (reliable network)...
...

        Part A: Moving shards


                  Your first job is to implement shardgrps and the
                  InitConfig, Query,
                  and ChangeConfigTo methods
                  when there are no failures.
                  We have given you the code for describing a
                  configuration, in
                  shardkv1/shardcfg.
                  Each shardcfg.ShardConfig has a unique
                  identifying number, a mapping from shard number to
                  group number, and a mapping from group number to the
                  list of servers replicating that group.
          There will usually be more shards than groups
          (so that each group serves more than one shard), in
          order that load can be shifted at a fairly fine
          granularity.


          Implement these two methods in
                  shardctrler/shardctrler.go:



        The InitConfig method receives the first
        configuration, passed to it by the tester as
                a shardcfg.ShardConfig.
                InitConfig should store the configuration in
                an instance of Lab 2's kvsrv.

         The Query method returns the current
        configuration; it should read the configuration from
                kvsrv, previously stored there by InitConfig.




                   Implement InitConfig and Query, and
                   store the configuration in kvsrv.
           You're done when your code passes the
           first test.  Note this task doesn't require any shardgrps.

$ cd ~/6.5840/src
$ make RUN="-run TestInitQuery5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestInitQuery5A
=== RUN   TestInitQuery5A
Test (5A): Init and Query ... (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     3 #Ops    0
--- PASS: TestInitQuery5A (0.13s)
PASS
ok      6.5840/shardkv1 1.143s


                       Implement InitConfig and Query
                        by storing and reading the initial
                        configuration from kvsrv:
                        use the Get/Put methods of
                        ShardCtrler.IKVClerk to talk to kvsrv,
                        use the String method
                        of ShardConfig to turn
                        a ShardConfig into a string that you
                        can pass to Put,
                        and use the shardcfg.FromString() function to
                        turn a string into a ShardConfig.



                  Implement an initial version of shardgrp
                  in shardkv1/shardgrp/server.go and a corresponding
                  clerk in shardkv1/shardgrp/client.go by
                  copying code from your Lab 4 kvraft solution.


                  Implement a clerk in shardkv1/client.go
                  that uses the Query method to find the
                  shardgrp for a key, and then talks to that
                  shardgrp.  You're done when your code passes the
                  Static test.

$ cd ~/6.5840/src
$ make RUN="-run TestStatiOneShardGroup5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestStaticOneShardGroup5A
=== RUN   TestStaticOneShardGroup5A
Test (5A): one shard group ... (reliable network)...
  ... Passed --  time  9.4s #peers 1 #RPCs   822 #Ops  420
--- PASS: TestStaticOneShardGroup5A (9.52s)
PASS
ok      6.5840/shardkv1 10.542s


                       Copy code from your kvraft client.go and
                server.go for Put and Get,
                and any other code you need from kvraft.

                  The code in shardkv1/client.go provides
                  the Put/Get clerk for the overall system: it finds out
                  which shardgrp holds the desired key's shard by
                  invoking the Query method,
                  and then talks to the shardgrp that holds
                  that shard.


                  Implement
          shardkv1/client.go, including its Put/Get methods.
          Use shardcfg.Key2Shard()
          to find the shard number for a key.
          The tester passes a
          ShardCtrler object to
          MakeClerk in shardkv1/client.go.
                  Retrieve the current configuration using the
                  Query method.

              To put/get a key from a shardgrp, the shardkv
              clerk should create a shardgrp clerk for the
              shardgrp by calling shardgrp.MakeClerk,
              passing in the servers found in the
              configuration and the shardkv
              clerk's ck.clnt. Use the
            GidServers() method from ShardConfig to get the group for
            a shard.

                     shardkv1/client.go's Put must
                     return ErrMaybe when the reply was maybe lost, but
                     this Put invokes shardgrp's Put to talk
                     a particular shardgrp.  The inner Put can signal
                     this with an error.

                      Upon creation, the first shardgrp (shardcfg.Gid1)
                        should initialize itself to own all shards.



  Now you should support movement of shards among groups by
  implementing
  the ChangeConfigTo method, which
  changes from an old configuration to a new configuration.  The new
  configuration may include new shardgrps that are not present in the
  old configuration, and may exclude shardgrps that were present
  in the old configuration. The controller should move shards
  (the key/value data) so that the set of shards stored by each
  shardgrp matches the new configuration.

                  The approach we suggest for
                moving a shard is for ChangeConfigTo
        to first "freeze" the shard at the source
        shardgrp, causing that shardgrp to
        reject Put's for keys in the moving
        shard. Then, copy (install) the shard to the destination
        shardgrp; then delete the frozen shard. Finally,
        post a new configuration so that clients can find the
        moved shard.  A nice property of this approach is that it
        avoids any direct interactions among the shardgrps.
        It also supports serving shards that are not affected
        by an ongoing configuration change.

                  To be able to order changes to the configuration,
                    each configuration has a unique
                    number Num
                    (see shardcfg/shardcfg.go).  The tester
                    in Part A invokes ChangeConfigTo
                    sequentially, and the configuration passed
                    to ChangeConfigTo will have
                    a Num one larger than the previous one;
                    thus, a configuration with a higher
                    Num is newer than one with a
                    lower Num.

                  The network may delay RPCs, and RPCs may arrive
                    out of order at the shardgrps.  To reject old
                    FreezeShard, InstallShard,
                    and DeleteShard RPCs, they should
                    include Num
                    (see shardgrp/shardrpc/shardrpc.go), and
                    shardgrps must remember the largest Num
                    they have seen for each shard.

          Implement ChangeConfigTo
                    (in shardctrler/shardctrler.go) and
                    extend shardgrp to support freeze,
                    install, and delete.  ChangeConfigTo
                    should always succeed in Part A because the tester
                    doesn't induce failures in this part. You will
                    need to implement
                 FreezeShard, InstallShard, and DeleteShard
                in shardgrp/client.go
                and shardgrp/server.go using the RPCs in
                the shardgrp/shardrpc package, and reject old RPCs
                based on Num. You will also need modify the
                shardkv clerk in shardkv1/client.go to
        handle ErrWrongGroup, which a shardgrp should
        return if it isn't reponsible for the shard.


                You have completed this task when
                 you pass the JoinBasic and DeleteBasic tests. These
                 tests focus on adding shardgrps; you don't have to
                worry about shardgrps leaving just yet.



                   A shardgrp should respond with
                  an ErrWrongGroup error to a
                  client Put/Get with a key that the
                  shardgrp isn't responsible for (i.e., for a key
                  whose shard is not assigned to the shardgrp).  You
                  will have to modify shardkv1/client.go to
                  reread the configuration and retry
                  the Put/Get.

           Note that you will have to run FreezeShard,
                  InstallShard, and DeleteShard through
                  your rsm package, just like Put
                  and Get.

          You can send an entire map as your state
        in an RPC request or reply, which may help
        keep the code for shard transfer simple.

         If one of your RPC handlers includes in its reply
              a map (e.g. a key/value map) that's part of your
              server's state, you may get bugs due to races. The
              RPC system has to read the map in order to send it
              to the caller, but it isn't holding a lock that
              covers the map. Your server, however, may proceed to
              modify the same map while the RPC system is reading
              it. The solution is for the RPC handler to include a
              copy of the map in the reply.



                  Extend ChangeConfigTo to
                    handle shard groups that leave; i.e., shardgrps
                    that are present in the current configuration but
                    not in the new one. Your solution should
                    pass TestJoinLeaveBasic5A now.  (You
                    may have handled this scenario already in the
                    previous task, but the previous tests didn't test
                    for shardgrps leaving.)

                    Make your solution pass all Part A
                 tests, which check that your sharded key/value
                 service supports many groups joining and leaving,
                 shardgrps restarting from snapshots, processing Gets
                 while some shards are offline or involved in a
                 configuration change, and linearizability when many
                 clients interact with the service while the tester
                 concurrently invokes the controller's ChangeConfigTo to
                 rebalance shards.



$ cd ~/6.5840/src
$ make RUN="-run 5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5A
Test (5A): Init and Query ... (reliable network)...
  ... Passed --  time  0.0s #peers 1 #RPCs     3 #Ops    0
Test (5A): one shard group ... (reliable network)...
  ... Passed --  time  5.1s #peers 1 #RPCs   792 #Ops  180
Test (5A): a group joins... (reliable network)...
  ... Passed --  time 12.9s #peers 1 #RPCs  6300 #Ops  180
Test (5A): delete ... (reliable network)...
  ... Passed --  time  8.4s #peers 1 #RPCs  1533 #Ops  360
Test (5A): basic groups join/leave ... (reliable network)...
  ... Passed --  time 13.7s #peers 1 #RPCs  5676 #Ops  240
Test (5A): many groups join/leave ... (reliable network)...
  ... Passed --  time 22.1s #peers 1 #RPCs  3529 #Ops  180
Test (5A): many groups join/leave ... (unreliable network)...
  ... Passed --  time 54.8s #peers 1 #RPCs  5055 #Ops  180
Test (5A): shutdown ... (reliable network)...
  ... Passed --  time 11.7s #peers 1 #RPCs  2807 #Ops  180
Test (5A): progress ... (reliable network)...
  ... Passed --  time  8.8s #peers 1 #RPCs   974 #Ops   82
Test (5A): progress ... (reliable network)...
  ... Passed --  time 13.9s #peers 1 #RPCs  2443 #Ops  390
Test (5A): one concurrent clerk reliable... (reliable network)...
  ... Passed --  time 20.0s #peers 1 #RPCs  5326 #Ops 1248
Test (5A): many concurrent clerks reliable... (reliable network)...
  ... Passed --  time 20.4s #peers 1 #RPCs 21688 #Ops 10500
Test (5A): one concurrent clerk unreliable ... (unreliable network)...
  ... Passed --  time 25.8s #peers 1 #RPCs  2654 #Ops  176
Test (5A): many concurrent clerks unreliable... (unreliable network)...
  ... Passed --  time 25.3s #peers 1 #RPCs  7553 #Ops 1896
PASS
ok      6.5840/shardkv1 243.115s
$

                 Your solution must continue serving shards
        that are not affected by an ongoing configuration
        change.

        Part B: Handling a failed controller

                The controller is a short-lived command, which an
                  administrator invokes: it moves shards and then
                  exits.  But, it may fail or lose network
                  connectivity while moving shards.  The main task in
                  this part of the lab is recovering from a controller
                  that fails to complete
                  ChangeConfigTo. The tester starts a new
                  controller and invokes its ChangeConfigTo
                  after partitioning the first controller; you have to
                  modify the controller so that the new one finishes
                  the reconfiguration.
                The tester calls InitController when
                starting a controller; you can modify that function
                to check whether an interrupted configuration
                change needs to be completed.

              A good approach to allowing a controller to
                finish a reconfiguration that a previous one started
                is to keep two configurations: a current one and a
                next one, both stored in the controller's kvsrv.
        When a controller starts a reconfiguration, it
        stores the next configuration.
                Once a controller completes the reconfiguration, it
                makes the next configuration the current one.
                Modify InitController
                to first check if there is a stored next
                configuration with a higher configuration number than
                the current one, and if so, complete the shard moves
                necessary to reconfigure to the next one.


                Modify shardctrler to implement the above approach.
                A controller that picks up the work from a
                failed controller may
                repeat FreezeShard, InstallShard,
                and Delete RPCs; shardgrps can use Num to
                detect duplicates and reject them.  You have completed
                this task if your solution passes the Part B tests.

$ cd ~/6.5840/src
$ make RUN="-run 5B" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5B
Test (5B): Join/leave while a shardgrp is down... (reliable network)...
  ... Passed --  time  9.2s #peers 1 #RPCs   899 #Ops  120
Test (5B): recover controller ... (reliable network)...
  ... Passed --  time 26.4s #peers 1 #RPCs  3724 #Ops  360
PASS
ok      6.5840/shardkv1 35.805s
$


                   The tester calls InitController
                  when starting a controller; you can implement
                  recovery in that method
                    in shardctrler/shardctrler.go.


        Part C: Concurrent configuration changes

        In this part of the lab you will modify the
        controller to allow for concurrent controllers. When a
        controller crashes or is partitioned, the tester will
        start a new controller, which must finish any work
        that the old controller might have in progress (i.e.,
        finishing moving shards like in Part B).  This means
        that several controllers may run concurrently and
        send RPCs to the shardgrps and the kvsrv that stores
        configurations.

        The main challenge is to ensure these controllers
        don't step on each other.  In Part A you already
        fenced all the shardgrp RPCs with Num so that
        old RPCs are rejected.  Even if several controllers
        pick up the work of an old controller concurrently,
        one of them succeeds and the others repeat all the
        RPCs, the shardgrps will ignore them.

                   Thus the challenging case left is to ensure that
          only one controller updates the next configuration
          to avoid that two controllers (e.g., a partitioned
          one and a new one) put different configurations in
          the next one.  To stress this scenario, the tester runs
          several controllers concurrently and each one
          computes the next configuration by reading the
          current configuration and updating it for a shardgrp
          that left or joined, and then the tester
          invokes ChangeConfigTo; thus multiple
          controllers may invoke ChangeConfigTo with
          different configuration with the same Num.
          You can use the version number of a key and
          versioned Puts to ensure that only one
          controller updates the next configuration and that
          the other invocations return without doing anything.


          Modify your controller so that only one controller
          can post a next configuration for a
          configuration Num.  The tester will start
          many controllers but only one should
          start ChangeConfigTo for a new
          configuation.
          You have completed this task
          if you pass the concurrent tests of
          Part C:

$ cd ~/6.5840/src
$ make RUN="-run TestConcurrentReliable5C" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestConcurrentReliable5C
Test (5C): Concurrent ctrlers ... (reliable network)...
  ... Passed --  time  8.2s #peers 1 #RPCs  1753 #Ops  120
PASS
ok      6.5840/shardkv1 8.364s
$ make RUN="-run TestAcquireLockConcurrentUnreliable5C" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestAcquireLockConcurrentUnreliable5C
Test (5C): Concurrent ctrlers ... (unreliable network)...
  ... Passed --  time 23.8s #peers 1 #RPCs  1850 #Ops  120
PASS
ok      6.5840/shardkv1 24.008s
$


                   See concurCtrler in test.go
                     to see how the tester runs controllers concurrently.



        In this exercise you will put recovery of an old
        controller together with a new controller: a new
        controller should perform recovery from Part B.  If
        the old controller was partitioned
        during ChangeConfigTo, you will have to make
        sure that the old controller doesn't interfere with the
        new controller.  If all the controller's updates are
        already properly fenced with Num checks from Part B,
        you don't have to write extra code.  You have completed
        this task if you pass the
                Partition tests.

$ cd ~/6.5840/src
$ make RUN="-run Partition" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run Partition
Test (5C): partition controller in join... (reliable network)...
  ... Passed --  time  7.8s #peers 1 #RPCs   876 #Ops  120
Test (5C): controllers with leased leadership ... (reliable network)...
  ... Passed --  time 36.8s #peers 1 #RPCs  3981 #Ops  360
Test (5C): controllers with leased leadership ... (unreliable network)...
  ... Passed --  time 52.4s #peers 1 #RPCs  2901 #Ops  240
Test (5C): controllers with leased leadership ... (reliable network)...
  ... Passed --  time 60.2s #peers 1 #RPCs 27415 #Ops 11182
Test (5C): controllers with leased leadership ... (unreliable network)...
  ... Passed --  time 60.5s #peers 1 #RPCs 11422 #Ops 2336
PASS
ok      6.5840/shardkv1 217.779s
$

                You have completed implementing a highly-available
                sharded key/value service with many shard groups for
                scalability, reconfiguration to handle changes in
                load, and with a fault-tolerant controller; congrats!

        Rerun all tests to check that your
                recent changes to the controller haven't broken earlier
                tests.


      Gradescope will rerun the Lab 3A-D and Lab 4A-C tests
      on your submission, in addition to the 5C tests.
      Before submitting, double check that your solution works:

$ make raft1
$ make kvraft1
$ make shardkv




        Part D: Extend your solution

                In this final part of the lab, you get to extend
                your solution in any way you like. You will have to
                write your own tests for whatever extensions you choose
                to implement.

        Implement one of the ideas below or
        come up with your own idea.  Write a paragraph in a
        file extension.md describing your extension,
        and upload extension.md to Gradescope.
        If you would like to do one of the harder, open-ended
        extensions, feel free to partner up with another
        student in the class.

                Here are some ideas for possible extension (the first
                few are easy and the later ones are more open ended):


                      Change the tester
                    to use kvraft instead of kvsrv
                    (i.e., replace the kvsrv.StartKVServer
                    in MakeTestMaxRaft in test.go
                    with kvraft.StartKVServer) so that the
                    controller uses your kvraft to store its
                    configuration.  Write a test that checks that the
                    controller can query and update the configuration
                    while one of the kvraft peers is down. The
                    existing code for the tester is distributed across
                    src/kvtest1, src/shardkv1, and
                    src/tester1.


                      Change kvsrv to implement
                      exactly-once semantics for Put/Get as in
                      Lab 2
                      from last year (see the dropped messages part).
                      You may be able to port over some tests from
                      2024 instead of having to write your own from
                      scratch.  Implement exactly-once also in your kvraft.


                    Change kvsrv to support a
                    Range function, which returns all keys in
                    the range low key to high key.
                    The lazy way to implement Range is to
                    iterate through the key/value map that the server
                    maintains; a better way is to use a data structure
                    that supports range searches (e.g.,
                    B-tree). Include a test that fails the lazy
                    solution but passes on the better solution.

                      Modify your kvraft
                      implementation to allow the leader to
                      serve Gets without running
                      the Get through rsm.  That is,
                      implement the optimization described at the end
                      of section 8 of the raft paper, including
                      leases, to ensure that kvraft maintains
                      linearizability. Your implementation should pass
                      the existing kvraft tests.  You should also add
                      a test that checks that your optimized
                      implementation is faster (e.g., by comparing the
                      number of RPCs) and a test that checks that term
                      switches are slower because a new leader must wait
                      until the lease has expired.

                    Support transactions
                     in kvraft so that a developer can
                     perform several Put and Gets
                     atomically.  Once you have transaction, you don't
                     need version Puts anymore; transactions subsume
                     versioned Puts.  Look at
                    etcd's
                    transactions for an example interface.  Write
                    tests to demonstrate your extension works.


              Modify shardkv to support transactions
              so that a developer can perform
              several Puts and Gets
              atomically across shards.  This requires
              implementing two-phase-commit and two-phase
              locking. Write tests to demonstrate that your
              extension works.



        Handin procedure



            Before submitting, please run all the tests
            one final time.


            Before submitting, double check that your solution
            works with:


$ make raft1
$ make kvraft1
$ make shardkv

6.5840 Lab submission instructions

Handin procedure

Before submitting, please run all tests one final time.

Use the make labnp command to package your lab
assignment and upload the generated tarball labnp-handin.tar.gz
to Gradescope.
n is the lab number (i.e., 1, 2, 3, or 4) and p the part
(e.g, A, B, or C), if the lab is in parts.

For example:

$ cd ~/6.5840
$ make lab1

Or:

$ cd ~/6.5840
$ make lab3a

Before you submit a later part, make sure the earlier parts still work.
That is run, all tests through the part you are submitting.

You may submit multiple times. We will use the timestamp of
your last submission for the purpose of
calculating late days.

Porcupine





        Clients

        Time



        Valid LP

        Invalid LP
        [ jump to first error ]

6.5840: Byzantine
Fault Tolerance
Lecture 21

Last week: security
Lecture 19: Fork Consistency (SUNDR)
Lecture 20: Decentralized payments (Bitcoin)


Last week: security
Lecture 19: Fork Consistency (SUNDR)
Lecture 20: Decentralized payments (Bitcoin)
•Can we replicate state in an open system?
•Solved problem thought to be impossible!


Last week: security
Lecture 19: Fork Consistency (SUNDR)
Lecture 20: Decentralized payments (Bitcoin)
•Can we replicate state in an open system?
•Solved problem thought to be impossible!
But, limitations…
•Throughput: ~1K pay/min           Latency: ~1hr (6-block depth)
•No linearizability guarantee!

Idea: RSM? (Raft)
But Raft has a linearizability guarantee…
•Network problems: wait to recover
Can we implement a RSM with malicious replicas?
•Throughput: ~1K pay/min           Latency: ~1hr (6-block depth)
•No linearizability guarantee!


Can we implement a RSM with malicious replicas?
                 “Byzantine”
•
(Lamport, Shostak, Pease ’82)

•“Academic problem”
•Ancestor of many of today’s cryptocurrency protocols
Can we implement a RSM with malicious replicas?
                 “Byzantine”
•
Practical Byzantine Fault Tolerance
(Castro + Liskov ’99)
(Lamport, Shostak, Pease ’82)

Aside: About me
•Class project: Implement PBFT

(6.5840 final project)
•Job: Implement BFT protocol at company

(Algorand, Inc.)
•PhD project: Implement PBFT without bugs

(Formal verification (c.f. Ironfleet, 6.5120)

w/ Frans Kaashoek + Nickolai Zeldovich)

PBFT solves harder
problem than Raft
Similar idea: RSM, but with malicious replicas
•Leaders ≈ Primaries
•Terms ≈ Views
•Timeouts

3.

PBFT solves harder
problem than Raft
Similar idea: RSM, but with malicious replicas
•Leaders ≈ Primaries
•Terms ≈ Views
•Timeouts
Additional ingredients needed
1.Authenticity of messages
2.More honest nodes
3.Leadership “fairness”

•Network
•Attacker can reorder messages
•Attacker can delay messages for limited time (denial of service)
•Nodes
•Attacker controls f server replicas
•Detail: client honest in paper
•Cryptography protects messages of honest machines
Assumptions

Aside: Digital signatures
•KeyGen(randomness) → (public key, private key)
•Sign(private key, message) → signature
•Verify(public key, message, signature) → ok?
Properties:
•Correctness: signature from non-faulty node 㱺 Verify ok
•Security: Verify ok 㱺 signature from non-faulty node
(see 6.5610 or 6.5620 for details)

•Network                                                                            Realistic?
•Attacker can reorder messages
•Attacker can delay messages for limited time (denial of service)
•Nodes
•Attacker controls f server replicas
•Detail: client honest in paper
•Cryptography protects messages of honest machines
Assumptions

Let’s build a PBFT
Start with one client

One-client protocol
Cannot tolerate >=N/3 faults! (i.e., N >= 3f+1)
Quorum: any Q where |Q| >= 2f+1
•Safety: Every Q has an honest majority (therefore unique)
•Liveness: |Q| <= N - f (no dependency on faulty replicas)
(inspired by Baudet, Danezis, Sonnino ’20)

Multiple clients
Need to elect a primary
Problem: what if the primary is faulty?

Faulty primaries
These make PBFT expensive
Recovery example

Faulty primaries impose
requirements
Add new all-to-all prepare round
Recovery messages must be justified: signature stapling

Multiple views
Like Raft: need primary with sufficiently-fresh view

    (NULL ~ view 0)
Unlike Raft: (liveness) can’t just elect any eligible candidate!
㱺 “term limits”: rotating primary
㱺 timeouts: View-change must be all-to-all as well

  (exponential)

Extending from op to RSM
•(Like Raft): Primary pipelines many client requests
•Low- and high-water mark prevent sequence # exhaustion
•Each view is a new “log”
•Checkpoints allow log compaction (c.f. Raft)
•Commit also all-to-all
•Clients get f+1 replies
•Sufficient: contains at least 1 honest replica

Optimizations
•Hash of values (c.f. Bitcoin)
•Tentative replies
•Read-only operations
•MACs vs. signatures
•Network NACKs

PBFT: RSM, securely
•Redundant state redundantly replicated for redundancy
•Never lose evidence of commit
•Cryptography limits attacker’s influence
•Under some assumptions!
•Design limits any node’s influence
•A good idea for robust systems, digital or not


PBFT: RSM, securely
•Redundant state redundantly replicated for redundancy
•Never lose evidence of commit
•Cryptography limits attacker’s influence
•Under some assumptions!
•Design limits any node’s influence
•A good idea for robust systems, digital or not
Questions?

Patterns and Hints
for Concurrency in Go
Russ Cox
MIT 6.5840 / “Spring” 2026

Concurrency is not Parallelism
Concurrency: composition of independently executing processes.
Parallelism: simultaneous execution of (possibly related)
computations.
Concurrency is about dealing with lots of things at once.
Parallelism is about doing lots of things at once.

Prologue:
Goroutines for State

/"([^"\\]|\\.)*"/

state := 0
for {
    c := read()
    switch state {
    case 0:
        if c != '"' {
            return false
        }
        state = 1
    case 1:
        if c == '"' {
            return true
        }
        if c == '\\' {
            state = 2
        } else {
            state = 1
        }
    case 2:
        state = 1
    }
}

state := 0
for {
    c := read()
    switch state {
    case 0:

        if c != '"' {
            return false
        }
        state = 1
    case 1:

        if c == '"' {
            return true
        }
        if c == '\\' {
            state = 2
        } else {
            state = 1
        }
    case 2:

        state = 1
    }
}
state := 0
for {

    switch state {
    case 0:
        c := read()
        if c != '"' {
            return false
        }
        state = 1
    case 1:
        c := read()
        if c == '"' {
            return true
        }
        if c == '\\' {
            state = 2
        } else {
            state = 1
        }
    case 2:
        read()
        state = 1
    }
}

state := 0
for {

    switch state {
    case 0:
        c := read()
        if c != '"' {
            return false
        }
        state = 1
    case 1:
        c := read()
        if c == '"' {
            return true
        }
        if c == '\\' {
            state = 2
        } else {
            state = 1
        }
    case 2:
        read()
        state = 1
    }
}




    state0:
        c := read()
        if c != '"' {
            return false
        }
        goto state1
    state1:
        c := read()
        if c == '"' {
            return true
        }
        if c == '\\' {
            goto state2
        } else {
            goto state1
        }
    state2:
        read()
        goto state1






    state0:
        c := read()
        if c != '"' {
            return false
        }
        goto state1
    state1:
        c := read()
        if c == '"' {
            return true
        }
        if c == '\\' {
            goto state2
        } else {
            goto state1
        }
    state2:
        read()
        goto state1





    state0:
        c := read()
        if c != '"' {
            return false
        }

    state1:
        c := read()
        if c == '"' {
            return true
        }
        if c == '\\' {
            goto state2
        } else {
            goto state1
        }
    state2:
        read()
        goto state1


       state0:         c := read()         if c != '"' {             return false         }             state1:         c := read()         if c == '"' {             return true         }         if c == '\\' {             goto state2         } else {             goto state1         }     state2:         read()         goto state1
       state0:         c := read()         if c != '"' {             return false         }              state1:         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()             goto state1         } else {             goto state1         }

       state0:         c := read()         if c != '"' {             return false         }              state1:         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()         }         goto state1
       state0:         c := read()         if c != '"' {             return false         }              state1:         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()             goto state1         } else {             goto state1         }

       state0:         c := read()         if c != '"' {             return false         }              state1:         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()         }         goto state1
         c := read()     if c != '"' {         return false     }              for {         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()         }     }

         c := read()     if c != '"' {         return false     }              for {         c := read()         if c == '"' {             return true         }         if c == '\\' {             read()         }     }
              if read() != '"' {         return false     }              var c rune     for c != '"' {         c = read()         if c == '\\' {             read()         }     }     return true






    if read() != '"' {
        return false
    }
    inEscape := false
    for {
        c := read()
        if inEscape {
            inEscape = false
            continue
        }
        if c == '"' {
            return true
        }
        if c == '\\' {
            inEscape = true
        }
    }








    if read() != '"' {
        return false
    }

    var c rune
    for c != '"' {
        c = read()
        if c == '\\' {
            read()
        }
    }
    return true







     func parse(read func() rune) bool {         if read() != '"' {         return false     }              var c rune     for c != '"' {         c = read()         if c == '\\' {             read()         }     }     return true }
state==0 →
state==1 →
state==2 →
Hint: Convert data state into code state
when it makes programs clearer.
← inEscape==false
← inEscape==true

type quoter struct {     state int }  func (q *quoter) Init() {     r.state = 0 }  func (q *quoter) Write(c rune) Status {     switch q.state {     case 0:         if c != '"' {             return BadInput         }         q.state = 1     case 1:         if c == '"' {             return Success         }         if c == '\\' {             q.state = 2         } else {             q.state = 1         }     case 2:         q.state = 1     }     return NeedMoreInput }

type quoter struct {     char   chan rune     status chan Status }  func (q *quoter) Init() {     q.char = make(chan rune)     q.status = make(chan Status)     go q.parse()     <-q.status // always NeedMoreInput }  func (q *quoter) Write(c rune) Status {     q.char <- c     return <-q.status }            Hint: Use additional goroutines
to hold additional code state.
func (q *quoteReader) parse() {     if q.read() != '"' {         q.status <- SyntaxError         return     }              var c rune     for c != '"' {         c = q.read()         if c == '\\' {             q.read()         }     }     q.status <- Done }  func (q *quoter) read() int {     q.status <- NeedMoreInput     return <-q.char }

package main  import (     "net/http"     _ "net/http/pprof" )  var c = make(chan int)  func main() {     for i := range 100 {         go f(0x10*i)     }     http.ListenAndServe("localhost:8080", nil) }  func f(x int) {     g(x+1) }  func g(x int) {     h(x+1) }  func h(x int) {     c <- 1     f(x+1) }
Hint: Know why and when
each goroutine will exit.

$ go run x.go
^\

SIGQUIT: quit
PC=0x105a17b m=0 sigcode=0

...

goroutine 18 [chan send]:
main.h(0x12)
    /tmp/x.go:26 +0x45
main.g(0x11)
    /tmp/x.go:22 +0x20
main.f(0x10)
    /tmp/x.go:18 +0x20
created by main.main
    /tmp/x.go:12 +0x42

goroutine 19 [chan send]:
main.h(0x22)
    /tmp/x.go:26 +0x45
main.g(0x21)
    /tmp/x.go:22 +0x20
main.f(0x20)
    /tmp/x.go:18 +0x20
created by main.main
    /tmp/x.go:12 +0x42

...

Hint: Type Ctrl-\ to kill a program and
dump all its goroutine stacks.

goroutine profile: total 106
100 @ 0x12d8715 0x12d86c0 0x12d8690 0x1058d61
#   0x12d8714   main.h+0x44 /tmp/x.go:26
#   0x12d86bf   main.g+0x1f /tmp/x.go:22
#   0x12d868f   main.f+0x1f /tmp/x.go:18

2 @ 0x11ddfcf 0x11dddcf 0x1248265 0x124f513 0x1253636 0x1058d61
#   0x11ddfce   net/textproto.(*Reader).readLineSlice+0x5e  go/src/net/textproto/reader.go:55
#   0x11dddce   net/textproto.(*Reader).ReadLine+0x2e       go/src/net/textproto/reader.go:36
#   0x1248264   net/http.readRequest+0xa4                   go/src/net/http/request.go:926
#   0x124f512   net/http.(*conn).readRequest+0x1b2          go/src/net/http/server.go:934
#   0x1253635   net/http.(*conn).serve+0x495                go/src/net/http/server.go:1763

1 @ 0x115a102 0x116b1cd 0x124dc92 0x1058d61
#   0x115a101   net.(*netFD).Read+0x51                      go/src/net/fd_unix.go:207
#   0x116b1cc   net.(*conn).Read+0x6c                       go/src/net/net.go:182
#   0x124dc91   net/http.(*connReader).backgroundRead+0x61  go/src/net/http/server.go:656

1 @ 0x12cfe22 0x12cfc20 0x12cc6e5 0x12d8051 0x12d8365 0x1254b84 0x1255fa0 0x1257312 0x1253845
0x1058d61
#   0x12cfe21   runtime/pprof.writeRuntimeProfile+0xa1      go/src/runtime/pprof/pprof.go:634
#   0x12cfc1f   runtime/pprof.writeGoroutine+0x9f           go/src/runtime/pprof/pprof.go:596
#   0x12cc6e4   runtime/pprof.(*Profile).WriteTo+0x3b4      go/src/runtime/pprof/pprof.go:310
#   0x12d8050   net/http/pprof.handler.ServeHTTP+0x1d0      go/src/net/http/pprof/pprof.go:232
#   0x12d8364   net/http/pprof.Index+0x1e4                  go/src/net/http/pprof/pprof.go:244
#   0x1254b83   net/http.HandlerFunc.ServeHTTP+0x43         go/src/net/http/server.go:1942
#   0x1255f9f   net/http.(*ServeMux).ServeHTTP+0x12f        go/src/net/http/server.go:2242
#   0x1257311   net/http.serverHandler.ServeHTTP+0x91       go/src/net/http/server.go:2572
#   0x1253844   net/http.(*conn).serve+0x6a4                go/src/net/http/server.go:1825
Hint: Use the HTTP server’s
/debug/pprof/goroutine
to inspect live goroutine stacks.

Pattern #1
Publish/subscribe server

type PubSub interface {
    // Publish publishes the event e to
    // all current subscriptions.
    Publish(e Event)

    // Subscribe registers c to receive future events.
    // All subscribers receive events in the same order,
    // and that order respects program order:
    // if Publish(e1) happens before Publish(e2),
    // subscribers receive e1 before e2.
    Subscribe(c chan<- Event)

    // Cancel cancels the prior subscription of channel c.
    // After any pending already-published events
    // have been sent on c, the server will signal that the
    // subscription is cancelled by closing c.
    Cancel(c chan<- Event)
}

type PubSub interface {
    // Publish publishes the event e to
    // all current subscriptions.
    Publish(e Event)

    // Subscribe registers c to receive future events.
    // All subscribers receive events in the same order,
    // and that order respects program order:
    // if Publish(e1) happens before Publish(e2),
    // subscribers receive e1 before e2.
    Subscribe(c chan<- Event)

    // Cancel cancels the prior subscription of channel c.
    // After any pending already-published events
    // have been sent on c, the server will signal that the
    // subscription is cancelled by closing c.
    Cancel(c chan<- Event)
}
Hint: Close a channel to signal
that no more values will be sent.

type Server struct {     mu sync.Mutex     sub map[chan<- Event]bool }  func (s *Server) Init() {     s.sub = make(map[chan<- Event]bool) }  func (s *Server) Publish(e Event) {     s.mu.Lock()     defer s.mu.Unlock()      for c := range s.sub {         c <- e     } }  func (s *Server) Subscribe(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if s.sub[c] {         panic("pubsub: already subscribed")     }     s.sub[c] = true }  func (s *Server) Cancel(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if !s.sub[c] {         panic("pubsub: not subscribed")     }     close(c)     delete(s.sub, c) }

type Server struct {     mu sync.Mutex     sub map[chan<- Event]bool }  func (s *Server) Init() {     s.sub = make(map[chan<- Event]bool) }  func (s *Server) Publish(e Event) {     s.mu.Lock()     defer s.mu.Unlock()      for c := range s.sub {         c <- e     } }  func (s *Server) Subscribe(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if s.sub[c] {         panic("pubsub: already subscribed")     }     s.sub[c] = true }  func (s *Server) Cancel(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if !s.sub[c] {         panic("pubsub: not subscribed")     }     close(c)     delete(s.sub, c) } Hint: Prefer defer for unlocking mutexes.

type Server struct {     mu sync.Mutex     sub map[chan<- Event]bool }  func (s *Server) Init() {     s.sub = make(map[chan<- Event]bool) }  func (s *Server) Publish(e Event) {     s.mu.Lock()     defer s.mu.Unlock()      for c := range s.sub {         c <- e     } }  func (s *Server) Subscribe(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if s.sub[c] {         panic("pubsub: already subscribed")     }     s.sub[c] = true }  func (s *Server) Cancel(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if !s.sub[c] {         panic("pubsub: not subscribed")     }     close(c)     delete(s.sub, c) }
Hint: Consider the effect
of slow goroutines.

Options for slow goroutines
•Slow down event generation.
•Drop events.
Examples: os/signal, runtime/pprof
•Queue an arbitrary number of events.
Hint: Think carefully before
introducing unbounded queuing.

type Server struct {     mu sync.Mutex     sub map[chan<- Event]bool }  func (s *Server) Init() {     s.sub = make(map[chan<- Event]bool) }
type Server struct {     publish   chan Event     subscribe chan subReq     cancel    chan subReq }  type subReq struct {     c  chan<- Event     ok chan bool }  func (s *Server) Init() {     s.publish = make(chan Event)     s.subscribe = make(chan subReq)     s.cancel = make(chan subReq)     go s.loop() }

func (s *Server) Publish(e Event) {     s.mu.Lock()     defer s.mu.Unlock()      for c := range s.sub {         c <- e     } }  func (s *Server) Subscribe(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if s.sub[c] {         panic("pubsub: already subscribed")     }     s.sub[c] = true }  func (s *Server) Cancel(c chan<- Event) {     s.mu.Lock()     defer s.mu.Unlock()      if !s.sub[c] {         panic("pubsub: not subscribed")     }     close(c)     delete(s.sub, c) }
func (s *Server) loop() {
    sub := make(map[chan<- Event]bool)
    for {
        select {
        case e := <-s.publish:
            for c := range sub {
                c <- e
            }

        case r := <-s.subscribe:
            if sub[r.c] {
                r.ok <- false
                break
            }
            sub[r.c] = true
            r.ok <- true

        case c := <-s.cancel:
            if !sub[r.c] {
                r.ok <- false
                break
            }
            close(r.c)
            delete(sub, r.c)
            r.ok <- true
        }
    }
}









func (s *Server) Publish(e Event) {
    s.mu.Lock()
    defer s.mu.Unlock()

    for c := range s.sub {
        c <- e
    }
}

func (s *Server) Subscribe(c chan<- Event) {
    s.mu.Lock()
    defer s.mu.Unlock()

    if s.sub[c] {
        panic("pubsub: already subscribed")
    }
    s.sub[c] = true
}

func (s *Server) Cancel(c chan<- Event) {
    s.mu.Lock()
    defer s.mu.Unlock()

    if !s.sub[c] {
        panic("pubsub: not subscribed")
    }
    close(c)
    delete(s.sub, c)
}




func (s *Server) Publish(e Event) {
    s.publish <- e
}






func (s *Server) Subscribe(c chan<- Event) {
    r := subReq{c: c, ok: make(chan bool)}
    s.subscribe <- r
    if !<-r.ok {
        panic("pubsub: already subscribed")
    }
}



func (s *Server) Cancel(c chan<- Event) {
    r := subReq{c: c, ok: make(chan bool)}
    s.cancel <- r
    if !<-r.ok {
        panic("pubsub: not subscribed")
    }
}








type Server struct {
    publish chan Event
    subscribe chan subReq
    cancel chan subReq
}

type subReq struct {
    c chan<- Event
    ok chan bool
}

func (s *Server) Init() {
    s.publish = make(chan Event)
    s.subscribe = make(chan subReq)
    s.cancel = make(chan subReq)
    go s.loop()
}

func (s *Server) Publish(e Event) {
    s.publish <- e
}

func (s *Server) Subscribe(c chan<- Event) {
    r := subReq{c: c, ok: make(chan bool)}
    s.subscribe <- r
    if !<-r.ok {
        panic("pubsub: already subscribed")
    }
}

func (s *Server) Cancel(c chan<- Event) {
    r := subReq{c: c, ok: make(chan bool)}
    s.cancel <- r
    if !<-r.ok {
        panic("pubsub: not subscribed")
    }
}
func (s *Server) loop() {
    sub := make(map[chan<- Event]bool)
    for {
        select {
        case e := <-s.publish:
            for c := range sub {
                c <- e
            }

        case r := <-s.subscribe:
            if sub[r.c] {
                r.ok <- false
                break
            }
            sub[r.c] = true
            r.ok <- true

        case c := <-s.cancel:
            if !sub[r.c] {
                r.ok <- false
                break
            }
            close(r.c)
            delete(sub, r.c)
            r.ok <- true
        }
    }
}








Hint: Convert mutexes
into goroutines
when it makes programs clearer

func helper(in <-chan Event,
            out chan<- Event) {
    var q []Event
    for {
        select {
        case e := <-in:
            q = append(q, e)
        case out <- q[0]:
            q = q[1:]
        }
    }
}




















func helper(in <-chan Event,
            out chan<- Event) {
    var q []Event
    for {
        select {
        case e := <-in:
            q = append(q, e)
        case out <- q[0]:
            q = q[1:]
        }
    }
}



















func helper(in <-chan Event,
            out chan<- Event) {
    var q []Event
    for {
        // Decide whether and what to send.
        var sendOut chan<- Event
        var next Event
        if len(q) > 0 {
            sendOut = out
            next = q[0]
        }

        select {
        case e := <-in:
            q = append(q, e)
        case sendOut <- next:
            q = q[1:]
        }
    }
}












func helper(in <-chan Event,
            out chan<- Event) {
    var q []Event
    for {
        // Decide whether and what to send.
        var sendOut chan<- Event
        var next Event
        if len(q) > 0 {
            sendOut = out
            next = q[0]
        }

        select {
        case e := <-in:
            q = append(q, e)
        case sendOut <- next:
            q = q[1:]
        }
    }
}











func helper(in <-chan Event,
            out chan<- Event) {
    var q []Event
    for in != nil || len(q) > 0 {
        // Decide whether and what to send.
        var sendOut chan<- Event
        var next Event
        if len(q) > 0 {
            sendOut = out
            next = q[0]
        }

        select {
        case e, ok := <-in:
            if !ok {
                in = nil // stop receiving from in
                break
            }
            q = append(q, e)
        case sendOut <- next:
            q = q[1:]
        }
    }
    close(out)
}







func (s *Server) loop() {
    sub := make(map[chan<- Event]bool)
    for {
        select {
        case e := <-s.publish:
            for c := range sub {
                c <- e
            }

        case r := <-s.subscribe:
            if sub[r.c] {
                r.ok <- false
                break
            }
            sub[r.c] = true
            r.ok <- true

        case c := <-s.cancel:
            if !sub[r.c] {
                r.ok <- false
                break
            }
            close(r.c)
            delete(sub, r.c)
            r.ok <- true
        }
    }
}






func (s *Server) loop() {
    sub := make(map[chan<- Event]bool)
    for {
        select {
        case e := <-s.publish:
            for c := range sub {
                c <- e
            }

        case r := <-s.subscribe:
            if sub[r.c] {
                r.ok <- false
                break
            }
            sub[r.c] = true
            r.ok <- true

        case c := <-s.cancel:
            if !sub[r.c] {
                r.ok <- false
                break
            }
            close(r.c)
            delete(sub, r.c)
            r.ok <- true
        }
    }
}





func (s *Server) loop() {
    sub := make(map[chan<- Event]chan<- Event)
    for {
        select {
        case e := <-s.publish:
            for _, h := range sub {
                h <- e
            }

        case r := <-s.subscribe:
            if sub[r.c] != nil {
                r.ok <- false
                break
            }
            h = make(chan Event)
            go helper(h, r.c)
            sub[r.c] = h
            r.ok <- true

        case c := <-s.cancel:
            if sub[r.c] == nil {
                r.ok <- false
                break
            }
            close(sub[r.c])
            delete(sub, r.c)
            r.ok <- true
        }
    }
}



Hint: Use goroutines
to let independent concerns
run independently.

Pattern #2
Work scheduler

func Schedule(servers []string, numTask int,
              call func(srv string, task int))

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     } }
Hint: Use a buffered channel
as a concurrent blocking queue.

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     }      for task := range numTask {         go func() {             srv := <-idle             call(srv, task)             idle <- srv         }()     } }
Hint: Use goroutines
to let independent concerns
run independently.

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     }      for task := range numTask {         go func() {             srv := <-idle             call(srv, task)             idle <- srv         }()     } }
Hint: Think carefully before
introducing unbounded queuing.

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     }      for task := range numTask {        go func() {             srv := <-idle             call(srv, task)             idle <- srv         }()     } }
                                       for task := range numTask {              srv := <-idle         go func() {             call(srv, task)             idle <- srv         }()     }

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     }      for task := range numTask {         srv := <-idle         go func() {             call(srv, task)             idle <- srv         }()     }      for range servers {         <-idle     } }

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      idle := make(chan string, len(servers))     for _, srv := range servers {         idle <- srv     }      for task := range numTask {         srv := <-idle         go func() {             call(srv, task)             idle <- srv         }()     }      for range servers {         <-idle     } }

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := 0      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done++     }      for _, srv := range servers {         go runTasks(srv)     }      for task := range numTask {         work <- task     }     close(work)      for done < len(servers) {         runtime.Gosched()     } }
Hint: Think carefully before
introducing unbounded queuing.
Hint: Close a channel to signal
that no more values will be sent.

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := 0      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done++     }      for _, srv := range servers {         go runTasks(srv)     }      for task := range numTask {         work <- task     }     close(work)      for done < len(servers) {         runtime.Gosched()     } }
$ go run -race /tmp/x.go
==================
WARNING: DATA RACE
Write at 0x00c0000121d8 by goroutine 6:
  main.Schedule.func1()
      /tmp/x.go:19 +0x80
  main.Schedule.gowrap1()
      /tmp/x.go:23 +0x48

Previous read at 0x00c0000121d8 by main goroutine:
  main.Schedule()
      /tmp/x.go:31 +0x27c
  main.main()
      /tmp/x.go:6 +0x90

Goroutine 6 (running) created at:
  main.Schedule()
      /tmp/x.go:23 +0x13c
  main.main()
      /tmp/x.go:6 +0x90
==================



Hint: Use the race detector,
for development and even production.

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := 0      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done++     }      for _, srv := range servers {         go runTasks(srv)     }      for task := range numTask {         work <- task     }     close(work)      for done < len(servers) {         runtime.Gosched()     } }
$ go run -race /tmp/x.go ================== WARNING: DATA RACE Write at 0x00c00019c008 by goroutine 7:   main.Schedule.func1()       /tmp/x.go:19 +0x80   main.Schedule.gowrap1()       /tmp/x.go:23 +0x48  Previous write at 0x00c00019c008 by goroutine 6:   main.Schedule.func1()       /tmp/x.go:19 +0x80   main.Schedule.gowrap1()       /tmp/x.go:23 +0x48|  Goroutine 7 (running) created at:   main.Schedule()       /tmp/x.go:23 +0x13c   main.main()       /tmp/x.go:6 +0x90  Goroutine 6 (finished) created at:   main.Schedule()       /tmp/x.go:23 +0x13c   main.main()       /tmp/x.go:6 +0x90 ==================

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := 0      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done++     }      for _, srv := range servers {         go runTasks(srv)     }      for task := range numTask {         work <- task     }     close(work)      for done < len(servers) {         runtime.Gosched()     } }
Hint:
Don’t communicate by sharing memory.
Share memory by communicating.
       work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done <- true     }      ...            for range servers {         <-done     }

func Schedule(servers []string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done <- true     }      for _, srv := range servers {         go runTasks(srv)     }      for task := range numTask {         work <- task     }     close(work)      for range servers {         <-done     } }
                    func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {                                                                go func() {             for srv := range servers {                 go runTasks(srv)             }         }()
Hint: Use goroutines
to let independent concerns
run independently.

func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)         }         done <- true     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for range numTask {         work <- task     }     close(work)      for range servers {         <-done     } }
                                       runTasks := func(srv string) {    for task := range work {         call(srv, task)         done <- true     } }                                                                                                                 for range numTask {     <-done }

func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }
Hint: Know why and when
each communication will proceed.
$ go run /tmp/x.go
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
main.Schedule(0xc4200120c0, 0x3, 0x3, 0x14, 0x106acc8)
 /tmp/x.go:26 +0x150
main.main()
 /tmp/x.go:4 +0x96

goroutine 5 [chan send]:
main.Schedule.func1(0x1066bc0, 0x1)
 /tmp/x.go:15 +0xba
created by main.Schedule.func2
 /tmp/x.go:21 +0x5f








func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for i := range numTask {         <-done     } }
                                                    i := 0 WorkLoop:     for task := range numTask {         for {             select {             case work <- task:                 continue WorkLoop             case <-done:                 i++             }         }     }     close(work)      for ; i < numTask; i++ {         <-done     }

func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }
                                                                                                                                               go func() {     for task := range numTask {         work <- task          }     close(work) }()
Hint: Use goroutines
to let independent concerns
run independently.

func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }
                work := make(chan int, numTask)
Hint: Think carefully before
introducing unbounded queuing.

func Schedule(servers chan string, numTask int,               call func(srv string, task int)) {      work := make(chan int, numTask)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }

func Schedule(servers chan string, numTask int,               call func(srv string, task int) bool) {      work := make(chan int, numTask)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }
                 runTasks := func(srv string) {     for task := range work {          if call(srv, task) {            done <- true                 } else {            work <- task         }       }     }       ...                                           for task := range numTask {      work <- task     }          for range numTask {     <-done } close(work)
Hint: Know why and when
each communication will proceed.

func Schedule(servers chan string, numTask int,               call func(srv string, task int) bool) {      work := make(chan int, numTask)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             call(srv, task)             done <- true         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }     close(work)      for range numTask {         <-done     } }
                 runTasks := func(srv string) {     for task := range work {          if call(srv, task) {            done <- true                 } else {            work <- task         }       }     }       ...                                           for task := range numTask {      work <- task     }          for range numTask {     <-done } close(work)          Hint: Close a channel to signal
that no more values will be sent.

func Schedule(servers chan string, numTask int,               call func(srv string, task int) bool) {      work := make(chan int, numTask)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             if call(srv, task) {                 done <- true             } else {                 work <- task             }         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }      for range numTask {         <-done     }     close(work) }

func Schedule(servers chan string, numTask int,               call func(srv string, task int) bool) {      work := make(chan int, numTask)     done := make(chan bool)      runTasks := func(srv string) {         for task := range work {             if call(srv, task) {                 done <- true             } else {                 work <- task             }         }     }      go func() {         for _, srv := range servers {             go runTasks(srv)         }     }()      for task := range numTask {         work <- task     }      for range numTask {         <-done     }     close(work) }
              work := make(chan int, numTask) done := make(chan bool) exit := make(chan bool)   ...         go func() {     for {                select {           case srv := <-servers:             go runTasks(srv)         case <-exit:             return         }     } }()   ...          for range numTask {     <-done } close(work)      exit <- true
Hint: Make sure you know
why and when each goroutine will exit.

Pattern #3
Replicated service client

type ReplicatedClient interface {
    // Init initializes the client to use the given servers.
    // To make a particular request later,
    // the client can use callOne(srv, args), where srv
    // is one of the servers from the list.
    Init(servers []string, callOne func(string, Args) Reply)

    // Call makes a request on any available server.
    // Multiple goroutines may call Call concurrently.
    Call(args Args) Reply
}




type Client struct {       servers []string     callOne func(string, Args) Reply      mu     sync.Mutex      prefer int }   func (c *Client) Init(servers []string, callOne func(string, Args) Reply) {       c.servers = servers     c.callOne = callOne }
Hint: Use a mutex if that is
the clearest way to write the code.

type Client struct {
    servers []string
    callOne func(string, Args) Reply

    mu     sync.Mutex
    prefer int
}

func (c *Client) Init(servers []string, callOne func(string, Args) Reply) {
    c.servers = servers
    c.callOne = callOne
}

func (c *Client) Call(args Args) Reply {
    type result struct {
        serverID int
        reply Reply
    }

    done := make(chan result, 1)
    id := ...

    go func() {
        done <- result{id, c.callOne(c.servers[id], args)}
    }()
}




Hint: Use goroutines
to let independent concerns
run independently.

func (c *Client) Call(args Args) Reply {          type result struct {         serverID int         reply Reply     }          const timeout = 1 * time.Second     t := time.NewTimer(timeout)     defer t.Stop()      done := make(chan result, 1)        id := ...      go func() {         done <- result{id, c.callOne(c.servers[id], args)}     }()      select {     case r := <-done:         return r.reply     case <-t.C:         // timeout     } }
Hint: Stop timers you don’t need.
(fixed in Go 1.23)

Hint: Know why and when
each goroutine will exit.

Hint: Know why and when
each communication will proceed.

func (c *Client) Call(args Args) Reply {          type result struct {         serverID int         reply Reply     }          const timeout = 1 * time.Second     t := time.NewTimer(timeout)     defer t.Stop()      done := make(chan result, len(c.servers))        for id := range c.servers {         go func() {             done <- result{id, c.callOne(c.servers[id], args)}         }()          select {         case r := <-done:             return r.reply         case <-t.C:             // timeout             t.Reset(timeout)         }     }      r := <-done     return r.reply }

    c.mu.Lock()     prefer := c.prefer     c.mu.Unlock()      var r result       for off := range c.servers {         id := (prefer + off) % len(c.servers)         go func() {             done <- result{id, c.callOne(c.servers[id], args)}         }()          select {         case r = <-done:             goto Done         case <-t.C:             // timeout             t.Reset(timeout)         }     }      r = <-done Done:     c.mu.Lock()     c.prefer = r.serverID     c.mu.Unlock()     return r.reply
Hint: Use a goto if that is
the clearest way to write the code.

Pattern #4
Protocol multiplexer

type ProtocolMux interface {
    // Init initializes the mux to manage messages to the given service.
    Init(Service)

    // Call makes a request with the given message and returns the reply.
    // Multiple goroutines may call Call concurrently.
    Call(Msg) Msg
}

type Service interface {
    // ReadTag returns the muxing identifier in the request or reply message.
    // Multiple goroutines may call ReadTag concurrently.
    ReadTag(Msg) int64

    // Send sends a request message to the remote service.
    // Send must not be called concurrently with itself.
    Send(Msg)

    // Recv waits for and returns a reply message from the remote service.
    // Recv must not be called concurrently with itself.
    Recv() Msg
}



type Mux struct {
    srv  Service
    send chan Msg

    mu sync.Mutex
    pending map[int64]chan<- Msg
}

func (m *Mux) Init(srv Service) {
    m.srv = srv
    m.pending = make(map[int64]chan Msg)
    go m.sendLoop()
    go m.recvLoop()
}














type Mux struct {
    srv  Service
    send chan Msg

    mu sync.Mutex
    pending map[int64]chan<- Msg
}

func (m *Mux) Init(srv Service) {
    m.srv = srv
    m.pending = make(map[int64]chan Msg)
    go m.sendLoop()
    go m.recvLoop()
}

func (m *Mux) sendLoop() {
    for args := range m.send {
        m.srv.Send(args)
    }
}








func (m *Mux) sendLoop() {     for args := range m.send {         m.srv.Send(args)     } }  func (m *Mux) recvLoop() {     for {         reply := m.srv.Recv()         tag := m.srv.ReadTag(reply)          m.mu.Lock()         done := m.pending[tag]         delete(m.pending, tag)         m.mu.Unlock()          if done == nil {             panic("unexpected reply")         }         done <- reply     } }

func (m *Mux) sendLoop() {     for args := range m.send {         m.srv.Send(args)     } }  func (m *Mux) recvLoop() {     for {         reply := m.srv.Recv()         tag := m.srv.Tag(reply)          m.mu.Lock()         done := m.pending[tag]         delete(m.pending, tag)         m.mu.Unlock()          if done == nil {             panic("unexpected reply")         }         done <- reply     } }
func (m *Mux) Call(args Msg) (reply Msg) {     tag := m.srv.ReadTag(args)     done := make(chan Msg, 1)      m.mu.Lock()     if m.pending[tag] != nil {         m.mu.Unlock()         panic("mux: duplicate call tag")     }     m.pending[tag] = done     m.mu.Unlock()      m.send <- args     return <-done }
Hint: Use goroutines, channels,
and mutexes together if that is
the clearest way to write the code.

Hints

Use the race detector, for development and even production.
Don’t communicate by sharing memory. Share memory by communicating.

Convert data state into code state when it makes programs clearer.
Convert mutexes into goroutines when it makes programs clearer.
Use additional goroutines to hold additional code state.
Use goroutines to let independent concerns run independently.

Consider the effect of slow goroutines.
Know why and when each communication will proceed.
Know why and when each goroutine will exit.
Type Ctrl-\ to kill a program and dump all its goroutine stacks.
Use the HTTP server’s /debug/pprof/goroutine to inspect live goroutine stacks.

Use a buffered channel as a concurrent blocking queue.
Think carefully before introducing unbounded queuing.
Close a channel to signal that no more values will be sent.

Use a mutex if that is the clearest way to write the code.
Prefer defer for unlocking mutexes.
Use a goto if that is the clearest way to write the code.
Use goroutines, channels, and mutexes together
if that is the clearest way to write the code.

package main

import (
    "fmt"
    "sync"
)

//
// Several solutions to the crawler exercise from the Go tutorial
// https://tour.golang.org/concurrency/10
//

//
// Serial crawler
//

func Serial(url string, fetcher Fetcher, fetched map[string]bool) {
    if fetched[url] {
        return
    }
    fetched[url] = true
    urls, err := fetcher.Fetch(url)
    if err != nil {
        return
    }
    for _, u := range urls {
        Serial(u, fetcher, fetched)
    }
}

//
// Concurrent crawler with shared state and Mutex
//

type fetchState struct {
    mu      sync.Mutex
    fetched map[string]bool
}

func (fs *fetchState) testAndSet(url string) bool {
    fs.mu.Lock()
    defer fs.mu.Unlock()
    r := fs.fetched[url]
    fs.fetched[url] = true
    return r
}

func ConcurrentMutex(url string, fetcher Fetcher, fs *fetchState) {
    if fs.testAndSet(url) {
        return
    }
    urls, err := fetcher.Fetch(url)
    if err != nil {
        return
    }
    var done sync.WaitGroup
    for _, u := range urls {
        done.Add(1)
        go func(u string) {
            ConcurrentMutex(u, fetcher, fs)
            done.Done()
        }(u)
    }
    done.Wait()
}

func makeState() *fetchState {
    return &fetchState{fetched: make(map[string]bool)}
}

//
// Concurrent crawler with channels
//

func worker(url string, ch chan []string, fetcher Fetcher) {
    urls, err := fetcher.Fetch(url)
    if err != nil {
        ch <- []string{}
    } else {
        ch <- urls
    }
}

func coordinator(ch chan []string, fetcher Fetcher) {
    n := 1
    fetched := make(map[string]bool)
    for urls := range ch {
        for _, u := range urls {
            if fetched[u] == false {
                fetched[u] = true
                n += 1
                go worker(u, ch, fetcher)
            }
        }
        n -= 1
        if n == 0 {
            break
        }
    }
}

func ConcurrentChannel(url string, fetcher Fetcher) {
    ch := make(chan []string)
    go func() {
        ch <- []string{url}
    }()
    coordinator(ch, fetcher)
}

//
// main
//

func main() {
    fmt.Printf("=== Serial===\n")
    Serial("http://golang.org/", fetcher, make(map[string]bool))

    fmt.Printf("=== ConcurrentMutex ===\n")
    ConcurrentMutex("http://golang.org/", fetcher, makeState())

    fmt.Printf("=== ConcurrentChannel ===\n")
    ConcurrentChannel("http://golang.org/", fetcher)
}

//
// Fetcher
//

type Fetcher interface {
    // Fetch returns a slice of URLs found on the page.
    Fetch(url string) (urls []string, err error)
}

// fakeFetcher is Fetcher that returns canned results.
type fakeFetcher map[string]*fakeResult

type fakeResult struct {
    body string
    urls []string
}

func (f fakeFetcher) Fetch(url string) ([]string, error) {
    if res, ok := f[url]; ok {
        fmt.Printf("found:   %s\n", url)
        return res.urls, nil
    }
    fmt.Printf("missing: %s\n", url)
    return nil, fmt.Errorf("not found: %s", url)
}

// fetcher is a populated fakeFetcher.
var fetcher = fakeFetcher{
    "http://golang.org/": &fakeResult{
        "The Go Programming Language",
        []string{
            "http://golang.org/pkg/",
            "http://golang.org/cmd/",
        },
    },
    "http://golang.org/pkg/": &fakeResult{
        "Packages",
        []string{
            "http://golang.org/",
            "http://golang.org/cmd/",
            "http://golang.org/pkg/fmt/",
            "http://golang.org/pkg/os/",
        },
    },
    "http://golang.org/pkg/fmt/": &fakeResult{
        "Package fmt",
        []string{
            "http://golang.org/",
            "http://golang.org/pkg/",
        },
    },
    "http://golang.org/pkg/os/": &fakeResult{
        "Package os",
        []string{
            "http://golang.org/",
            "http://golang.org/pkg/",
        },
    },
}

package main

import (
    "fmt"
    "log"
    "net"
    "net/rpc"
    "sync"
)

//
// Common RPC request/reply definitions
//

type PutArgs struct {
    Key   string
    Value string
}

type PutReply struct {
}

type GetArgs struct {
    Key string
}

type GetReply struct {
    Value string
}

//
// Client
//

func connect() *rpc.Client {
    client, err := rpc.Dial("tcp", ":1234")
    if err != nil {
        log.Fatal("dialing:", err)
    }
    return client
}

func get(key string) string {
    client := connect()
    args := GetArgs{ key }
    reply := GetReply{}
    err := client.Call("KV.Get", &args, &reply)
    if err != nil {
        log.Fatal("error:", err)
    }
    client.Close()
    return reply.Value
}

func put(key string, val string) {
    client := connect()
    args := PutArgs{ key, val }
    reply := PutReply{}
    err := client.Call("KV.Put", &args, &reply)
    if err != nil {
        log.Fatal("error:", err)
    }
    client.Close()
}

//
// Server
//

type KV struct {
    mu   sync.Mutex
    data map[string]string
}

func server() {
    kv := &KV{data: map[string]string{}}
    rpcs := rpc.NewServer()
    rpcs.Register(kv)
    l, e := net.Listen("tcp", ":1234")
    if e != nil {
        log.Fatal("listen error:", e)
    }
    go func() {
        for {
            conn, err := l.Accept()
            if err == nil {
                go rpcs.ServeConn(conn)
            } else {
                break
            }
        }
        l.Close()
    }()
}

func (kv *KV) Get(args *GetArgs, reply *GetReply) error {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    reply.Value = kv.data[args.Key]

    return nil
}

func (kv *KV) Put(args *PutArgs, reply *PutReply) error {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    kv.data[args.Key] = args.Value

    return nil
}

//
// main
//

func main() {
    server()

    put("subject", "6.5840")
    fmt.Printf("Put(subject, 6.5840) done\n")
    fmt.Printf("get(subject) -> %s\n", get("subject"))
}

6.5840 2026 Lecture 11: Distributed Transactions

Topics:
  distributed transactions = concurrency control + atomic commit

where are we in the course?
  so far, mostly distribution for fault tolerance
    multiple servers trying to look like one reliable server
  now, many servers for performance
    [diagram: clients, servers, data sharded by key]
    split the data up (shard) over multiple servers, for parallelism
    fine as long as clients use data items one at a time
    what if an application operation involves records in different shards?
      failures? atomicity?
    important problem, we'll see a number of approaches

client application actions often involve multiple reads and writes
  bank transfer: debit and credit
  install bi-directional links in a social graph
  insert new record, add to index
  we'd like to hide interleaving and failure from application writers

this is an old problem in databases
  the traditional solution: transactions
  programmer marks beginning/end of sequences of code as transactions
  the system automatically provides good behavior

example transactions
  x and y are bank balances -- records in database tables
    both start out as $10
  T1 and T2 are transactions
    T1: transfer $1 from y to x
    T2: audit, to find the total amount of money in the bank
  T1:             T2:
  BEGIN-X         BEGIN-X
    add(x, 1)       tmp1 = get(x)
    add(y, -1)      tmp2 = get(y)
  END-X             print tmp1, tmp2
                  END-X

the "END-X" indicates that the transaction would like to commit
  it asks the transaction system to attempt to commit
  as we'll see, the commit may succeed, or it may fail

what is correct behavior for a transaction?
  usually called "ACID"
    Atomic -- all writes or none, despite failures
    Consistent -- obeys application-specific invariants
    Isolated -- no interference between xactions -- serializable
    Durable -- committed writes are permanent
  ACID transactions are magic!
    programmer writes straightforward serial code
    system automatically adds locking!
    system automatically adds fault tolerance!
  of course we need to implement this magic

a note on trends
  some storage systems provide transactions, some don't
  some applications benefit a lot from transactions, some don't
  SQL databases provide transactions
  but transactions are slow, particularly for sharded data
    so for a while simple key/value stores gained popularity
    just put and get on single records
  but transactions are coming back

today: ACID for distributed transactions
  with data sharded over multiple servers

What does serializable mean?
  you execute some concurrent transactions, which yield results
    "results" means both output and changes in the DB
  the results are serializable if:
    there exists a serial execution order of the transactions
    that yields the same results as the actual execution
  (serial means one at a time; wait for one to finish before starting the next)
  (this definition should remind you of linearizability)

You can test whether an execution's result is serializable by
  looking for a serial order that yields the same results.
  for our example, the possible serial orders are
    T1; T2
    T2; T1
  so the correct (serializable) results are:
    T1; T2 : x=11 y=9 "11,9"
    T2; T1 : x=11 y=9 "10,10"
  the results for the two differ; either is OK
  no other result is OK for a serializable system
  the implementation might have executed T1 and T2 in parallel
    but it must still yield results as if in a serial order

what if T1's operations run entirely between T2's two get()s?
  would the result be serializable?
  T2 would print 10,9
  but 10,9 is not one of the two serializable results!
what if T2 runs entirely between T1's two adds()s?
  T2 would print 11,10
  but 11,10 is not one of the two serializable results!
what if x's server does the increment but y's server can't?
  x=11 y=10 is not one of the serializable results!

a transaction can "abort" if something goes wrong
  an abort un-does any modifications
  the transaction might voluntarily abort,
    e.g. if the account doesn't exist, or y's balance is <= 0
  the system may force an abort, e.g. to break a locking deadlock
  server failure can result in abort
  result of abort should be as if xaction never executed!!!
    must un-do, or not apply, all updates
  the application might (or might not) try the transaction again

serializable transactions are nice for application programmers
  write transaction as if there is nothing else going on
  ordinary serial code, programmer doesn't need to worry about locks

now: implementing distributed transactions

two main components:
  concurrency control (to provide isolation/serializability)
  atomic commit (to provide atomicity despite failure)

first, concurrency control
  the goal: isolated/serializable execution of concurrent transactions
  for now, on a single DB server (not distributed)

two classes of concurrency control for transactions:
  pessimistic:
    lock records before use
    conflicts cause delays (waiting for locks)
  optimistic:
    use records without locking
    commit checks if reads/writes were serializable
    conflict causes abort+retry
    called Optimistic Concurrency Control (OCC)
  pessimistic is faster if conflicts are frequent
  optimistic is faster if conflicts are rare

today: pessimistic concurrency control
in a few weeks: optimistic concurrency control (FaRM)

"Two-phase locking" is one way to implement serializability
  each database record has a lock
  2PL rules:
    a transaction must acquire a record's lock before using it
    a transaction must hold its locks until *after* commit or abort

2PL for our example
  suppose T1 and T2 start at the same time
  the transaction system automatically acquires locks as needed
  so first of T1/T2 to use x will get the lock
  the other waits until the first completely finishes (reaches END-X)
  this prohibits the non-serializable interleavings

details:
  an executing transaction acquires locks as needed, at the first use
    add() and get() implicitly acquire record's lock
    END-X commits or aborts, then releases all locks
  all locks are exclusive (for this discussion, no reader/writer locks)
  the full name is "strong strict two-phase locking"
    strong strict = hold locks until after commit
  much more structured than e.g. Go's Mutexes:
    programmer must supply BEGIN-X/END-X
    DB locks automatically, on first use of each record
    DB unlocks automatically, at transaction end
    DB may automatically abort to resolve deadlock
  (consider: could Go mutexes usefully be automatic like this?)

Why hold locks until after commit/abort?
  why not release as soon as done with the record?
  example of a resulting problem:
    suppose T2 releases x's lock after get(x)
    T1 could then execute between T2's get()s
    T2 would print 10,9
    not a serializable execution: neither T1;T2 nor T2;T1

Two-phase locking can produce deadlock, e.g.
  T3      T4
  get(x)  get(y)
  get(y)  get(x)
The system must resolve deadlocks, usually by aborting
  Detect waits-for cycles -- maybe too hard when locks are distributed
  Time out
  Wound-wait (pre-emptively abort newer xaction rather than waiting)

Performance with 2PL
  good: parallelism for transactions that use disjoint records
  bad: locking has overhead even if lock is available
  bad: locking conflicts require waiting

The Question: describe a situation where Two-Phase Locking yields
higher performance than Simple Locking. Simple locking: lock *every*
record before *any* use; release after abort/commit.

Next topic: distributed transactions versus failures

how can distributed transactions cope with failures?
  suppose, for our example, x and y are on different storage servers
  suppose x's server adds 1, but y's crashes before subtracting?
  or x's server adds 1, but y's realizes the account doesn't exist?
  or x and y both can do their part, but aren't sure if the other will?
  it's a hard problem!

We want "atomic commit":
  A bunch of computers are cooperating on some task
  Each computer has a different role
  We want atomicity: all execute, or none execute
  Challenges: failures, performance

We're going to look at a protocol called "two-phase commit"
  Used by distributed databases for multi-server transactions

The setting
  Data is sharded among multiple servers
  Each transaction runs on a "transaction coordinator" (TC)
  For each read/write, TC sends RPC to relevant shard server
    Each shard server is a "participant"
    Each participant manages locks for its shard of the data
  There may be many concurrent transactions, many TCs
    TC assigns unique transaction ID (TID) to each transaction
    Every message, every piece of xaction state tagged with TID
    To avoid confusion

Two-phase commit without failures:
  [time diagram: A, TC, B]
  [state (just for B): DATA, TID LOCKS TEMP PREPARED]
  TC sends put(), get(), &c RPCs to A, B
    A and B lock records (and wait if already locked).
    Modifications are tentative, on a copy, only installed if commit.
  TC gets to END-X.
  TC sends PREPARE messages to A and B.
  If A is able to commit,
    A responds YES.
    then A is in "prepared" state.
  otherwise, A responds NO.
  Same for B.
  If both A and B said YES, TC sends COMMIT messages to A and B.
  If either A or B said NO, TC sends ABORT messages.
  A/B commit if they get a COMMIT message from the TC.
    I.e. they copy tentative records to the real DB.
    And release the transaction's locks on their records.
  A/B send ACK (acknowledge) to say they have finished committing.

Why is this correct so far?
  Neither A nor B can commit unless they both agreed.

What if B crashes and restarts?
  If B crashed *before* it got the PREPARE, it can forget the transaction.
  If B crashed *after* sending YES, B must remember (despite crash)!
    Because A might have received a COMMIT and committed.
    So B must be able to commit (or not) even after a reboot.

PREPARE requires participants to save state that survives a crash.
  Typically by writing to disk (slow).
  B must save xaction state on disk before saying YES,
    including locks and tentative data updates.
  If B reboots, and disk says PREPARED but didn't receive COMMIT from TC,
    B must ask TC, or wait for TC to re-send.
  And meanwhile, B must continue to hold the transaction's locks.

What if TC crashes and restarts?
  If TC might have sent COMMIT before crash, TC must remember!
    Since one participant may already have committed.
  Thus TC must write COMMIT to disk before sending COMMIT msgs.
  And repeat COMMIT if it crashes and reboots,
    or if a participant asks (i.e. if A/B didn't get COMMIT msg).
  Participants must filter out duplicate COMMITs (using TID).

What if TC never gets a YES/NO from B?
  Perhaps B crashed and didn't recover; perhaps network is broken.
  TC can time out, and tell participants to abort,
    since TC has not sent any COMMIT msgs.
  Good: allows servers to release locks.

What if B times out or crashes while waiting for PREPARE from TC?
  B has not yet responded to PREPARE, so TC can't have decided commit
  so B can unilaterally abort, and release locks
  respond NO to future PREPARE

What if B replied YES to PREPARE, but doesn't receive COMMIT or ABORT?
  Can B unilaterally decide to abort?
    No! TC might have gotten YES from both,
    and sent COMMIT to A, but crashed before sending to B.
    So then A would commit and B would abort: incorrect.
  B can't unilaterally commit, either:
    A might have voted NO.

So: if B voted YES, it must "block": wait for TC decision.

When can TC completely forget about a committed transaction?
  If it sees an ACK from every participant for the COMMIT.
  Then no participant will ever need to ask again.

When can participant completely forget about a committed transaction?
  After it ACKs the TC's COMMIT message.
  If it gets another COMMIT, and has no record of the transaction,
    it must have already committed and forgotten, and can send ACK (again).

Two-phase commit perspective
  Used in sharded DBs when a transaction uses data on multiple shards
  But it has a bad reputation:
    slow: multiple rounds of messages
    slow: disk writes
    locks are held over the prepare/commit exchanges; blocks other xactions
    TC crash can cause indefinite blocking, WITH LOCKS HELD
  Thus usually used only in a single small domain
    E.g. not between banks, not between airlines, not over wide area
  Faster distributed transactions are an active research area.

Raft and two-phase commit solve different problems!
  Use Raft to get high availability by replicating
    i.e. to be able to operate when some servers are crashed
    the servers all do the *same* thing
  Use 2PC when each participant does something different
    And *all* of them must do their part
  2PC does not help availability
    since all servers must be up to get anything done
  Raft does not ensure that all servers do something
    since only requires a majority to execute.

What if you want high availability *and* atomic commit?
  Here's one plan.
  [diagram]
  The TC and servers should each be replicated with Raft
  Run two-phase commit among the replicated services
  Then you can tolerate failures and still make progress
  Spanner uses this arrangement (after break).

Next:
  midterm on Thursday! in Walker.
  read the papers, review lecture notes, look at old exams, look at your lab code

---

http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-phase.html

6.5840 2024 Lecture 22:  Security: Byzantine Fault Tolerance

Paper: "Practical byzantine fault tolerance" by Castro and Liskov (OSDI'99)

why are we reading this paper?
  impressive result
    solves a strictly harder problem than Raft
    namely state-machine replication in the presence of malicious replicas
  many follow-on papers
  BFT is not widely-used today
    people rely on prevention and detection of compromised nodes
  BFT is seeing a come back for bitcoin-like systems
    Bitcoin solves consensus with malicious participants
      But, proof-of-work and long delays to resolve forks
    Stellar generalizes PBFT for federated deployments
    IBM's hyperledger uses PBFT

we've considered many fault-tolerance protocols
  have always assumed "fail-stop" failures -- like power failure
  i.e. servers follow the protocol
  hard enough: crash vs network down; network partition

can one handle a larger class of failures?
  buggy servers, that compute incorrectly rather than stopping?
  servers that *don't* follow the protocol?
  servers that have been modified by an attacker?
  often called "Byzantine" faults

the paper's approach:
  replicated state machine
  assumes 2f+1 of 3f+1 are non-faulty
  use voting to select the right results
  not as easy as it might sound

let's assume the worst case:
  a single attacker controls the f faulty replicas
  and is actively trying to break the system
  if we can handle this, we can handle bugs in f replicas too

what are the attacker's powers?
  supplies the code that faulty replicas run
  knows the code the non-faulty replicas are running
  knows the faulty replicas' crypto keys
  can read network messages
  can temporarily force messages to be delayed via DoS

what faults *can't* happen?
  no more than f out of 3f+1 replicas can be faulty
  no client failure -- clients never do anything bad
  no guessing of crypto keys or breaking of cryptography

example use scenario:
  RM:
    echo A > grade
    echo B > grade
    tell FK "the grade file is ready"
  FK:
    cat grade

a faulty system could:
  totally make up the file contents
  execute write("A") but ignore write("B")
  show "B" to RM and "A" to FK
  execute write("B") only only some of the replicas

let's try to design our own byzantine-fault-tolerant RSM
  start simple (and broken), work towards paper's design

design 1:
  [client, n servers]
  n servers
    servers don't have the same exploits
    different OSes, different service implementations, etc.
  client and servers have public-key pairs
    every message is signed by sending
    recipients authenticate a message
  client sends request to all servers
  waits for all n to reply
    authenticates replies
  only proceeds if all n agree

what's wrong with design 1?
  one server may have been compromised
  that one server can stop progress by disagreeing

design 2:
  let's have replicas vote
  2f+1 servers, assume no more than f are faulty
  client waits for f+1 matching replies
    if only f are faulty, and network works eventually, must get them!

what's wrong with design 2's 2f+1?
  f+1 matching replies might be f bad nodes and just 1 good
    so maybe only one good node got the operation!
  *next* operation also waits for f+1
    might *not* include that one good node that saw op1
  example: (see drawing below)
    f g1 g2 (f is faulty/bad)
    everyone hears and replies to write("A")
    f and g1 reply to write("B"), but g2 misses it
      client can't wait for g2 since it may be the one faulty server
    f and g2 reply to read(), but g1 misses it
    so read() yields "A"
  result: client tricked into accepting a reply based on out-of-date state
    e.g. TA reads A instead of B from grades file
    e.g. Breaks correctness; it should provide linearizability (return B)

       Put k=A       Put k=B     Get k   ->   A   (f lies, sending A too)
    C --------------------------------------------------
           ^ ^ ^      \   ^  ^    \   ^     ^
        \ / / /        \/  /       \ /     /
    f --------------------------------------------------
         \/  /           \/         \    /
    g1 --------------------------------------------------
           \/                         \/
    g2 --------------------------------------------------
            A

    ==> Need to to ensure that replies include majority of good nodes, so that
    at least one will return B later

design 3:
  3f+1 servers, of which at most f are faulty
  client waits for 2f+1 matching replies
    == f bad nodes plus a majority of the good nodes
    so all sets of 2f+1 overlap in at least one good node
  example (see below):
    f g1 g2 g2 (f is faulty/bad)
    everyone hears write("A")
    f, g1, g2 process write("B"), g3 misses it
    now the read()
      client will wait for 2f+1=3 matching replies
      f and g3 will reply "A"
      g1 and g2 will reply "B"
    client doesn't know what to believe (neither is 2f+1)
      but it is guaranteed to see there's a problem
  so client can *detect* that some good nodes missed an operation
    we'll see how to repair in a bit

       Put k=A        Put k=B       Get k: A  B    B  A
    C --------------------------------------------------
           ^ ^ ^  ^     \  ^  ^ ^     \   ^  ^    ^ ^
        \ / / /  /       \/  / /       \ /   /   / /
    f --------------------------------------------------
         \/  /  /         \/  /          \ /   / /
    g1 --------------------------------------------------
           \/  /           \/              \/  /
    g2 --------------------------------------------------
            \/                              \/
    g3 --------------------------------------------------


what about handling multiple clients?
  non-faulty replicas must process operations in the same order!

let's have a primary to pick order for concurrent client requests
  but we have to worry about a faulty primary

what can a faulty primary do?
  1. ignore a client op
  2. different ops to different replicas
  3. send wrong result to client

general approach to handling faulty primary
  1. clients notify replicas of each operation, as well as primary
  2. replicas exchange info about ops sent by primary
  3. replicas send results direct to client
  4. each replica watches progress of each operation
     if no progress, force change of primary

can a replica execute an operation when it first receives it from primary?
  no: maybe primary gave different ops to different replicas
  if we execute before we're sure, we've wrecked the replica's state
  need 2nd round of messages to make sure all good replicas got the same op

design 4 (with primary):
  3f+1 servers, one is primary, f faulty, primary might be faulty
  client sends request to primary AND to each replica
  primary chooses next op and op #
  primary sends PRE-PREPARE(op, n) to replicas
  each replica sends PREPARE(op, n) to all replicas
  if replica gets matching PREPARE(op, n) from 2f+1 replicas (incl itself)
    and n is the next operation #
    execute the operation, possibly modifying state
    send reply to client
  else:
    keep waiting
  client is happy when it gets f+1 matching replies

    op                                                                  f+1  matching replies
    C ---------------------------------------------------------------------------------------
                                                       ^      ^
        \ pre-P (op,n)      PREPARE   2f+1 matching replies p /      /
    p ---------------------------------------------------------------------------------------
         \       \    ^ ^    \   ^ ^   / \  2f+1 matching replies  /
    g1 --------------------------------------------------------------------------------------
           \       \ / /      \ / /    \  /
    g2 --------------------------------------------------------------------------------------
            \       \/         \/       \/
    g3 --------------------------------------------------------------------------------------


remember our strategy:
  primary follows protocol => progress
  no progress => replicas detect and force change of primary

if the primary is non-faulty, can faulty replicas prevent correct progress?
  they can't forge primary msgs
  they can delay msgs, but not forever
  they can do nothing: but they aren't needed for 2f+1 matching PREPAREs
  they can send correct PREPAREs
    and DoS f good replicas to prevent them from hearing ops
    but those replicas will eventually hear the ops from the primary
  worst outcome: delays

if the primary is faulty, will replicas detect any problem?
  or can primary cause undetectable problem?
  primary can't forge client ops -- signed
  it can't ignore client ops -- client sends to all replicas
  it can try to send in different order to different replicas,
    or try to trick replicas into thinking an op has been
    processed even though it hasn't
  will replicas detect such an attack?

results of the primary sending diff ops to diff replicas?
  case 1: all good nodes get 2f+1 matching PREPAREs
    did they all get the same op?
    yes: everyone who got 2f+1 matching PREPAREs must have gotten same op
      since any two sets of 2f+1 share at least one good server
    result: all good nodes will execute op, client happy
  case 2: >= f+1 good nodes get 2f+1 matching PREPARES
    again, no disagreement possible
    result: f+1 good nodes will execute op, client happy
    BUT up to f good nodes don't execute
      can they be used to effectively roll back the op?
      i.e. send the write("B") to f+1, send read() to remaining f
      no: won't be able to find 2f+1 replicas with old state
      so no enough PREPAREs
  case 3: < f+1 good nodes get 2f+1 matching PREPAREs
    result: client never gets a reply
    result: system will stop, since f+1 stuck waiting for this op

how to resume operation after faulty primary?
  need a view change to choose new primary
  (this view change only chooses primary; no notion of set of live servers)

when does a replica ask for a view change?
  if it sees a client op but doesn't see 2f+1 matching PREPAREs
  after some timeout period

is it OK to trigger a view change if just one replica asks?
  no: faulty replicas might cause constant view changes

let's defer the question of how many replicas must ask for
  a view change

who is the next primary?
  need to make sure faulty replicas can't always make themselves next primary
  view number v
  primary is v mod n
  so primary rotates among servers
  at most f faulty primaries in a row

view change design 1 (not correct)
  replicas send VIEW-CHANGE requests to *new* primary
  new primary waits for enough view-change requests
  new primary announces view change w/ NEW-VIEW
    includes the VIEW-CHANGE requests
    as proof that enough replicas wanted to change views
  new primary starts numbering operations at last n it saw + 1

will all non-faulty replicas agree about operation numbering across view change?

problem:
  I saw 2f+1 PREPAREs for operation n, so I executed it
  new primary did not, so it did not execute it
  thus new primary may start numbering at n, yielding two different op #n

can new primary ask all replicas for set of operations they have executed?
  doesn't work: new primary can only wait for 2f+1 replies
    faulty replicas may reply, so new primary may not wait for me

solution:
  don't execute operation until sure a new primary will hear about it
  add a third phase: PRE-PREPARE, PREPARE, then COMMIT
  only execute after commit

operation protocol:
  client sends op to primary
  primary sends PRE-PREPARE(op, n) to all
  all send PREPARE(op, n) to all
  after replica receives 2f+1 matching PREPARE(op, n)
    send COMMIT(op, n) to all
  after receiving 2f+1 matching COMMIT(op, n)
    execute op

view change:
  each replica sends new primary 2f+1 PREPAREs for recent ops
  new primary waits for 2f+1 VIEW-CHANGE requests
    new primary fixes up the log based on info in view-change requests
  new primary sends NEW-VIEW msg to all replicas with
    complete set of VIEW-CHANGE msgs
    list of every op for which some VIEW-CHANGE contained 2f+1 PREPAREs
      if op doesn't have 2f+1 PREPAREs, make the op a no-op
    i.e. list of final ops from last view

if a replica executes an op, will new primary will know of that op?
  replica only executed after receiving 2f+1 COMMITS
  maybe f of those were lies, from faulty replicas, who won't tell new primary
  but f+1 COMMITs were from replicas that got 2f+1 matching PREPAREs
  new primary waits for view-change requests from 2f+1 replicas
    ignoring the f faulty nodes
    f+1 sent COMMITs, f+1 sent VIEW-CHANGE
    must overlap

can the new primary omit some of the reported recent operations?
  no, NEW-VIEW must include signed VIEW-CHANGE messages

paper also discusses
  checkpoints and logs to help good nodes recover
  various cryptographic optimizations
  optimizations to reduce # of msgs in common case
  fast read-only operations

what are the consequences of more than f corrupt servers?
  can the system recover?

what if the client is corrupt?

suppose an attacker can corrupt one of the servers
  exploits a bug, or steals a password, or has physical access, &c
  why can't the attacker corrupt them all?

References:
  PhD thesis: https://dspace.mit.edu/bitstream/handle/1721.1/86581/48116479-MIT.pdf
  Proactive recovery:  http://dl.acm.org/citation.cfm?id=571640>
  BASE: http://dl.acm.org/citation.cfm?id=859718
  Stellar: https://www.stellar.org/papers/stellar-consensus-protocol.pdf
  Hyperledger: https://www.zurich.ibm.com/dccl/papers/cachin_dccl.pdf
  A funny perspective on BFT: https://www.usenix.org/system/files/login-logout_1305_mickens.pdf

6.5840 2026 Lecture 20: Bitcoin

Bitcoin: A Peer-to-Peer Electronic Cash System, by Satoshi Nakamoto, 2008

why this paper?
  solves a problem that seemed obviously impossible
    built entirely on a collection of untrusted participants
    you don't know who they are
    some are certain to be corrupt
    yet Bitcoin is secure enough for financial transactions
  contains some very clever ideas!

related to SUNDR straw man:
  log of signed operations, linked by cryptographic hashes
  agreement on log -> agreement on state
  forks are a key danger
  unlike SUNDR: Bitcoin automatically resolves forks

the key danger: double spending

notation:
  pub(new): public key of new owner
  H(prev): cryptographic hash of this coin's previous transaction record
  sig(old): signature over this transaction by previous owner's private key

a simple (broken) transaction-chain scheme:
  X has previously paid a coin to Y:
    T6: pub(X), ...
    T7: pub(Y), H(T6), sig(X)
  Y buys a sandwich from Z and pays with this coin
    Z sends public key to Y
    Y creates a new transaction and signs it
    T8: pub(Z), H(T7), sig(Y)
  Y sends transaction T8 to Z
  Z verifies T8:
    really pub(Z)
    T7 exists, hash is correct
    T8's sig(Y) verifies using T7's pub(Y)
      verify(T8, T8.sig(), T7.pub()) == ok
  Z gives sandwich to Y

Bitcoin only records transactions, not coins or accounts or balances
  Z's "balance": unspent transactions for which Z knows private key
  the "identity" of a coin is the (hash of) its most recent xaction

can anyone other than the owner spend a coin?
  current owner's private key needed to sign next transaction
  danger: perhaps attacker can steal Z's private key
    e.g. from PC or smartphone or online exchange
    this is a serious problem in practice, and hard to solve well

can a coin's owner spend it twice in this scheme?
  Y creates two transactions for same coin: Y->Z, Y->Q
    both with H(T7)
  Y shows different transactions to Z and Q
  both transactions look good, including signatures and hash
  now both Z and Q will give sandwiches to Y
  double-spending is the most fundamental problem that bitcoin solves

why is double-spending possible in my simple scheme?
  Z and Q didn't know about the other transaction

a "public ledger" would reveal Y's double-spend
  publish a log of all transactions
  ensure everyone sees the same log, in the same order
  ensure no-one can un-publish or modify a log entry
  Z, Q only believe transactions seen in log
  result:
    Q will see Y->Z is already in log, and will reject Y->Q
  how to create such a ledger?

the BitCoin peer network
  lots of peer computers participate in agreement
    [net diagram]
    each has a complete copy of the whole chain
    each has TCP connections to a few other peers -- a "mesh overlay"
    new chain blocks flooded to all peers, by forwarding over TCP
    proposed transactions also flooded to all peers
    flooding helps ensure all peers are aware of all transactions
  anyone can run a Bitcoin peer!
    currently tens of thousands
    so some peers will be malicious
    design assumes honest >> malicious

the BitCoin block chain
  the block chain contains all transactions on all coins
    so that double-spending will be visible
  each block:
    H(prevblock)
    reward transaction
    list of transactions (pub, H, sig)
    "nonce"
    block creation time (wall clock timestamp)
  note that the block is not signed!
  new block every 10 minutes containing xactions since prev block
  payee believes transaction when seen in the block chain

who creates each new block?
  this is "mining" via "proof-of-work"
  requirement: H(block) has N leading zeros
    each peer tries random nonce values until this works out
  trying one nonce is fast, but most nonces won't work
    it's like flipping a zillion-sided coin until it comes up heads
    each flip has an independent small chance of success
    mining a block is *not* a specific fixed amount of work
  it would likely take one CPU months to create one block
  but thousands of peers are working on it
  such that average time to first to find is about 10 minutes
    though the variance is high
  the winner floods the new block to all peers

how does a Y->Z transaction work w/ block chain?
  start: all peers know ...<-B5
    and are mining block B6 (trying different nonces)
  Y sends Y->Z transaction to peers, which flood it
  peers buffer the transaction until B6 computed
  peers that heard Y->Z include it in next block
  so eventually ...<-B5<-B6<-B7, where B7 includes Y->Z

Q: could there be *two* different successors to B6?
A: yes:
   1) two peers find nonces at about the same time, or
   2) slow network, 2nd block found before 1st is flooded to all
  two simultaneous blocks will be different
    miners know about slightly different sets of new transactions, &c.
  if two successors, the blockchain temporarily forks
    peers mine a successor to whichever block they heard first
    but switch to longer chain if they hear one

how is a fork resolved?
  each peer initially believes the first new (and valid) block it sees
  tries to mine a successor
  if more saw Bx than By, more will mine for Bx,
    so Bx successor likely to be created first
  even if exactly half-and-half, one fork likely to be extended first
    since significant variance in mining time
  peers switch to the longest fork once they see it
    so longer fork gets more mining power extending it
    so agreement on a block tends to be re-enforced
  what about transactions in the abandoned fork?
    most will be in both forks
    but some may be in just the abandoned fork -- appear, then disappear!

what if Y->Z ends up in an abandoned transient fork?
  that might allow Y to double-spend!
  Z sees Y->Z; then it disappears; then Y can broadcast Y->Q

thus:
  double spending is possible, due to forks
  but a transient fork is highly likely to be resolved quickly
  if Z sees Y->Z with a few blocks after it,
    it's very unlikely that it could be overtaken by a
    different fork containing Y->Q
  if Z is selling a high-value item, Z should wait for a few
    blocks before shipping it
  if Z is selling something cheap, maybe OK to wait just for some peers
    to see Y->Z and validate it (even before Y->Z is in a block)

could attacker start a fork from an old block, with Y->Q instead of Y->Z?
  yes -- but fork must be longer in order for peers to accept it
  since attacker's fork starts behind main fork,
    attacker must mine blocks *faster* than total of other peers
  with just one CPU, will take months to create even a few blocks
    by that time the main chain will be much longer
    no peer will switch to the attacker's shorter chain

"51% attack"
  if the attacker has more CPU power than all the honest
    bitcoin peers -- then the attacker can create the longest fork,
    everyone will switch to it, allowing the attacker to double-spend.

one way to think about bitcoin's proof-of-work mining
  random choice over peers for who gets to choose which fork to extend
    weighted by CPU power
  if most participants are honest,
    they will re-inforce agreement on longest fork
  random choice means a (small) attacker won't get many
    chances to try to switch agreement to a different fork
  surprising that random choice is possible without knowing
    participant identities or even how many!

what motivates miners?
  critical that there be many miners, to make 51% attacks harder
  each new block pays miner a few newly created bitcoins -- a reward
    block contains public key that gets the new bitcoins
    this is incentive for people to operate bitcoin peers
  hardness (# of leading zeros) automatically adjusted
    to make avg interval be 10 minutes
  consequences:
    arms race, since more hardware -> more reward
    special hardware
    pools of miners that collaborate
    energy waste

how is it possible to create reward money out of nothing?
  the rules are determined by the code in (a majority) of peers
  the peers all have code that honors the special reward transactions

validation checks:
  peer, new xaction:
    previous transaction exists
    no other transaction spends the same previous transaction
    signature is by private key of pub key in previous transaction
    then will add transaction to txn list for next block to mine
  peer, new block:
    hash value has enough leading zeroes (i.e. nonce is right, proves work)
    previous block hash exists
    all transactions in block are valid
    peer switches to new chain if longer than current longest
  Z:
    (client does above checks, or relies on a peer to do them)
    Y->Z is in a block
    Z's public key / address is in the transaction
    there's several more blocks in the chain
  (other stuff has to be checked as well, lots of details)

Q: why an open / "permissionless" / decentralized peer structure?

Q: why is it reasonable to assume the majority of peers are honest?

Q: if one started a new bitcoin-like cryptocurrency, with only a few peers,
   would it be reasonable to trust it?

Q: what transaction rate can bitcoin support?
   10 minutes * 1 MB blocks -> about 5 transactions/second
   credit card system handles about 5000 / second

Q: 10 minutes is annoying; could it be made much shorter?

Q: what would happen if I modified my copy of the Bitcoin s/w?
   e.g. to reduce the proof-of-work hardness?

Q: could Bitcoin have been just a ledger w/o a new currency?
   e.g. have dollars be the currency?
   since the currency part is pretty awkward.
   how to incentivize miners?
   how to transfer actual dollars?

Q: why don't more people use Bitcoin for ordinary transactions?
   high fees; not reversible; 10 minute delay; volatile value

Q: are transactions anonymous?

Q: how could one steal someone else's bitcoins?

Q: if I steal bitcoins, is it safe to spend them?

Q: what if the block format needs to be changed?
   esp if new format wouldn't be acceptable to previous s/w version?
   "hard fork"

Q: how do peers find each other?

Q: what if a peer has been tricked into only talking to corrupt peers?

Q: what if a peer rejoins after a long period of disconnection?

Q: why does it make sense for the mining reward to decrease with time?

Q: is it a problem that there will be a fixed number of coins?
   what if the real economy grows (or shrinks)?

Q: why do bitcoins have value?
   e.g. people seem willing to pay $76,000 per bitcoin (April 2026).

Q: if lots of miners join, will blocks be created at a higher rate?

Q: can bitcoins be forged, i.e. a totally fake coin created?

weak points in the design?
  proof-of-work uses a lot of electricity
  too bad it's a new currency as well as a payment system
  transaction confirmation takes at least 10 minutes, or 60 for high confidence
  flooding limits performance, may be a point of attack
  maximum block size plus 10 minutes limits max transactions per second
  vulnerable to majority attack
  not very anonmyous
  anonymous enough to attract illegal activity
  users have trouble securing private keys

key idea: block chain
  public agreed-on ledger is a great idea
  mining is a clever way to resolve forks / ensure agreement

next week:
  BFT, another way to obtain security from possibly-insecure servers

---- References ----

https://www.ieee-security.org/TC/SP2015/papers-archived/6949a104.pdf
https://en.bitcoin.it/wiki/Script
https://www.oreilly.com/library/view/mastering-bitcoin/9781491902639/ch08.html

6.5840 2026 Lecture 13: Chain Replication

Chain replication for supporting high throughput and availability
(OSDI 2004) by Renesse and Schneider

a couple of topics today:
  replicated state machines
  revisit primary/backup replication
  chain replication as a better primary/backup
  compare p/b vs chain vs quorum
  structure of sharded systems

two approaches to replicated-state machine
  1. run all ops through Raft/Paxos (lab 3)
  2. configuration server plus primary-backup
      GFS master + chunk replication
      VM-FT: test-and-set + P/B replication
      zookeeper + primary-backup
    configuration server must be fault-tolerant

why approach 2?
  separation of concerns
    configuration srv stores configuration info
      who is primary, which server has what shard
      who is the head of the chain
      must handle split-brain syndrome
        often uses paxos/raft
        can proceed with majority
    data replication protocol
      primary-backup, chain replication or whatever
      often simpler than raft (figure 2); doesn't need to handle split brain
      can proceed if only 1 server is available (after reconfiguration)

primary/backup (paper's Section 4)
  old, widely used in various forms
  GFS uses a (non-strict) p/b scheme for chunks
  CR paper is an improved p/b

the basic primary/backup arrangement
  [diagram: CFG, clients, primary, two backups, state = key/value table]
  primary numbers ops, to impose order if concurrent client ops arrive
  primary forwards each to backups (in parallel)
  primary waits for response from ALL backups, replies to client
  primary can respond to reads w/o sending to backups
  if primary fails, one of the backups becomes new primary
    can proceed if even a single replica survives
    so N replicas can survive N-1 failures -- better than Raft
  a separate configuration service (CFG) manages failover
    configuration = identity of current primary and backups
    usually built on Paxos or Raft or ZooKeeper
    pings all servers to detect failures
    CFG must include at least one replica from
      previous configuration in any new configuration.
  if a backup fails, CFG must remove it from the configuration,
    b/c the primary has to wait for all backups

sometimes called ROWA (Read One Write All), as opposed to quorum
  the "write all" is what allows us to tolerate N-1 failures
    since every replica has all committed values, unlike quorums

a few key properties we need to check for any strong replication scheme:
  1. never reveal uncommitted data
     i.e. data that might not survive a tolerated failure
  2. after failure recovery, all replicas must agree!
     if one saw last msg from primary, but the other did not
  3. no split brain if network partitions
     only one new primary; old primary must stop
  4. able to recover after a complete failure
  5. handle retried client requests

#1 for primary/backup?
  example bad situation:
    put(x, 99) arrives at primary
    get(x) immediately follows it
      before primary gets backups' responses to put(x,99)
    we said primary can serve reads w/o talking to backups
    but it's not safe to return 99!
  answer: either respond with old value,
    or block read until all backups respond to put(x,99).

#2 for primary/backup?
  new primary could merge all replicas' last few operations.
  or, elect as primary the backup that received the highest # op.
  and new primary must start by forcing backups to agree with it

#3 for primary/backup?
  how to prevent backup from taking over if partitioned from primary?
    and primary is still alive.
    *don't* have backups make this decision!
    only the CFG can declare a new primary + backups.
      based on CFG's opinion of which servers it can reach.
  what if the old primary is actually alive, but CFG can't talk to it?
    how to prevent old primary from serving requests?
    for updates:
      old primary cannot respond to client requests until all
        replicas in its configuration reply
      at least one replica must be part of the new configuration
      so replicas must be careful not to reply to an old primary!
    for reads:
      primary doesn't have to talk to backups for reads
      so CFG must grant lease to primary

#5 for primary/backup?
  server does duplicate detection keeps a reply table with sent responses
    the client clerk stamps each request with a nonce (unique id)
    server checks if already responded to that nonce
  other solution:
    add version# to put and store version # with key  (lab 2)

Chain Replication was published at a time when only a few people
  understood the detailed design of strongly consistent replication;
  it's been influential, and a fair number of real-world systems build
  on it.

what p/b problems does Chain Replication aim to fix?
  1. primary has to do a lot of work
  2. primary has to send a lot of network data
  3. re-sync after primary failure is complex, since backups may differ

the basic Chain Replication idea:
[clients, S1=head, S2, S3=tail, CFG (= master)]
(can be more replicas)
  clients send updates requests to head
    head picks an order (assigns sequence numbers)
    head updates local replica, sends to S1
    S1 updates local replica, sends to S2
    S2 updates local replica, sends to S3
    S3 updates local replica, sends response to client
    updates move along the chain in order:
      at each server, earlier updates delivered before later ones
  clients send read requests to tail
    tail reads local replica and responds to client

benefit: head sends less network data than a primary

benefit: client interaction work is split between head and tail

The Question: Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that are not
  linearizable.

what if the head fails?
  the CFG (master) is in charge of recovering from failures
  CFG tells 2nd chain server to be new head
    and tells clients who the new head is
  will all (remaining) servers still be exact replicas?
    they will soon!
    in-order delivery means each server is identical
      to the one before it, just missing last few updates
    so each server needs to compare notes with successor and
      just send those last few updates
  some client update requests may be lost
    if only the failed head knew about them
    clients won't receive responses
    and will eventually re-send to new head

what if the tail fails?
  CFG tells next-to-last server to be new tail
    and tells clients, for read requests
  next-to-last is at least as up to date as the old tail
  for updates that new tail received but old tail didn't,
    system won't send responses to clients.
    clients will time out and re-send
    Section 2 says clients are responsible for checking
      whether timed-out operations have actually already
      been executed (often harder than it sounds).

what if an intermediate server fails?
  CFG tells previous/next servers to talk to each other
  previous server may have to re-send some updates that
    it had already sent to failed server

note that servers need to remember updates even after forwarding
  in case a failure requires them to re-send
  when to free?
  tail sends ACKs back up the chain as it receives updates
    when a server gets an ACK, it can free all through that op

what's the argument that CR won't reveal an uncommitted update?
  i.e. could client read a value,
    but it then disappears due to a tolerated failure?
  or could a client get a "yes" response to an update,
    but then it's not there after a failure?
  reads come from the tail
    the tail only sees an update after every other server sees it
    so after a failure, every server still has that update
  update is responded to after it gets to the tail
    at which point every server has it

how to add a new server? ("extend the chain")
  you need to do this to restore replication level after a failure.
  again, CFG manages this
  new server is added at the tail
  a slow possibility:
    tell the old tail to stop processing updates
    tell the old tail to send a complete copy of its data to the new tail
    tell the old tail to start acting as an intermediate server,
      forwarding to the new tail
    tell the new tail to start acting as the tail
    tell clients about new tail
  slow b/c we're pausing all updates for minutes or hours
  better is to transfer a snapshot of the state in advance
    then freeze the system just long enough to send the
      last few updates to the new tail
    then reconfigure and un-freeze
    perhaps use ZooKeeper's fuzzy snapshot idea

partition situation is much as in p/b
  CFG makes all decisions
    it will pick a single new head &c
    based on its view of server liveness -- i.e. just in CFG's partition
  new head is old 2nd server, it should ignore
    updates from the old head, to cope with "what if old
    head is alive but CFG thinks it has failed"
  CFG needs to grant tail a lease to serve client reads,
    and not designate a new tail until lease has expired

p/b versus chain replication?
  p/b may have lower latency (for small requests)
  chain head has less network load than primary
    important if data items are big (as with GFS)
  chain splits work between head and tail
    primary does it all, maybe more of a bottleneck
  chain has simpler story for which server should take over if head fails,
    and how ensure servers get back in sync

chain (or p/b) versus Raft/Paxos/Zab (quorum)?
  p/b can tolerate N-1 of N failures, quorum only N/2
  p/b simpler, maybe faster than quorum
  p/b requires separate CFG, quorum self-contained
  p/b must wait for reconfig after failure, quorum keeps going
  p/b slow if even one server slow, quorum tolerates temporary slow minority
  p/b CFG's server failure detector hard to tune:
    any failed server stalls p/b, so want to declare failed quickly!
    but over-eager failure detector will waste time copying data to new server.
    quorum system handles short / unclear failures more gracefully

for a long time p/b (and chain) dominated data replication
  Paxos was viewed as too complex and slow for high-performance DBs
  recently quorum systems have been gaining ground
    due to good toleration of temporarily slow/flaky replicas

what if you have too much data to fit on a single replica group?
  e.g. millions of objects
  you need to "shard" across many "replica groups"

sharding diagram:
  [CFG, G1, G2, .., Gn, clients]
  GFS looked like this
  modern system might use ZK for CFG, chain or p/b or Raft for data

how to lay out chains on server in a big sharding setup?
  the paper's 5.2 / 5.3 / 5.4

a not-so-great chain or p/b sharding arrangement:
  each set of three servers serves a single shard / chain
    shard A: S1 S2 S3
    shard B: S4 S5 S6
  problem: some servers will be more loaded than others
    the primary in each group will be slow while the others have idle capacity
    the head and tail will be more loaded than the middle
    the under-loaded servers waste money!
  problem: replacing a failed replica takes a long time!
    the new server must fetch a whole disk of data over the network
      from one of the remaining replicas
    a terabyte at a gigabit/second takes two hours!
    significant risk of remaining replicas failing before completion!

a better plan ("rndpar" in Section 5.4):
  split data into many more shards than servers
    (so each shard is much smaller than in previous arrangement)
  each server is a replica in many shard groups
    shard A: S1 S2 S3
    shard B: S2 S3 S1
    shard C: S3 S1 S2
    (this is a regular arrangement, but in general would be random)
  for p/b, a server is primary in some groups, backup in other
  for chain, a server is head in some, tail in others, middle in others
  now request processing work is likely to be more balanced

how does rndpar do for repair speed?
  suppose one server fails.
  say it participated in M replica groups, for M shards.
  instead of designating a single replacement server, let's
    choose M replacement servers, a different one for each shard.
    these are existing servers, which we're giving a new responsibility.
  now repair of the M shards can go on in parallel!
    instead of taking a few hours, it will take 1/M'th that time.

how does rndpar do if three random servers fail?
  as the number of shards on each server increases, it gets more
    likely that *some* shard had its three replicas on
    the three random servers that failed.
  this is not ideal.
  rndpar gives us fast repair, but it's somewhat undermined
    by higher probability that a few failures wipes out all
    replicas for some shard (Figure 7).

conclusion
  Chain Replication is one of the clearest descriptions of a ROWA scheme
  it does a good job of balancing work
  it has a simple approach to re-syncing replicas after a failure
  influential: used in EBS, Ceph, Parameter Server, COPS, FAWN.
  it's one of a number of designs (p/b, quorums) with different properties

-----------------

5.1 take-away:
  chain *throughput* as high as p/b b/c limited by head/primary CPU
  for 0% updates, chain limited by tail alone, and p/b limited by prim alone
  for 100% updates, chain limited by head alone, and p/b limited by prim alone
  only in the middle is there a difference b/c chain splits work
    between head and tail
  BUT network communication assumed to be free; in real life
    p/b has a problem b/c prim must send to all backups
  BUT write latency is a problem in the real world, since
    client often waiting for "committed" reply

5.2 take-away:
  assume 1000s of chains and dozens of servers.
  idea: split your data over many chains, and the chains
    over a modest number of servers, so that every server
    is in many chains, some as head, some as tail, some
    in the middle
  this balances the load of being head and tail vs middle.
  Figure 5 doesn't seem to say much, maybe just that if
    you have only 25 clients, then there's not much point
    in having more than about 25 servers.

5.3 take-away:
  assume 1000s of chains and dozens of servers.
  repair time is a big deal, since a single server can store
    so much data that it takes hours to transfer over a
    single network link.
  when a *single* server fails, need to spread responsibility
    for the data it replicated over *multiple* other servers,
    to get fast parallel repair.

5.4 take-away:
  assume 1000s of chains and dozens of servers.
  best for parallel recovery speed is if no constraints and
    random placement, so that both sources and destinations
    of recovery traffic are evenly-ish spread after a single
    failure.
  BUT if chains are randomly spread over servers, then *any*
    combination of three (if chainlen=3) random server failures
    has a good chance of destroying all replicas of *some* chain.
  the ring topologies are a compromise: you can't spread
    recovery load very widely, but a few random server failures
    are less likely to destroy all of any one chain's replicas.
  for their setup, speed of reconstruction seems to be more
    important than simultaneous failures wiping out a chain.
  however, they assume MTBF of a single server of 24 hours,
    which does mean fast repair is crucial when repair can
    take hours, but 24 hours seems unrealistically short.

6.5840 2026 Lecture 14: FaRM, Optimistic Concurrency Control

why are we reading about FaRM?
  another take on transactions+replication+sharding
    this is still an open research area!
  optimistic concurrency control
  exploiting huge performance potential of RDMA NICs

the overall setup
  all in one data center
  configuration manager, using ZooKeeper, chooses primaries/backups
  sharded w/ primary/backup replication
    P1 B1
    P2 B2
    ...
    can recover as long as at least one replica of each shard
    i.e. f+1 replicas tolerate f failures
  transaction clients (which they run in the servers)
  transaction code acts as two-phase-commit Transaction Coordinator (TC)

the goal:
  millions of distributed transactions per second
  so the time budget is tens of microseconds
  very challenging!

how do they get high performance?
  sharding over many servers (90 in the evaluation)
  data must fit in total RAM (so no disk reads)
  non-volatile RAM (so no disk writes)
  one-sided RDMA (fast cross-network access to RAM)
  fast user-level access to NIC
  transaction+replication protocol that exploits one-sided RDMA

NVRAM (non-volatile RAM)
  FaRM writes go to RAM, not disk -- eliminates a huge bottleneck
  RAM write takes 200 ns, hard drive write takes 10 ms, SSD write 100 us
    ns = nanosecond, ms = millisecond, us = microsecond
  but RAM loses content in power failure! not persistent by itself.
  why not just write to RAM of f+1 machines, to tolerate f failures?
    might be enough if failures were always independent
    but power failure is not independent -- may strike 100% of machines!
  so:
    batteries in every rack, can run machines for a few minutes
    power h/w notifies s/w when main power fails
    s/w halts all transaction processing
    s/w writes FaRM's RAM to SSD; may take a few minutes
    then machine shuts down
    on re-start, FaRM reads saved memory image from SSD
    "non-volatile RAM"
  what if crash prevents s/w from writing SSD?
    e.g bug in FaRM or kernel, or cpu/memory/hardware error
    FaRM copes with single-machine crashes with replication
    crashes (other than power failure) must be independent!
  summary:
    NVRAM eliminates persistence write bottleneck
    leaving network and CPU as remaining bottlenecks

why is the network often a performance bottleneck?
  FaRM assumes single data-center, so low speed-of-light delay
  but CPU cost of network data handling is often large!
  the usual setup for RPC over TCP over LAN:
    app                       app
    ---                       ---
    socket buffers            buffers
    TCP                       TCP
    NIC driver                driver
    NIC  -------------------- NIC
  lots of expensive CPU operations:
    system calls
    copy messages
    interrupts
    context switches
  slow:
    hard to build RPC than can deliver more than a few 100,000 / second
    wire b/w (e.g. 10 gigabits/second) is rarely the limit for short RPC
    per-packet CPU costs traditionally limit performance for small messages

FaRM uses two networking ideas:
  Kernel bypass
  RDMA

Kernel bypass
  [diagram: FaRM user program, CPU cores, DMA queues, NIC]
  application directly interacts with NIC -- no system calls, no kernel
  NIC DMAs into/out of user RAM
  FaRM s/w polls DMA areas to check for incoming messages
  NIC polls DMA areas to check for outgoing messages

RDMA (remote direct memory access)
  [src host, NIC, switch, NIC, target memory, target CPU]
  remote NIC directly reads/writes memory
    Sender provides memory address
    Remote CPU is not involved!
    This is "one-sided RDMA"
    Reads an entire cache line, atomically
  RDMA NICs use reliable protocol, with ACKs
  one server's throughput: 10+ million/second (Figure 2)
  latency: 5 microseconds (from their NSDI 2014 paper)

Performance would be amazing if clients could directly access
  DB records on servers via one-sided RDMA!

How to combine one-sided RDMA with replication and transactions?
  The protocols we've seen so far require active server participation.
  e.g. to check and set locks,
       to check lease,
       to indicate when safely persisted.
  Not immediately compatible with one-sided RDMA.

two classes of concurrency control for transactions:
  pessimistic (two-phase locking):
    wait for lock on first use of object; hold until commit/abort
    conflicts cause delays
  optimistic:
    read objects without locking
    don't install writes until commit
    commit "validates" to see if other xactions conflicted
    valid: commit the writes
    invalid: abort
    called Optimistic Concurrency Control (OCC)

FaRM uses OCC
  the reason:
    OCC lets FaRM read using one-sided RDMA reads
    so server needn't actively participate in reads
  how does FaRM's OCC validate? we'll look at Figure 4 in a minute.

FaRM transaction API (simplified):
  txCreate()
  o = txRead(oid)  -- RDMA
  o.f += 1
  txWrite(oid, o)  -- purely local
  ok = txCommit()  -- Figure 4

what's an oid?
  <region #, address>
  region # indexes a mapping to [ primary, backup1, ... ]
  target RDMA NIC uses address directly to read RAM

server memory layout
  regions, each an array of objects
  object layout
    header with version #, and lock flag in high bit of version #
  for each other server
    incoming log
    incoming message queue
    (senders write via RDMA, local FaRM reads via polling)
  all this in non-volatile RAM (i.e. written to SSD on power failure)

Figure 4: transaction execution / commit protocol
  let's consider steps in Figure 4 one by one
  focus on concurrency control (not fault tolerance)

Execute phase
  TC (the client) reads the objects it needs from servers
    including records that it will write
    using one-sided RDMA reads
    without locking
    this is the optimism in Optimistic Concurrency Control
  TC remembers the version numbers
  TC buffers writes locally

now TC commits; two big goals:
  atomic distributed commit -- all writes or none
  serializability -- as if entirely before or after every other transaction

LOCK (first message in commit protocol)
  TC sends to primary of each written object
  TC uses RDMA to append to its log at each primary
  LOCK record contains oid, version # xaction read, new value
  LOCK is now logged in primary's NVRAM
    will survive a power failure
  LOCK message is both a write-ahead log entry,
    and an RPC request to the primary

what does primary do on receipt of LOCK?
  FaRM s/w polls incoming logs in RAM, sees our LOCK
  if object locked, or version != what xaction read,
    send "no" reply to TC
  otherwise set the object's lock flag and reply "yes"
    but don't yet modify the data!
  lock check, version check, and lock set are atomic
    using atomic compare-and-swap instruction
    "locked" flag is high-order bit in object's version number
    in case other CPU also processing a LOCK, or a client is reading w/ RDMA

TC waits for all LOCK reply messages
  if any "no", abort
    append ABORT to primaries' logs so they can release locks
    returns "no" from txCommit()

let's ignore VALIDATE and COMMIT BACKUP for now

at this point primaries need to know TC's decision

TC appends COMMIT-PRIMARY to primaries' logs
  TC only waits for RDMA hardware acknowledgement (ack)
    does not wait for primary to process log entry
    hardware ack means safe in primary's NVRAM
  TC returns "yes" from txCommit()

when primary processes COMMIT-PRIMARY in its log:
  copy new value to object's memory
  increment object's version #
  clear object's lock flag

the commit point is when the first COMMIT-PRIMARY is written
  since at that point the transactions results can be revealed

example:
  T1 and T2 both want to increment x
    x = x + 1
  what results does serializability allow?
    i.e. what outcomes are possible if run one at a time?
    x = 2, both clients told "success"
    x = 1, one client told "success", other "aborted"
    x = 0, both clients told "aborted"

what if T1 and T2 are exactly in step?
  T1: Rx0  Lx  Cx
  T2: Rx0  Lx  Cx
  what will happen?

or
  T1:    Rx0 Lx Cx
  T2: Rx0          Lx  Cx

or
  T1: Rx0  Lx  Cx
  T2:             Rx0  Lx  Cx

intuition for why FaRM's OCC provides serializability:
  i.e. checks "was execution same as one at a time?"
  if there was no conflicting transaction:
    the versions won't have changed
  if there was a conflicting transaction:
    one or the other will see a lock or changed version #

what about VALIDATE in Figure 4?
  it is an optimization for objects that are just read by a transaction
  VALIDATE = one-sided RDMA read to re-fetch object's version # and lock flag
  if lock set, or version # changed since read, TC aborts
  does not set the lock, thus faster than LOCK+COMMIT

VALIDATE example:
x and y initially zero
T1:
  if x == 0:
    y = 1
T2:
  if y == 0:
    x = 1
(this is a classic test example for transactions)
T1,T2 yields y=1,x=0
T2,T1 yields x=1,y=0
aborts could leave x=0,y=0
but serializability forbids x=1,y=1

suppose simultaneous:
  T1:  Rx  Ly  Vx  Cy
  T2:  Ry  Lx  Vy  Cx
  what will happen?
  the LOCKs will both succeed!
  the VALIDATEs will both fail, since lock bits are both set
  so both will abort -- which is OK

how about:
  T1:  Rx  Ly  Vx      Cy
  T2:  Ry          Lx  Vy  Cx
  T1 commits
  T2 aborts since T2's Vy sees T1's lock or higher version
but we can't have *both* V's before the other L's
so VALIDATE seems correct in this example
  and fast: one-sided VALIDATE read rather than LOCK+COMMIT writes

a purely read-only FaRM transaction uses only one-sided RDMA reads
  no writes, no log records, no locking
  very fast!

what about fault tolerance?
  suppose some computers crash and don't reboot
  most interesting if TC and some primaries crash
  but we assume one backup from each shard survives

the critical issue:
  if a transaction was interrupted by a failure,
    and a client could have been told a transaction committed,
    or a committed value could have been read by another xaction,
  then the transaction must be preserved and completed during recovery.

look at Figure 4.
a committed write might be revealed as soon the
  first COMMIT-PRIMARY is sent (since primary writes and unlocks).
so by then, all of the transaction's writes must be on all
  f+1 replicas of all relevant shards.
the good news: LOCK and COMMIT-BACKUP achieve this.
  LOCK tells all primaries the new value(s).
  COMMIT-BACKUP tells all backups the new value(s).
  TC doesn't send COMMIT-PRIMARY until all LOCKs and COMMIT-BACKUPS complete.
  backups may not have processed COMMIT-BACKUPs, but in NVRAM logs.

similarly, TC doesn't return to client until at least one
 COMMIT-PRIMARY is safe in primary log.
  without the COMMIT-PRIMARY, the risky case is:
    TC replies "yes" to app after COMMIT-BACKUPS (before COMMIT-PRIMARY).
    TC and all backups then fail.
    now the only evidence left is the LOCK records.
    but even a complete set of LOCK records doesn't tell us if TC committed
      maybe TC aborted due to failed VALIDATE!
  writing the COMMIT-PRIMARY handles the risk, because the TC's
     decision will survive f failures of any shard.
  since there's one shard with a full set of COMMIT-BACKUP and COMMIT-PRIMARY.
  any of which is evidence that the primary decided to commit.

FaRM is very impressive; does it fall short of perfection?
  * works best if few conflicts, due to OCC.
  * data must fit in total RAM.
  * replication only within a datacenter (no geographic distribution).
  * the data model is low-level; would need e.g. SQL library.
  * requires somewhat unusual RDMA and NVRAM hardware.

how does FaRM differ from Spanner?
  both shard, replicate, and use two-phase commit (2pc) for transactions
  Spanner:
    focuses on coping with network delay due to geographic replication
    Paxos tolerates delay
    TrueTime lets them read from local replicas
    performance: r/w xaction takes 10 to 100 ms (Tables 3 and 6)
  FaRM
    focuses on reducing CPU costs
    RDMA, direct NIC access, NVRAM to avoid disk writes
    RDMA leads them to Optimistic Concurrency Control (OCC)
    performance: 58 microseconds for simple transactions (6.3, Figure 7)
      i.e. 100 times faster than Spanner

summary
  super high speed distributed transactions
  hardware is exotic (NVRAM and RDMA) but may be common soon
  use of OCC for speed and to allow fast one-sided RDMA reads

6.5840 2026 Lecture 3: GFS

The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
SOSP 2003

Why are we reading this paper?
  GFS paper touches on many themes of 6.5840
    parallel performance, fault tolerance, replication, consistency
  good systems paper -- details from apps all the way to network
  successful real-world design

Important distributed system ideas
  Build fault-tolerant applications using fault-tolerant storage
  Sharding
  Primary-backup replication protocol
  Leases to ensure 1 primary
  Checksums for detecting faulty hardware/software
  Sacrifice consistency for performance/simplicity

GFS context
  Many Google services needed a big fast unified storage system
    Mapreduce, crawler, indexer, log storage/analysis
  Shared among multiple applications e.g. crawl, index, analyze
  Huge capacity
  Huge performance
  Fault tolerant
  But:
    just for internal Google use
    aimed at batch big-data performance, not interactive

GFS overview
  100s/1000s of clients (e.g. MapReduce worker machines)
  100s of chunkservers, each with its own disk
  one coordinator

Capacity story?
  big files split into 64 MB chunks
  each file's chunks striped/sharded over chunkservers
    so a file can be much larger than any one disk
  each chunk in a Linux file

Throughput story?
  clients talk directly to chunkservers to read/write data
  if lots of clients access different chunks, huge parallel throughput
  read or write

Fault tolerance story?
  each 64 MB chunk stored (replicated) on three chunkservers
  client writes are sent to all of a chunk's copies
  a read just needs to consult one copy

What are the steps when client C wants to read a file?
  1. C sends filename and offset to coordinator (CO) (if not cached)
     CO has a filename -> array-of-chunkhandle table
     and a chunkhandle -> list-of-chunkservers table
  2. CO finds chunk handle for that offset
  3. CO replies with chunkhandle + list of chunkservers
  4. C caches handle + chunkserver list
  5. C sends request to nearest chunkserver
     chunk handle, offset
  6. chunk server reads from chunk file on disk, returns to client

Clients only ask coordinator where to find a file's chunks
  clients cache name -> chunkhandle info
  coordinator does not handle data, so (hopefully) not heavily loaded

What about writes?
  Client knows which chunkservers hold replicas that must be updated.
  How should we manage updating of replicas of a chunk?

A bad replication scheme
  (This is *not* what GFS does)
  [diagram: C, S1, S2, S3]
  Client sends update to each replica chunkserver
  Each chunkserver applies the update to its copy

What can go wrong?
  *Two* clients write the same data at the same time
    i.e. "concurrent writes"
    Chunkservers may see the updates in different orders!
    Again, the risk is that, later, two clients may read different content

Idea: primary/secondary replication
  (or primary/backup)
  For each chunk, designate one server as "primary".
  Clients send write requests just to the primary.
    The primary alone manages interactions with secondary servers.
    (Some designs send reads just to primary, some also to secondaries)
  The primary chooses the order for all client writes.
    Tells the secondaries -- with sequence numbers -- so all replicas
    apply writes in the same order, even for concurrent client writes.
  There are still many details to fill in, and we'll
    see a number of variants in upcoming papers.

What are the steps when C wants to write a file at some offset?
  paper's Figure 2
  1. C asks CO about file's chunk @ offset
  2. CO tells C the primary and secondaries
  3. C sends data to all (just temporary...), waits for all replies (?)
  4. C asks P to write
  5. P checks that lease hasn't expired
  6. P writes its own chunk file (a Linux file)
  7. P tells each secondary to write (copy temporary into chunk file)
  8. P waits for all secondaries to reply, or timeout
     secondary can reply "error" e.g. out of disk space
  9. P tells C "ok" or "error"
  10. C retries from start if error

What data may a client read after a failed write?
  Replicas may return different data
  Primary P updated its own state.
  But secondary S1 did not update (failed? slow? network problem?).
  Client C1 reads from P; Client C2 reads from S1.
    they will see different results!
  Such a departure from ideal behavior is an "anomaly".
    in non-replicated systems, this couldn't happen!
    data can be *inconsistent*
  Note: a successful writes doesn't lead to inconsistency

What is the result of concurrent writes?
  Clients break writes spanning chunks into two writes
  C1 writes chunks x and y at offset 0 and 1, respectively
  C2 writes chunks a b at offset 0 and 1
  A possible state of the replicas (without errors): [a y]
    x was overwritten by a
    b was overwritten with y
    x and b are lost!

GFS solution: atomic append of records
  Primary picks the offset and pads if record spans chunks
  Result for the above case: some interleaving of [a, b, x, and y]
  No data lost, but "empty" areas due to padding
  Note: client doesn't determine where the data is in the file
    primary picks the offset
    append returns offset where client's record is written
  Note: requires changing applications to use append instead of write

What data may a client read after a failed append?
  Read from a backup that has the record
  Read from a backup that didn't apply the append: a hole
  Read from a backup that recorded the initial and retried append
    duplicate records

Consistency: guarantees offered by a storage system to applications
  GFS consistency is complex! (Table 1)
  if primary tells client that a write succeeded,
    and no other client is writing the same part of the file,
    all readers will see the write.
    "defined"
  if successful concurrent writes to the same part of a file,
    and they all succeed,
    all readers will see the same content,
    but maybe it will be a mix of the writes.
    "consistent"
    E.g. C1 writes "ab", C2 writes "xy", everyone might see "xb".
  if primary doesn't tell the client that the write succeeded,
    different readers may see different content, or none.
    "inconsistent"

Why are these anomalies OK?
  They only intended to support a certain subset of their own applications.
    Written with knowledge of GFS's behavior.
  Probably mostly single-writer and Record Append.
  Writers could include checksums and record IDs.
    Readers could use them to filter out junk and duplicates.
  Later commentary by Google engineers suggests that it
  might have been better to make GFS more consistent.
  http://queue.acm.org/detail.cfm?id=1594206

What might better consistency look like?
  There are many possible answers.
  Trade-off between easy-to-use for client application programmers,
    and easy-to-implement for storage system designers.
  Maybe try to mimic local disk file behavior.
  Perhaps:
    * atomic writes: either all replicas are updated, or none,
      even if failures.
    * read sees latest write.
    * all readers see the same content (assuming no writes).
  We'll see more precision later.

Let's think about how GFS handles crashes of various entities.

A client crashes while writing?
  Either it got as far as asking primary to write, or not.

A secondary crashes just as the primary asks it to write?
  1. Primary may retry a few times, if secondary revives quickly
     with disk intact, it may execute the primary's request
     and all is well.
  2. Primary gives up, and returns an error to the client.
     Client can retry -- but why would the write work the second time around?
  3. Coordinator notices that a chunkserver is down.
     Periodically pings all chunk servers.
     Removes the failed chunkserver from all chunkhandle lists.
     Perhaps re-replicates, to maintain 3 replicas.
     Tells primary the new secondary list.

Re-replication after a chunkserver failure may take a Long Time.
  Since a chunkserver failure requires re-replication of all its chunks.
  80 GB disk, 10 MB/s network -> an hour or two for full copy.
  So the primary probably re-tries for a while,
    and the coordinator lets the system operate with a missing
    chunk replica,
    before declaring the chunkserver permanently dead.
  How long to wait before re-replicating?
    Too short: wasted copying work if chunkserver comes back to life.
    Too long: more failures might destroy all copies of data.

What if a primary crashes?
  Remove that chunkserver from all chunkhandle lists.
  For each chunk for which it was primary,
    wait for lease to expire,
    grant lease to another chunkserver holding that chunk.

What is a lease?
  Permission to act as primary for a given time (60 seconds).
  Primary promises to stop acting as primary before lease expires.
  Coordinator promises not to change primaries until after expiration.
  Separate lease per actively written chunk.

Why are leases helpful?
  The coordinator must be able to designate a new primary if the present
    primary fails.
  But the coordinator cannot distinguish "primary has failed" from
    "primary is still alive but the network has a problem."
  What if the coordinator designates a new primary while old one is active?
    two active primaries!
    C1 writes to P1, C2 reads from P2, doesn't seen C1's write!
    called "split brain" -- a disaster
  Leases help prevent split brain:
    Coordinator won't designate new primary until the current one is
    guaranteed to have stopped acting as primary.

What if the coordinator crashes?
  Two strategies.
  1. Coordinator writes critical state to its disk.
     If it crashes and reboots with disk intact,
     re-reads state, resumes operations.
  2. Coordinator sends each state update to a "backup coordinator",
     which also records it to disk; backup coordinator can take
     over if main coordinator cannot be restarted.

What information must the coordinator save to disk to recover from crashes?
  Table mapping file name -> array of chunk handles.
  Table mapping chunk handle -> current version #.
  What about the list of chunkservers for each chunk?
    A rebooted coordinator asks all the chunkservers what they store.
  A rebooted coordinator must also wait one lease time before
    designating any new primaries.

* Who/what decides the coordinator is dead, and chooses a replacement?
  Paper does not say.
  Could the coordinator replicas ping the coordinator,
    and automatically take over if no response?

* Suppose the coordinator reboots, and polls chunkservers.
  What if a chunkserver has a chunk, but it wasn't a secondary?
    I.e. the current primary wasn't keeping it up to date?
  Coordinator remembers version number per chunk, on disk.
    Increments each time it designates a new primary for the chunk.
  Chunkserver also remembers its version number per chunk.
  When chunkserver reports to coordinator, coordinator compares
    version number, only accepts if current version.

* What if a client has cached a stale (wrong) primary for a chunk?

* What if the reading client has cached a stale server list for a chunk?

* What if the primary crashes before sending append to all secondaries?
  Could a secondary that *didn't* see the append be chosen as the new primary?
  Is it a problem that the other secondary *did* see the append?

What would it take to have no anomalies -- strict consistency?
  I.e. all clients see the same file content.
  Too hard to give a real answer, but here are some issues.
  * All replicas should complete each write, or none -- "atomic write".
    Perhaps tentative writes until all promise to complete it?
    Don't expose writes until all have agreed to perform them!
  * Primary should detect duplicate client write requests.
  * If primary crashes, some replicas may be missing the last few ops.
    They must sync up.
  * Clients must be prevented from reading from stale ex-secondaries.
  You'll see solutions in Labs 2, 3, and 4!

* Are there circumstances in which GFS will break its guarantees?
  e.g. write succeeds, but subsequent readers don't see the data.
  All coordinator replicas permanently lose state (permanent disk failure).
    Read will fail.
  All chunkservers holding the chunk permanently lose disk content.
    Read will fail.
  CPU, RAM, network, or disk yields an incorrect value.
    checksum catches some cases, but not all
    Read may say "success" but yield the wrong data!
    Above errors were "fail-stop", but this is a "byzantine" failure.
  Time is not properly synchronized, so leases don't work out.
    So multiple primaries, maybe write goes to one, read to the other.
    Again, read may yield "success" but wrong data -- byzantine failure.

Performance (Figure 3)
  large aggregate throughput for read
    94 MB/sec total for 16 clients + 16 chunkservers
      or 6 MB/second per client
      is that good?
      one disk sequential throughput was about 30 MB/s
      one NIC was about 10 MB/s
    Close to saturating inter-switch link's 125 MB/sec (1 Gbit/sec)
    So: multi-client scalability is good
    Table 3 reports 500 MB/sec for cluster A, which was a lot
  writes to different files lower than possible maximum
    authors blame their network stack (but no detail)
  concurrent appends to single file
    limited by the server that stores last chunk
  hard to interpret after 15 years, e.g. how fast were the disks?

Retrospective interview with GFS engineer:
  http://queue.acm.org/detail.cfm?id=1594206
  file count was the biggest problem
    eventual numbers grew to 1000x those in Table 2 !
    hard to fit in coordinator RAM
    coordinator scanning of all files/chunks for GC is slow
  1000s of clients -> too much CPU load on coordinator
  coordinator fail-over initially manual, 10s of minutes, too long.
  applications had to be designed to cope with GFS semantics
    and limitations.
    more painful than expected.
  BigTable is one answer to many-small-files problem
  and Colossus apparently shards coordinator data over many coordinators

Summary
  case study of performance, fault-tolerance, consistency
    specialized for MapReduce applications
  good ideas:
    global cluster file system as universal infrastructure
    separation of naming (coordinator) from storage (chunkserver)
    sharding for parallel throughput
    huge files/chunks to reduce overheads
    primary to choose order for concurrent writes
    leases to prevent split-brain
  not so great:
    single coordinator performance
      ran out of RAM and CPU
    chunkservers not very efficient for small files
    lack of automatic fail-over to coordinator replica
    maybe consistency was too relaxed

---

http://queue.acm.org/detail.cfm?id=1594206
https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system

6.5840 2026 Lecture 15:  Verification: IronFleet

IronFleet: Proving Practical Distributed Systems Correct (2015)

Claim: my raft labs are bug-free because
(1) I passed all the test cases
(2) Raft thesis has proofs
Q: Convincing?
A: No.
   Tests are incomplete, proof could be wrong, proof is not about code.  It
   would be nice if there was another way. Today's lecture will be about another
   way to achieving correctness.

IronFleet
  Formal verification for distributed systems
  Think about all the papers, all the "what-if"s. This is both an answer to "how
  to reason about those systems", and also might help more deeply understand why
  distributed systems are tricky to reason about.
  By the end of studying this paper, the goal is to know the fundamental
  distributed systems verification techniques.

Basic mechanized verification
  Picture:
  implementation
  specification: definition of what system should provide (trusted)
  proof: steps for showing that spec is satisfied (untrusted)
  proof checker: tool that reads spec, proof, and impl, and outputs "yes" or
  "no" (trusted)
  TCB (trusted computing base) includes spec and verifier

IronFleet methodology.

Hoare logic:
  demo
  precondition, postcondition, SMT solver (e.g. Z3), assertions, loop invariants
    sort function
    - spec (pre+post condition)
    - invariant
    - introduce a bug in code -> dafny verification fails
    Dafny can compile the verified code.
  Lesson: specification is trusted.
  (Further topics: weakest preconditions, separation logic)
  What about distributed systems though?

State machines
  Specify overall distributed systems behavior
    State: per-machine state, network state
    Init ⊆ State
    Next ⊆ State × State
  Picture:
  Spec state machine for lock
  Implementation state machine for lock
    Concurrency -> interleaving transitions
    Transfer packets
  State machines are just a tool---can be at whatever level

IronFleet-style refinement
  Low-level behaviors are subset of high-level behaviors; only care about
  "externally visible" events.
  Picture: lockservice refinement
  Could use stutters (v.s. IronFleet's more complex picture)
  Refinement is transitive

Invariants:
  Property that holds throughout execution
  Can capture important guarantees, or serve as an intermediate proof step
  e.g. only one leader per term in Raft implementation
  True after all "atomic" transitions
  Helps prove refinement


IronFleet methodology:
  Figure 3.
  Complex implementation state machine
  Slightly simpler protocol state machine
    mathematical integers, unbounded sequences, immutable types, sending
    high-level messages (rather than bytes)
  Trusted, simpler spec state machine
  Q: How to connect pre+postconditions of methods to state machine
  transitions?
  Q: how to prove refinement (for lock, need a refinement mapping)?

How to apply IronFleet methodology to Raft?
  Figure 3.
  Top-level spec: State is log. Init: empty. Transitions: Commit(op).
  Prelecture question: where would the leader election restriction go wrong?
  A: Could go wrong at any step: if the protocol state machine is incorrectly
  written, could prove refinement impl -> protocol, but get stuck at protocol ->
  spec. However, if protocol is correctly written, then impl -> protocol would
  get stuck (while protocol -> spec might be provable).

Induction:
  Fundamental rule in program verification
  Loop invariants use induction
  Inductive invariants for distributed systems
  Conceptual picture.
  E.g.
    (exists a valid Transfer packet for e and no node holds lock
     \/ all transfer packets are stale and one node holds lock)

Proof engineering:
  Figure 12, Proof effort.
  Automation; lots of different decision procedures, tradeoffs
  Triggers
  Libraries
  Opaqueness (e.g. temporal logic formulas)

Liveness v.s. safety
  Temporal Logic of Actions
  (Network is eventually good) -> every node will hold the lock infinitely often
  Must show:
  - for any infinite execution (subject to assumptions), there is a "good" state
  Assumptions are "fairness" assumptions
  Reasoning principles like "WF1"

10min Q&A

Limitations:
  No local concurrency => round robin scheduler
  No crash recovery
  No (verified) client libraries
  Not clear how to use a verified IronFleet system as part of a bigger verified
  system.

--------------------------------------------------------------------------------
sort.dfy example

predicate sorted(s: seq<int>) {
  forall i, j :: 0 <= i < j < |s| ==> s[i] <= s[j]
}

predicate pivoted(s: seq<int>, p: int) {
  forall i, j :: 0 <= i <= p < j < |s| ==> s[i] <= s[j]
}

method insertion_sort(a: array<int>)
  modifies a
  ensures sorted(a[..])
  ensures multiset(a[..]) == multiset(old(a[..]))
{
  var i := 1;
  while i < a.Length
    invariant 1 <= i
    invariant sorted(a[..(if i < a.Length then i else a.Length)])
    invariant multiset(a[..]) == multiset(old(a[..]))
  {
    var j := i;
    while j > 0 && a[j-1] > a[j]
      invariant 0 <= j <= i
      invariant multiset(a[..]) == multiset(old(a[..]))
      invariant sorted(a[..j])
      invariant sorted(a[j..i+1])
      invariant pivoted(a[..i+1], j)
    {
      a[j-1], a[j] := a[j], a[j-1];
      j := j - 1;
    }
    i := i + 1;
  }
}

6.5840 2026 Lecture 8: Consistency, Linearizability

today's topic: consistency models, specifically linearizability

consistency model:
  a spec for the relationship of different clients' views of a service
  I'll focus on key/value storage with network clients
    given a bunch of put/get calls, what outcome(s) are valid?
  assuming individual operations, in isolation, are correct

in ordinary sequential programming, there's nothing to talk about:
  we expect a read to yield the last written value

when might there be any question about what's correct?
  [simple client/server diagrams]
  read simultaneous with write
  replicas
  caches
  failure, recovery
  lost messages + retransmission

why does a storage system need an explicit consistency model?
  for applications, hard to be correct w/o guarantees from storage
    e.g. producer computes, then executes
      put("result", 27)
      put("done", true)
    consumer executes
      while get("done") == false:
        pause
      v = get("result")
    is v guaranteed to be 27?
  for services, hard to design/optimize w/o a specification
    e.g. OK for clients to read from GFS replicas? from Raft followers?

there are lots of consistency models
  today: linearizability
  but we'll also see:
    eventual consistency
    causal consistency
    fork consistency
    serializability
  driving force: tradeoffs among
    performance
    simplicity
    fault tolerance

linearizability
  it's a specification -- a requirement for how a service must behave
    from clients' point of view: from outside the service
  it's usually what people mean by "strong consistency".
    few "anomalies" / surprises
    a gold standard
    but rules out many optimizations.
  important for us:
    Raft is intended for building fault-tolerant linearizable services
    Lab 4 will be a linearizable key/value store on Raft

starting point
  we assume that there's a serial spec for what individual operations do
  serial = a single server executing operations one at a time
  db[]
  put(k, v):
    db[k] = v
    return true
  get(k):
    return db[k]
  no surprises here.

what about concurrent client operations?
  "concurrent" = overlap in time
  a client sends an RPC request;
    takes some time crossing the network;
    server computes, talks to replicas, &c;
    reply moves through network;
    client receives reply
  other clients may send/receive/wait during that time!
  the serial spec doesn't tell us what results are legal
  we need a way to describe concurrent scenarios,
    so we can talk about which results are/aren't allowed

definition: a history
  describes a time-line of possibly-concurrent operations
  each operation tagged w/ client start and finish times
    time client sent RPC request; time received reply
    as well as argument and return values
  example:
    C1: |-Wx1-| |-Wx2-|
    C2:   |---Rx2---|
  the x-axis is real time
    |- indicates the time at which client sent request
    -| indicates the time at which the client received the reply
  "Wx1" means put(x, 1)
  "Rx2" means get(x) -> 2
  records assumed to start with some undefined value
  C1 sent put(x, 1), recv reply, sent put(x, 2), recv reply
    writes have responses, signifying completion
  C2 sent get(x), recv reply=2

a history is a trace of what clients saw (or might see)
  used to check whether the execution was linearizable
  used by designers in "would this be OK" thought experiments

definition: a history is linearizable if
  * you can find a point in time for each operation
    between its start and finish, and
  * the history's result values are the same as serial
    execution in point order.

example history 1:
  |--Wx1--| |--Wx2--|
     |----Rx2----|
       |--Rx1--|

is this history linearizable?
  can we find a linearization point for each operation?
  we may need to try a few different point assignments.
  this order of points satisfies the rules:
    Wx1 Rx1 Wx2 Rx2
  1. each point lies between start and finish.
  2. the sequence satisfies the serial put/get spec.
  so: yes, this history is linearizable
    (and it's an answer to The Question)

note: either read could have returned either 1 or 2.
  so linearizability often allows multiple different outcomes.
  so we often can't predict in advance, but we can check afterwards.

note: the service may not have executed the operations at those points!
  we're not concerned here with how the service operated internally
  we only care that the client-visible results could have
    resulted from execution in some point order

what can we do with the linearizability definition?
  for designer: could this optimization result in non-linearizable results?
  for programmer: what can I assume / expect as a client?
  for testing: generate requests, check observed history.

why is it called "linearizability"?
  the linearization points turn concurrent operations
    into a serial execution -- "linear".
  thus "linearizable" in the sense that the results are the same
    as some linear execution of the operations.

example 2:
  |-Wx1-| |----Wx2----|
                  |-Rx1-|
  linearizable?
    if yes, linearization points?
  so: concurrent read/write can go either way

example 3: like example 2, but...
  |-Wx1-| |----Wx2----|
    |---Rx2---|
                |-Rx1-|
we can try a few assignments of linearization points.
  how about Wx1 Wx2 Rx2 Rx1?
  not valid because "Wx2 Rx1" doesn't conform to serial spec.
how to show something *isn't* linearizable?
  show that no assignment of points works.
  i.e. breaks either time rule or value rule.
no assignment works for example 2!
  Wx2's point must be before Rx2's point
  so Wx2's point is also before Rx1's point
  so the second read got an impossible value
thus, if a system can produce this history, we know the
  system isn't linearizable: has a bug, or never promised linearizability.
the Rx1 *would* have been legal if there had been no Rx2
  reads -- not just writes -- can affect what's subsequently legal
so, if we want linearizability:
  once any read sees a write, all strictly-subsequent reads must also see it.
    rules out split-brain
  can't forget a revealed write
    rules out e.g. forgetting data due to a crash

GFS is not linearizable: it can produce example 3 since
  the Rx1 could come from a replica that hasn't yet been updated.
  if we wanted GFS to be linearizable,
    one approach is to have client reads go through the primary too.
    would be slower!

example 4:
|--Wx0--|  |--Wx1--|
            |--Wx2--|
C1:      |-Rx2-| |-Rx1-|
C2:      |-Rx1-| |-Rx2-|
can there be a serial order?
  C1 needs Wx2 Rx2 Wx1 Rx1
  C2 needs Wx1 Rx1 Wx2 Rx2
  we can't have both Wx2 before Wx1, and Wx2 after Wx1.
  so not linearizable.
so:
  service can choose either order for concurrent writes
  but all clients must see the writes in the same order
  this is important when there are replicas or caches
    they must all appear to execute operations in the same order

example 5:
|-Wx1-|
        |-Wx2-|
                |-Rx1-|
no order is possible -- not linearizable
so:
  linearizability rules out "stale" reads
  even if the reader doesn't know there was a write
    the time rule requires reads to see the latest completed update
  "no stale data" restricts caching and replication designs

linearizability outlaws many design possibilities / anomalies:
  split brain (two active leaders)
  forgetting completed writes after a crash+reboot
  reading from lagging replicas or out-of-date caches

example 6:
[client / network / server diagram]
C1 sends put(x, 1)
C2 sends put(x, 2)
service receives C1's request;
  network drops response;
  C1's RPC library re-sends request
is it legal for service to execute *both* of C1's request messages?
we then might see this if C3 reads three times:
C1: |--------Wx1---------| (due to retransmission)
C2:        |-Wx2-|
C3:   |-Rx1-| |-Rx2-|  |-Rx1-|
this history is not linearizable!
so, if we want linearizability:
  repeated requests from retransmissions should only execute once!
  the Raft paper mentions this issue (and a solution) in Section 8

linearizable systems are not limited to just read and write operations
  increment
  append
  test-and-set (to implement locks)
  any self-contained single-object operation

application programmers like linearizability:
  * reads see latest data -- never stale
  * all clients see the same data (when there aren't writes)
  * all clients see data changes in the same order
    so my put(v,27); put(done,true); example works
  these benefits will be clearer when we look at weaker consistencies.

can we have better semantics than linearizability?
  or, what might you want that linearizability won't support?
  * transactions: atomic *groups* of operations on different objects
    linearizability only thinks about individual operations, not groups of ops
  * operations that themselves invoke other services

how can we implement linearizability?
  depends on how much replication, caching, and fault-tolerance we want.

single serial server that doesn't crash.
  [diagram: clients, server, op queue, state]
  server picks an order for concurrently arriving client requests.
  executes them in that order, one at a time,
    replies to each before starting the next.
  plus duplicate request suppression

this simple implementation directly implements linearizability definition
  executes in serial order, one at a time
  executes an operation after it is received
    and doesn't reply until after operation is finished
  (thus can't eagerly reply to put()s)

good news: server does not have to reason about histories,
  linearization points, or concurrency.

what about fault-tolerant linearizability?
  primary-backup replication
    like GFS but would need modifications e.g. read only via primary
  Raft!
  and we'll see more examples

what about the performance of linearizable systems?

a fundamental performance limit imposed by no-stale-reads rule:
  [W, E]
  a write on the west coast, which completes
  then a read on the east coast
  there *must* be communication of the written value
    *and* either reader or writer must wait for communication
  speed of light is a serious problem:
    dozens of milliseconds cross-country
    can limit serial throughput to a few tens per second

generally:
  bad news: serial aspect makes it hard to get parallel speedup for individ objs
  bad news: if replication, then lots of communication and waiting
  bad news: if replication, replicas must be reachable,
            limiting fault tolerance
  good news: you can shard by key

what about other consistency models?
  can they allow better performance?
  do they have intuitive semantics?

example: eventual consistency -- a weak model
  [diagram: two copies, clients]
  multiple copies of the data (e.g. in different datacenters, for speed)
  a read consults any replica (e.g. closest)
  a write updates any replica (e.g. closest)
    replica sends response when that one update is done
  replicas synchronize updates in the background
    eventually, other replicas will see my update

many systems provide eventual consistency
  faster than linearizability
    especially if replicas are in different cities for fault-tolerance
  and more available -- any one replica will do
    no waiting for primary/backup communication or Raft quorum
  Amazon's Dynamo; Cassandra; GFS (sort of)

but eventual consistency exposes some anomalies to application programmer:
  * reads can see stale data
    a problem for password change, ACL change
  * writes may appear out of order
    breaks my result/done example
  * different clients may see different data
  * concurrent writes to same item need to be resolved somehow!
    C1: put(x, 1)
    C2: put(x, 2)
    may initially be applied at different replicas
    only later will they be pushed to other replicas
    how to merge concurrent new values?
    how to ensure all replicas choose the same final value?
      so that, eventually, they are identical?
  * eventual consistency cannot support e.g. test-and-set

A general pattern: you can usually choose only one of these:
  Strong consistency
  Maximum availability

But not both.
  Strong consistency makes you wait to update replicas,
    and can't proceed if too many replicas are unavailable.
    Thus poor availability.
  Eventual consistency can proceed even if no other replicas are reachable.
    But has poor consistency.

Next week:
  ZooKeeper, an interesting service built on a Raft-like protocol
    combines strong and weak consistency

---

https://jepsen.io/consistency/models

6.5840 2026 Lecture 16: Scaling Memcache at Facebook

Scaling Memcache at Facebook, by Nishtala et al, NSDI 2013

why are we reading this paper?
  it's an experience paper
  how did the authors scale up a big system?
  problems? solutions?
  a window into the real world
  performance vs consistency vs practicality

the big facebook infrastructure picture
  lots of data: friend lists, status, posts, likes, photos
  fresh/consistent data not critical -- humans are tolerant
  read-heavy (helpful)
  little locality (not helpful)
  high load: billions of storage operations per second
    much higher than a single storage server can handle
      ~100,000 simple queries/s for mysql
      ~1,000,000 get/puts/s  for memcached
  multiple data centers (at least west and east coast)
  [diagram]
  each data center -- "region":
    "real" data sharded over MySQL DBs -- ACID, but slow
    memcached layer (mc) -- in RAM: fast but limited size
    web servers (clients of memcached and DB) -- "stateless"
  each data center's DBs contain full replica
  west coast is primary, others are replicas via MySQL async log replication

let's talk about performance first
  much of paper is about avoiding stale cached data
  but staleness arose from efforts to increase performance

what is memcached?
  a simple key/value server: put(k,v), get(k), delete(k).
  in RAM: fast, not durable, no replication.
  LRU eviction, since RAM size limited
  stores what clients tell it to.
  usually lots of memcached servers are deployed.
    clients decide what to store where.

how do FB apps use mc? Figure 1.
  FB uses mc as a "look-aside" cache
    real data is in the DB
    application talks separately to mc and (if miss or write) to DB
    mc doesn't know about the DB
  read(k):
    h = hash(k) % n  -- hash chooses which memcache server to talk to
    v = mc[h].get(k)
    if v is nil:
      v = fetch from DB
      put(k, v)
  write(k,v):
    send k,v to DB
    h = hash(k) % n
    mc[h].delete(k)

memcached very popular!
  look-aside makes it easy to add to existing web applications
  cache anything from DB rows to entire computed html pages
  flexible, simple, fast

what is the benefit of using mc?
  it's only helpful for reads -- but that's by far the majority of operations
  high hit rate -> reduces load on DB servers
    Table 2 says about 99% hit rate, i.e. 100x reduction in DB read load
    wow; but watch out: a 1% decrease in hit rate *doubles* DB load
  this caching is not about reducing user-visible delay,
    it's about protecting the DB servers from massive overload.

lots of mc servers are needed to handle the total load
  CPU/network parallelism
  total RAM
  how to divide the load among the memcache servers?

the client hash function determines how keys are assigned to mc servers
  can shard (partition), or replicate, or some combination
  all web servers use the same hash(k) function
    so if C1 caches key k, C2 will see it!
  central configuration manager tells clients how to hash

will sharding or replication yield most mc throughput?
  this is a central concern in many designs
  [two little diagrams]
  sharding: divide keys over mc servers
  replicate: divide clients over mc servers
  sharding:
    + memory-efficient (only one copy of each k/v pair)
    - not effective if a few keys are extremely popular
    - each web server must talk to many mc servers (high packet overhead)
    - risk of "in-cast congestion"
  replication:
    + useful if a few keys are very popular for reads
    + can pack many requests/responses per packet (low overhead)
    + few servers need be contacted, reducing per-packet costs
    + easier to design supporting network (2 x-capacity nets easier than 1 2x)
    - uses more memory, so fewer distinct items can be cached
    - writes are more expensive

performance and multiple regions (Section 5)

[diagram: west, db primary shards, mc servers, clients |
 east, db secondary shards, ...,
 feed from db primaries to secondaries ]

Q: what is the point of regions -- multiple complete replicas?
   lower RTT to users (east coast, west coast)
   quick local reads, from local mc and DB
   (though writes are expensive: must be sent to primary region)
   hot replica in case primary site fails

Q: why not divide users over regions?
   i.e. why not east-coast users' data in east-coast region, &c
   then no need to replicate: might cut hardware costs in half!
   but: social net -> not much locality
   might work well for e.g. e-mail

Q: why OK performance despite writes sent to the primary region?
   writes are much rarer than reads
   users do not wait for writes to finish

performance within a region (Section 4)

[diagram: db shards, multiple clusters, each w/ mc's and clients ]

multiple mc clusters *within* each region
  cluster = complete set of mc cache servers + web servers
  each web server hashes keys over just the mc servers in its cluster

why multiple clusters per region?
  why not a single big cluster in each region?
    divide the load among many parallel mc servers?
  1. more mc servers don't help very popular keys
     replicating (one copy per cluster) does help
  2. more mcs in cluster -> sharded more finely ->
       each web view sends more packets (to more mc servers)
     and more in-cast congestion from replies
     client requests fetch 20 to 500 keys! over many mc servers
     MUST request in parallel (total latency too large if serial)
     but then all replies come back at the same time
     network switches, NIC run out of buffers
  3. hard to build network for single big cluster
     any-to-any client/server access
     so cross-section b/w must be large -- expensive
     two clusters -> 1/2 the cross-section b/w

but -- replicating is a waste of RAM for less-popular items
  "regional pool" shared by all clusters
  unpopular objects (no need for many copies)
  the application s/w decides what keys to put in regional pool
  frees mc servers to replicate more popular objects

bringing up new mc cluster is a performance problem
  new cluster has 0% hit rate
  so its clients could generate big spike in DB load
  thus the clients of new cluster first get() from existing cluster (4.3)
    and put() into new cluster
    basically lazy copy of existing cluster to new cluster

another overload problem: thundering herd
  one client updates DB and delete()s a key
  lots of clients get() but miss
    they all fetch the same data from DB
  not good: needless DB load
  solution: mc gives just the first missing client a "lease"
    lease = permission to refresh from DB
    mc remembers set of valid leases
    mc tells others "try get() again in a few milliseconds"
  effect: only one client reads the DB and does put()
    others re-try get() later and hopefully hit

what if an mc server fails (Section 3.3)?
  can't have DB servers handle the misses -- too much load
  can't shift load to another mc server -- too much load
  Gutter -- pool of idle mc servers, clients only use after mc server fails
    separate Gutter per cluster
  after a while, failed mc server will be replaced
  as long as only a few mc servers are down at any one time,
    a small Gutter pool can act as backups for a large set of mc servers

The Question:
  why aren't invalidates (deletes) sent to Gutter servers?
    from web servers and MySQL/McSqueal
  my guess:
    Gutter can hold *any* key
    so all invalidates would have to be sent to Gutter
    this at least doubles delete traffic
    and may place a heavy load on small # of Gutter servers

let's talk about consistency now

what is the paper's consistency plan?
  writes go direct to primary DB, with transactions, so DB stays consistent
    e.g. incrementing a "like" count will be correct
  what about reads?
    reads not guaranteed to see the latest write
    different clients not guaranteed to see the same values
    but not too stale! only a few seconds
    i.e. eventual consistency
  *and* "read-your-own-writes"

this is a common pattern:
  updates are ACID -- and slow
  reads are not very consistent -- but fast

why is it OK that reads can yield stale data?
  the data is news feed items, postings, likes, &c
  users may see web pages with content that lags the DB a little
  few people will notice or care as long as it's only a little
  next time they look, mc will likely have caught up to the DB

what does the paper mean by "consistency"?
  they mean how out-of-date a read might be
  "more consistent" means reads don't lag recent writes by too much
  it's a given that reads can be stale; just trying to limit how stale
  this is a user-experience view of consistency
    it is not about correctness / guaranteed properties

how are DB replicas kept in sync across regions?
  one region is primary
  all clients send updates only to primary region's DB servers
  primary DBs distribute log of updates to DBs in secondary regions
  secondary DBs apply
  secondary DBs are complete replicas (not caches)
  DB replication delay can be considerable (many seconds)

Q: why do clients send updates only to primary region's DB servers?
   why not to local region DB server?

what do they do about now-stale cached data when DB is written?
  there can be many cached copies of an item in a given region:
    one per cluster
  1. DBs send invalidates (delete()s) to relevant mc servers in region
     this is McSqueal in Figure 6
  2. writing client also invalidates mc in local cluster
     for read-your-own-writes
  secondary DBs hear updates, send out invalidates

they ran into a number of DB-vs-mc consistency problems
  due to concurrent updates affecting different cached copies in different orders
  dangerous if can lead to permanently stale cached data

example race (Section 2, Figure 1):
  suppose client write(k,v) looked like (this is broken):
    send k,v to DB
    put(k,v) in mc -- rather than delete(k)
  what if two clients write the same key at the same time?
    updates might arrive at DB in one order
    but at mc server in the other order!
    leading to perhaps-permanent cached incorrect data
  solution: they delete(k), not update; delete is correct in either order

example race (Section 3.2.1):
  k not in cache
  C1 get(k), misses
  C1 v1 = read k from DB
    C2 writes k = v2 in DB
    C2 delete(k)
  C1 put(k, v1)
  now mc has stale data, delete(k) has already happened
  will stay stale indefinitely, until k is next written
  solved with leases:
    mc gives C1 a lease on k with the "miss" -- permission to write k.
    C2's delete(k) invalidates C1's lease.
    so mc ignores C1's put(k).
    key still missing, so next reader will refresh it from DB

Q: aren't the consistency problems caused by clients copying DB data to mc?
   why not have only DB install values in mc, and never clients?
     then there would be no racing client updates &c, just ordered writes

A: that's correct in principle, but:
  1. DB doesn't generally know how to compute values for mc
     generally client app code computes cached items from DB results,
       i.e. mc content is often not simply a literal DB record
  2. DB doesn't know what's cached, would end up sending lots
     of values for keys that aren't cached

FB/mc lessons for storage system designers?
  cache is vital for surviving high load, not just to reduce latency
  need flexible tools for controlling partition vs replication
  linearizability is too much; eventual often not enough

next tuesday:
  guest lecture from an AWS designer!

--- references

http://cs.cmu.edu/~beckmann/publications/papers/2020.osdi.cachelib.pdf
https://engineering.fb.com/2008/08/20/core-data/scaling-out/
https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf

6.5840 2026 Lecture 4: Fault-Tolerant Agreement, Paxos

From Paxos Made Simple, by Leslie Lamport, 2001

remember GFS coordinator
  what if the coordinator fails?
  GFS paper did not have much to say about what happens
  many services built on GFS, so ought to be automatic and reliable
  what to do?

good overall goals:
  high availability via replication
    continue even if one server has crashed or can't be contacted
    "no single point of failure"
  strongly consistent
    look as if a single server

why designing a fault-tolerant coordinator is hard
  suppose we want to use primary/backup replication
    we want clients to switch to backup if primary fails
  a broken scheme:
  [C1, C2, net, S1, S2]
  clients ordinarily send all ops to S1
    S1 replies, and also forwards to S2 (for replication)
    S1 is primary, S2 is backup
  if client gets no response from S1, re-send to S2
    so the system can tolerate the failure of S1
  problem:
    suppose network failure:
      C1 and S1 can communicate;
      C2 and S2 can communicate;
      but C1/S1 can't talk to C2/S2
    on both sides, it seems like the other failed
    so both S1 and S2 will independently act as primary
    "network partition"
    "split brain"
  computers can't distinguish "server crashed" from "network broken"
    all they can observe is "no response to my request"
  this comes up again and again in fault-tolerant designs!

for a while automated fault-tolerant fail-over seemed impossible
  due to possibility of partition
  special external agent (a human) was used to switch primaries
  but that's a single point of failure
  not very scalable for big systems

around 1989 a few solutions appeared -- surprise!
  Paxos is the simplest, easiest to understand
  inspired many widely used schemes
    we'll see Paxos in multiple papers later in the course
  incorporates ideas worth knowing

Paxos provides fault-tolerant "agreement", "consensus"
  to allow a set of computers to agree on a single value
  for example, who the current primary is
  if some Paxos participants agree on X,
    no participant will think some other value was agreed
  despite computer or network failures
  no split brain even if partition

Paxos performs just one agreement
  Real systems need a sequence of Paxos agreements
    e.g. a new agreement to recover after each coordinator failure

a key idea: quorums
  [diagram: three servers]
  an odd number of servers, e.g. 3
  responses from a majority are required to do anything -- 2 out of 3
  if cannot contact a majority, cannot make progress
  (quorum idea is much older than Paxos)

a useful property of majorities is that any two intersect
  if some information is known by a majority,
    any later majority will contain at least one server that knows
  so each step in a quorum system typically looks like
    send message to all participants
    gather a majority of responses (but don't wait for more)
      look at responses to find latest state
    send updated state message to all participants
  nice:
    tolerates minority of failed or slow servers
    overlap ensures state changes aren't lost

another property of majorities:
  at most one network partition can contain a majority of servers
  servers in the minority partitions won't get a quorum, won't proceed
  so at most one partition will perform operations
    helps avoid split brain

note: majority is out of all servers, not just out of live ones

we say 2f+1 servers can tolerate f failed servers
  since the remaining f+1 is a majority of 2f+1
  3 servers can tolerate 1 failed server
  5 servers can tolerate 2 failed
  so you can get more availability (at some expense!)

What properties does Paxos guarantee?
  correctness:
    if agreement reached, all agreeing servers agree on same value
    once any agreement reached, never changes its mind
  fault-tolerance:
    can proceed if f+1 of 2f+1 (a majority) can communicate
      i.e. can tolerate <= f failed servers
    no majority -> cannot proceed
      but will resume correctly once f+1 are again available
      e.g. can survive a site-wide power failure
  liveness:
    will reach agreement when a majority can communicate for long enough
    (this is a weak property)

let's look at the Paxos algorithm
  this is a pseudo-code version of the paper

to cope with failures during agreement protocol,
  Paxos may go through multiple numbered rounds,
    each driven by a proposer.
  any server can propose if it thinks agreement hasn't yet been reached.
  two message exchanges per round:
    prepare, prepare_ok
    accept, accept_ok

definition: S accepts n/v
  S responded accept_ok to accept(n, v)

definition: n/v is chosen
  a majority accepted n/v

the crucial property:
  if v is chosen, any subsequent choice will also == v
    protocol will not change its mind
    maybe a different proposer &c, but same value!
  tricky b/c "chosen" is system-wide property
    e.g. what if majority accepts, then proposer crashes?
    no server can tell locally that a value was chosen

----------------

proposer(v):
  choose n, unique and higher than any n seen so far
  send prepare(n) to all servers including self
  if prepare_ok(n, n_a, v_a) from majority:
    v' = v_a with highest n_a; choose own v otherwise
    send accept(n, v') to all
    if accept_ok(n) from majority:
      send decided(v') to all

acceptor state:
  must persist across reboots
  n_p (highest prepare seen)
  n_a, v_a (highest accept seen)

acceptor's prepare(n) handler:
  if n > n_p
    n_p = n
    reply prepare_ok(n, n_a, v_a)
  else
    reject

acceptor's accept(n, v) handler:
  if n >= n_p
    n_p = n
    n_a = n
    v_a = v
    reply accept_ok(n)
  else
    reject

-----------------

example 1 (normal operation):
  S1, S2, S3
  but S3 is dead or slow
  S1 starts proposal, n=5 v=A
S1: p5    a5A    dA
S2: p5    a5A    dA
S3: dead...
"p5" means Sx receives prepare(n=5)
"a5A" means Sx receives accept(n=5, v=A)
"dA" means Sx receives decided(v=A)
these diagrams are not specific about who the proposer is
  we only care about what acceptors saw and replied

Proposer only needs to wait for a majority
  so it can continue even though S3 was down

What would happen if S3 was alive, but network partitioned?
  and S3 wanted to propose value B?
  S3's prepare would not assemble a majority

the homework question:
  How does Paxos ensure that the following sequence of events can't
  happen? What actually happens, and which value is ultimately chosen?
  S1 wants to propose v=X, crashes after sending two accepts
  S2 wants v=Y
  S1: p1 a1X
  S2: p1     p2 a2?
  S3: p1 a1X p2 a2?
  S3's prepare_ok to prepare(2) would have included "X"
    thus a2X
    good: X had been chosen in the first round; chosen again in 2nd
  the point:
    if the system has already reached agreement, majority will know value.
    any new majority of prepares will intersect that majority.
    so subsequent proposer will learn of already-agreed-on value,
    and send it in accept msgs

example 2 (concurrent proposers):
S1 starts proposing n=10
S1 sends first accept v=X; meanwhile
S3 starts proposing n=11
  but S1 does not receive its proposal
  S3 only has to wait for a majority of prepare responses
S1: p10 a10X
S2: p10        p11
S3: p10        p11  a11Y
S1 is still sending out accept messages...
has a value been chosen?
what will happen?
  what will S2 do if it gets a10X accept msg from S1?
  what will S1 do if it gets a11Y accept msg from S3?
what if S3 were to crash after just one a11Y (and not restart)?

important design pattern:
  if there's evidence that agreement *might* have been reached already,
    must act as if it had been.
  proposer may not *know*,
    since its majority may overlap by just one with previous.
  nevertheless, must be conservative.

how about this:
S1: p10  a10X               p12
S2: p10          p11  a11Y
S3: p10          p11        p12   a12X
has the system agreed to a value at this point?
  after all, a majority have accepted value "X"

what's the "commit point"?
  i.e. exactly when has agreement been reached?
  i.e. at what point can a server safely act on the agreement?
  i.e. at what point might *some other* server have already acted?
  after a majority has the same v_a? no -- why not?  above counterexample
  after a majority has the same v_a/n_a? yes

why does the proposer need to pick v_a with highest n_a?
S1: p10  a10A               p12
S2: p10          p11  a11B
S3: p10          p11  a11B  p12   a12?
n=11 already agreed on vB
n=12 sees both vA and vB, but must choose vB
why this makes sense: two cases:
  1. a value had been chosen before n=11
     n=11's prepares would have seen value and re-used it
     so it's safe for n=12 to re-use n=11's value
  2. no value had been chosen before n=11
     n=11 might have obtained a majority
     so it's required for n=12 to re-use n=11's value

why does accept handler check n >= n_p?
  w/o n >= n_p check, you could get this bad scenario:
  S1: p1 p2 a1A
  S2: p1 p2 a1A a2B
  S3: p1 p2     a2B
  oops, for a while A was chosen, then changed to B!

why does accept handler update n_p = n?
  required to prevent earlier n's from being accepted
  server can receive accept(n,v) even though it never saw prepare(n)
  without n_p = n, can get this bad scenario:
  S1: p1    a2B a1A p3 a3A
  S2: p1 p2         p3 a3A
  S3:    p2 a2B
  oops, for a while B was chosen, then changed to A!

what if proposer S2 chooses n < S1's n?
  e.g. S2 didn't see any of S1's messages
  S2 won't make progress, so no correctness problem

what if an acceptor crashes+reboots after sending accept_ok?
S1: p1  a1X
S2: p1  a1X reboot  p2  a2?
S3: p1              p2  a2?
we know X was chosen; but will proposal #2 preserve X?
the story:
  S2 is the only intersection between p1's and p2's majorities
  thus the only evidence that Paxos already chose X
  so S2 *must* return X in prepare_ok(n=2)
  so S2 must be able to recover its pre-crash n_a/v_a (and n_p)
thus: if S2 wants to re-join this Paxos instance after crash,
  it must remember its n_p/v_a/n_a on disk.
  if lost, do not re-join!

what if an acceptor reboots after sending prepare_ok?
  does it have to remember n_p on disk?
  if n_p not remembered, this could happen:
  S1: p10            a10X
  S2: p10 p11 reboot a10X a11Y
  S3:     p11             a11Y
  11's did not receive a prepare_ok w/ X, so 11 proposed its own value Y
  but just before that, X had been chosen!
  b/c S2 did not remember to ignore a10X
  so: persisting n_p prevents this

can Paxos get stuck?
  yes, if there is not a majority that can communicate
  once net/computers repaired, Paxos will continue

Performance?
  viewed as slow due to multiple communication rounds
    but lots of schemes for optimizing/batching
  viewed as expensive: need 3 replicas (not 2)
  but hard to avoid in serious fault-tolerant systems

-----------------

Many systems use Paxos internally, in a couple of styles.
  To agree on a new coordinator when the old one doesn't respond.
  To agree on when the backup should take over from the primary.
  To agree on the sequence of operations in an RSM.

How to build a database RSM using Paxos
  [diagram: clients, 3 replicas, paxos layer, log, DB layer]
  clients can send operations to any replica
  servers use Paxos to *agree* on each next op to append to the log
  servers apply agreed+logged operations to state (e.g. k/v DB)
    in log order

example:
  client sends "x=1" to S1
  S1 picks log entry 3
  S1 uses Paxos to get all servers to agree that entry 3 holds x=1
  after agreement ("commit"), all DBs execute x=1

example:
  client sends "x?" to S2
  S2 picks log entry 4
  S2 uses Paxos to get all servers to agree that entry 4 holds "x?"
  after agreement, S2's DB executes "x?"
    and replies to the client

why a log?
  help cope with concurrency
    need to hold later operations until earlier ones have committed
  help replicas catch up
    not in majority, briefly offline, &c
  usually only the tail of the log is kept, along with "state"

next:
  guest lecture by a Go wizard
  then Raft
  Lab 1 due tomorrow

6.5840 Lecture 10: lab 3 A+B Q&A

Two different structures of Raft library
  multi-threaded with locks
  single-threaded state-machine
  neither one is strictly better than the other
  other plans are possible

Correctness in distributed systems:
 safety: never returning an incorrect result
 liveness: eventually returning a result
 Q: give an example of a safety and liveness bug in your solution

Getting started on a lab
  make first test case work
   fill out VoteRequest structs
   do an RPC
  then much easier to get the lay of the land
  other strategy: read all guides, read test code, etc.

Debugging
  Run with race detector
  Log all action/messages in easy searchable way
   standard format: src, dst, opcode, raft state,..
  Run test case
    if ok:
      next test case
    if fail:
      repeat:
        study test case
        formulate hypothesis about what might be wrong
        study log and figure 2, run with race detector
        modify code and try test again

Multi-threaded Raft library
  many threads (Start() thread, thread reading from applych)
  many RPC threads to talk to peers in parallel
  many RPC handler threads (started by RPC package)
  raft state with lock
  one thread writing to applych
    use condvar to signal it

Locking
  Raft lock serializes operations
  RPC handlers hold Raft lock so are atomic
    little parallelism
  Threads don't hold lock during RPC
    risk: deadlock if locks are held during RPC

Single-threaded state-machine
  observation: little parallelism anyway
    main use: sending RPCs in parallel
  one thread runs Raft protocol
    receives input events (e.g., tick, RPC request, reply)
    process event, updating raft structure, without locks
      collecting output events (persist, send RPCs)
    performs output events
      first persist, then RPC requests
  one thread for the applier
    need to separate state between applier and state-machine
    protect shared state with lock (e.g., the log)

Code tour
  Raft struct
  Ticker
  Election timeout
  Start election
  VoteRequest handling
  becomeLeader
  send appends
  AppendEntries request handling
  AppendEntries reply handling
  commit
  applier
  Start()

6.5840 2026 Lecture 6: RSM and Raft (1)

this lecture
  today: replicated state machine, and Raft elections (Lab 3A)
  next: Raft persistence, client behavior, snapshots (Lab more 3B, 3C, 3D)

Goal: high availability
  even if a machine fails, deliver service
    i.e., no down time even if a machine fails
  approach: replication

What kinds of failures can replication deal with?
  Replication is good for "fail-stop" failure of a single replica
    fan stops working, CPU overheats and shuts itself down
    someone trips over replica's power cord or network cable
    software notices it is out of disk space and stops
  Replication may not help with bugs or operator error
    Often not fail-stop
    May be correlated (i.e. some input causes all replicas to crash)
  How about earthquake or city-wide power failure?
    Only if replicas are physically separated

How many replicas?
  Often you want few because many is costly
  But enough to survive failures during repair
    Many systems run with 3-5 replicas, as we will see
  This paper runs with only 2 replicas
    resilient to 1 failure at the time

*** topic: replicated state machine

A popular approach to replication
  Clients send operations to primary,
      primary sequences and sends to backups
  All replicas execute all operations
  If same start state,
    same operations,
    same order,
    deterministic,
    then same end state.
  Example: primary backup in GFS

What if primary fails?
  * Coordinator picks new primary in GFS
  * What if coordinator fails?

Can we have the replicas elect a new primary
  how about two servers, S1 and S2
  if both are up, S1 is in charge, forwards decisions to S2
  if S2 sees S1 is down, S2 takes over as coordinator
  what could go wrong?
  network partition! split brain!

the problem: computers cannot distinguish "server crashed" from "network broken"
  the symptom is the same: no response to a query over the network
  this difficulty seemed insurmountable for a long time
  seemed to require outside agent (a human) to decide when to switch servers
  we'd prefer an automated scheme!

Two partition-tolerant replication schemes were invented around 1990,
  Paxos and View-Stamped Replication
  called "consensus" or "agreement" protocols
  in the last 15 years this technology has seen a lot of real-world use
  the Raft paper is a good introduction to modern techniques

*** topic: state-machine replication with raft

state machine replication with Raft -- Lab 2 + 4 as example:
  [diagram: clients, 3 replicas, k/v layer + state, raft layer + logs]
  Raft is a library included in each replica

time diagram of one client command
  [C, L, F1, F2]
  client sends Put/Get "command" to k/v layer in leader
  k/v layer calls Start() to invoke Raft
    leader's Raft layer adds command to log
    leader sends AppendEntries RPCs to followers
    followers add command to log
  leader waits for replies from a bare majority (including itself)
  entry is "committed" if a majority put it in their logs
    committed means won't be forgotten even if failures
    majority -> will be seen by the next leader's vote requests
    leader "piggybacks" commit info in next AppendEntries
  leader and follower hand commands to k/v layer once entry is committed
    ApplyMsg and applyCh in lab
  leader sends response to client

why the logs?
  the service keeps the state machine state, e.g. key/value DB
    the log is an alternate representation of the same information!
    why both?
  the log orders the commands
    to help replicas agree on a single execution order
    to help the leader ensure followers have identical logs
  the log stores tentative commands until committed
  the log stores commands in case leader must re-send to followers
  the log stores commands persistently for replay after reboot

are the servers' logs exact replicas of each other?
  no: some replicas may lag
  no: we'll see that they can temporarily have different entries
  the good news:
    they'll eventually converge to be identical
    the commit mechanism ensures servers only execute stable entries

Implementation challenges:
  Failures
     network partitions, lost messages, server crashes
  Concurrency
     within a server and between servers
  Result: many possible executions and many corner cases
    many details to work -- figure 2
  Today: electing a new leader, which must handle these challenges

*** topic: leader election (Lab 3A)

why a leader?
  ensures all replicas execute the same commands, in the same order
  (some designs, e.g. Paxos, don't have a leader)

Raft numbers the sequence of leaders
  new leader -> new term
  a term has at most one leader; might have no leader
  the numbering helps servers follow latest leader, not superseded leader

when does a Raft peer start a leader election?
  when it doesn't hear from current leader for an "election timeout"
  increments local currentTerm, tries to collect votes
  note: this can lead to un-needed elections; that's slow but safe
  note: old leader may still be alive and think it is the leader

how to ensure at most one leader in a term?
  (Figure 2 RequestVote RPC and Rules for Servers)
  leader must get "yes" votes from a majority of servers
  each server can cast only one vote per term
    if candidate, votes for itself
    if not a candidate, votes for first that asks (within Figure 2 rules)
  at most one server can get majority of votes for a given term
    -> at most one leader even if network partition
    -> election can succeed even if some servers have failed
  note: again, majority is out of all servers (not just the live servers)

how does a server learn about a newly elected leader?
  the leader sends out AppendEntries heart-beats
    with the new higher term number
  only the leader sends AppendEntries
    only one leader per term
    so if you see AppendEntries with term T, you know who the leader for T is
  the heart-beats suppress any new election
    leader must send heart-beats more often than the election timeout

an election may not succeed for two reasons:
  * less than a majority of servers are reachable
  * simultaneous candidates split the vote, none gets majority

what happens if an election doesn't succeed?
  no heartbeats -> another timeout -> a new election for a new term
  higher term takes precedence, candidates for older terms quit

without special care, elections will often fail due to split vote
  all election timers likely to go off at around the same time
  every candidate votes for itself
  so no-one will vote for anyone else!
  so everyone will get exactly one vote, no-one will have a majority

how does Raft avoid split votes?
  each server adds some randomness to its election timeout period
  [diagram of times at which servers' timeouts expire]
  randomness breaks symmetry among the servers
    one will choose lowest random delay
  hopefully enough time to elect before next timeout expires
  others will see new leader's AppendEntries heartbeats and
    not become candidates
  randomized delays are a common pattern in network protocols

how to choose the election timeout?
  * at least a few heartbeat intervals (in case network drops a heartbeat)
    to avoid needless elections (which can cause an unnecessary pause)
  * short enough to react quickly to failure, avoid long pauses
  * short enough to allow a few re-tries before tester gets upset
    tester requires election to complete in 5 seconds or less
  * random part long enough to let one candidate succeed before next starts

what if old leader isn't aware a new leader is elected?
  perhaps old leader didn't see election messages
  perhaps old leader is in a minority network partition
  new leader means a majority of servers have incremented currentTerm
  either old leader will see new term in a AppendEntries reply and step down
  or old leader won't be able to get a majority of replies
    so old leader won't commit or execute any new log entries
  thus no split brain
  but a minority may accept old server's AppendEntries
    so logs may diverge at end of old term

example log divergence
  a leader crashes before sending AppendEntries to all
    S1: 3
    S2: 3 3
    S3: 3 3
  (the 3s are the term number in the log entry)
  worse: logs might have different commands in same entry!
    after a series of leader crashes, e.g.
        10 11 12 13  <- log entry #
    S1:  3
    S2:  3  3  4
    S3:  3  3  5

  how could this happen?
    S2 is leader in term 3
      appends 10 to S1, S2, and S3
      appends 11 to S2 and S3 (S1 crashed)
    S2 crashes, reboots quickly, and leader in term 4
      appends 12 to its log, and crashes.
    S3 becomes leader in term 5 (with help of S1)
      appends a different entry for 12 to its log

next lecture we will see how Raft handles log divergence
  including an additional restriction on leader election
  you can pass 3A without this restriction

6.5840 2026 Lecture 7: Raft (2)

Raft
  library to build replicated state machines
    tolerant to network partition
    automatic fail-over to a new leader/primary
    key idea: quorum
  building block for large-scale distributed systems; examples:
    Raft library is used in Etcd, a configuration service
      etcd in turn is used to build services (e.g., Kubernetes)
    Raft is used in CockroadDB, a sharded distributed database
    ...and many others

what do we want to ensure?
  if any server executes a given command in a log entry,
    then no server executes something else for that log entry
  (Figure 3's State Machine Safety)
  why? if the servers disagree on the operations, then a
    change of leader might change the client-visible state,
    which violates our goal of mimicing a single server.
  example:
    S0: 101 | 102
    S1: 101 | 103
    can't allow both to execute their 2nd log entries!

Last lecture: election safety
  a single leader per term
  as long as the leader stays up:
    clients only interact with the leader
    clients don't see follower states or logs
  but, logs of replicas may diverge
    made-up example in last lecture

Today: repair log divergence, persistence, and compacting log

*** topic: the Raft log (Lab 2B)

TestRejoin3B: one test that test log divergence

  make RUN="-run Rejoin3B" raft1
    what does the test do?
      commits one entry
      disconnects old leader
      old leader appends to its log (102 at index 2)
      new leader commits 103 at index 2
      new leader disconnects and old leader connects
        the remaining follower becomes leader!
        old leader's log overwritten with new leader's log!

  here is a trace for a correct implementation
    other traces are possible too

  === RUN   TestRejoin3B
  Test (3B): rejoin of partitioned leader (reliable network)...
  1  0: Start log {[{0 <nil>} {1 101}] 0}
  2        1: append args Term 1 Leader 0 PrvLogIndex 0 PrvLogTerm 0 Entries [{T 1}] LeaderCommit 0
  3        1: append log {[{0 <nil>} {1 101}] 0}
  4              2: append args Term 1 Leader 0 PrvLogIndex 0 PrvLogTerm 0 Entries [{T 1}] LeaderCommit 0
  5              2: append log {[{0 <nil>} {1 101}] 0}
  test: leader 0 disconnect
  6  0: Start log {[{0 <nil>} {1 101} {1 102}] 0}
  7  0: Start log {[{0 <nil>} {1 101} {1 102} {1 103}] 0}
  8  0: Start log {[{0 <nil>} {1 101} {1 102} {1 103} {1 104}] 0}
  9              2: Start log {[{0 <nil>} {1 101} {2 103}] 0}
  10       1: append args Term 2 Leader 2 PrvLogIndex 1 PrvLogTerm 1 Entries [{T 2}] LeaderCommit 1
  11       1: append log {[{0 <nil>} {1 101} {2 103}] 0}
  12       1: append args Term 2 Leader 2 PrvLogIndex 1 PrvLogTerm 1 Entries [{T 2}] LeaderCommit 1
  13       1: append log {[{0 <nil>} {1 101} {2 103}] 0}
  test: disconnect new leader and connect old leader
  14  0: Start log {[{0 <nil>} {1 101} {1 102} {1 103} {1 104} {1 104}] 0}
  15  0: append args Term 3 Leader 1 PrvLogIndex 0 PrvLogTerm 0 Entries [{T 1} {T 2}] LeaderCommit 2
  16  0: append log {[{0 <nil>} {1 101} {2 103}] 0}
  17       1: Start log {[{0 <nil>} {1 101} {2 103} {3 104}] 0}
  18  0: append args Term 3 Leader 1 PrvLogIndex 2 PrvLogTerm 2 Entries [{T 3}] LeaderCommit 2
  19  0: append log {[{0 <nil>} {1 101} {2 103} {3 104}] 0}
  test: connect all
  20       1: Start log {[{0 <nil>} {1 101} {2 103} {3 104} {3 105}] 0}
  21  0: append args Term 3 Leader 1 PrvLogIndex 3 PrvLogTerm 3 Entries [{T 3}] LeaderCommit 3
  22              2: append args Term 3 Leader 1 PrvLogIndex 2 PrvLogTerm 2 Entries [{T 3} {T 3}] LeaderCommit 3
  23  0: append log {[{0 <nil>} {1 101} {2 103} {3 104} {3 105}] 0}
  24              2: append log {[{0 <nil>} {1 101} {2 103} {3 104} {3 105}] 0}

  draw timing diagram on board

trace high lights
  print statement in
    Start:
      when leader appends to log (Start log)
    AppendEntries handler:
      request received (append args)
      handler appends to log (append log)
  at line 1-5:
    server 0 is leader and appends 101 to all servers
      server 0 may deliver 101 to application, assuming it received the reply from 1 and 2
  at line 6 leader 0 is disconnected but append to its own log
    results in an election because other servers don't receive heartbeats
  at line 9 server 2 becomes leader and appends 103 in index 2
    server 0 and server 1 have different logs
  at line 11 server 1 appends 103 in index 2
    server 1 and 2 may send 103 to the app
      103 might commit
      so, leader 0 should definitely not deliver 102 to application in index 2
        it won't because it isn't on a majority of servers yet
  at line 14 tester asks old leader 0 to start another attempt for 104
    doesn't work; 1 and 2 will reject the append request
  at line 15, new leader for term 3 (server 1)
    why new term? leader for term 2 is disconnected
    new leader forces 0 to agree on its log
    server 0 forgets about some log entries
      ok because they didn't commit
    why didn't server 0 become leader---it has a longer log
  line 20 and up
    add 105 with all servers online

leader 1 in term 3 forces logs to be identical
  each live follower deletes tail of log that differs from leader
  then each live follower accepts leader's entries after that point
  now followers' logs are identical to leader's log

could new leader roll back *committed* entries from end of previous term?
  i.e. could a committed entry be missing from the new leader's log?
  this would be a disaster -- old leader might have already said "yes" to a client
  so: Raft needs to ensure elected leader has all committed log entries

why not elect the server with the longest log as leader?
  it would overwrite 103 at 2, which could be committed by 1 and 2
  who should be next leader?
    end of 5.4.1 explains the "election restriction"
    RequestVote handler only votes for candidate who is "at least as up to date":
      candidate has higher term in last log entry, or
      candidate has same last term and same length or longer log
  so:
    S1 and S2 won't vote for P0
    S1 and S2 will vote for each other
  so only S1 or S2 can be leader, will force S0 to discard 102, 103, 104
    ok since those are not on majority -> not committed -> reply never sent to clients
    -> clients will resend the discarded commands

the point:
  "at least as up to date" rule ensures new leader's log contains
    all potentially committed entries
  so new leader won't roll back any committed operation

"a committed operation" has two meanings in 6.5840:
  1) the op cannot be lost, even due to (allowable) failures.
     in Raft: when a majority of servers persist it in their logs.
     this is the "commit point" (though see Figure 8).
  2) the system knows the op is committed.
     in Raft: leader saw a majority in *current* term

again:
  we cannot reply "yes" to client before commit.
  we cannot forget an operation that may have been committed.


The Question (from last lecture)
  figure 7, top server is dead; which can be elected?

who could become leader in figure 7? (with top server dead)
  need 4 votes to become leader
  a: yes -- a, b, e, f
  b: no -- b, f
    e has same last term, but its log is longer
  c: yes -- a, b, c, e, f
  d: yes -- a, b, c, d, e, f
  e: no -- b, f
  f: no -- f

why won't d prevent a from becoming leader?
  after all, d's log has higher term than a's log
  a does not need d's vote in order to get a majority
  a does not even need to wait for d's vote

why is Figure 7 analysis important?
  choice of leader determines which entries are preserved vs discarded
  critical: if service responded positively to a client,
    it is promising not to forget!
  must be conservative: if client *could* have seen a "yes",
    leader change *must* preserve that log entry.
    Election Restriction does this via majority intersection.
  why OK to discard e's last 4,4?
  why OK to (perhaps) preserve c's last 6?
    could client have seen a "yes" for them?

how to roll back quickly
  the Figure 2 design backs up one entry per RPC -- slow!
  lab tester may require faster roll-back
  paper outlines a scheme towards end of Section 5.3
    no details; here's my guess; better schemes are possible
      Case 1      Case 2       Case 3
  S1: 4 5 5       4 4 4        4
  S2: 4 6 6 6 or  4 6 6 6  or  4 6 6 6
  S2 is leader for term 6, S1 comes back to life, S2 sends AE for last 6
    AE has prevLogTerm=6
  rejection from S1 includes:
    XTerm:  term in the conflicting entry (if any)
    XIndex: index of first entry with that term (if any)
    XLen:   log length
  Case 1 (leader doesn't have XTerm):
    nextIndex = XIndex
  Case 2 (leader has XTerm):
    nextIndex = leader's last entry for XTerm
  Case 3 (follower's log is too short):
    nextIndex = XLen

*** topic: persistence (Lab 2C)

what would we like to happen after a server crashes?
  Raft can continue with one missing server
    but failed server must be repaired soon to avoid dipping below a majority
  two repair strategies:
  * replace with a fresh (empty) server
    requires transfer of entire log (or snapshot) to new server (slow)
    we must support this, in case failure is permanent
  * or reboot crashed server, re-join with state intact, catch up
    requires state that persists across crashes
    we must support this, for simultaneous power failure
  let's talk about the second strategy -- persistence

if a server crashes and restarts, what must Raft remember?
  Figure 2 lists "persistent state":
    log[], currentTerm, votedFor
  a Raft server can only re-join after restart if these are intact
  thus it must save them to non-volatile storage
    non-volatile = disk, SSD, battery-backed RAM, &c
    save after each point in code that changes non-volatile state
    or before sending any RPC or RPC reply
  why log[]?
    if a server was in leader's majority for committing an entry,
      must remember entry despite reboot, so next leader's
      vote majority includes the entry, so Election Restriction ensures
      new leader also has the entry.
  why votedFor?
    to prevent a client from voting for one candidate, then reboot,
      then vote for a different candidate in the same term
    could lead to two leaders for the same term
  why currentTerm?
    avoid following a superseded leader.
    avoid voting in a superseded election.

some Raft state is volatile
  commitIndex, lastApplied, next/matchIndex[]
  why is it OK not to save these?

persistence is often the bottleneck for performance
  a hard disk write takes 10 ms, SSD write takes 0.1 ms
  so persistence limits us to 100 to 10,000 ops/second
  (the other potential bottleneck is RPC, which takes << 1 ms on a LAN)
  lots of tricks to cope with slowness of persistence:
    batch many new log entries per disk write
    persist to battery-backed RAM, not disk
    be lazy and risk loss of last few committed updates

how does the service (e.g. k/v server) recover its state after a crash+reboot?
  easy approach: start with empty state, re-play Raft's entire persisted log
    lastApplied is volatile and starts at zero, so you may need no extra code!
    this is what Figure 2 does
  but re-play will be too slow for a long-lived system
  faster: use Raft snapshot and replay just the tail of the log

*** topic: log compaction and Snapshots (Lab 2D)

problem:
  log will get to be huge -- much larger than state-machine state!
  will take a long time to re-play on reboot or send to a new server

luckily:
  a server doesn't need *both* the complete log *and* the service state
    the executed part of the log is captured in the state
    clients only see the state, not the log
  service state usually much smaller, so let's keep just that

what log entries *can't* a server discard?
  committed but not yet executed
  not yet known if committed

solution: service periodically creates persistent "snapshot"
  [diagram: service state, snapshot on disk, raft log (in mem, on disk)]
  copy of service state as of execution of a specific log entry.
    e.g. k/v table.
  service hands snapshot to Raft, with last included log index.
  Raft persists its state and the snapshot.
  Raft then discards log before snapshot index.
  every server snapshots (not just the leader).

what happens on crash+restart?
  service reads snapshot from disk
  Raft reads persisted log from disk
  Raft sets lastApplied to snapshot's last included index
    to avoid re-applying already-applied log entries

problem: what if follower's log ends before leader's log starts?
  because follower was offline and leader discarded early part of log
  nextIndex[i] will back up to start of leader's log
  so leader can't repair that follower with AppendEntries RPCs
  thus the InstallSnapshot RPC

philosophical note:
  state is often equivalent to operation history
  one or the other may be better to store or communicate
  we'll see examples of this duality later in the course

practical notes:
  Raft's snapshot scheme is reasonable if the state is small
  for a big DB, e.g. if replicating gigabytes of data, not so good
    slow to create and write entire DB to disk
  perhaps service data should live on disk in a B-Tree
    no need to explicitly snapshot, since on disk already
  dealing with lagging replicas is hard, though
    leader should save the log for a while
    or remember which parts of state have been updated

*** read-only operations (end of Section 8)

Q: does the Raft leader have to commit read-only operations in
   the log before replying? e.g. Get(key)?

that is, could the leader respond immediately to a Get() using
  the current content of its key/value table?

A: no, not with the scheme in Figure 2 or in the labs.
   suppose S1 thinks it is the leader, and receives a Get(k).
   it might have recently lost an election, but not realize,
   due to lost network packets.
   the new leader, say S2, might have processed Put()s for the key,
   so that the value in S1's key/value table is stale.
   serving stale data is not linearizable; it's split-brain.

so: Figure 2 requires Get()s to be committed into the log.
    if the leader is able to commit a Get(), then (at that point
    in the log) it is still the leader. in the case of S1
    above, which unknowingly lost leadership, it won't be
    able to get the majority of positive AppendEntries replies
    required to commit the Get(), so it won't reply to the client.

but: many applications are read-heavy. committing Get()s
  takes time. is there any way to avoid commit
  for read-only operations? this is a huge consideration in
  practical systems.

idea: leases
  modify the Raft protocol as follows
  define a lease period, e.g. 5 seconds
  after each time the leader gets an AppendEntries majority,
    it is entitled to respond to read-only requests for
    a lease period without adding read-only requests
    to the log, i.e. without sending AppendEntries.
  a new leader cannot execute Put()s until previous lease period
    has expired
  so followers keep track of the last time they responded
    to an AppendEntries, and tell the new leader (in the
    RequestVote reply).
  result: faster read-only operations, still linearizable.

note: for the Labs, you should commit Get()s into the log;
      don't implement leases.

in practice, people are often (but not always) willing to live with stale
  data in return for higher performance

----

https://decentralizedthoughts.github.io/2020-12-12-raft-liveness-full-omission/

6.5840 2026 Lecture 18: Ray

Ownership: A Distributed Future System For Fine-Grained Tasks by
Stephanie Wang et al., NSDI 2021

Why are we reading this paper?
  A modern version of MapReduce, Spark, etc.
  Moving large data efficiently with futures
    Quite different from RPCs in lab
  Managing distributed futures efficiently using ownership
  Widely-used open-source project (e.g., by OpenAI)
    anyscale

Evolving parallel applications
  Combine functional and stateful
  Low latency requirements
  Examples:
    Model serving (3a)
      quick response
      large data (client images)
      router and model replicas maintain state between invocations
    On-line video processing (video stabilizing in 3b)
      compute trajectory of an object
        start processing frame before seeing next frame
       actor for storing the decoded frame
   MapReduce and Spark aren't good at these

Three ideas:
  futures (handle for the result of a computation)
  actors (to hold state between invocations)
  shard state about futures based on ownership

Futures (API table 1)
  program can invoke a function asynchronously, which return a future
  program can pass the reference to a future as an argument to other functions
  programs can force a future to evaluate
  benefit: system can decide where to run future and when data is moved

Example of borrowing: fig 2
  f1 = C()
  f2 = C()
  f3 = Add(shared(f1), shared(f2))  # pass f1+f2 by reference
  c = get(f3)

Example of borrowing: fig 6
  def A():
    x = B()
    y = C(shared(x))  # pass x by reference
  f1 = A()

Ray example of remote execution with futures:
  # see https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html
  import ray
  ray.init()

  @ray.remote
  def g(i):
    return i

  f = g.remote(10)  # f is a future and ray invokes g asynchronously

  // the 4 g invocations run in parallel
  future_ids = [g.remote(i) for i in range(4)]
  results = ray.get(future_ids)

Ray example of actors and object refs
  @ray.remote
  class Counter:
    def __init__(self):
        self.value = 0
    def increment(self):
        self.value += 1
        return self.value

  counter = Counter.remote()            # Instantiate as a remote actor
  f = counter.increment.remote()        # future corresponding to an object ref
  print(ray.get(f))

Implementation challenges:
  keep track of futures and objects
    incl. garbage collection of future and objects
  worker machine crashes while running a future
    Ray: transparent recovery of idempotent futures (fig 5)

Sketch of simple/hypothetical implementation: centralized coordinator
  Coordinator maintains a table (id, task, refs, worker, value)
    see example below
  Invoking future task:
    scheduler picks a worker
    adds future to table
  Get() contacts coordinator
    If value present, return value
    If not present, block at coordinator until worker fills in value
    If value is large, store it in object store and make the value the object id
      caller fetches object directly from object store
      objects are immutable
  Task can garbage collect its futures and objects
  If a worker crashes, driver/task restarts invoked tasks
    "lineage reconstruction"
       re-execute tasks and descendants
       which recreate futures, and further descendants
    use second copies of objects to avoid recomputation

Example: state when both B and C are running (fig 6b)

    ID  |   Task   | Refs |  Locations
    ----------------------------------
    x       B()      W1,W3      W2
    y       C(X)     W1         W3

    where worker 1 is running A, 2 running B, and 3 running C

  refs for reference counting
    if C returns, remove 3 from Refs for x
      but cannot GC x yet, since A is using it
    if A returns, no refs to x and y
      safe to delete x and y

  recovery though lineage reconstruction
    ex: re-executing A, re-executes B and C too

Implementation challenge with centralized plan
  one round-trip for invocation and get
  many futures
    some run for a short time (a few ms)
    coordinator is scalability bottleneck

Alternative: shard the table (e.g., by obj id)
  still one round-trip with coordinator
  GC maybe involve multiple shards

Ray solution: shard by ownership
  Example
    A owns "x"
    C "borrows" x
      it may pass it further
      does it reference counting
    The caller of function is the owner of a returned future
      But value of object is stored at the worker that runs the task
      Ex. x:
        A  is owner of x,
        value of x is store at the worker that runs B (2)
  advantage
    A can invoke C without any communication with B

Example: state when both B and C are running
  Owner table at 1:
  ID  Task  Owner   Ref    Loc  Val
  x   B()    W1     W1,W3  W2
  y   C(x)   W1     W1     W3

  Owner table at 2:
  ID  Owner Ref  Loc   Val
  x     W1        W2

  Owner table at 3:
  ID  Owner Ref   Loc   Val
  x    W1          W2
  y    W1          W3

structure of application guides sharding
  if one tasks invokes many futures, split task up into several
    each subtask being in charge of some of the futures

Ownership implementation
  owner = (IP, port, workerid)
  taskId = parentTaskId + task_index
  objectId = taskId + obj_index

distributed scheduler (figure 8)
 schedule(t)
   id = local worker
   while true:
     ok, id = reserve @id using resource request
     if ok:
        l = lease(id)
        break
   table[t].Loc = l
 optimization: reuse leased worker

Memory management (figure 9)
  API obj store: Create, Get, Pin, Release
    Get blocks until obj is created
    Initial Create pins obj
  tab[oid].Location = locations in object store
    may have secondary copies
      C's worker has a copy of x after fetching it from B's worker
      if x is large

Failure recovery: fate sharing
  check locations in table
    loss of an owned object
      re-execute tasks following lineage
      use secondary copy to avoid recomputing
    loss of an owner: risk of dangling reference
      fate sharing
      example: if 2 crashes, 3 may have a dangling ref
        3 fails and 1 fails

Example:
  if worker 3 fails (after starting C but before finishing)
    worker 1 (A) will learn about it
    it is the owner of y (and has the Task info) and asks Ray to re-execute C(X)
  if worker 1 and 2 fail, worker 3 has a dangling ref to x
    it will never be resolved to a value
    worker 3 "shares fate" with the owner of "x"
      i.e., it terminates itself, pretending a crash
    the caller of A will re-submit A

Homework: if C() in figure 6(a) is as follows:
  def C(x):
    z = D(X)
    return get(z)   # return value of future z

  Suppose the worker than runs D fails before finishing, which worker would
  initiates the re-execution of D()?

  who owns z?
    the caller: C
    who resubmits D? C

References
  "Ray: A Distributed Framework for Emerging AI Applications" (OSDI 2018)

6.5840 2026 Lecture 2: Threads and RPC

Topic: implementing distributed systems
  ... and Go programming for the labs
  Go threads, and the web crawler
  Go RPC
  Details are Go specific but the concepts are important and widely used

Why Go?
  good support for threads
  convenient RPC
  type- and memory- safe
  garbage-collected (no use after freeing problems)
    threads + GC is particularly attractive!
  not too complex
  Go is often used in distributed systems

After the tutorial, use https://golang.org/doc/effective_go.html

Threads
  a useful structuring tool, but can be tricky
  Go calls them goroutines; everyone else calls them threads

Thread = "thread of execution"
  threads allow one program to do many things at once
  each thread executes serially, just like a non-threaded program
  the threads share memory
  each thread includes some per-thread state:
    program counter, registers, stack

Why threads?
  I/O concurrency
    Client sends requests to many servers in parallel and waits for replies.
    Server processes many simultaneous client requests.
      Each request may block.
      While waiting for the disk to read data for client X,
        process a request from client Y.
  Multicore performance
    Execute code in parallel on several cores.
  Convenience
    In background, once per second, check whether each worker is still alive.

Is there an alternative to threads?
  Yes: write code that explicitly interleaves activities, in a single thread.
    Usually called "event-driven."
  Keep a table of state about each activity, e.g. each client request.
  One "event" loop that:
    checks for new input for each activity (e.g. arrival of reply from server),
    does the next step for each activity,
    updates state.
  Event-driven can get you I/O concurrency,
    and eliminates thread costs (which can be substantial),
    but doesn't get multi-core speedup,
    and is painful to program.

Threading challenges:
  sharing data safely
    what if two threads do n = n + 1 at the same time?
      or one thread reads while another increments?
    this is a "race"
      = two threads use same memory at same time, one (or both) writes
      often a bug
    -> use locks (Go's sync.Mutex)
    -> or avoid sharing mutable data
  coordination between threads
    one thread is producing data, another thread is consuming it
      how can the consumer wait (and release the CPU)?
      how can the producer wake up the consumer?
    -> use Go channels or sync.Cond or sync.WaitGroup
  deadlock
    a cycle of threads waiting for each other
    via locks, or channels, or RPC

Let's look at the tutorial's web crawler as a threading example.

What is a web crawler?
  goal: fetch all web pages, e.g. to feed to an indexer
  you give it a starting web page
  it recursively follows all links
  [diagram: pages, links, a DAG, a cycle]
  but don't fetch a given page more than once
    and don't get stuck in cycles

Crawler challenges
  Exploit I/O concurrency
    Network latency is more limiting than network capacity
      internet latency: maybe 0.1 seconds, due to speed of light &c
      internet throughput: maybe MB/sec or GB/sec
    Fetch many pages in parallel
      To increase URLs fetched per second
    => Use threads for concurrency
  Fetch each URL only *once*
    avoid wasting network bandwidth
    avoid link cycles
    be nice to remote servers
    => Need to remember which URLs visited
  Know when finished

We'll look at three solutions [crawler.go on schedule page]
  Serial
  Concurrent, coordination via shared data
  Concurrent, coordination via channels

Serial crawler:
  performs depth-first exploration via recursive Serial calls
  the "fetched" map avoids repeats, breaks cycles
    a single map, passed by reference, caller sees callee's updates
  finished when all [recursive] links are explored: easy
  but: fetches only one page at a time -- slow
    can we just put a "go" in front of the Serial() call?
    what will happen?
    let's try it... what happened?

ConcurrentMutex crawler:
  Creates a thread for each page fetch
    Many concurrent fetches, higher fetch rate
  the "go func" creates a goroutine and starts it running
    func... is an "anonymous function"
  The threads share the fs.fetched map
    So only one thread will fetch any given page
  Why the Mutex (Lock() and Unlock()) in testAndSet()?
    One reason:
      Two threads make simultaneous calls to ConcurrentMutex() with same URL
        Due to two different pages containing link to same URL
      T1 reads fetched[url], T2 reads fetched[url]
      Both see that url hasn't been fetched (fetched[url] = false)
      Both fetch, which is wrong
      The mutex causes one to wait while the other does both check and set
        So only one thread sees fetched[url]==false
      We say "the lock protects fs.fetched[]"
        But note Go does not enforce any relationship between locks and data!
      The code between lock/unlock is often called a "critical section"
    Another reason:
      Internally, map is a complex data structure (tree? expandable hash?)
      Concurrent update/update may wreck internal invariants
      Concurrent update/read may crash the read
    defer...
    What if I comment out Lock() / Unlock()?
      go run crawler.go
        Does it always work? Always fail? Why?
      go run -race crawler.go
        Detects races even when output is correct!
    What if I forget to Unlock()?  deadlock
  How does the ConcurrentMutex crawler decide it is done?
    sync.WaitGroup -- it's basically a counter
    Wait() waits for all Add()s to be balanced by Done()s
      i.e. waits for all child threads to finish
    [diagram: tree of goroutines, overlaid on cyclic URL graph]
    there's a WaitGroup per node in the tree
  How many concurrent threads might there be?

ConcurrentChannel crawler
  a Go channel:
    a channel is an object
      ch := make(chan int)
    a channel lets one thread send an object to another thread
    ch <- x
      the sender waits until some goroutine receives
    y := <- ch
      a receiver waits until some goroutine sends
    also: for y := range ch
    channels both communicate and synchronize
    several threads can send and receive on a channel
    send+recv takes less than a microsecond -- fairly cheap
    remember: sender blocks until the receiver receives!
      "synchronous"
      watch out for deadlock
  ConcurrentChannel coordinator()
    coordinator() creates a worker goroutine to fetch each page
    worker() sends slice of page's URLs on a channel
      multiple workers send on the single channel
    coordinator() reads URL slices from the channel
  At what line does the coordinator wait?
    Does the coordinator use CPU time while it waits?
  Note: there is no recursion here; coordinator() creates all workers.
  Note: no need to lock the fetched map, because it isn't shared!
  How does the coordinator know it is done?
    Keeps count of workers in n.
    Each worker sends exactly one item on channel.
  The channel does two things:
    1. communication of values.
    2. notification of events (e.g. thread termination).

Why is it safe for multiple threads use the same channel?

Is this a race:
  Worker thread modifies (creates) url slice, coordinator uses it?
  * worker only writes slice *before* sending
  * coordinator only reads slice *after* receiving
  So they can't use the slice at the same time, so there's no race.

Why does ConcurrentChannel() create a goroutine just for "ch <- ..."?
  Let's get rid of the goroutine...

When to use sharing and locks, versus channels?
  Most (all?) problems can be solved in either style
  What makes the most sense depends on how the programmer thinks
    state -- sharing and locks
    communication -- channels
  For the 6.824 labs, I recommend sync.Mutex/sync.Cond for shared state

Remote Procedure Call (RPC)
  a key piece of distributed system machinery; all the labs use RPC
  goal: easy-to-program client/server communication
  hide details of network protocols
  convert data (strings, arrays, maps, &c) to "wire format"
  portability / interoperability

RPC message diagram:
  Client             Server
    request--->
       <---response

Software structure
  client app        handler fns
   stub fns         dispatcher
   RPC lib           RPC lib
     net  ------------ net

Go example: kv.go on schedule page
  A toy key/value storage server -- Put(key,value), Get(key)->value
  Uses Go's RPC library
  Common:
    Declare Args and Reply struct for each server handler.
  Client:
    connect()'s Dial() creates a TCP connection to the server
    get() and put() are client "stubs"
    Call() asks the RPC library to perform the call
      you specify connection, function name, arguments, place to put reply
      library marshalls args, sends request, waits, unmarshalls reply
      return value from Call() indicates whether it got a reply
      usually you'll also have a reply.Err indicating service-level failure
  Server:
    Go requires server to declare an object with methods as RPC handlers
    Server then registers that object with the RPC library
    Server accepts TCP connections, gives them to RPC library
    The RPC library
      reads each request
      creates a new goroutine for this request
      unmarshalls request
      looks up the named object (in table create by Register())
      calls the object's named method (dispatch)
      marshalls reply
      writes reply on TCP connection
    The server's Get() and Put() handlers
      Must lock, since RPC library creates a new goroutine for each request
      read args; modify reply

A few details:
  Binding: how does client know what server computer to talk to?
    For Go's RPC, server name/port is an argument to Dial
    Big systems have some kind of name or configuration server
  Marshalling: format data into packets
    Go's RPC library can pass strings, arrays, objects, maps, &c
    Go passes pointers by copying the pointed-to data
    Cannot pass channels or functions
    Marshals only exported fields (i.e., fields w/ CAPITAL letter)

RPC problem: what to do about failures?
  e.g. lost packet, broken network, slow server, crashed server

What does a failure look like to the client RPC library?
  Client never sees a response from the server
  Client does *not* know if the server saw the request!
    [diagram of losses at various points]
    Maybe server never saw the request
    Maybe server executed, crashed just before sending reply
    Maybe server executed, but network died just before delivering reply
  Remote procedure call doesn't behave the same as procedure call on a single machine!
    A recurring challenge in implementing distributed systems

Simplest failure-handling scheme: "best-effort RPC"
  Call() waits for response for a while
  If none arrives, re-send the request
  Do this a few times
  Then give up and return an error

Q: is "best effort" easy for applications to cope with?

A particularly bad situation:
  client executes
    Put("k", 10);
    Put("k", 20);
  both succeed
  what will Get("k") yield?
  [diagram, timeout, re-send, original arrives late]

Q: is best effort ever OK?
   read-only operations
   operations that it's harmless to repeat
     e.g. DB checks if record has already been inserted

Other common semantics: at-most-once
  For example, Go RPC is a simple form of "at-most-once"
    open TCP connection
    write request to TCP connection
    Go RPC never re-sends a request
      So server won't see duplicate requests
    Go RPC code returns an error if it doesn't get a reply
      perhaps after a timeout (from TCP)
      perhaps server didn't see request
      perhaps server processed request but server/net failed before reply came back
  Labs explore others way of implementing at-most-once
    No retry is too restrictive for replicated servers
    Like to retry at another replica if first replica fails

6.5840 2026 Lecture 12: Spanner

Why this paper (Google Spanner, OSDI 2012)?
  Unusually ambitious for its time:
    Wide-area distributed transactions.
    Consistent wide-area replication.
  Neat ideas:
    Two-phase commit over Paxos.
    Read/only transactions w/o locking, via Snapshot Isolation.
    Read/only reads from local replica, but fresh.
    Synchronized time for external consistency.
  Spanner is used a lot inside Google.
    And influential, e.g. CockroachDB.

What was the motivating use case?
  Google F1 advertising database (Section 5.4).
  Previously sharded over many MySQL and BigTable DBs; awkward.
  Needed:
    Synchronous replication over data centers (failure zones).
    Cross-shard transactions.
    External consistency / linearizability / serializability.
    Automatic re-sharding.
  Workload is dominated by read-only transactions (Table 6).

The basic organization:
  Datacenter A:
    "clients" are web servers e.g. for gmail
    data is sharded over multiple servers:
      a-m
      n-z
  Datacenter B:
    has its own local clients
    and its own copy of the data shards
      a-m
      n-z
  Datacenter C:
    same setup

Replication managed by Paxos; one Paxos group per shard.
  Paxos similar to Raft -- replicas, leader, quorum, log.
  Replicas in different data centers, hopefully fail independently.

Why this arrangement?
  Sharding allows huge total throughput via parallelism.
  Replicas at different data centers cope with whole-site failures.
  Paxos requires only a majority -- tolerate slow/distant replicas.
  Clients can read local replica -- fast!

What are the challenges?
  Entirely local read-only transactions pose some problems:
    Can we have multi-record read-only transactions without locking?
    Data on local replica may not be fresh if not in Paxos majority.
  A transaction may involve multiple shards -> multiple Paxos groups.

Spanner treats read/write and read/only transactions differently.

First, read/write transactions.

Example read/write transaction (bank transfer):
  BEGIN
    x = x + 1
    y = y - 1
  END
We don't want any read or write of x or y sneaking between our two ops.
After commit finishes, all reads should see our updates.

Summary: two-phase commit (2pc) with Paxos-replicated participants.
  (Omitting timestamps for now.)
  (This is for r/w transactions, not r/o.)
  Diagram:
    1. C sends Rx, Ry to A, B.
    2. A, B update lock tables.
    3. C commits, sends Wx Wy to A and B.
    4. A, B log prepare.
    5. A, B send "yes" to TC.
    6. TC logs commit.
    7. TC sends commit to A, B, C.
    8. A, B update tables, release locks.
  Client sends each read to Paxos leader of relevant shard (2.1).
    Each shard first acquires a lock on the relevant record.
      May have to wait.
    Separate lock table per shard, in shard leader.
    Read locks are not replicated via Paxos, so leader failure -> abort.
  Client keeps writes private until commit.
  When client commits (4.2.1):
    Chooses a Paxos group to act as 2pc Transaction Coordinator (TC).
    Sends writes to relevant shard leaders.
    Each written shard leader:
      Acquires/upgrades lock(s) on the written record(s).
      Log a "prepare" record via Paxos, to replicate lock and new value.
      Tell TC it is prepared.
      Or tell TC "no" if crashed and lost lock table.
  Transaction Coordinator:
    Decides commit or abort.
    Logs the decision to its group via Paxos.
    Tell participant leaders and client the result.
  Each participant leader:
    Log the TC's decision via Paxos.
    Perform writes (values already logged during commit).
    Release the transaction's locks.

Some points about the design so far (just read/write transactions).
  Locking (two-phase locking) ensures serializability.
  2pc widely hated b/c it blocks with locks held if TC fails.
    Replicating the TC with Paxos solves this problem!
  r/w transactions take a long time.
    Inter-data-center messages, sync disk writes.
    Tables 4 and 6 suggest 10ms to 100ms.
  But lots of parallelism: many clients, many shards.
    So total throughput can be high if busy.

Now for read-only (r/o) transactions.
  These can involve multiple reads, from multiple shards.
  Table 6 shows 99.9% of transactions are read-only!
  So we want to make them fast, even at expense of r/w transactions.
  But still strictly serializable.

Spanner eliminates three big costs for r/o transactions:
  No two-phase commit, no transaction manager, no locks (!).
    To avoid slow inter-data-center msg to Paxos leader.
    And to avoid forcing r/w transactions to wait via locks.
  Read from local replicas, again to avoid Paxos and cross-datacenter msgs.
    Note local replica may not be up to date!
  Tables 3 and 6 show r/o cleverness yields 10x lower latency than r/w!
    This is a big deal.
  How to be correct despite cutting these corners?

Correctness constraints on r/o transactions:
  Serializable:
    Same results as if transactions executed one-by-one.
      Even though they may actually execute concurrently.
    I.e. an r/o xaction must essentially fit between r/w xactions.
      See all writes from prior transactions, nothing from subsequent.
    Even though *concurrent* with r/w xactions! And not locking!
  Externally consistent (== linearizable == "strict" serializability):
    If T1 completes before T2 starts, T2 must see T1's writes.
    "Before" refers to real (wall-clock) time.
    Even though T2 doesn't know of T1, diff data ctr, no locks.

Why not have r/o transactions just read the latest committed values?
  Suppose two bank transfers, and a transaction that reads both balances.
    T1:  Wx  Wy  C
    T2:                 Wx  Wy  C
    T3:             Rx             Ry
  We want T3 to see both of T2's writes, or none.
  We want T3's reads to *all* occur at the *same* point relative to T1/T2.
  (Locking would fix this; T3's Rx would force T2 to wait in Wx...)

Idea: Snapshot Isolation (SI):
  DB stores multiple versions for each record.
    Each labeled with a time -- called "time-stamp".
  A write adds a new version (doesn't overwrite).
  Assign every transaction a time-stamp.
  All of a r/w transactions's writes get the xaction's time-stamp.
  An r/o transaction's reads see versions as of its time-stamp.
    The record version with the highest time-stamp less than the xaction's.
  Called Snapshot Isolation.

Our example with Snapshot Isolation:
                      x@10=9         x@20=8
                      y@10=11        y@20=12
    T1 @ 10:  Wx  Wy  C
    T2 @ 20:                 Wx  Wy  C
    T3 @ 15:             Rx             Ry
  "@ 10" indicates the time-stamp.
  Now T3's reads will both be served from the @10 versions.
    T3 won't see T2's write even though T3's read of y occurs after T2.
  Now the results are serializable: T1 T3 T2
    The same as if executed serially in time-stamp order.

Why OK for T3 to read the *old* value of y even though there's a newer value?
  T2 and T3 are concurrent, so external consistency allows either order.
  Remember: r/o transactions need to read values
    as of their timestamp, and *not* see later writes.

Nice:
  T3 can get transactional (serializable) reads without locking!
    Reduces communication and blocking of r/w transactions.
  The cost (so far) is storing multiple versions for Snapshot Isolation.

Problem: what if T3 reads x from a replica that hasn't seen T1's write?
  Because the replica wasn't in the Paxos majority?

Solution: replica "safe time".
  Paxos leaders send writes in timestamp order.
  Before serving a read at time 20, replica must see Paxos write for time > 20.
    So it knows it has seen all writes < 20.
  So the replica may have to delay its response to a read.
  Must also delay if prepared but uncommitted r/w transactions (Section 4.1.3).
  Thus: r/o transactions can read from local replica -- usually fast.

Problem: what if clocks are not correct?
  How to ensure external consistency for r/o transactions?

What goes wrong if clocks aren't correct?
  R/O TS too large:
    Its TS will be higher than replica safe times, and reads will block.
    Correct but slow -- delay increased by amount of clock error.
  R/O TS too small:
    It will miss writes that committed before the r/o xaction started.
    Since its low TS will cause it to use old versions of records.
    This violates external consistency!
  R/W TS too large:
    Old writes may seem to r/o like latest data.
  R/W TS too small:
    New writes may seem old to r/o, and ignored.

Example of problem if r/o xaction's TS is too small:
  r/w T1 @ 10:         Wx2 C
  r/o T2 @  5:                   Rx?
(C for commit)
This would cause T2 to read the version of x before T1.
But T2 started after T1 committed (in real time),
  so external consistency requires that T2 see x=2.
So we need a way to deal with incorrect clocks!

What goes wrong with computer clocks?
  The absolute time is never quite right -- offset.
  Tick rate varies, never exactly right -- drift.

You can ask a better clock what time it it.
  Correct your offset, measure and compensate for drift.
  But comparisons have error due to variable communication delays.

Google's time reference system (Section 5.3)
  [UTC(USNO), GPS satellites, masters, servers]
  A tree, periodic network queries.
  Uncertainty due to network delays, drift between queries.

TrueTime
  Time service yields a TTinterval = [ earliest, latest ].
  The correct time is somewhere in the interval, with high probability.
  Interval width computed from measured network delays,
    measured clock drift, time since last sync.
  Figure 6: intervals are usually < 1 millisecond, but sometimes 10+ ms.
  So: server clocks aren't exactly synchronized.
  But some useful guarantees:
    TS > latest is guaranteed to be in the future.
    TS < earliest is guaranteed to be in the past.

How Spanner ensures that if r/w T1 finishes before r/o T2 starts, TS2 > TS1.
  So that T2's snapshot reads see T1's writes.
  And are thus externally consistent.

Two rules for r/w transactions (4.1.2):
  Start rule:
    TS = TT.now().latest, when commit begins
    this is the time-stamp it will write in the DB records
  Commit wait:
    Before releasing locks or replying to client,
      delay until TS < TT.now().earliest
    Guarantees that TS *is* in the past.

Rule for r/o transctions (4.1.4, 4.2.2):
  TS = TT.now().latest
  and read that version (really, version with highest timestamp < TS)
  TS guaranteed *not* to be in the past

Example updated with intervals and commit wait:
  The scenario is T1 finishes, then T2 starts, T2 must see T1's writes.
  I.e. we need TS1 < TS2.

               TS1=10
               [1,10]     [11,20]
  r/w T1:        C..........W
                                   TS2=15
                                   [5,15]
  r/o T2:                            Rx
  "C" for start of commit, "W" for end of commit wait
  Remember, the assumption is that T2 starts after T1 finishes,
    so that T2 needs to see T1's writes.

Why commit wait provides external consistency for r/o transactions:
  Given that T1 finishes before T2 starts.
  Commit wait means TS1 is guaranteed to be in the past.
  After commit wait:
    TS1 guaranteed to be in the past.
    TS2 = TT.now().latest guaranteed to be in the future.
    So TS2 > TS1.
    So T2 will see T1's writes.

Why it matters:
  I change my password, wait for finish, then log in via another data ctr.
  External consistency guarantees I'll encounter my new password.

Why does an r/w transaction use TS = TT.now().latest?
  (above reasoning allows any TS that commit-wait forces into the past)
  Suppose sequential r/w transactions: T3, and afterwards T4.
  We must have TS3 < TS4 (so that an r/o that sees T4 also sees T3).
  The same reasoning as for r/o will cause TS4 to be greater than TS3.
  So all r/w transactions use TS = TT.now().latest.

More generally:
  Snapshot Isolation gives you serializable r/o transactions.
    Timestamps set an order.
    Snapshot versions implement consistent reads at a timestamp.
    Xaction sees all writes from lower-TS xactions, none from higher.
  Synchronized time + commit-wait yields external consistency.
    Even among transactions at different data centers.
  Safe time allows reading from local replicas that might lag.

Why is all this useful?
  Fast r/o transactions:
    Read from local replicas!
    No locks!
    Thus the 10x latency r/o vs r/w improvement in Table 3 (and Table 6).
  Although:
    r/o transaction reads may pause due to safe time, to catch up.
    r/w transaction commits pause in Commit Wait.
    Accurate (small interval) TrueTime minimizes these delays.
      (This is The Question)

Summary:
  Rare to see deployed systems offer distributed transactions
    over geographically distributed data.
  Spanner was a surprising demonstration that it can be practical.
  Timestamping scheme is the most interesting aspect.
  Widely used within Google; a commercial Google service; influential.

Next:
  Chain replication: simpler than Raft/Paxos.

---

Spanner: Becoming a SQL System, 2017, https://research.google/pubs/pub46103.pdf
https://www.cockroachlabs.com/blog/living-without-atomic-clocks/
https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/
https://sookocheff.com/post/time/truetime/
https://communities.actian.com/s/article/Using-MVCC-Multi-Version-Concurrency-Control
https://cloud.google.com/spanner/docs/replication

6.5840 2026 Lecture 19: Secure Untrusted Data Repository (SUNDR) (2004)

We've mostly ignored security
  A big topic, multiple MIT courses
  A few lectures now, intersection with consistency and replication

Why this paper?
  We routinely trust storage services: github, gmail, AFS, Dropbox.
  Google &c may mean well, but:
    Maybe the server s/w or h/w is buggy, even exploitable.
    Maybe an attacker guessed server admin password, modified s/w.
    Maybe an employee of the cloud provider is corrupt or sloppy.

The problem is real!
  Paper cites successful attacks on open source code repositories.
  Troubling: attacker can then modify source, perhaps w/o detection.

Can we obtain trustworthy storage from non-trustworthy servers?
  [diagram: clients, server]

We want "integrity"
  Readers can check that they retrieve correct data.
  Server can't forge or hide updates or change update order.
    e.g. can't hide a critical security patch.

SUNDR goal is integrity -- not secrecy.
  Uses cryptography only to verify data is correct.

What tools does SUNDR use for integrity?
  cryptographic hashes
  digital signatures

Cryptographic hash, e.g. SHA-1
  h = hash(data)
  h is small, data can be large
  for SHA-1, h is 160 bits
  secure = not practical to find two different inputs that have the same hash
  used to name data and/or to verify integrity

Secure storage with cryptographic hashes
  suppose:
    you're using a cloud provider's key/value server
    you want to detect if server corrupts your data
    here's one way
  client write(v):
    k = hash(v)
    put(k, v)  -- RPC to server
    return k
  client read(k):
    v = get(k)  -- RPC to server
    if hash(v) != k: reject!
    return v
  note k is used in two ways: as a name, and as hash for validation

from where does a reading client get the key?
  can't choose keys -- key must be hash(v)
    so reader can't predict keys -- must be told
  can e.g. publish on secure web pages
  often used for e.g. secure software distributions

this is secure storage with an untrusted server
  if server corrupts data, client check will fail
  "content-hash storage"

we can build secure list- and tree-shaped structures from content-hash storage
  [diagram: files... <- directory <- root key]
  root key allows client to retrieve and check the whole structure
  even if blocks are stored in untrusted server
  paper calls this a "hash tree"

content-hash storage is not enough for a read/write file system:
  a key's value cannot change, since key = hash(v)
  "immutable"

Digital signatures (e.g. RSA) allow mutable storage
  each user has a public/private key pair
  sig = sign(data, k_priv)
  ok = verify(data, sig, k_pub)
  client write(k,v) OR UPDATE(k,v):
    put(k, v+sign(v, k_priv))
  client read(k):
    v+sig = get(k)
    if verify(v, sig, k_pub) != true: reject!
    return v
  note k and k_priv are totally separate
  the value can be modified w/o changing k
  anyone who knows k and k_pub can fetch the value and
    can check that owner of k_priv signed that value
    (though not quite "v is the correct value for k"!)

Are digital signatures alone enough?
  the server cannot forge a signature, so it cannot create arbitrary fake values
  the server *can* return an old correctly-signed value
    so it can hide an update
    or show different signed versions to different readers
    or show old versions for some keys, new for other keys
    these are serious security problems
  also we'd like a file system, not just individual data items
    file names, directories, multiple users, permissions, &c

Example of a stale value causing a problem:
  Open source project, two files: x.c, NEWS
  Storage server owned by someone else
  X: x.c -- v1
  A: x.c -- v2 fixes a bug
  B: NEWS -- announce "bug fixed!"
  C: read NEWS
  C: read x.c
  Server could provide new NEWS but *old* x.c!
    Signatures would be correct! But it's the wrong x.c.
  SUNDR can prevent this!

One of SUNDR's ideas:
  Each update includes digital signature over entire file-system (FS).
  Thus B's update to NEWS includes a signature reflecting A's updated x.c
  So if C sees B's updated NEWS,
    signature will only check if C also sees A's new x.c

Strawman design (Section 3.1)
  Server stores a complete log of file-system modifications
    create, write, rename, delete, mkdir, &c
  Clients ask server to append new operations.
  Clients reconstruct FS content by fetching and playing entire log.
  Clients also ask server to append "fetch" log entries.

Strawman details
  Log entries: fetch or modify, user, sig.
    Signature covers the entire log up to that point.
  Client step:
    Fetch entire log (other clients now wait).
    Check the log:
      Correct signature in each entry, covering prior log.
      Each operation is allowed by permissions.
      This client's last log entry is present (so client must remember!).
    Construct FS state by replaying logged operations, from start.
    Append client's operation with signature over whole resulting log.
    Upload log (then other clients can proceed).
    Client remembers its signed new entry (on local disk).
  Inefficient but simple to reason about.

Example straw-man log:
  mod(x.c), X, sig   -- X creates x.c
  mod(x.c), A, sig   -- A applies a crucial security patch
  mod(NEWS), B, sig  -- B announces the patch

Could the server forge a log entry?
  No: it would need to know the private key of an authorized user.

How do clients know the public keys of authorized users?
  1. All clients must be told public key of root directory owner.
  2. File system itself stores the public key of each user.

Could the server hide an entry from C?
  Yes.
  Hiding and showing are the only bad things a SUNDR server can do.

What if the server hides A's and B's updates from C?
  C will read,
  see just X's mod(x.c),
  ask to append a fetch() with a signature over what it saw,
    fetch(), C, sig
  and remember its last operation (fetch(), C, sig).
  [diagram: start of a forked log]

Server can continue to let C extend a "forked" log
  I'm assuming server is malicious, and has been modified to trick C

Suppose later the server wants to show B's updated NEWS to C?
  It won't work for the server to show C the "real" log:
    mod(x.c), X, sig
    mod(x.c), A, sig
    mod(NEWS), B, sig
    because C requires that its last operation (its fetch) be in the log.
  It won't work for the server to insert NEWS before C's fetch:
    mod(x.c), X, sig
    mod(NEWS), B, sig
    fetch(), C, sig
    Because C's fetch signature didn't include A's mod(x.c).
  Nor NEWS after C's fetch:
    mod(x.c), X, sig
    fetch(), C, sig
    mod(NEWS), B, sig
    Because B's mod()'s signature didn't include C's fetch.

So the server cannot ever show C B's mod(NEWS).
  Or any future B operation, since B will generate signatures
  that include its mod(NEWS).

The server can't show B any future C operation either.
  Since it can't show B C's fetch(), since it's not compatible
  with B's mod(NEWS).
  And any future C operations will have signature covering C's fetch.

But the server *can* continue to give C a separate view of the file system.
  Containing only C's future operations.
  But not B's operations.

This is SUNDR's "fork consistency":
  A server can permanently fork users, concealing each others' updates.
  But the server cannot heal a fork -- forked forever.

Is fork consistency a good outcome?
  It's not ideal that SUNDR allows fork attacks.
  But a fork will be disruptive, since hiding an update hides
    all future updates from that user.
    So likely to be detected after a while.
  Users can detect if they can communicate outside of SUNDR.
    e.g. e-mail asking "what do you think of my last update?"
  Given assumptions, it seems the best one can do.

The Question: why include fetch() in the log?

Strawman is not practical: the log keeps growing!

Here's a simplified overview of Section 3.3 (Serialized SUNDR).

Idea: store current file-system (FS) state, not log of operations.
  (Figure 2)
  A hash tree mimicing an FS directory/file tree.
    [diagram: files, directories, handle w/ signed root hash]
    Can't be directly updated, but...
  A signed "handle" object containing the current root hash.
    Under a known unchanging key.
  To update: create a new hash tree, sign root hash and update handle,
    delete old tree.

But: multiple users, each with their own public/private key.
     users might not fully trust each other.
     so a single writeable-by-all root is too fragile.

Idea: a tree per user (and signed handle), FS is union of trees.
  Each user's tree holds just that user's files.
  A read fetches all trees, merge to form single FS.

But: we need to enforce fork consistency.
  i.e. detect if server is trying to heal a fork.
  i.e. detect if clients saw different sequences of updates.

User X's tree needs to declare what other trees X saw
  when X last updated or read the file-system.
  So clients can check that different user's trees form a linear sequence.
  (like straw-man's signatures over the whole log)

Idea:
  Each user's tree goes through a sequence of numbered versions.
  User X's signed handle includes (paper's Version Structure, Figure 3):
    X's root hash.
    X's version number.
    A "version vector" with one number for each other user.
      when X last read, the version it saw of each other user's tree.
    signature (just on the handle block)
  Then can compare numbers in X's and Y's version vectors to decide
    do tree updates form a sequence, with each seeing previous?
    is server trying to reveal a previously hidden tree?

Example (when server is well-behaved):
  "A1" is a new signed tree
  "1 0 0" is the version vector in A1
  time flows downward

    A  B  C
    -------
A1  1  0  0
B1  1  1  0
C1  1  1  1
B2  1  2  1

The version vectors advance one slot (user) at a time.
The fact that B's latest vv is "1 2 1" means B saw
  A's version 1 and C's version 1.
Anyone who sees B's latest update will then know to
  demand at least those versions for A's and C's tree.
The server can't forge vv's b/c everything is signed.
  But it can hide the latest trees: serve old signed handles.

Note in example that vv's can be ordered (if server is well-behaved):
  Each is >= than the one before in all positions.
  Meaning that the updates occured in order,
    and each saw the previous updates.
  Clients check that vv's are orderable.

What if the server conceals a tree update?
  suppose it conceals B1 from C (it can do that).

    A  B  C
    -------
A1  1  0  0
B1  1  1  0   C1  1 0 1

The server cannot afterwards reveal C's update to B.
  That would reveal C's vv to B.
  B would see that B's and C's vv's cannot be ordered.
  That is, B1 then C1 is not correct, since then B1 was hid from C.
  And C1 then B1 is not correct, since then C1 was hid from B.

And indeed SUNDR's version vector scheme enforces fork consistency.
  And it's much cheaper than the straw-man signatures over the whole log.

Take-away ideas:
  The dream of trustworthy service from untrusted servers.
  Fork attacks.
  Hash trees.
  Version vectors.

Next lecture:
  Bitcoin versus fork attacks.

6.5840 2026 Lecture 9: Zookeeper Case Study

Reading: "ZooKeeper: Wait-free coordination for Internet-scale
systems", Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, Benjamin
Reed. 2010 USENIX Annual Technical Conference.

today's lecture considers ZooKeeper from two angles:
  * a simpler foundation for fault-tolerant applications.
  * an example use of Raft-like replication
  ZooKeeper is very widely used, so worth paying attention
  Etcd, another popular coordination service, is influenced by ZooKeeper's design

if we wanted to make a fault-tolerant service like MR coordinator,
  we could replicate with Raft, and that would be OK!
  [diagram: Raft-replicated MR coordinator, workers]

but building directly on Raft is hard
  a replicated state machine is awkward to program
  everything framed as events, commit them, then execute them
  is there a simpler way?

you can think of state machine replication (Raft) as replicating
  the computation; the state is replicated as a side-effect.

can we have fault-tolerance without replicating computation?
  yes!
  ordinary non-replicated server
  server maintains state in fault-tolerant storage service
  server crash -> new server, reads state from storage

ZooKeeper is designed to be the required fault-tolerant storage
  [MR coord, workers, ZK black box]
  MR coordinator can be written in ordinary straight-line code
    write state updates to ZK
    much like saving in a file

what might MR coord store in ZK?
  coord's IP addr, set of jobs, status of tasks, set of workers, assignments
  update data in ZK on each change
  (but big data itself in GFS, not ZK)
  workers can read coord's IP address, maybe even task assignments, from ZK
  MR using ZK for "configuration management"
    keep track of a collection of servers
    help servers find each other

what if MR coord fails?
  we weren't replicating it on a backup coord server
  but we don't need one!
  just pick any computer, start MR coord s/w on it,
    have it read state from ZK.
  new coord can pick up where failed one left off.
  makes a lot of sense in a big cloud
    easy to allocate a replacement server

challenges
  * failure detection (of MR coord)
  * election (just one MR coord at a time -- no split brain!)
  * recover/repair state from ZK (old MR coord might
    have been in the middle of updating it)
  * deal with possibility old MR coord is still alive and active!
  * performance
  ZK helps with all of these

ZooKeeper server arrangement
  [ZK leader, ZK followers, clients, writes, ZXIDs, reads, watches]
  Raft-like leader
  Raft-like log, commit, replicated execution and state
  ZK leader chooses order for incoming writes,
    assigns ZXIDs,
    all followers execute writes in the same order
  ZK followers execute client reads (reads are not sent to ZK leader)

for now I'll treat ZK as a black box

Zookeeper data model (Figure 1)
  a file-system-like tree of znodes
  znode names, znode content, children, path names
    names and hierarchy help different apps avoid interfering
  each znode has a version number
  types of znodes:
    regular
    ephemeral
    sequential: name + seqno

Operations (Section 2.2)
  s = openSession()
  create(s, path, data, flags)
    exclusive -- fails if path already exists
  exists(s, path, watch)
    watch=true asks for notification if path is later created/deleted
  getData(s, path, watch) -> data, version
  setData(s, path, data, version)
    if znode.version = version, then update
    (same version scheme as Lab 2)
  getChildren(s, path, watch)
  exception if the ZK server says it has terminated the session
    so that application won't continue

ZooKeeper API designed for synchronization and concurrent access:
  + exclusive znode creation; exactly one concurrent create returns success
  + getData()/setData(x, version) supports mini-transactions
  + sessions/ephemeral help cope with client failure
  + sequential znodes create order among multiple clients
  + watches avoid costly polling

Example: MapReduce coordinator election
  this is the paper's Simple Lock in Section 2.4
    s = openSession()
    while true:
      if create(s, "/mr/c", ephemeral=true)
        // we won this election and are now coordinator
        setData(s, "/mr/ip", ...)
        setData(s, "/mr/...", ...)
    ...
    break
      else if exists(s, "/mr/c", watch=true)
        // we lost this election
        wait for watch event

note:
  exclusive create
    if multiple clients concurrently attempt, only one will succeed
  ephemeral znode
    coordinator failure automatically lets new coordinator be elected
  watch
    potential replacement coordinators can wait w/o polling

what do we want if the elected coordinator fails?
  * elect a replacement
  * cope with crash in the middle of updating state in ZK
  * cope with possibility that the coordinator *didn't* fail!
  even though /mr/c looks like a lock,
    the possibility of coordinator failure makes the situation
    different from e.g. Go sync.Mutex

what does ZK do on failure of client (e.g. MR coordinator)?
  client failure -> client stops sending keep-alive messages to ZK
  no keep-alives -> ZK leader times out and terminates the session
  session termination -> ZK leader deletes session's ephemeral znodes
                         *and* ignores further requests from that session
                         (ephemeral deletions are A-linearizable ZK ops)
  now a new MR coordinator can elect itself

what if the MR coordinator crashed while updating state in ZK?
  this is crash recovery, related to DB logging; requires care.
  simplest: MR coord stores all info in a single ZK znode
    individual setData() calls are atomic (all or nothing vs failure)
  what if MR coord stores state in multiple znodes?
    use paper's "ready" znode scheme (Section 2.3)
      delete "ready"; update znodes; create "ready"
      leader:                          worker:
         delete(s, "ready")
     setData(s, z1)
     setData(s, z2)                if exits("ready", watch=ready):
     create(s, "ready)                read z1
                                  read z2

    newly elected MR coord can then tell if update was partial
  better: write entirely new set of znodes, then update
    a znode that indicates which set is current
  all three end with "single committing write"

what if the old coordinator is alive and thinks it is still coordinator?
  but ZK has decided it is dead and deleted its ephemeral /mr/c znode?
  and a new coordinator is elected?
  will two computers think they are the coordinator?
    this could happen.
  can the old coordinator modify state in ZK?
    this cannnot happen!
  when ZK times out a client's session, two things happen atomically:
    ZK deletes the client's ephemeral nodes.
    ZK stops listening to the session -- will reject all operations.
  so old coordinator can no longer modify or read data in ZK!
    if it tries, its client ZK library will raise an exception
    forcing the client to realize it is no longer coordinator

"Fencing" is a term for ignoring requests from a client declared dead
  even if it is actually alive

an important pattern in distributed systems:
  a single entity (e.g. ZK) decides which computers are alive or dead
    "failure detector"
  it may not be correct, e.g. if the network drops messages
  but everyone obeys its decisions
  agreement is more important than being right, to avoid split brain
  but possibility of being wrong => need to fence
    thus ZK's session termination

how is ZK designed for good performance?
  optimized primarily for read/watch performance
    write performance is secondary
  [diagram: leader, lots of followers, clients talk to followers]
  1) many ZK follower servers; clients are spread over them for parallelism
     client sends all operations to its ZK follower
     ZK follower executes reads locally, from its replica of ZK data
       to avoid loading the ZK leader
     ZK follower forwards only writes to ZK leader
  2) watch, not poll
     the ZK follower (not the ZK leader) keeps watch info
  3) clients of ZK can launch async operations
     i.e. send request; completion notification delivered to code separately
          unlike RPC
     a client can launch many ops without waiting
     ZK processes async ops efficiently in a batch; fewer msgs, disk writes
     client library numbers them, ZK executes them in that order
     e.g. to update a bunch of znodes then create "ready" znode

a ZK read may not see latest completed writes!
  since client's follower may be behind (not in write's majority)
  when is it ok for reads not to see recent writes?
    data merely displayed to humans
    read-only data
    data that can be checked e.g. GFS chunk-server
  when is it not ok?
    read-modify-write e.g. to increase a counter
    when a group of items needs to be consistent

ZK does provide some guarantees for reads
  every client sees writes appear in the same order (ZXID)
  a client read sees all of its own previous writes
    so follower may have to delay a read
  a client's reads move only forward in time (by ZXID)
    even if client switches ZK followers!
  "client FIFO order"
     good enough the examples above

Some implementation details related to performance:
  Data must fit in memory, so reads are fast (no need to read disk).
    So you can't store huge data in ZooKeeper.
  ZK logs writes to disk.
    So committed updates aren't lost in a crash or power failure.
    Hurts performance; batching can help throughput.
  Periodically, ZK writes complete snapshots to disk.
    So it can truncate on-disk log.
    Fuzzy technique allows snapshotting concurrently with write operations.

How is the performance?

Figure 5 -- throughput.
  Overall, can handle 10s of thousands of operations / second.
    Is this a lot? Enough?
  Why do the lines go up as they move to the right?
  Why does the x=0 performance go down as the number of servers increases?
  Why does the "3 servers" line change to be worst at 100% reads?
  What might limit it at x=0 to 20,000?
    Each op is a 1000-byte write...

What about latency?
  Why might we care about latency? What's a good target?
  Table 2 / Section 5.2 implies 1.3 milliseconds (1 / 776).
    For a single worker (client) waiting after each write request.
  Where might the 1.3 milliseconds come from?
    Disk writes? Communication? Computation?
    (How can it be this fast, given mechanical disk rotation times?)
  Why only ~2000 req/s rather than Figure 5's 20,000?

How long to recover from a ZK server failure?
  Is this an important consideration?
  Figure 8
  Leader failure -> a pause for timeout and election.
    Visually, on the order of a few seconds.
  Follower failure -> brief decrease in total throughput.
    Why not a long pause for timeout?
  What are the leader recovery time tradeoffs/obstacles likely to be?

ZooKeeper has been very successful
  good foundation for building fault-tolerant applications
  see ZooKeeper's Wikipedia page for a list of projects that use it

Areas it could be improved?
  too bad reads aren't linearizable
  sessions are pretty blunt; maybe better to have per-znode leases
  not easy to shard:
    znode tree has no obvious places to slice
    sessions are global
  multi-znode transactions would be nice
  see etcd and consul for other design decisions

Next week:
  distributed transactions

References:
  https://zookeeper.apache.org/doc/r3.4.8/api/org/apache/zookeeper/ZooKeeper.html
  ZAB: http://dl.acm.org/citation.cfm?id=2056409
  https://zookeeper.apache.org/
  https://cs.brown.edu/~mph/Herlihy91/p124-herlihy.pdf  (wait free, universal
  objects, etc.)

6.5840 2026 Lecture 1: Introduction

6.5840: Distributed Systems Engineering

A "distributed system":
  a group of computers cooperating to provide a service

Examples:
  popular apps' back-ends, e.g. for messaging
  big web sites
  cloud providers

Focus here is distributed infrastructure:
  storage
  transaction systems
  "big data" processing frameworks

Hard to build:
  concurrency
  complex interactions
  performance bottlenecks
  partial failure

Why useful?
  to increase capacity via parallel processing
  to tolerate faults via replication
  to match distribution of physical devices e.g. sensors
  to increase security via isolation

Why take this course?
  interesting -- hard problems, powerful solutions
  big demand -- driven by the rise of big Web sites
  active research area -- important unsolved problems
  challenging -- the labs

COURSE STRUCTURE

http://pdos.csail.mit.edu/6.5840

Course staff:
  Frans Kaashoek and Robert Morris, lecturers
  Baltasar Dinis, TA
  Ayana Alemayehu, TA
  Upamanyu Sharma, TA
  Yun-Sheng Chang, TA
  Danny Villanueva, TA
  Brian Shi, TA
  Nour Massri, TA
  Beshr Islam Bouli, TA

Lectures:
  paper discussion, context, lab guidance

Papers:
  one per lecture
  research papers, some classic, some new
  ideas, problems, implementation details, evaluation
  please read papers before class!
  web site has a question about each paper
    submit your answer before start of lecture
  optionally, submit a question for us

Exams:
  Mid-term exam in class
  Final exam during finals week
  papers, lectures, and labs
  You must attend the exams!

Labs:
  goal: deeper grasp of some important techniques
  goal: experience with distributed programming
  first lab is due a week from Friday
  one per week after that for a while

Lab 1: distributed big-data framework (like MapReduce)
Lab 2: client/server vs unreliable network
Lab 3: fault tolerance using replication (Raft)
Lab 4: a fault-tolerant database
Lab 5: scalable database performance via sharding

We grade the labs using a set of tests
  we give you all the tests; none are secret

Optional final project at the end, in groups of 2 or 3.
  The final project substitutes for Lab 5.
  You think of a project and clear it with us.
  Code, short write-up, demo on last day.

Warning: debugging the labs can be time-consuming
  start early
  ask questions on Piazza
  TA office hours

MAIN TOPICS

This is a course about infrastructure.
  * Storage.
  * Communication.
  * Computation.

A big goal: hide the complexity of distribution from applications.

Topic: fault tolerance
  1000s of servers, big network -> constant failures
    We'd like to hide these failures.
    "High availability": service continues despite failures
  Big idea: replication.
    If one server crashes, can proceed using the other(s).

Topic: consistency
  General-purpose infrastructure needs well-defined behavior.
    E.g. "read(x) yields the value from the most recent write(x)."
  Guaranteeing specified behavior is hard!
    e.g. "replica" servers are hard to keep identical.

Topic: performance
  A common goal: scalable throughput
    Nx servers -> Nx total throughput via parallel CPU, RAM, disk, net.
  Scaling gets harder as N grows
    e.g. load imbalance.

Topic: tradeoffs
  Fault-tolerance, consistency, and performance are enemies.
  Fault tolerance and consistency require communication
    e.g., send data to backup server
    e.g., check if cached data is up-to-date
    but communication is often slow and hard to scale up
  Many designs sacrifice consistency to gain speed.
    e.g. read(x) might *not* yield the latest write(x)!
    Painful for application programmers (or users).
  We'll see many consistency/performance design points.

Topic: implementation
  RPC, threads, concurrency control.
  The labs...

CASE STUDY: MapReduce

Let's talk about MapReduce (MR)
  a good illustration of 6.5840's main topics
  hugely influential
  the focus of Lab 1

Context: multi-hour computations on multi-terabyte data-sets
  e.g. build search index, or sort, or analyze structure of web
  only practical with 1000s of computers

A big goal: easy for non-specialist programmers
  programmer just defines Map and Reduce functions
  often simple sequential code
  MR manages, and hides, all aspects of distribution!
  MR is a framework / library; "application" is just Map()/Reduce()

Abstract view of a MapReduce job -- word count
  Input1 -> Map -> a,1 b,1
  Input2 -> Map ->     b,1
  Input3 -> Map -> a,1     c,1
                    |   |   |
                    |   |   -> Reduce -> c,1
                    |   -----> Reduce -> b,2
                    ---------> Reduce -> a,2
  1) input is (already) split into M pieces
  2) MR calls Map() for each input split, produces list of k,v pairs
     "intermediate" data
     each Map() call is a "task"
  3) when Maps are done,
     MR gathers all intermediate v's for each k,
     and passes each key + values to a Reduce call
  4) final output is set of <k,v> pairs from Reduce()s

Word-count code
  Map(d)
    chop d into words
    for each word w
      emit(w, "1")
  Reduce(k, v[])
    emit(len(v[]))

MapReduce scales well:
  N "worker" computers (might) get you Nx throughput.
    Maps()s can run in parallel, since they don't interact.
    Same for Reduce()s.
  Thus more computers -> more throughput -- very nice!

MapReduce hides much complexity:
  sending map+reduce code to servers
  tracking which tasks have finished
  "shuffling" intermediate data from Maps to Reduces
  balancing load over servers
  recovering from crashed servers

To get these benefits, MapReduce restricts applications:
  Only one pattern (Map -> shuffle -> Reduce).
  No interaction or state (other than via intermediate output).
  Only batch: no real-time or streaming processing.

Some details (paper's Figure 1)

Input and output are stored on the GFS cluster file system
  MR needs huge parallel input and output throughput.
  GFS splits files over many servers, many disks, in 64 MB chunks
    Maps read in parallel
    Reduces write in parallel
  GFS replicates data on 2 or 3 servers, for fault tolerance
  GFS is a big win for MapReduce

MR writes Map() output to local disk
  MR splits into files by hash(key) mod R
  each "hash bucket" contains multiple keys
  The map workers all hash the same way

The shuffle
  each Reduce task processes one hash bucket
  MR fetches each Reduce tasks' bucket from every Map worker
  merge, sort by key, call Reduce() for each key
  each Reduce task writes a separate output file on GFS

The "Coordinator" manages all the steps in a job.
  tracks state of each task
  hands out tasks to worker machines

What will limit performance?
  We care since that limit is the thing to optimize.
  CPU? memory? disk? network?

In 2004 authors were limited by network speed.
  What does MR send over the network?
    Maps read input from GFS.
    Reduces fetch Map intermediate output.
      Often as large as input, e.g. for sorting.
    Reduces write output files to GFS.

How fast was the paper's network?
  Section 5.1: 1800 machines, two-level switched network
  [diagram: root switch, 2nd level of switches, machines]
    each switch must have had ~42 ports (square root of 1800)
  MR's shuffle requires every worker to fetch data from every other
    Only 1/42nd stays in local switch
  So MR's shuffle sends most data through root switch.
  Paper's root switch: 100 to 200 gigabits/second, total
    1800 machines, so ~55 megabits/second/machine.
    55 is small: less than disk or RAM speed.

How does MR minimize network use?
  Coordinator tries to run each Map task on GFS server that stores its input.
    All computers run both GFS and MR workers
    So Map input is usually read from GFS data on local disk, not over network.
  Intermediate data goes over network just once.
    Map worker writes to local disk.
    Reduce workers read from Map worker disks over the network.
    (Storing it in GFS would require at least two trips over the network.)

How does MR get good load balance?
  Why do we care about load balance?
    If one server has more work than others, or is slower,
    then other servers will lie idle (wasted) at the end, waiting.
  So ideally MR divides work so that all workers finish at same time.
  But tasks vary in size, and computers vary in speed.
  Solution: many more tasks than worker machines.
    Coordinator hands out new tasks to workers who finish previous tasks.
    So faster servers do more tasks than slower ones.
    And slow servers are given less work, reducing impact on total time.

What about fault tolerance?
  What if a worker computer crashes?
  We want MR framework to hide failures.
  Does MR have to re-run the whole job from the beginning?
    Why not?
  Coordinator re-runs just the failed Map()s and Reduce()s.

Suppose MR runs a Map task twice, one Reduce sees first run's output,
    but another Reduce sees the second run's output?
  The two Map executions had better produce identical intermediate output!
  Map and Reduce should be pure deterministic functions:
    they are only allowed to look at their arguments/input.
    no state, no file I/O, no interaction, no external communication,
      no random numbers.
  Programmer is responsible for ensuring this determinism.

Other failures/problems:
  * What if the coordinator gives two workers the same Map() task?
    perhaps the coordinator incorrectly thinks one worker died.
    it will tell Reduce workers about only one of them.
  * What if the coordinator gives two workers the same Reduce() task?
    they will both try to write the same output file on GFS!
    atomic GFS rename prevents mixing; one complete file will be visible.
  * What if a single worker is very slow -- a "straggler"?
    perhaps due to flakey hardware.
    coordinator starts a second copy of last few tasks.
  * What if a worker computes incorrect output, due to broken h/w or s/w?
    too bad! MR assumes "fail-stop" CPUs and software.
  * What if the coordinator crashes?

Performance?
  Figure 2
  X-Axis is time
  Y-Axis is total rate at which a "grep"-style job reads its input
  A terabyte (1000 GB) of input
  1764 workers
  30,000 MB/s (30 GB/s) is huge!
  Why 30,000 MB/s?
    17 MB/s per worker machine -- 140 megabits/second
    more than our guess (55 mbit/s) of net bandwidth
    input probably read direct from two local GFS disks
    so each disk probably could read at about 9 MB/second
  Why is the main period of activity about 30 seconds?
  Why does it take 50 seconds for throughput to reach maximum?

Current status?
  Hugely influential (Hadoop, Spark, Lab 1, &c).
  Probably no longer in use at Google.
    Replaced by Flume / FlumeJava (see paper by Chambers et al).
    GFS replaced by Colossus (no good description), and BigTable.

Next lecture:
  Programming: Go, Threads, RPC

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda(specificallycontainer loading and snapstart)
Marc BrookerVP/Distinguished Engineermbrooker@amazon.com

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources.”

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why?•Building scalable, fault-tolerant systems is hard.•Driving high hardware utilization is hard.•Multi-tenancy makes it easier.•The cloud needs glue.•Code close to data is more efficient?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“… simple …”“… easy …”“… fast …”

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“… complex …”“… complicated …”“… difficult …”

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BeforeAfter250MB max10GB maxCode or .zipContainer imageCustom toolsStandard tools

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BeforeAfter250MB max10GB maxCode or .zipContainer imageCustom toolsStandard toolswithout slower cold starts!

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
150 Pb/s

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Harter et al: Only 6.4% of container data is needed at startup!
Harter et al, Slacker: Fast Distribution with Lazy Docker Containers, FAST’16

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Find a way, invisible to applications, to load data on demand and deduplicate common data.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer’s Function CodeMicroVM
Firecracker
virtio block
Agache et al, “Firecracker: Lightweight Virtualization for Serverless Applications”, AWS, 2020

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer’s Function CodeMicroVM
Firecracker
virtio block
Insert lies here

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
But…Container images aren’t filesystems.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DeterministicFlatteningExt4filesystem image
512kBchunks


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
But…Real storage is really fast (<100μs), so loading on demand must be really fast too.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda workerReally fast storage with the whole dataset.
<1ms

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda worker


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
This is fake data, aimed to give the general idea, because I can’t share the real numbers.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On-worker L1
In-AZ L2In S3 (aka L3)
This is fake data, aimed to give the general idea, because I can’t share the real numbers.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer’s CodeMicroVMFirecracker
Local CacheL1
Lambda Worker
AZ-LocalShared CacheL2Authority: S3L3


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Big?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jim Gray and Franco Putzolu, IBM Research, 1977 (picture thanks to Pat Helland)
cost of caching = cost of reloading

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Max(size needed for cost,size needed for latency)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What about failures?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Figure shamelessly borrowed from Decandia et al, Dynamo: Amazon’s Highly Available Key-value Store, SOSP’07

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
•One Copy•Not durable enough.•Two Copies•2x the cost! (or, half the effective cache size)

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Erasure Code“Any 4 of 6”

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
p99

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What about chunks of other stuff?Like memory?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brooker, et al, Restoring Uniqueness in MicroVM Snapshots, https://arxiv.org/pdf/2102.12892

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Brooker, et al, Restoring Uniqueness in MicroVM Snapshots, https://arxiv.org/pdf/2102.12892


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
userfaultfd - create a file descriptor for handling page faults in user space

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Not only cold starts!Tenant Isolation (for multi-tenant services handling untrusted code).Session isolation (e.g. for AI agents).

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions46


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Erasure CodingvsComplexity

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
•Deterministic Flattener•Can only put chunks, not read them.•Worker•Can only read chunks for the functions it is running.•Everybody else•Can do basically nothing.

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chunk data
Chunk name = SHA2()

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chunk data
Chunk name = SHA2(AES())KeyIV

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chunk ciphertext = AESgcm(     key = f(     iv = [0,…])Chunk data,Chunk data),Douceur et al, “Reclaiming Space from Duplicate Files in a Serverless Distributed File System”, Microsoft, 2002

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
chunk ciphertext
Chunk name = SHA2()

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
chunk ciphertext
Chunk name = SHA2()Q: Why not use the GCM tag?

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
chunk 1, name, keychunk 2, name, key…chunk N, name, keyCustomer’s KMS Key

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
chunk 1, name, keychunk 2, name, key…chunk N, name, keyCustomer’s KMS Key


© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
•Deterministic Flattener•Can only put chunks, not read them.•Worker•Can only read chunks for the functions it is running.•Everybody else•Can do basically nothing.

--- Paxos Proposer ---

     1  proposer(v):
     2    while not decided:
     2      choose n, unique and higher than any n seen so far
     3      send prepare(n) to all servers including self
     4      if prepare_ok(n, na, va) from majority:
     5        v' = va with highest na; choose own v otherwise
     6        send accept(n, v') to all
     7        if accept_ok(n) from majority:
     8          send decided(v') to all


        --- Paxos Acceptor ---

     9  state on each node (persistent):
    10   np     --- highest prepare seen
    11   na, va --- highest accept seen

    12  prepare(n) handler:
    13   if n > np
    14     np = n
    15     reply prepare_ok(n, na, va)
    16   else
    17     reply reject


    18  accept(n, v) handler:
    19   if n >= np
    20     np = n
    21     na = n
    22     va = v
    23     reply accept_ok(n)
    24   else
    25     reply reject

This paper is included in the Proceedings of the
2023 USENIX Annual Technical Conference.
July 10–12, 2023 • Boston, MA, USA
978-1-939133-35-9
Open access to the Proceedings of the
2023 USENIX Annual Technical Conference
is sponsored by
On-demand Container Loading in AWS Lambda
Marc Brooker, Mike Danilov, Chris Greenwood, and
Phil Piwonka, Amazon Web Services
https://www.usenix.org/conference/atc23/presentation/brooker

On-demand Container Loading in A WS Lambda
Marc Brooker
Amazon Web Services
Mike Danilov
Amazon Web Services
Chris Greenwood
Amazon Web Services
Phil Piwonka
Amazon Web Services
Abstract
AWS Lambda is a serverless event-driven compute service,
part of a category of cloud compute offerings sometimes
called Function-as-a-service (FaaS). When we first released
AWS Lambda, functions were limited to 250MB of code and
dependencies, packaged as a simple compressed archive. In
2020, we released support for deploying container images
as large as 10GiB as Lambda functions, allowing customers
to bring much larger code bases and sets of dependencies
to Lambda. Supporting larger packages, while still meeting
Lambda’s goals of rapid scale (adding up to 15,000 new con-
tainers per second for a single customer, and much more in
aggregate), high request rate (millions of requests per second),
high scale (millions of unique workloads), and low start-up
times (as low as 50ms) presented a significant challenge.
We describe the storage and caching system we built, op-
timized for delivering container images on-demand, and our
experiences designing, building, and operating it at scale. We
focus on challenges around security, efficiency, latency, and
cost, and how we addressed these challenges in a system
that combines caching, deduplication, convergent encryption,
erasure coding, and block-level demand loading.
Since building this system, it has reliably processed hun-
dreds of trillions of Lambda invocations for over a million
AWS customers, and has shown excellent resilience to load
and infrastructure failures.
1 Introduction
AWS Lambda is a serverless event-driven compute service,
part of a category of cloud compute offerings sometimes
called Function-as-a-service (FaaS). First launched in 2015,
today AWS Lambda functions run millions of times per sec-
ond over millions of unique customer workloads. One factor
that attracts customers to Lambda is its ability to scale up
to handle increased load, typically in less than one second
(and often as quickly as 50ms). This scale-up time, which
customers have come to refer to as cold-start time, is one of
the most important metrics that determine the customer ex-
perience in FaaS systems. When we launched AWS Lambda,
we recognized that reducing data movement during these cold
starts was critical. Customers deployed functions to Lambda
in compressed archives (.zip files), which were unpacked as
each function instance was provisioned. As Lambda evolved,
and customers increasingly looked to deploy more complex
applications, there was significant demand for larger deploy-
ments, and the ability to use container tooling (such asDocker)
to create and manage these deployment images. Customers
also wanted Lambda to support these images without compro-
mising on cold-start performance.
Adding container support to AWS Lambda without regress-
ing on cold-start time presented a significant technical chal-
lenge for our team. The core challenge is simply one of data
movement. Today, Lambda can start up to 15,000 containers a
second [18] for production workloads, and we expect to scale
further for future workloads. Simply moving and unpacking
a 10GiB image for each of these 15,000 containers would
require 150Pb/s of network bandwidth. To achieve scalability
and cold-start latency goals, we needed to take advantage of
three factors which simplify this problem:
Cacheability While Lambda serves hundreds of thousands
of unique workloads, large scale-up spikes tend to be
driven by a smaller number of images, suggesting that
the workload is highly cacheable.
Commonality Many popular images are based on common
base layers (such as our own AWS base layers, or open
source offerings like Alpine). Caching and deduplicating
these common base layers reduce data movement for all
containers that build on them.
Sparsity Most container images contain a lot of files, and
file contents, that applications don’t need at startup (or
potentially never need). Harter et al [15] found that on
average only 6.4% of container data is needed at startup.
Our solution combines caching, deduplication, erasure cod-
ing, and sparse loading to take advantage of our needs. With-
USENIX Association 2023 USENIX Annual Technical Conference    315

WorkersWorkers
Frontend
Worker
Manager
Workers
Function
Metadata
Figure 1: Architecture of the AWS Lambda invoke path
out adding any customer visible complexity (they simply up-
load a container image to a convenient repository), we were
able to achieve our scale and cold-start latency goals, while
having significant headroom for future scaling.
In this section, we present the existing architecture of AWS
Lambda, and the overall architecture of our system. Section 2
presents the low-level implementation of our sparse loading
solution. The cache architecture, and use of erasure coding
to improve scalability and tail latency is presented in Section
4. Section 3 presents our convergent encryption-based secure
deduplication architecture. Finally Section 6 compares our
solution to other approaches from academia and industry.
1.1 Existing Architecture Overview
To reduce risk and optimize time-to-market, we wanted to
introduce these new capabilities to Lambda with the minimum
amount of change to the existing architecture, as shown in
Figure 1. Requests to execute a certain function (we call
these invokes) arrive via a load-balanced stateless frontend
service. This service loads the metadata associated with the
request, performs authentication and authorization, and then
sends a request to the Worker Manager, requesting capacity.
Worker Manager is a stateful, sticky, load balancer. For every
unique function in the system, it keeps track of what capacity
is available to run that function, where that capacity is in
the fleet, and predicts when new capacity may be needed.
If capacity is available, the Worker Manager instructs the
frontend to forward the request payload to a Worker, where
the function is executed. If no capacity is available, the Worker
Manager identifies a Worker with available CPU and RAM,
and sends a request to start a sandbox for the relevant function.
Once this is complete, the frontend is notified and the function
is executed.
Each Lambda worker, as shown in Figure 2, includes a
small controller process, the Micro Manager, some additional
agents for logging and monitoring, and a large number of
MicroVMs. Each MicroVM, based on our Firecracker [3]
hypervisor, contains the code for a single Lambda function
for a single customer. Inside the MicroVM is a minimized
Customer Code
Firecracker
λ Shim
Linux Kernel
virtio
Micro
Manager
MicroVM “slot”Monitoring,
Logging, etc.
Figure 2: Architecture of the AWS Lambda worker
Linux guest kernel, a small shim that provides Lambda’s
programming model, any provided runtime (e.g. the JVM for
Java or CoreCLR for .NET), and the customer’s code and
libraries. As described in our Firecracker paper [3], the key
concern here is security: customer code and data is not trusted,
and the only communication between the workload inside
the MicroVM and the shared worker components is over a
simple, well tested, and formally verified implementation of
virtio [27, 32] (specificallyvirtio-net and virtio-blk).
In the first generation architecture (before this work), when
a new MicroVM is created with new capacity for a particular
function, the Worker downloads the function image (a.zip file
up to 250MiB in size) from Amazon S3, and unpacks it into
the MicroVM guest’s filesystem. This model is simple, and
works well for small images, but requires the full archive to be
downloaded and unpacked before the new MicroVM can do
any work. To support larger images, we wanted to avoid this
blocking download, and avoid the storage cost of unpacking
the entire archive if only part of it is used.
2 Block-Level Loading
To take advantage of the sparsity property of containers, we
needed to allow the system to load (and store) only the data the
application needs, ideally at the time it needs it. Approaches
like Slacker [15] and Starlight [8] have approached this prob-
lem at the filesystem level - a natural fit for containers, which
are built as an overlayed stack of file-level archives. This ap-
proach isn’t the right one for our environment. We believed
that the inherent complexity of filesystems, and additional
complexity of overlaying multiple filesystems, would unac-
ceptably increase the attack surface of the shared components
in Lambda. Instead, we decided to keep the block-level virtio-
blk interface between the MicroVM guest and the hypervisor,
perform all filesystem operations inside the guest. This re-
quires performing sparse loading at the block, rather than file,
level.
Figure 3 shows our high-level architecture, showing the
Lambda worker (shown in detail in Figure 4) where cus-
tomer’s code is run, container registry which contains the
primary copy of customer’s container images, and the chunk
316    2023 USENIX Annual Technical Conference USENIX Association

Container
Registry
Deterministic
Flatten
Chunk Origin
(S3)
Lambda
Worker
Lambda
Worker
Distributed
Cache
Key Store
(KMS)
Keys Chunks Container
Images
Scale: per
new function
Scale: per
invoke
Figure 3: High-level system architecture.
creation and caching infrastructure.
Our first step in supporting block-level loading is to col-
lapse the container image into a block device image. As de-
scribed in the OCI image specification [1], a container image
is a stack of tarball layers. In the typical container stack, these
layers are overlayed at at runtime using overlayfs. In our im-
plementation, we perform this overlaying operation at the
time the function is initially created, following a determinis-
tic flattening process which applies each tarball in order to
create a single ext4 filesystem. Function creation is a low-rate
control-plane process, that is typically only triggered by cus-
tomers when they make changes to their code, configuration,
or architecture. Even the most aggressive adoptees of con-
tinuous integration only make these changes on the order of
minutes, while function invocation can happen up to millions
of times a second.
The flattening process is designed so that blocks of the
filesystem that contain unchanged files will be identical, al-
lowing for block-level deduplication of the flattened images
between containers that share common base layers. We’ll
revisit this in Section 3, but the high-level reason is that differ-
ences between functions (and even more so between versions
of the same function) are typically much smaller than the
functions themselves. The flattening process proceeds by un-
packing each layer onto an ext4 filesystem, using a modified
filesystem implementation that performs all operations deter-
ministically. Most filesystem implementations take advantage
of concurrency to improve performance, introducing non-
determinism. Ours is serial, and deterministically chooses
normally-variable parameters like modification times.
Following the flattening process, the flattened filesystem
is broken up into fixed-size chunks, and those chunks are
uploaded to the origin tier of a three-tiered cache for later use
Customer Code
Firecracker
λ Shim
Guest Linux Kernel
virtio
MicroVM “slot”
ext4 Filesystem
Local Agent
Per-function resources
Worker Local
Cache
To shared cache
Worker
Figure 4: Lambda worker with per-worker, per-customer, and
in-guest components
(we use S3 as this origin tier). Chunks in the shared storage
are named according the their content, ensuring that chunks
with the same content have the same name and can be cached
once. This scheme, described in detail in Section 3, allows
efficient deduplication of chunk content in storage and cache
layers without requiring a central directory or index of chunks.
Each fixed-size chunk is 512KiB. Smaller chunks lead to
better deduplication by minimizing false-sharing, and can ac-
celerate loading for workloads with highly random access pat-
terns. Larger chunks reduce metadata size, reduce the number
of requests needed to load data (hence improving throughput),
and provide natural read-ahead for sequential workloads. The
optimal value will change over time as the system evolves,
and we expect that future iterations of the system may choose
a different chunk size as our understanding of how customers
use the system evolves.
2.1 Per-MicroVM Snapshot Loading
Once chunks are created, the system needs to be able to access
the data they require from the chunks that contain that data.
As shown in Figure 4, we added two new components to
support this loading:
• A per-function local agent which presents a block device
to the per-function Firecracker hypervisor (via FUSE),
which is then forwarded using the existing virtio inter-
face into the guest, where it is mounted by the guest
kernel.
• A per-worker local cache which caches chunks of data
that are frequently used on the worker, and interacts with
the remote cache (see Section 4 for details)
When a new Lambda function is started on a worker, the
Micro Manager creates a new local agent, and a new Fire-
cracker MicroVM which contains two virtio block devices: a
USENIX Association 2023 USENIX Annual Technical Conference    317

root device which is the same for all MicroVMs, and a block
device backed by the FUSE filesystem exposed by the local
agent. The MicroVM boots, starts some supervisory com-
ponents, and then starts executing the customer code in the
container image. Each IO that this code performs (unless it
can be served from the page cache kept by the guest kernel)
turns into a virtio-blk request, which is then processed by
Firecracker, and handed off to the local agent.
The local agent handles reads by reading directly from the
local cache, if the chunk that contains the requested offset
is already present there. If not, the relevant chunk is fetched
from the tiered cache, as described in Section 4. The local
agent handles write by writing them to block overlay, backed
by encrypted storage on the worker. A bitmap is maintained
at page granularity, indicating whether data should be read
from the overlay, or from the backing container image. The
page granularity of the bitmap requires a read-modify-write
for writes from the guest which don’t cover an entire page.
This page-level copy-on-write approach allows the Mi-
croVM guest to handle both reads and writes, while keeping
the data in the local cache (and all other caching tiers) im-
mutable, allowing it to be shared across multiple guests.
3 Deduplication Without Trust
Base container images, such as the official Docker alpine,
ubuntu, and nodejs are extremely widely used: each boasts
over a billion aggregate downloads from the popular Dock-
erHub container repository1. Starting from one of these base
images, and customizing it to the special needs of the applica-
tion, is a common way to create new container images. When
a popular base image is used, the deterministic flattening pro-
cess described in Section 2 produces unique chunks for the
customized parts, and chunks for the common parts that are
identical to those produced for other images with the same
base. These shared chunks create a significant opportunity for
deduplication: if only a single copy of these chunks is stored,
less data movement is needed, less storage is consumed, and
caches are more effective.
Approximately 80% of newly uploaded Lambda functions
result in zero unique chunks, and are just re-uploads of images
that had been uploaded in the past. This appears to be pri-
marily driven by automated testing and deployment (CI/CD)
systems. Of the remaining 20% of functions that create at
least one unique chunk (and therefore aren’t just trivial re-
uploads), the mean upload contains 4.3% unique chunks, and
the median 2.5% unique chunks. Trivial all-zero chunks are
not included in these numbers: they are excluded entirely
from images at creation time.
Figure 5 shows the distribution of deduplication effective-
ness, for the top quartile (by image size) and remainder of the
population. This breakdown shows that the majority of func-
1statistics from https://hub.docker.com/, accessed July 2022
Figure 5: Empirical CDF of deduplication effectiveness at
chunk creation time, among functions that aren’t trivial re-
uploads.
tions of all sizes are heavily deduped, and a significant tail
where deduplication is not as effective. While large functions
are still effectively deduplicated, they have a smaller tail of
unique chunks. This data clearly suggests that deduplication
is worth the complexity, reducing storage by as much as 23x,
and improving effectiveness of the cache tiers (how much
cache effectiveness is improved depends on the correlation
between probability of deduplication and frequency of ac-
cess). While the 80% of functions with no unique chunks
aren’t statistically interesting, deduplicating these has a large
practical benefit, including reducing storage costs by another
5x, and boosting cache effectiveness.
3.1 Convergent Encryption
Deduplication of plaintexts is relatively straightforward.
Venti [30], dating back to 2002, used a hash of block con-
tents and a separate index to deduplicate blocks. Introducing
encryption, however, significantly complicates deduplication.
As Storer, et al [33] write:
Unfortunately, deduplication exploits identical con-
tent, while encryption attempts to make all content
appear random; the same content encrypted with
two different keys results in very different cipher-
text. Thus, combining the space efficiency of dedu-
plication with the secrecy aspects of encryption is
problematic.
One solution is to have a shared key, or keys, that can be
used to decrypt shared blocks, but this either introduces single
keys that can access a large number of blocks, or a significant
key management problem. Perhaps the hardest problem is
318    2023 USENIX Annual Technical Conference USENIX Association

minimizing trust. While AWS Lambda runs user code with
strong isolation [3], we still wish to restrict each Lambda
worker host to only being able to access the data it needs for
the functions that have been sent to it.
The authors of Farsite [2,11] developed convergent encryp-
tion as a solution to this problem. A cryptographic hash of
each block (in the case of Farsite a file block, in our case a
chunk of a flattened container image) is used to determinis-
tically derive a cryptographic key that is used for encryptng
the block. We follow this same scheme, but mix additional
metadata into the key derivation (as described in Section 3.3).
The flattening process described in Section 2 takes each
chunk, derives a key from it by computing its SHA256 digest,
and then encrypts the block using AES-CTR (with the derived
key). Here, AES-CTR is used with a deterministic (all zero)
IV , ensuring that the same ciphertext always leads to the same
plaintext. Using a deterministic IV in this context is safe,
because due to the collision resistance of SHA256, a key, IV
pair is only used on for one plaintext block [12]. A manifest of
chunks is then created, containing the offset, unique key, and
SHA256 hash of each chunk2. The manifest is then encrypted,
using AES-GCM, using a unique per-customer key managed
by AWS Key Management Service (AWS KMS). Chunks are
then named based on a function of the hash of their ciphertext,
and uploaded to the backing store (AWS S3) using that name
if no chunk of that name already exists.
In our scheme, we do not encrypt the entire manifest with
the customer’s unique key. Instead, only the key table (the
keys of each encrypted chunk) is encrypted, and the whole
document is authenticated (i.e. included in the calculation
of the AES-GCM tag as additional data). This allows the
garbage collection process to access the list of chunks in the
manifest, while having no access to the chunk keys. The size
of manifests, stored in an efficient binary format, is negligi-
ble: less than 3MiB for a 16GiB container image, or 0.02%
overhead.
This approach provides a number of desirable properties:
• Data can be deduplicated with no sharing of keys: the
keys to decrypt the customer’s manifest are unique to that
customer, and access to them (via AWS KMS) is only
provided to the workers that that particular customer’s
functions are placed on.
• Data can be deduplicated with no coordination or spe-
cial access provided to the flattening process. Flattening
processes operate independently, and the only special
operation they need is "upload this file to storage if it
doesn’t already exist".
• The scheme provides strong end-to-end integrity protec-
tion for chunks. Workers check the chunks they down-
2It may appear attractive to use an AEAD mode like AES-GCM rather
than the more expensive SHA256 in this application, but these modes do not
commonly provide collision resistance against attackers who know the data
key [10], an important property in our security scheme.
load against the MAC in the manifest, ensuring that mod-
ified ciphertexts can be detected and rejected.
3.2 Compression
Our system does not compress chunk plaintexts prior to en-
cryption. This is for two reasons. First, given the network
bandwidth available to our caches and workers the additional
latency of decompression, and difficulty of allowing random
access to compressed data, makes the latency benefit of com-
pression marginal. Second, compression before encryption
allows potential attackers to infer plaintext contents from com-
pressed sizes, a compression side channel. This risk, and the
relatively small expected benefit, means that we decided not
to implement compression (beyond trivial elision of all-zero
chunks).
3.3 Limiting Blast Radius
While deduplication has value in cost and cache performance,
it also adds some risks. Some popular chunks are widely ref-
erenced, meaning that anything that causes access to those
chunks to break or become slow, also has a very wide impact
on the system. Risks include partial (gray) failures of cache
nodes, operational issues that cause unavailability of data,
bugs in garbage collection, or corruption of data in the cache
hierarchy. Highly popular chunks also cause hot-spotting in
distributed storage. While our cryptographic scheme detects
corruption and will prevent readers from seeing corrupt data,
it does not correct it, and so corrupted data will become un-
available.
To solve this problem, we include a varying salt in the key
derivation step of our convergent encryption scheme. This
salt value can vary in time, with chunk popularity, and with
infrastructure placement (such as using different salts in dif-
ferent availability zones or datacenters). Otherwise-identical
chunks with different salt values will end up with different
keys, and therefore difference ciphertexts, and will not dedu-
plicate against each other. By controlling the frequency with
which the salt is rotated, we can continuously trade off dedu-
plication efficiency with blast radius. Salt allows us to encap-
sulate the control of deduplication entirely within the chunk
creation layer, without any other component needing to be
aware of its decisions. Salt rotation is an operational concern,
and is not needed for the security of the deduplication scheme.
3.4 Garbage Collection
A key challenge of any distributed deduplication scheme is
garbage collection: removing data from the backing store
when it is no longer actively referenced. Garbage collecting
the wrong chunk could cause wide impact across multiple cus-
tomers. Our deduplication scheme does not maintain a central
USENIX Association 2023 USENIX Annual Technical Conference    319

active
read &
write
retired
read
only
expired
alarm on
access
R1
active retired expiredR2
Copy active data
deleted
deleted
Figure 6: Lifecycle of data chunks used by the generational
garbage collector.
directory of chunk references or manifests, making exact ref-
erence counting infeasible. Past experience with distributed
garbage collection has taught us that the problem is both
complex (because the tree of chunk references is changing
dynamically) and uniquely risky (because it is the one place
in our system where we delete customer data). The approach
we took to garbage collection is based on this experience.
Our approach to garbage collection is based on the con-
cept of roots. A root is a self-contained manifest and chunk
namespace, analogous to the roots used in traditional garbage
collection algorithms. Unlike traditional GC roots, in our
system we periodically create new roots (which then get all
new data), and retire old roots (after moving any still-needed
data into a fresh root).When a customer’s container image
is converted, the manifest and set of chunks are placed in an
active root, for example R1. An active root handles both reads
and writes of data. Periodically, a new root R2 is created and
becomes active, while root R1 enters a retired state at which
point it only serves reads of data. While R1 is retired, any
manifest that is still referenced in R1 is migrated, along with
any chunks it references, to R2. Over time the manifests and
chunks in R1 that are in active use will be migrated to R2,
allowing R1 to be safely deleted. This process is repeated: R2
is retired and R3 becomes the active root and so on. Figure 6
shows this lifecycle. Moving chunks along with their manifest
ensures that if a manifest exists in root R, then all the chunks
it references do to. A unique identifier for the currently active
root is also included in the deduplication salt (Section 3.3),
ensuring that newly-created chunks in the active root are not
shared with previous roots.
Instead of deleting roots immediately after data migration
is complete, we put them into an expired state. In this state,
data is still allowed to be read, but any attempt to access data
leads to an alarm. These alarms both engage an operator and
automatically stop further deletion of data. This approach
allows us to robustly detect garbage collection issues (espe-
cially incomplete copying) in production, and quickly and
automatically stop any data from being deleted. While this
mechanism is inexact (data could be accessed after the period
the root is expired), it provides a valuable additional layer of
protection against data loss. While software bugs are rare, and
we test garbage collection changes carefully, multiple layers
of protection against customer data loss are critical in any
distributed storage system.
Having data in multiple roots does drive up storage costs,
however that additional cost is palatable for Lambda as cus-
tomers often update their functions and a large majority of
data is never migrated to a new root. The system is also ca-
pable of having multiple roots active simultaneously, which
reduces the blast radius of bugs and provides the ability to
roll out new garbage collection changes and algorithms to a
subset of manifests and their chunks.
4 Tiered Caching
When workers don’t have chunks in their local cache, they
attempt to pull them from a remote availability-zone-level
(AZ-level) shared cache (as shown in Figure 3). If chunks
aren’t in this cache, workers download them from S3, and
upload them into the cache. This AZ-level cache is a custom
implementation of a fairly standard design: chunks are fetched
over HTTP2, data storage is two-tiered with an in-memory
tier for hot chunks and a flash tier for colder chunks, and evic-
tion is LRU-k [29] (a scan-resistant variant of Least Recently
Used). Chunks are distributed to the AZ-level cache using a
variant of a consistent hashing [19] scheme, with optimiza-
tions to improved load spreading (similar to the approach of
Chen et al [7]). The caching tier improves fetch performance
considerably: from the worker’s perspective, a hit on the AZ-
level cache takes a median time of 550µs, versus 36ms for a
fetch from the origin in S3 (99.9th percentile 3.7ms versus
175ms).
Figure 7 shows the effectiveness of these three cache tiers.
Over a week of production usage in one large AWS region,
a median of 67% of chunks were loaded from the on-worker
cache, 32% from the AZ-level distributed cache, and the re-
maining 0.06% from the backing store.
The per-worker cache has a median hit rate of 67%, and a
10th percentile low hit rate over the week in question of 65%.
The in-AZ cache is even more effective, with a median hit
rate of 99.9% and 10th percentile low hit rate over the week
of 99.4%. Figure 8 shows the empirical CDF of the hit rate
of the in-AZ cache over the week, measured in one-minute
buckets across one at-scale production availability zone. The
left tail of the distribution is associated with large spikes in
traffic to newly created functions. We are evaluating priming
the in-AZ caches during the chunk creation process to flatten
this left tail and further improve hit rates, primarily with the
goal of reducing load-time latency for new functions.
4.1 Optimizing for Tail Latency
While data in the AZ-level cache is not required to be durable
(durability is ensured using S3 as the origin), a simple unrepli-
cated cache scheme (where each object is stored in a single
node) didn’t meet our needs for three reasons.
320    2023 USENIX Annual Technical Conference USENIX Association

Figure 7: One week of hit rates on each of the cache tiers:
on-worker (L1), distributed in-AZ (L2), and backing store
(L3)
Figure 8: Empirical CDF of in-AZ cache hit rate
Tail latency A single slow cache server can cause
widespread impact. Slowness could be caused by conges-
tion at the host or in the network, or by partial hardware
or software failure.
Hit Rate Drops Having each item cached in a single server
means that the hit rate drops if that server fails, or is
taken down for deployment.
Throughput Bounds Having each item cached in a single
server means that the bandwidth available to fetch the
object is bounded by a single server’s bandwidth.
Of these, tail latency is the largest practical concern. Our
experience operating these types of systems suggests that
debugging slowness and partial failure is much harder than
debugging outright failure. Even if this slowness is only in the
long tail, it still matters in production because each container
start needs to fetch a large number of chunks. For example,
a start which fetches 1000 chunks will experience the 99.9th
percentile tail latency of the cache on 63% of tasks. The
difference is material: in one deployment of the cache we
observe a median client-measured latency of 500µs, and a
99.9th percentile latency of 4ms.
Replication, combined with redundant requests is a well-
established [13, 37, 39] technique to drive down tail latency,
and would also solve our throughput and hit-rate problems.
Unfortunately, replication increases costs proportionally to
the replication factor, an important concern in a primarily in-
memory cache. Instead, we chose erasure coding, following
a similar scheme to EC-Cache [31]. Erasure coding is not
widely used in caches, but provides compelling solutions for
all three of our concerns. When a worker misses the cache,
it fetches the chunk it needs from the origin, then uploads
erasure-coded stripes of that chunk into the cache. When a
worker needs to fetch a chunk, it requests more stripes than
are strictly needed to reconstruct the chunk, and then recon-
structs the chunk as soon as enough stripes are returned. Our
current production deployment uses a 4 of 5 code, achieving
25% storage overhead, and a 25% increase in request rate in
exchange for a significant decrease in tail latency. Figure 9
compares the empirical latency CDF of the 4 of 5 code versus
a hypothetical 4 of 4 scheme using latency measurements
from one deployment of our production system.
This scheme prevents any drop in hit rate from occurring
when cache nodes fail, or are taken down for deployment.
A common approach in similar systems is to use retries to
hide the effects of deployments and failed nodes, an approach
which is known to lead to metastable failure modes in large
systems [5, 17]. Erasure coding allows us to achieve a similar
level of resiliency while performing the same amount of work
in success and failure cases (a design philosophy we call
constant work [23]).
4.2 Stability and Metastability
Caches with high hit rates, such as ours, are desirable from a
latency and efficiency perspective, but have a hidden down-
side. If the cache becomes empty (such as due to power loss or
operational issue), or the hit rate suddenly drops (such as due
to a change in customer behavior), the downstream services
can see significantly more traffic than they are used to. In the
case of our cache, with an end-to-end hit rate typically exceed-
ing 99.8%, this downstream traffic increase could be up to 500
times normal. S3 is an extremely scalable backing store, and
can tolerate the full uncached load. However, the increased
latency leads to higher concurrency demand from customer’s
applications (due to Little’s Law [21]), and therefore higher
demand for new Lambda slots, increasing load and changing
USENIX Association 2023 USENIX Annual Technical Conference    321

Figure 9: Comparative empirical CDFs of client-side latency
of 4-of-4 parallel cache load, versus 4-of-5 erasure coded
cache load.
the size and composition of the system’s working set. This
can lead to metastable behavior [5, 6, 17], where the system
isn’t able to refill the cache when it is empty3.
We have built mitigations for this risk into higher layers of
Lambda. Primarily, the system is designed to be concurrency-
limited. When container starts slow down and the number of
concurrent tasks exceeds this limit, new starts are rejected un-
til in-flight ones complete. We also actively test the system’s
ability to cold start from an empty cache at the maximum
concurrency. This testing allows us to be confident that the
system is able to restart from a cold cache, or tolerate work-
load changes that significantly reduce hit rate.
4.3 Cache Eviction and Sizing
Traditional cache replacement policies like Least Recently
Used (LRU) and First In First Out (FIFO) are simple and
easy to implement, but have a significant downside for this
application: a lack of scan resistance. In our case, this means
that a large number of infrequently used functions starting up
3Related effects have been observed in computer systems since at least the
1960s. In the 1968 paper ‘The Working Set Model for Program Behavior’ [9],
Peter J Denning observed a similar effect in paging systems:
This can create a self-intensifying crisis. Programs, deprived
of still-needed pages, generate a plethora of page faults; the
resulting traffic of returning pages displaces still other useful
pages, leading to more page faults, and so on.
can replace all the hot entries in the cache with recently-used
entries belonging to those functions, dropping cache hit rates
for more frequently-used entries, and filling the cache with
entries that will never be read again. This happens periodically
in our environment, driven by weekly, daily, and hourly spikes
of periodic cron job functions. These functions are large in
number, but each runs at a low scale (typically only using one
sandbox), making caching their chunks relatively unimportant.
To avoid the hit-rate drops caused by this periodic work, we
use the LRU-k [29] eviction algorithm, which tracks the last
k times an item in the cache was used, rather than only the
most recent time.
Eviction and hit rates are also related to the size of our local
and AZ-level caches. Following the logic of Gray and Put-
zolu’s classicFive Minute Rule [14], the minimum desirable
cache size is the one that makes the cost of cache retention
equal to the cost of fetching chunks from S3. However, be-
cause our cache is not only aimed at reducing costs but also
improving customer-observed latency, we also set a hit rate
goal and increase the cache size if we fall below that goal.
The total cache size, then, is the larger of the size needed
to achieve our hit rate goal, and the size needed to optimize
costs.
5 Implementation and Production Experience
We built the local agent (the FUSE implementation that backs
the sparse block device for each MicroVM), the worker-local
cache, and the remote cache server in the Rust programming
language. We used the tokio runtime, and reqwest and hyper
for HTTP. At the time we started this project, the invoke path
of AWS Lambda includes components written in Java, Go,
C, and Rust. We chose Rust because of our good experiences
with the Rust components we had built in the past, especially
around performance and stability, and have again been happy
with our choice of Rust, encountering no major production
bugs in the libraries we chose. We were also attracted to Rust
because of the successes other AWS teams (such as the Ama-
zon S3 team [4]) have had applying formal methods to verify
code correctness in Rust, even with non-expert programmers.
One interesting stumbling block with Rust (version 1.46.0,
current at the time of implementation) is brittle optimization,
especially autovectorization, of hotspots. Unsurprisingly, we
found that the parity calculations we use for erasure coding
are nearly 5x faster when performed 64 bytes at a time (with
A VX512) or 32 bytes at a time (with A VX or NEON) than
when performed 8 bytes at a time, and 10x faster than when
performed byte-at-a-time. Unfortunately, the naive Rust loop
emitted the byte-at-a-time code (as shown in Listing 1), de-
spite the compiler being capable of autovectorization. Small
changes to the code would change autovectorization behav-
ior, even changes outside the function of interest. Reluctant
to move to assembly for this code, we finally settled on the
code in Listing 2, which robustly emits appropriately unrolled
322    2023 USENIX Annual Technical Conference USENIX Association

A VX, A VX512, or vectorized ARM code depending on the tar-
get platform. Seemingly small changes to this function (such
as removing the assert, changing any of the assignments, or
allowing it be inlined) cause autovectorization to be disabled.
This is a small issue with Rust, and one that we expect to be
improved in future compiler versions.
Listing 1 Naive byte-by-byte x86 assembly code as emitted
by the Rust compiler for straightforward loop implementa-
tion (with annotations by perf showing percent of runtime).
Note significant missed opportunities for optimizations like
vectorization and loop unrolling.
0.08 |350: cmp %rax,%rsi
| ↓ jae 3f4
49.18 | movzbl (%rdi,%rsi,1),%ebx
0.13 | xor %bl,(%rcx,%rsi,1)
50.52 | lea 0x1(%rsi),%rbp
0.08 | mov %rbp,%rsi
| cmp %rax,%rbp
| ↑ jb 350
Listing 2 Implementation of parity calculation in Rust, show-
ing extra lines needed for reliable autovectorization.
#[inline(never)]
fn parity(target: &mut [u8], source: &[u8]) {
assert_eq!(source.len(), target.len());
let len = target.len();
let _ = target[len-1];
let _ = source[len-1];
for i in 0..len {
target[i] ^= source[i];
}
}
On the other hand, the Rust ecosystem’s support for build-
time microbenchmarks (such as with the criterion crate)
makes it fast and easy to iterate on this type of performance
work, and even assert at build time that autovectorization has
succeeded (effectively stopping regressions from entering pro-
duction). This is a significant boon in a cloud environment,
where performance regressions can cause production outages,
and performance is tied to both cost and carbon efficiency.
5.1 Latency and Multimodality
As with any storage system, performance was an important
goal for the design and implementation of our snapshot chunk
loading system. While throughput, CPU efficiency, and other
bandwidth measures contribute to the cost of running the
system, its scale-out nature make latency and scalability the
Figure 10: Empirical CDF of server-side measured latency of
the L2 cache server
most important factors of performance. The local agent and
on-worker caches trivially scale out, due to the fact that they
do not communicate off their worker, except in interacting
with S3 (to pull chunks from the origin), and the L2 AZ-level
cache.
Figure 10 shows the latency for GETs and PUTs on this
cache, measured from the server side, across all of the cache
nodes in a production deployment over the course of one
week. Each GET or PUT is of a 512kB chunk. As discussed
in Section 4, the L2 cache is a flash-based cache with a sig-
nificant local memory tier (about 10% of cache size). GET
latency is very consistent, with a median of below 50µs. PUT
latency is less consistent, with some multi-modality appar-
ently caused by writeback behavior on the cache host. Despite
this multi-modality, performance is still excellent, with a me-
dian latency of 125µs, a 99th percentile latency below 300µs,
and a 99.99th percentile of 413µs4. When building this cache
server, we chose HTTP2 as a wire protocol for convenience
with the intention of replacing it with an efficient binary pro-
tocol later. In production, we’ve found the overhead of HTTP
(implemented with hyper and reqwest) so low that we have
not yet been motivated to replace the protocol.
Figure 11 shows the end-to-end latency for returning a read
from the perspective of the local agent (that is the FUSE im-
plementation). This doesn’t show the end-to-end IO latency
experienced by guests, because it’s from the perspective of
the worker and does not include the (significant) hit rate on
the page cache maintained by the MicroVM guest’s kernel,
and read-ahead performed by the guest to populate that cache.
Like the L2 server latency, this end-to-end latency shows sig-
4Having a 99.99th percentile at less than 4x the median is a very desirable
property, and difficult to achieve with garbage collected languages like Java
and Go
USENIX Association 2023 USENIX Annual Technical Conference    323

Figure 11: Empirical CDF of end-to-end read latency ob-
served at the local agent (FUSE implementation).
nificant multi-modality: a mode below100µs which represent
local cache hits, a mode around 2.75ms which represent L2
hits (and the subsequent work like decryption), and mode
(trimmed from the graph) showing rare fetches from the ori-
gin (see Figure 7 for the relative frequencies of these modes).
We are working on an optimized cryptographic scheme which
reduces the latency of decryption.
Multi-modality like this is the norm in storage systems,
but presents a few practical challenges to operators. First,
as discussed in Section 7 a small change in the relative fre-
quencies of each mode can significantly change the mean
latency observed by clients (and so change the concurrency
and throughput of the system). Second, latency percentiles and
trimmed means are the summary statistics most commonly
used by operators at AWS, and they tend to obscure multi-
modality. Plots like empirical CDFs (eCDFs, as presented
here) can be valuable, but don’t show change-over-time as
time series of summary statistics do. We have experimented
with heat maps, day-over-day eCDFs, and others, but have
yet to find a succinct way to present these data to operators.
Third, multi-modality makes the decision of where to spend
optimization resources more complex. Which mode should
the team work to improve? Or should they work to reduce the
relative frequencies of higher modes?
5.2 Production experience with FUSE
Our experiences with FUSE match those reported by Vangoor
et al [36], showing relatively little throughput overhead when
well tuned. However, we have found that the choice to use
FUSE to present a file which is then subsequently used as a
block device by Firecracker’s virtio-blk implementation, has
introduced significant overhead. When an application running
in a MicroVM reads a new chunk, control is passed to the
guest kernel, then Firecracker, then the host kernel’s FUSE
layer, then the local agent, before flowing back through the
same path. This introduces context switch overhead, but more
importantly requires four different threads to be scheduled
by the host kernel’s scheduler. This introduces inefficiency in
steady state, and significant jitter under load. We are moving
away from FUSE for this application, primarily due to this
effect. Our new implementation uses userfaultfd and mmap,
removing two layers from the architecture.
We don’t regret starting with FUSE. It provided a conve-
nient interface, a clear security and operational isolation story,
and allowed a team without deep systems-level programming
experience to build an acceptably high performance system.
6 Related work
Mirroring the rise in popularity of serverless and contain-
ers accelerated container loading has been a highly active
area of research, and industry implementation, over the last
decade. Before that, accelerating VM loading through faster
disc image movement was an active area of research. For
example, Frisbee [16] in 2003. Amazon EC2 has taken ad-
vantage of common data to accelerate VM image loading,
through tracking lineage of EBS snapshot chunks [28], since
2009. With Slacker [15] Harter et al studied access patterns in
container loading, and presented a system which takes advan-
tage of these patterns by performing layer-level lazy loading.
Starlight [8] takes a fairly similar filesystem-orientated ap-
proach, optimized for loading at the edge where minimizing
round-trips to the datacenter is a significant contributor to
performance. eStargz [35] extends common container image
formats to make lazy loading at the layer level more efficient,
building on the approach of Google’s CRFS.
DADI [20] uses a block-level approach fairly similar to our
own, but with a peer-to-peer approach rather than a dedicated
cache layer, and without the ability to deduplicate as widely
as our system is able to. FaaSNet [38] approaches a similar
problem to the one we were solving, but works on the layer
level (rather than flattening images as we do), and does not
appear to perform deduplication. Cntr [34] and Yolo [26] take
the approach of breaking down container images into different
classes of data, some needed urgently on start up and some
likely to be accessed less urgently. This explicit approach
may be more efficient than the simple block-based approach,
but also requires a deeper introspection of the contents of
the container. Wharf [41] and CFS [22] take the distributed
filesystem approach, showing that can significantly improve
loading performance at the cost of increased coordination
between containers.
Accelerating storage performance and loading with dedu-
plication has an even longer history, for example in 2001
with Muthitacharoen et al [25] and 2002 with Venti [30], and
Farsite [2].
324    2023 USENIX Annual Technical Conference USENIX Association

7 Conclusion
We present AWS Lambda’s solution for accelerated loading
of container images, and approach that combines deduplica-
tion, erasure coding, tiered caching, userspace filesystems,
and convergent encryption. We have operated this system for
several years, and are extending its use into other areas of
AWS. While our solution on the surface appears to have a lot
of moving parts, it is optimized for what we believe to be the
realities of building massive scale cloud systems: failures are
frequent, failures are often partial and complex, and security
is the top priority.
7.1 Broader Lessons and Future Work
While Lambda’s snapshot loading infrastructure is a special-
ized system for a rather specialized application, we believe
that there are some broader lessons from our experiences that
apply to the systems community as a whole.
• Containers are most popularly used by Lambda cus-
tomers as “static linking in the large” dependency clo-
sures. Customers want to build, test, and deploy a func-
tion with all its dependencies in one atomic unit, but
traditional static linking is either unavailable or inconve-
nient. However, containers are also highly inefficient in
this context, necessitating the deduplication and sparse
loading we describe here. We believe that there is a sig-
nificant need for a lighter-weight dependency closure
mechanism, which comes closer to traditional static link-
ing in the size of the artifacts that it creates.
• Caches reduce costs, improve latency, and reduce load
on durable storage, and are a critical component of nearly
any stateful system. However, they also introduce risks
such as metastable failures (due to unexpectedly empty
caches, or sudden shifts in workloads), and challenges
for users like multi-modal latency distributions. While
work such Yang et al [40], and Huang et al [17] have
made steps towards deeply understanding these effects,
we believe that significantly more work is needed to
understand the dynamic behaviors of caching in large
systems, and to develop patterns to mitigate the risks of
caches.
• MicroVMs provide an isolation mechanism which is
nearly as lightweight as containers, or even processes [3,
24], while providing additional interfaces for plugging
in both local and distributed operating system logic. Mi-
croVMs provide a powerful new tool in the operating
system researcher’s or builder’s toolbox. We believe that
operating system support for virtualization, and virtual-
ization support for applications, operating systems, and
databases are ripe areas of research which are not yet
receiving sufficient attention.
Our future work is focused on optimizing the system further
for cost, performance, and especially customer-experienced
cold-start latency. This same system is used in Lambda Snap-
Start, a feature of AWS Lambda which reduces cold-start
latency using memory snapshots, to store and load memory
snapshot contents. That use-case is especially latency sensi-
tive, motivating significant investments in both average case
and tail latency. We expect this work to include optimizing
cache retention and data placement policies, optimizing client
and server performance, and completing the migration from
FUSE to userfaultfd.
Acknowledgements
Any system of this size requires a team to build and op-
erate, and in this case we’re deeply thankful to the AWS
Lambda team for their work and contributions. Holly Mesro-
bian, David R. Richardson, Ajay Nair, and David Nasi were
instrumental in supporting this work. Shay Gueron, Osman
Surkatty, and Derek Manwaring helped ground our crypto-
graphic ambitions, and provided valuable feedback.
References
[1] Oci image format specification. Accessed: 2022-04-15.
URL: https://github.com/opencontainers/imag
e-spec.
[2] Atul Adya, William J. Bolosky, Miguel Castro, Gerald
Cermak, Ronnie Chaiken, John R. Douceur, Jon Howell,
Jacob R. Lorch, Marvin Theimer, and Roger P. Wat-
tenhofer. FARSITE: Federated, available, and reliable
storage for an incompletely trusted environment. In
5th Symposium on Operating Systems Design and Im-
plementation (OSDI 02), Boston, MA, December 2002.
USENIX Association. URL: https://www.usenix.o
rg/conference/osdi-02/farsite-federated-a
vailable-and-reliable-storage-incompletel
y-trusted-environment.
[3] Alexandru Agache, Marc Brooker, Alexandra Iordache,
Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and
Diana-Maria Popa. Firecracker: Lightweight virtual-
ization for serverless applications. In 17th {USENIX}
Symposium on Networked Systems Design and Imple-
mentation ({NSDI} 20), pages 419–434, February 2020.
[4] James Bornholt, Rajeev Joshi, Vytautas Astrauskas,
Brendan Cully, Bernhard Kragl, Seth Markle, Kyle
Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran, Jacob
Van Geffen, and Andrew Warfield. Using lightweight
formal methods to validate a key-value storage node in
amazon s3. In Proceedings of the ACM SIGOPS 28th
Symposium on Operating Systems Principles, SOSP ’21,
USENIX Association 2023 USENIX Annual Technical Conference    325

page 836–850, New York, NY , USA, 2021. Association
for Computing Machinery. doi:10.1145/3477132.34
83540.
[5] Nathan Bronson, Abutalib Aghayev, Aleksey Charapko,
and Timothy Zhu. Metastable failures in distributed
systems. In Proceedings of the Workshop on Hot Top-
ics in Operating Systems , HotOS ’21, page 221–227,
New York, NY , USA, 2021. Association for Computing
Machinery. doi:10.1145/3458336.3465286.
[6] Marc Brooker. Some risks of coordinating only some-
times. In High Performance Transaction Systems 2019
(HPTS’19), November 2019.
[7] John Chen, Ben Coleman, and Anshumali Shrivastava.
Revisiting consistent hashing with bounded loads, 2019.
URL: https://arxiv.org/abs/1908.08762 , doi:
10.48550/ARXIV.1908.08762.
[8] Jun Lin Chen, Daniyal Liaqat, Moshe Gabel, and Eyal
de Lara. Starlight: Fast container provisioning on the
edge and over the W AN. In19th USENIX Symposium on
Networked Systems Design and Implementation (NSDI
22), pages 35–50, Renton, WA, April 2022. USENIX
Association. URL: https://www.usenix.org/confe
rence/nsdi22/presentation/chen-jun-lin.
[9] Peter J. Denning. The working set model for program
behavior. Commun. ACM, 11(5):323–333, may 1968.
doi:10.1145/363095.363141.
[10] Yevgeniy Dodis, Paul Grubbs, Thomas Ristenpart, and
Joanne Woodage. Fast message franking: From invisible
salamanders to encryptment. In Advances in Cryptology
– CRYPTO 2018: 38th Annual International Cryptology
Conference, Santa Barbara, CA, USA, August 19–23,
2018, Proceedings, Part I, page 155–186, Berlin, Hei-
delberg, 2018. Springer-Verlag. doi:10.1007/978-3
-319-96884-1_6 .
[11] John R. Douceur, Atul Adya, William J. Bolosky, Dan
Simon, and Marvin Theimer. Reclaiming space from
duplicate files in a serverless distributed file system. In
Proceedings of the 22 Nd International Conference on
Distributed Computing Systems (ICDCS’02) , ICDCS
’02, page 617, USA, 2002. IEEE Computer Society.
[12] Morris J Dworkin. NIST SP 800-38D. recommendation
for block cipher modes of operation: Galois/counter
mode (gcm) and gmac . National Institute of Standards
& Technology, 2007.
[13] Kristen Gardner, Samuel Zbarsky, Sherwin Doroudi,
Mor Harchol-Balter, and Esa Hyytia. Reducing la-
tency via redundant requests: Exact analysis. SIGMET-
RICS Perform. Eval. Rev. , 43(1):347–360, jun 2015.
doi:10.1145/2796314.2745873.
[14] Jim Gray and Franco Putzolu. The 5 minute rule for
trading memory for disc accesses and the 10 byte rule
for trading memory for cpu time. In Proceedings of
the 1987 ACM SIGMOD international conference on
Management of data, pages 395–398, 1987.
[15] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C.
Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau.
Slacker: Fast distribution with lazy docker containers.
In 14th USENIX Conference on File and Storage
Technologies (F AST 16), pages 181–195, Santa Clara,
CA, February 2016. USENIX Association. URL:
https://www.usenix.org/conference/fast16/t
echnical-sessions/presentation/harter.
[16] Mike Hibler, Leigh Stoller, Jay Lepreau, Robert Ricci,
and Chad Barb. Fast, scalable disk imaging with
frisbee. In 2003 USENIX Annual Technical Confer-
ence (USENIX ATC 03), San Antonio, TX, June 2003.
USENIX Association. URL: https://www.usenix.o
rg/conference/2003-usenix-annual-technical
-conference/fast-scalable-disk-imaging-fri
sbee.
[17] Lexiang Huang, Matthew Magnusson, Abishek Ban-
galore Muralikrishna, Salman Estyak, Rebecca Isaacs,
Abutalib Aghayev, Timothy Zhu, and Aleksey Chara-
pko. Metastable failures in the wild. In 16th USENIX
Symposium on Operating Systems Design and Imple-
mentation (OSDI 22), pages 73–90, Carlsbad, CA, July
2022. USENIX Association. URL: https://www.us
enix.org/conference/osdi22/presentation/hu
ang-lexiang.
[18] Vlad Ionescu. Scaling containers on aws in 2022. Ac-
cessed: 2022-04-15. URL: https://www.vladionesc
u.me/posts/scaling-containers-on-aws-in-2
022/.
[19] David Karger, Eric Lehman, Tom Leighton, Rina Pani-
grahy, Matthew Levine, and Daniel Lewin. Consistent
hashing and random trees: Distributed caching proto-
cols for relieving hot spots on the world wide web. In
Proceedings of the Twenty-Ninth Annual ACM Sympo-
sium on Theory of Computing, STOC ’97, page 654–663,
New York, NY , USA, 1997. Association for Computing
Machinery. doi:10.1145/258533.258660.
[20] Huiba Li, Yifan Yuan, Rui Du, Kai Ma, Lanzheng Liu,
and Windsor Hsu. DADI: Block-Level image service
for agile and elastic application deployment. In 2020
USENIX Annual Technical Conference (USENIX ATC
20), pages 727–740. USENIX Association, July 2020.
URL: https://www.usenix.org/conference/atc2
0/presentation/li-huiba.
326    2023 USENIX Annual Technical Conference USENIX Association

[21] John DC Little. A proof for the queuing formula: L= λ
w. Operations research, 9(3):383–387, 1961.
[22] Haifeng Liu, Wei Ding, Yuan Chen, Weilong Guo, Shuo-
ran Liu, Tianpeng Li, Mofei Zhang, Jianxing Zhao,
Hongyin Zhu, and Zhengyi Zhu. Cfs: A distributed
file system for large scale container platforms. In Pro-
ceedings of the 2019 International Conference on Man-
agement of Data, SIGMOD ’19, page 1729–1742, New
York, NY , USA, 2019. Association for Computing Ma-
chinery. doi:10.1145/3299869.3314046.
[23] Colm MacCárthaigh. Reliability, constant work, and a
good cup of coffee, 2020. URL: https://aws.amaz
on.com/builders-library/reliability-and-c
onstant-work/.
[24] Filipe Manco, Costin Lupu, Florian Schmidt, Jose
Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata,
Costin Raiciu, and Felipe Huici. My vm is lighter (and
safer) than your container. In Proceedings of the 26th
Symposium on Operating Systems Principles, SOSP ’17,
page 218–233, New York, NY , USA, 2017. Association
for Computing Machinery. doi:10.1145/3132747.31
32763.
[25] Athicha Muthitacharoen, Benjie Chen, and David Maz-
ières. A low-bandwidth network file system. InProceed-
ings of the Eighteenth ACM Symposium on Operating
Systems Principles, SOSP ’01, page 174–187, New York,
NY , USA, 2001. Association for Computing Machinery.
doi:10.1145/502034.502052.
[26] Thuy Linh Nguyen, Ramon Nou, and Adrien Lebre.
Yolo: Speeding up vm and docker boot time by reduc-
ing i/o operations. In European Conference on Parallel
Processing, pages 273–287. Springer, 2019.
[27] OASIS. Virtual i/o device (virtio) version 1.0, March
2016.
[28] Marc Olson and Prarthana Karmakar. Amazon ebs under
the hood: A tech deep dive, December 2021. URL: ht
tps://www.youtube.com/watch?v=kaWzAEVZ6k8.
[29] Elizabeth J O’neil, Patrick E O’neil, and Gerhard
Weikum. The lru-k page replacement algorithm for
database disk buffering.Acm Sigmod Record, 22(2):297–
306, 1993.
[30] Sean Quinlan and Sean Dorward. Venti: A new approach
to archival data storage. In Conference on File and
Storage Technologies (F AST 02), Monterey, CA, January
2002. USENIX Association. URL: https://www.us
enix.org/conference/fast-02/venti-new-app
roach-archival-data-storage .
[31] K. V . Rashmi, Mosharaf Chowdhury, Jack Kosaian, Ion
Stoica, and Kannan Ramchandran. EC-Cache: Load-
Balanced, Low-Latency cluster caching with online era-
sure coding. In 12th USENIX Symposium on Operating
Systems Design and Implementation (OSDI 16) , pages
401–417, Savannah, GA, November 2016. USENIX As-
sociation. URL: https://www.usenix.org/confere
nce/osdi16/technical-sessions/presentation
/rashmi.
[32] Rusty Russell. Virtio: Towards a de-facto standard for
virtual i/o devices. SIGOPS Oper . Syst. Rev., 42(5):95–
103, July 2008. URL: http://doi.acm.org/10.114
5/1400097.1400108 , doi:10.1145/1400097.1400
108.
[33] Mark W. Storer, Kevin Greenan, Darrell D.E. Long,
and Ethan L. Miller. Secure data deduplication. In
Proceedings of the 4th ACM International Workshop
on Storage Security and Survivability , StorageSS ’08,
page 1–10, New York, NY , USA, 2008. Association for
Computing Machinery. doi:10.1145/1456469.1456
471.
[34] Jörg Thalheim, Pramod Bhatotia, Pedro Fonseca, and
Baris Kasikci. Cntr: Lightweight OS containers. In2018
USENIX Annual Technical Conference (USENIX ATC
18), pages 199–212, Boston, MA, July 2018. USENIX
Association. URL: https://www.usenix.org/confe
rence/atc18/presentation/thalheim.
[35] Kohei Tokunaga. Startup containers in lightning speed
with lazy image distribution on containerd. Accessed:
2022-04-15. URL: https://medium.com/nttlabs
/startup-containers-in-lightning-speed-wit
h-lazy-image-distribution-on-containerd-2
43d94522361.
[36] Bharath Kumar Reddy Vangoor, Vasily Tarasov, and
Erez Zadok. To FUSE or not to FUSE: Performance of
User-Space file systems. In15th USENIX Conference on
File and Storage Technologies (F AST 17), pages 59–72,
Santa Clara, CA, February 2017. USENIX Association.
URL: https://www.usenix.org/conference/fast
17/technical-sessions/presentation/vangoor.
[37] Ashish Vulimiri, Philip Brighten Godfrey, Radhika Mit-
tal, Justine Sherry, Sylvia Ratnasamy, and Scott Shenker.
Low latency via redundancy. In Proceedings of the
Ninth ACM Conference on Emerging Networking Exper-
iments and Technologies, CoNEXT ’13, page 283–294,
New York, NY , USA, 2013. Association for Computing
Machinery. doi:10.1145/2535372.2535392.
[38] Ao Wang, Shuai Chang, Huangshi Tian, Hongqi Wang,
Haoran Yang, Huiba Li, Rui Du, and Yue Cheng. FaaS-
Net: Scalable and fast provisioning of custom serverless
USENIX Association 2023 USENIX Annual Technical Conference    327

container runtimes at alibaba cloud function compute. In
2021 USENIX Annual Technical Conference (USENIX
ATC 21), pages 443–457. USENIX Association, July
2021. URL: https://www.usenix.org/conferenc
e/atc21/presentation/wang-ao.
[39] Zhe Wu, Curtis Yu, and Harsha V . Madhyastha.
CosTLO: Cost-Effective redundancy for lower latency
variance on cloud storage services. In 12th USENIX
Symposium on Networked Systems Design and Imple-
mentation (NSDI 15) , pages 543–557, Oakland, CA,
May 2015. USENIX Association. URL: https://
www.usenix.org/conference/nsdi15/technical
-sessions/presentation/wu.
[40] Juncheng Yang, Yao Yue, and K. V . Rashmi. A large
scale analysis of hundreds of in-memory cache clusters
at twitter. In 14th USENIX Symposium on Operating
Systems Design and Implementation (OSDI 20) , pages
191–208. USENIX Association, November 2020. URL:
https://www.usenix.org/conference/osdi20/p
resentation/yang.
[41] Chao Zheng, Lukas Rupprecht, Vasily Tarasov, Douglas
Thain, Mohamed Mohamed, Dimitrios Skourtis, Amit S.
Warke, and Dean Hildebrand. Wharf: Sharing docker
images in a distributed file system. In Proceedings of
the ACM Symposium on Cloud Computing , SoCC ’18,
page 174–185, New York, NY , USA, 2018. Association
for Computing Machinery. doi:10.1145/3267809.
3267836.
328    2023 USENIX Annual Technical Conference USENIX Association

Q: Do any systems use a BFT protocol today?

A: For the particular setting targeted in the paper, developers tend
to focus on preventing and detecting compromised nodes instead of
running their systems with some BFT-like protocol that can handle
compromised nodes (which is quite expensive).  In open systems where
there is no central authority to cleanup bad nodes (e.g., in public
ledgers), BFT protocols have made a come back.

Q: Would it be practical to use BFT to implement a "permissioned" bitcoin where
the membership set is fixed? How fast and realistic (in the real world) would
such a system be?

A: There is an interesting connection to Bitcoin-like systems.  Bitcoin solves a
consensus problem with malicious nodes, but it can have long forks.  Checkout
Stellar and Hyperledger for ledger systems based or inspired by PBFT.

Q: Is primary-backup or Raft the only kind of system where we need to handle
Byzantine faults? Are there other systems?

A: Any computer that is compromised can become byzantine because its under the control of the attacker.

Q: How does Google or other companies handle Byzatine failures in their systems?

A: I don't know what Google does, but my guess is that they focus on preventing
and detecting compromised nodes instead of running their systems with some
BFT-like protocol that can handle compromised nodes.

Q:  Is byzantine fault tolerance a strictly harder problem than solving network
partitions?

A: The paper tackles the same problem as Raft (including handling network
partitions), but in the presence of malicious replicas.  So, the PBFT paper
solves a more challenging problem than Raft.

Q: How does replicas deal with lost messages or messages out of order?

A: The messages contain enough information for recipients to discover that they
missed a message or that a message is out of order.  See 4th paragraph of Sec
6.1 for the details.

Q: In term of leadership in BFT, does BFT follow a strong leader principle? Has
there designs on weaker form of leader to allow for more availability?

A: BFT, like Raft, has a designated leader for a view, which determines the
order of messages.  If the leader fails or doesn't follow the protocol, there is
a view change, which comes along with a new leader.

There are many papers following on the practical BFT paper.  The
high-performance ones I know about have a designated leader; for example, see
the following paper:
https://www.cs.utexas.edu/~lorenzo/papers/kotla07Zyzzyva.pdf

Q: The number of messages proposed by paper seems to be O(n^2) where n is the
number of replicas. Is there any literature that establishes that you need
O(n^2) messages to handle byzantine failures?

A: There is a large literature on BFT protocols. One of the most efficient ones
I know about is Zyzzyva
(https://www.cs.utexas.edu/~lorenzo/papers/kotla07Zyzzyva.pdf), which speculates
that every node is honest and sends in that case 2n messages (see Fig. 1).

The number of messages is only one factor in BFT protocols, however. And is
often not the most important one because in many protocols messages are sent in
parallel or use multicast as BFT does.  Other metrics are the number of servers
on the critical path for latency, the number of messages per second (which can
be increased by batching, and thus sacrificing latency), and so on.


Q: How does PBFT do key management for MACs?

A: Each client shares a secret session key with each replica.  Then, there is
some additional protocol machinery to allow replicas to verify the authenticity
of 2f+1 responses from other replicas, which takes advantage of the presence of
a primary.  The full details are in chapter 3 of the doctoral thesis:
https://dspace.mit.edu/bitstream/handle/1721.1/86581/48116479-MIT.pdf


Q: Do sequence numbers never skip, no matter what?

A: I believe that is true, although during a view change some sequence numbers
will become a no-op.

Q: How can a given request be committed in two different views, but with the
same sequence number? What sequence of events can cause this?

A: What can happen is that the new primary may not know about a message that was
committed in the previous view (because it missed it).  But, the
normal-operation protocol will guarantee that the primary will learn about that
message during the view change protocol, because 2f+1 good replicas have
committed to the message in the previous view.  The new primary then starts from
the last stable checkpoint and fixes up the log, inserting proofs of stability
for messages.  Then, it changes to a new view.

Q: Why can some sequence numbers turn into in a null request during a
view change?

A: The intuition is that the new primary decides the result of all
sequence numbers of the previous view as part of fixing up its log; if
a sequence number doesn't have a sufficient number of prepares in the
previous view, then the new primary makes it a null request.

Q: Is it possible for a non-primary replica to receive (and accept) a prepare
message *before* it receives the corresponding pre-prepare message from the
primary? Why is this okay?

A: A replica can learn about 2f+1 prepares before it has seen the pre-prepare
from the primary (because of out of order delivery or lost messages), but it
will notice that it hasn't seen a pre-prepare for that sequence number yet.  I
don't know what the protocol exactly does in that case, but most likely it waits
for the pre-prepare, because that contains the log entry itself. The prepares
contain only digests of the entry.

Q: From the bottom of page 2, why does a system have to rely on synchrony to provide liveness in cases when it does not rely on synchrony to provide safety?

A: There is an impossibility results that in an asynchronous distributed system
(where messages to don't have a bounded delay) with one faulty node, one cannot
achieve consensus in a bounded amount of time.  This is called the FLP result
(there is a discussion of FLP here:
http://the-paper-trail.org/blog/a-brief-tour-of-flp-impossibility/ or on
wikipedia).

The PBFT authors are just pointing out they aren't circumventing this result,
and instead that they are assuming a weak form of synchrony (i.e., delay(t)
doesn't grow faster than t indefinitely).

Q: Also, why does PBFT require at most 1/3 faulty machines and why
couldn't they relax this constraint to 1/2?

A: There is no protocol possible if more than 1/3 of the nodes is
malicious.  The authors have a short proof that 3f+1 is the minimum
number of replicas necessary (see p3, second paragraph).

Open ledger systems that use PBFT-like protocols have similar bounds
(e.g., Algorand inherits PBFT bound).  If you are willing to settle
for fork consistency, you can do better; take a look at
https://www.usenix.org/conference/nsdi-07/beyond-one-third-faulty-replica\
s-byzantine-fault-tolerant-systems

Q: The paper mentions deterministic state machine changes in order to prevent
divergence. They mention that replicas must be able to decide deterministically
whether a value is correct and what to do if it is not. How do replicas achieve
this? I understand the next paragraph regarding the replicas' participation in
choosing the correct value as a special case, but in the general case, how does
a replica know if a value is correct or not?

A: The general case requires an additional phase in the protocol, which asks for
2f+1 values and then using a deterministic computation to compute the value to
use (e.g., median of the 2f+1 values).  The authors point that this extra phase
can be avoided in the common case, because replicas can check if the value is
correct. For example, in case of a time stamp, the primary can choose it and the
replicas can validate that the time stamp is close enough, by comparing the time
stamp supplied by the primary is within a window from their local clock.


Q: Are view changes analagous to elections in Raft? I don't quite understand
though how a server "wins" the view and becomes the primary?

A: View changes are like changing leaders in Raft, but there is no election.
The primary in the next is pre-determined, it is v mod n (where v is the view
number and n the number of replicas).  So the primary rotates among servers and
there are at most f faulty primaries in a row (since no more than f nodes can be
faulty).

Q: Is the set of faulty nodes fixed in PBFT?

A: Yes, in the system described in the paper there can be no more
than f bad nodes during the lifetime of the system. The thesis
describes a recovery protocol that allows PBFT to recover nodes so
that it can handle more than f over the lifetime of the system:
https://dspace.mit.edu/bitstream/handle/1721.1/86581/48116479-MIT.pdf.

Q: How would a system decide on a value for f? How do you predict how many
servers you expect to fail in a malicious attack?

A: This is the Achilles heel of PBFT as described.  In later work, the authors
extended the PBFT to include proactive recovery so that the algorithm can
tolerate any number of faults over the lifetime of the system provided fewer
than 1/3 of the replicas become faulty within a small window of vulnerability
(see http://dl.acm.org/citation.cfm?id=571640).



Q: In the Introduction, the authors go out of their way to talk about how their
algorithm is _practical_, but then state later that their only restriction is
that each node in their network should run a different implementation of the
service code (in order to fail independently). How realistic is this?

A: This is indeed a tricky issue: the different replicas cannot have the same bugs, and thus the implementation must be different. The paper mentions N-version programming, but in later work the authors extended PBFT with BASE, a replication technique,  which uses abstraction to improve its ability to mask software errors and reuse of off-the-shelf service implementations. It repairs each replica periodically using an abstract view of the state stored by correct replicas, and each replica can run distinct or nondeterministic service implementations, which reduces the probability of common mode failures (see http://dl.acm.org/citation.cfm?id=859718 for the details).

Q: What is a digest?

A: A message digest refers to the result of computing collision-resistant
cryptographic hash function, such as SHA256, over the message.

Q: I recently read "The Saddest Moment"
(https://www.usenix.org/system/files/login-logout_1305_mickens.pdf), and wonder
what the practical applications of BFT would be? It seems that either the
environment is controlled and BFT is unnecessary, or the environment is
uncontrolled and BFT is intractable.

A: I don't know what they are, but we are reading the paper 1) because it is an
impressive result; and 2) it is likely that it might come back.  For example,
Bitcoin solves a similar problem as BFT but has long forks; perhaps in the
future we will systems that combine BFT and Bitcoin ideas.

6.824 Bitcoin FAQ

Q: I don't understand why the blockchain is so important. Isn't the
requirement for the owner's signature on each transaction enough to
prevent bitcoins from being stolen?

A: The signature is not enough, because it doesn't prevent the owner
from spending money twice: signing two transactions that transfer the
same bitcoin to different recipients. The blockchain acts as a
publishing system to try to ensure that once a bitcoin has been spent
once, lots of participants will know, and will be able to reject a
second spend.

Q: Why does Bitcoin need to define a new currency? Wouldn't it be more
convenient to use an existing currency like dollars?

A: The new currency (Bitcoins) allows the system to motivate miners
with freshly created money; this would be harder with dollars because
it's illegal for ordinary people to create fresh dollars. One would
also need a way to connect balances and transactions on a
dollar-denominated blockchain with dollars in the external world. But
have a look at "stablecoins" such as Tether.

Q: What is mining?

A: The proof-of-work process by which new blocks are found, by trying
many different nonces until one yields a block hash with enough
leading zero bits.

Q: Why is the purpose of proof-of-work?

A: It makes it hard for an attacker to convince the system to switch
to a blockchain fork in which a coin is spent in a different way than
in the main fork. You can view proof-of-work as making a random choice
over the participating CPUs of who gets to choose which fork to
extend. If the attacker controls only a few CPUs, the attacker won't
be able to extend a new malicious fork fast enough to overtake the
main blockchain.

Q: Could a Bitcoin-like system use something less wasteful than
proof-of-work?

A: Proof-of-work is hard to fake or simulate, a nice property in a
totally open system like Bitcoin where you cannot trust anyone to
follow rules. The main alternative is proof-of-stake, as used in
Ethereum, Algorand, and Byzcoin. Have a look at
https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/ In a
smallish closed system, in which the participants are known and mostly
trusted, Byzantine agreement protocols could be used, as in
Hyperledger, or variants of it as in Stellar.

Q: Can Alice spend the same coin twice by sending "pay Bob" and "pay
Charlie" to different subsets of miners?

A: Suppose Alice does that. One of the two subsets of miners is likely
to find the nonce for a new block first. Let's assume the first block
to be found is B50 and it contains "pay Bob". This block will be
flooded to all miners, so the miners working on "pay Charlie" will
switch to mining a successor block to B50. These miners validate
transactions they place in blocks, so they will notice that the "pay
Charlie" coin was spent in B50, and they will ignore the "pay Charlie"
transaction. Thus, in this scenario, double-spend won't work.

There's a small chance that two miners find blocks at the same time,
perhaps B50' containing "pay Bob" and B50'' containing "pay Charlie".
At this point there's a fork in the block chain. These two blocks will
be flooded to all the nodes. Each node will start mining a successor
to one of them (the first it hears). Again the most likely outcome is
that a single miner will finish significantly before any other miner,
and flood the successor, and most peers will switch to that winning
fork. The chance of repeatedly having two miners simultaneously find
blocks gets very small as the forks get longer. So eventually all the
peers will switch to the same fork, and in that fork there will be
only one spend of the coin.

The possibility of accidentally having a short-lived fork is the
reason that careful clients wait until there are a few successor
blocks before believing a transaction.

Q: It takes an average of 10 minutes for a Bitcoin block to be
validated. Does this mean that the parties involved aren't sure if the
transaction really happened until 10 minutes later?

A: Yes. The 10 minutes is awkward. But it's not always a problem. For
example, suppose you buy a toaster oven with Bitcoin from a web site.
The web site can check that the transaction is known by a few servers,
though not yet in a block, and show you a "purchase completed" page.
Before shipping it to you, they should check that the transaction is
in a block. For low-value in-person transactions, such as buying a cup
of coffee, it's probably enough for the seller to ask a few peers to
check that the bitcoins haven't already been spent (i.e. it's
reasonably safe to not bother waiting for the transaction to appear in
the blockchain at all). For a large in-person purchase (e.g., a car),
it is important to wait for sufficiently long to be assured that the
block will stay in the block chain before handing over the goods.

Q: What can be done to speed up transactions on the blockchain?

A: I think the constraint here is that 10 minutes needs to be much
larger (i.e. >= 10x) than the time to broadcast a newly found block to
all peers. The point of that is to minimize the chances of two peers
finding new blocks at about the same time, before hearing about the
other peer's block. Two new blocks at the same time is a fork; forks
are bad since they cause disagreement about which transactions are
real, and they waste miners' time. Since blocks can be pretty big (up
to a megabyte), and peers could have slow Internet links, and the
diameter of the peer network might be large, it could easily take a
minute to flood a new block. If one could reduce the flooding time,
then the 10 minutes could also be reduced.

Q: Bitcoin is limited to processing no more than a few transactions
per second, due to the 10 minutes and the limit of one megabyte per
block. Why the one megabyte limit?

A: I think the worry was that huge blocks would take a long time to
flood to peers with low-speed links. Here's a discussion:
https://en.wikipedia.org/wiki/Bitcoin_scalability_problem

When Bitcoin was originally designed, the intent was that lots of
ordinary people do the mining on ordinary computers, connected with
relatively slow home Internet links. Such a system could not have
supported a huge transaction rate, and would have taken a long time to
flood new blocks. The original design has proved difficult to modify,
since there's no central authority that can change things.

Q: The entire blockchain needs to be downloaded before a node can
participate in the network. Won't that take an impractically long time
as the blockchain grows?

A: It's true that it takes a while for a new node to get all the
transactions. But once a given server has done this work, it can save
the block chain, and doesn't need to fetch it again. It only needs to
know about new blocks, which is not a huge burden. On the other hand
most ordinary users of Bitcoin don't run full Bitcoin nodes; instead
they trust a few full nodes to answer questions about whether coins
have already been spent.

Q: Is it feasible for an attacker to gain a majority of the computing
power among peers? What are the implications for bitcoin if this happens?

A: It may be feasible; some people think that big cooperative groups
of miners have been close to a majority at times:
http://www.coindesk.com/51-attacks-real-threat-bitcoin/

If >50% of compute power is controlled by a single entity, they can
double-spend bitcoins: transfer a coin to one payee, and then generate
a new fork from before that transaction in which the transaction
doesn't exist. Bitcoin's security would be broken if this happened.

Q: From some news stories, I have heard that a large number of bitcoin
miners are controlled by a small number of companies.

A: True. See here: https://blockchain.info/pools. It looks like two or
three mining pools together hold about half of the compute power
today.

Q: Are there any ways for Bitcoin mining to do useful work, beyond simply
brute-force calculating SHA-256 hashes?

A: Maybe -- here are two attempts to do what you suggest:
https://www.cs.umd.edu/~elaine/docs/permacoin.pdf
http://primecoin.io/

Q: There is hardware specifically designed to mine Bitcoin. How does
this type of hardware differ from the type of hardware in a laptop?

A: Mining hardware has a lot of transistors dedicated to computing
SHA256 quickly, but is not particularly fast for other operations.
Ordinary server and laptop CPUs can do many things (e.g. floating
point division) reasonably quickly, but don't have so much hardware
dedicated to SHA256 specifically. Some Intel CPUs do have instructions
specifically for SHA256; however, they aren't competitive with
specialized Bitcoin hardware that massively parallelizes the hashing
using lots of dedicated transistors.

Q: The paper estimates that the disk space required to store the block
chain will grow by 4.2 megabytes per year. That seems very low!

A: The 4.2 MB/year is for just the block headers, and is still the
actual rate of growth. The current 730+ GB is for full blocks.

Q: Would the advent of quantum computing break the bitcoin system?

A: Here's a plausible-looking article:
http://www.bitcoinnotbombs.com/bitcoin-vs-the-nsas-quantum-computer/
Quantum computers might be able to forge bitcoin's digital signatures
(ECDSA).

Q: Bitcoin uses the hash of the transaction record to identify the
transaction, so it can be named in future transactions. Is this
guaranteed to lead to unique IDs?

A: The hashes are technically not guaranteed to be unique. But in
practice the hash function (SHA-256) is believed to produce different
outputs for different inputs with fantastically high probability.

Q: It sounds like anyone can create new Bitcoins. Why is that OK?
Won't it lead to forgery or inflation?

A: Only the person who first computes a proper nonce for the current
last block in the chain gets the 3.125-bitcoin reward for "mining" it. It
takes a huge amount of computation to do this. If you buy a computer
and have it spend all its time attempting to mine bitcoin blocks, you
will not make enough bitcoins to pay for the computer.

Q: The paper mentions that some amount of fraud is admissible; where
does this fraud come from?

A: This part of the paper is about problems with the current way of
paying for things, e.g. credit cards. Fraud occurs when you buy
something on the Internet, but the seller keeps the money and doesn't
send you the item. Or if a merchant remembers your credit card number,
and buys things with it without your permission. Or if someone buys
something with a credit card, but never pays the credit card bill. Or
if someone buys something with a stolen credit card.

Q: Has there been fraudulent use of Bitcoin?

A: Yes. I think most of the problems have been at web sites that act
as wallets to store peoples' bitcoin private keys. Such web sites,
since they have access to the private keys, can transfer their
customers' money to anyone. So someone who works at (or breaks into)
such a web site can steal the customers' Bitcoins.

If people used Bitcoin to buy things, we'd probably see fraud in the
form of sellers accepting payment but not delivering the goods.

Q: Satoshi's paper mentions that each transaction has a transaction
fee that is given to whoever mined the block. Why would a miner not
simply try to mine blocks with transactions with the highest
transaction fees?

A: Miners do favor transactions offering higher fees. You can read about
typical approaches here:
https://en.bitcoin.it/wiki/Transaction_fees
And here's a graph (the red line) of how long your transaction waits
as a function of how high a fee you offer:
https://bitcoinfees.github.io/misc/profile/

Q: Why would a miner bother including transactions that yield no fee?

A: I think many don't mine no-fee transactions any more.

Q: How are transaction fees determined/advertised?

A: Have a look here:
https://en.bitcoin.it/wiki/Transaction_fees
Different miners use different strategies to decide
which transactions to include in blocks.
It sounds like (by default) wallets look in the block chain at the
recent correlation between fee and time until a transaction is
included in a mined block, and choose a fee that correlates with
relatively quick inclusion.
Fees are typically around a dollar or two per transaction.

Q: What are some techniques for storing my personal bitcoins, in
particular the private keys needed to spend my bitcoins? I've heard of
people printing out the keys, replicating them on USB, etc. Does a
secure online repository exist?

A: Any scheme that keeps the private keys on a computer attached to
the Internet is a tempting target for thieves. On the other hand, it's
a pain to use your bitcoins if the private keys are on a sheet of
paper. So my guess is that careful people store the private keys for
small amounts on their computer, but for large balances they store the
keys offline.

Q: What other kinds of virtual currency were there before and after
Bitcoin (I know the paper mentioned hashcash)? What was different
about Bitcoin that led it to have more success than its predecessors?

A: There were many previous proposals for digital cash systems, none
with any noticeable success. It's tempting to think that Bitcoin has
succeeded because its design is better than others: that it has
just the right blend of incentives and decentralization and ease of
use. But there are too many forgotten yet apparently well-designed
technologies out there for me to believe that.

Q: What happens when more (or fewer) people mine Bitcoin?

A: Bitcoin adjusts the difficulty to match the measured compute power
devoted to mining. So if more and more computers mine, the mining
difficulty will get harder, but only hard enough to maintain the average
inter-block interval at 10 minutes. If lots of people stop mining, the
difficulty will decrease. This mechanism won't prevent new blocks from
being created, it will just ensure that they are created about every
10 minutes on average.

Q: Is there any way to make Bitcoin completely anonymous?

A: Have a look here: https://en.wikipedia.org/wiki/Zerocoin

Q: If I lose the private key(s) associated with the bitcoins I own,
how can I get my money back?

A: You can't.

Q: What do people buy and sell with bitcoins?

A: Much of the action in bitcoin is speculation: people buying bitcoin
in the hope that its price will go up. There seems to be a fair amount
of illegal activity that exploits Bitcoin's relative anonymity (buying
illegal drugs, demanding ransom). You can buy some ordinary (legal)
stuff on the Internet with Bitcoin too.
It's a bit of a pain, though, so I don't imagine many non-enthusiasts
would use bitcoin in preference to a credit card.

Q: Why is bitcoin illegal in some countries?

A: Here are some guesses.

Many governments adjust the supply of money in order to achieve
certain economic goals, such as low inflation, high employment, and
stable exchange rates. Widespread use of bitcoin may make that harder.

Many governments regulate banks (and things that function as banks) in
order to prevent problems, e.g. banks going out of business and
thereby causing their customers to lose deposits. This has happened to
some bitcoin exchanges. Since bitcoin can't easily be regulated, maybe
the next best thing is to outlaw it.

Bitcoin seems particularly suited to certain illegal transactions
because it is fairly anonymous. Governments regulate big transfers of
conventional money (banks must report big transfers) in order to track
illegal activity; but you can't easily do this with bitcoin.

Q: Why do bitcoins have any value at all? Why do people accept it as
money?

A: Because other people are willing to sell things in return for
bitcoins, and are willing to exchange bitcoins for ordinary currency
such as dollars. This is a circular argument, but has worked many
times in the past; consider why people view baseball trading cards as
having value, or why they think paper money has value.

Q: How is the price of Bitcoin determined?

A: The price of Bitcoin in other currencies (e.g. euros or dollars) is
determined by supply and demand. If more people want to buy Bitcoins
than sell them, the price will go up. If the opposite, then the price
will go down. There is no single price; instead, there is just recent
history of what prices people have been willing to buy and sell at on
public exchanges. The public exchanges bring buyers and sellers
together, and publish the prices they agree to:

  https://bitcoin.org/en/exchanges

Q: Why is the price of bitcoin so volatile?

A: The price is driven partially by people's hopes and fears. When
they are optimistic about Bitcoin, or see that the price is rising,
they buy so as not to miss out, and thus bid the price up further.
When they read negative news stories about Bitcoin or the economy in
general, they sell out of fear that the price will drop and cause them
to lose money. This kind of speculation happens with many goods;
there's nothing special about Bitcoin in this respect. For example:

  https://en.wikipedia.org/wiki/Tulip_mania

Q: How should we think about which parts of Bitcoin's design solve
which problems?

A: Here's one way to think about problems and solutions.

Because Bitcoin transactions are signed, they can't be forged without
the owner's private key, so money can't directly be stolen. The
remaining attacks are theft of private keys, and double-spending by
causing different parties to see different transactions or by causing
transactions to disappear.

All Bitcoin peers flood all the blocks they know of to other peers, so
it's likely that all peers will know of all transactions. Thus it's
hard to reveal a transaction to one peer, but conceal it from another.

Each block contains a hash over what the block's creator thought was
the previous block in the chain, and thus implicitly over the whole
chain back to the origin. This means there's a unique and agreed-on
sequence of blocks (and transactions) leading up to each block, but
leaves open the possibility that the blocks form a tree with multiple
forks, rather than a single chain.

One fork could contain a transaction transferring funds to one
recipient, and another fork could contain a transaction transferring
the same funds to a different recipient. So it's important that the
peers agree on which is the "real" fork, so that they agree on who
owns what. It's also important that it be hard for someone to cause
agreement to switch from one fork to another; such a switch would
allow double-spending.

Part of Bitcoin's plan for agreement is that peers always use the fork
with the longest path from the origin. But what if two leaves are the
same distance from the origin? Bitcoin's proof-of-work mining
mechanism makes it likely that one peer will mine a new block
extending one of these leaves significantly before any other peer
completes mining; and will flood that block to all peers; and then
most peers will agree that the newly mined block forms the longest
fork, breaking the tie. The winning miner is effectively randomly
selected, making it likely that it is honest and will follow the rules
and actually flood the block.

An attacker could try to double-spend by transferring funds one way,
and then causing an alternate fork with a different spend to grow
longer than the original fork; the longest-fork rule would then cause
everyone to switch forks. Bitcoin's proof-of-work mining makes it hard
to force a switch from the longest fork to an initially shorter fork,
since more peers will be working to extend the longest fork than to
extend the fraudulent fork. Because a switch from one fork to another
is not impossible if the longest fork is only one or two blocks
longer, careful recipients only believe in a transaction if it's still
in the longest fork after a few more blocks have been mined.

Bitcoin: A Peer-to-Peer Electronic Cash System
Satoshi Nakamoto
satoshin@gmx.com
www.bitcoin.org
Abstract.  A purely peer-to-peer version of  electronic cash would allow online
payments to be sent directly from one party to another without going through a
financial institution.  Digital signatures provide part of the solution, but the main
benefits are lost if a trusted third party is still required to prevent double-spending.
We propose a solution to the double-spending problem using a peer-to-peer network.
The network timestamps transactions by hashing them into an ongoing chain of
hash-based proof-of-work, forming a record that cannot be changed without redoing
the proof-of-work.  The longest chain not only serves as proof of the sequence of
events witnessed, but proof that it came from the largest pool of CPU power.  As
long as a majority of CPU power is controlled by nodes that are not cooperating to
attack the network, they'll generate the longest chain and outpace attackers.  The
network itself requires minimal structure.  Messages are broadcast on a best effort
basis, and nodes can leave and rejoin the network at will, accepting the longest
proof-of-work chain as proof of what happened while they were gone.
1. Introduction
Commerce on the Internet has come to rely almost exclusively on financial institutions serving as
trusted third parties to process electronic payments.  While the system works well enough for
most  transactions,  it  still  suffers  from  the  inherent  weaknesses  of  the  trust  based  model.
Completely non-reversible transactions are not really possible, since financial institutions cannot
avoid  mediating  disputes.   The  cost  of  mediation  increases  transaction  costs,  limiting  the
minimum practical transaction size and cutting off the possibility for small casual transactions,
and there is a broader cost in the loss of ability to make non-reversible payments for non-
reversible services.  With the possibility of reversal, the need for trust spreads.  Merchants must
be wary of their customers, hassling them for more information than they would otherwise need.
A certain percentage of fraud is accepted as unavoidable.  These costs and payment uncertainties
can be avoided in person by using physical currency, but no mechanism exists to make payments
over a communications channel without a trusted party.
What is needed is an electronic payment system based on cryptographic proof instead of trust,
allowing any two willing parties to transact directly with each other without the need for a trusted
third party.  Transactions that are computationally impractical to reverse would protect sellers
from fraud, and routine escrow mechanisms could easily be implemented to protect buyers.  In
this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed
timestamp server to generate computational proof of the chronological order of transactions.  The
system  is  secure  as  long  as  honest  nodes  collectively  control  more  CPU  power  than  any
cooperating group of attacker nodes.
1

2. Transactions
We define an electronic coin as a chain of digital signatures.  Each owner transfers the coin to the
next by digitally signing a hash of the previous transaction and the public key of the next owner
and adding these to the end of the coin.  A payee can verify the signatures to verify the chain of
ownership.
The problem of course is the payee can't verify that one of the owners did not double-spend
the coin.  A common solution is to introduce a trusted central authority, or mint, that checks every
transaction for double spending.  After each transaction, the coin must be returned to the mint to
issue a new coin, and only coins issued directly from the mint are trusted not to be double-spent.
The  problem with  this solution  is that  the  fate of the entire  money system depends on the
company running the mint, with every transaction having to go through them, just like a bank.
We need a way for the payee to know that the previous owners did not sign any earlier
transactions.  For our purposes, the earliest transaction is the one that counts, so we don't care
about later attempts to double-spend.  The only way to confirm the absence of a transaction is to
be aware of all transactions.  In the mint based model, the mint was aware of all transactions and
decided which arrived first.  To accomplish this without a trusted party, transactions must be
publicly announced [1], and we need a system for participants to agree on a single history of the
order in which they were received.  The payee needs proof that at the time of each transaction, the
majority of nodes agreed it was the first received.
3. Timestamp Server
The solution we propose begins with a timestamp server.  A timestamp server works by taking a
hash  of a  block  of  items  to  be  timestamped  and  widely  publishing  the  hash,  such  as  in  a
newspaper or Usenet post [2-5].  The timestamp proves that the data must have existed at the
time, obviously, in order to get into the hash.  Each timestamp includes the previous timestamp in
its hash, forming a chain, with each additional timestamp reinforcing the ones before it.
2
Block
Item Item ...
Hash
Block
Item Item ...
Hash
Transaction
Owner 1's
Public Key
Owner 0's
Signature
Hash
Transaction
Owner 2's
Public Key
Owner 1's
Signature
Hash
 Verify
Transaction
Owner 3's
Public Key
Owner 2's
Signature
Hash
 Verify
Owner 2's
Private Key
Owner 1's
Private Key
Sign  Sign
Owner 3's
Private Key

4. Proof-of-Work
To implement a distributed timestamp server on a peer-to-peer basis, we will need to use a proof-
of-work system similar to Adam Back's Hashcash [6], rather than newspaper or Usenet posts.
The proof-of-work involves scanning for a value that when hashed, such as with SHA-256, the
hash begins with a number of zero bits.  The average work required is exponential in the number
of zero bits required and can be verified by executing a single hash.
For our timestamp network, we implement the proof-of-work by incrementing a nonce in the
block until a value is found that gives the block's hash the required zero bits.  Once the CPU
effort has been expended to make it satisfy the proof-of-work, the block cannot be changed
without redoing the work.  As later blocks are chained after it, the work to change the block
would include redoing all the blocks after it.
The proof-of-work also solves the problem of determining representation in majority decision
making.  If the majority were based on one-IP-address-one-vote, it could be subverted by anyone
able  to  allocate  many  IPs.   Proof-of-work  is  essentially  one-CPU-one-vote.   The  majority
decision is represented by the longest chain, which has the greatest proof-of-work effort invested
in it.  If a majority of CPU power is controlled by honest nodes, the honest chain will grow the
fastest and outpace any competing chains.  To modify a past block, an attacker would have to
redo the proof-of-work of the block and all blocks after it and then catch up with and surpass the
work of the honest nodes.  We will show later that the probability of a slower attacker catching up
diminishes exponentially as subsequent blocks are added.
To compensate for increasing hardware speed and varying interest in running nodes over time,
the proof-of-work difficulty is determined by a moving average targeting an average number of
blocks per hour.  If they're generated too fast, the difficulty increases.
5. Network
The steps to run the network are as follows:
1) New transactions are broadcast to all nodes.
2) Each node collects new transactions into a block.
3) Each node works on finding a difficult proof-of-work for its block.
4) When a node finds a proof-of-work, it broadcasts the block to all nodes.
5) Nodes accept the block only if all transactions in it are valid and not already spent.
6) Nodes express their acceptance of the block by working on creating the next block in the
chain, using the hash of the accepted block as the previous hash.
Nodes always consider the longest chain to be the correct one and will keep working on
extending it.  If two nodes broadcast different versions of the next block simultaneously, some
nodes may receive one or the other first.  In that case, they work on the first one they received,
but save the other branch in case it becomes longer.  The tie will be broken when the next proof-
of-work is found and one branch becomes longer; the nodes that were working on the other
branch will then switch to the longer one.
3
Block
Prev Hash Nonce
Tx Tx ...
Block
Prev Hash Nonce
Tx Tx ...

New transaction broadcasts do not necessarily need to reach all nodes.  As long as they reach
many nodes, they will get into a block before long.  Block broadcasts are also tolerant of dropped
messages.  If a node does not receive a block, it will request it when it receives the next block and
realizes it missed one.
6. Incentive
By convention, the first transaction in a block is a special transaction that starts a new coin owned
by the creator of the block.  This adds an incentive for nodes to support the network, and provides
a way to initially distribute coins into circulation, since there is no central authority to issue them.
The steady addition of a constant of amount of new coins is analogous to gold miners expending
resources to add gold to circulation.  In our case, it is CPU time and electricity that is expended.
The incentive can also be funded with transaction fees.  If the output value of a transaction is
less than its input value, the difference is a transaction fee that is added to the incentive value of
the  block  containing  the  transaction.   Once  a  predetermined  number  of  coins  have  entered
circulation, the incentive can transition entirely to transaction fees and be completely inflation
free.
The incentive may help encourage nodes to stay honest.  If a greedy attacker is able to
assemble more CPU power than all the honest nodes, he would have to choose between using it
to defraud people by stealing back his payments, or using it to generate new coins.  He ought to
find it more profitable to play by the rules, such rules that favour him with more new coins than
everyone else combined, than to undermine the system and the validity of his own wealth.
7. Reclaiming Disk Space
Once the latest transaction in a coin is buried under enough blocks, the spent transactions before
it can be discarded to save disk space.  To facilitate this without breaking the block's hash,
transactions are hashed in a Merkle Tree [7][2][5], with only the root included in the block's hash.
Old blocks can then be compacted by stubbing off branches of the tree.  The interior hashes do
not need to be stored.
A block header with no transactions would be about 80 bytes.  If we suppose blocks are
generated every 10 minutes, 80 bytes * 6 * 24 * 365 = 4.2MB per year.  With computer systems
typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of
1.2GB per year, storage should not be a problem even if the block headers must be kept in
memory.
4
BlockBlock
Block Header (Block Hash)
Prev Hash Nonce
Hash01
Hash0 Hash1 Hash2 Hash3
Hash23
Root Hash
Hash01
Hash2
Tx3
Hash23
Block Header (Block Hash)
Root Hash
Transactions Hashed in a Merkle Tree After Pruning Tx0-2 from the Block
Prev Hash Nonce
Hash3
Tx0 Tx1 Tx2 Tx3

8. Simplified Payment Verification
It is possible to verify payments without running a full network node.  A user only needs to keep
a copy of the block headers of the longest proof-of-work chain, which he can get by querying
network nodes until he's convinced he has the longest chain, and obtain the Merkle branch
linking the transaction to the block it's timestamped in.  He can't  check the transaction for
himself, but by linking it to a place in the chain, he can see that a network node has accepted it,
and blocks added after it further confirm the network has accepted it.
As such, the verification is reliable as long as honest nodes control the network, but is more
vulnerable  if  the  network  is overpowered  by an  attacker.  While  network  nodes can  verify
transactions for themselves, the  simplified  method  can be  fooled by an attacker's fabricated
transactions for as long as the attacker can continue to overpower the network.  One strategy to
protect against this would be to accept alerts from network nodes when they detect an invalid
block, prompting  the  user's  software  to  download  the  full  block  and alerted  transactions to
confirm the inconsistency.  Businesses that receive frequent payments will probably still want to
run their own nodes for more independent security and quicker verification.
9. Combining and Splitting Value
Although it would be possible to handle coins individually, it would be unwieldy to make a
separate transaction for every cent in a transfer.  To allow value to be split and combined,
transactions contain multiple inputs and outputs.  Normally there will be either a single input
from a larger previous transaction or multiple inputs combining smaller amounts, and at most two
outputs: one for the payment, and one returning the change, if any, back to the sender.
It should be noted that fan-out, where a transaction depends on several transactions, and those
transactions depend on many more, is not a problem here.  There is never the need to extract a
complete standalone copy of a transaction's history.
5
Transaction
In
...
In Out
...
Hash01
Hash2 Hash3
Hash23
Block Header
Merkle Root
Prev Hash Nonce
Block Header
Merkle Root
Prev Hash Nonce
Block Header
Merkle Root
Prev Hash Nonce
Merkle Branch for Tx3
Longest Proof-of-Work Chain
Tx3

10. Privacy
The traditional banking model achieves a level of privacy by limiting access to information to the
parties involved and the trusted third party.  The necessity to announce all transactions publicly
precludes this method, but privacy can still be maintained by breaking the flow of information in
another place: by keeping public keys anonymous.  The public can see that someone is sending
an amount to someone else, but without information linking the transaction to anyone.  This is
similar to the level of information released by stock exchanges, where the time and size of
individual trades, the "tape", is made public, but without telling who the parties were.
As an additional firewall, a new key pair should be used for each transaction to keep them
from being linked to a common owner.  Some  linking is still unavoidable with multi-input
transactions, which necessarily reveal that their inputs were owned by the same owner.  The risk
is that if the owner of a key is revealed, linking could reveal other transactions that belonged to
the same owner.
11. Calculations
We consider the scenario of an attacker trying to generate an alternate chain faster than the honest
chain.  Even if this is accomplished, it does not throw the system open to arbitrary changes, such
as creating value out of thin air or taking money that never belonged to the attacker.  Nodes are
not going to accept an invalid transaction as payment, and honest nodes will never accept a block
containing them.  An attacker can only try to change one of his own transactions to take back
money he recently spent.
The race between the honest chain and an attacker chain can be characterized as a Binomial
Random Walk.  The success event is the honest chain being extended by one block, increasing its
lead by +1, and the failure event is the attacker's chain being extended by one block, reducing the
gap by -1.
The probability of an attacker catching up from a given deficit is analogous to a Gambler's
Ruin problem.  Suppose a gambler with unlimited credit starts at a deficit and plays potentially an
infinite number of trials to try to reach breakeven.  We can calculate the probability he ever
reaches breakeven, or that an attacker ever catches up with the honest chain, as follows [8]:
p = probability an honest node finds the next block
q = probability the attacker finds the next block
qz = probability the attacker will ever catch up from z blocks behind
qz={
1 if p≤q
q/ pz
if pq}
6
Identities Transactions Trusted
Third Party Counterparty Public
Identities Transactions Public
New Privacy Model
Traditional Privacy Model

Given our assumption that p > q, the probability drops exponentially as the number of blocks the
attacker has to catch up with increases.  With the odds against him, if he doesn't make a lucky
lunge forward early on, his chances become vanishingly small as he falls further behind.
We now consider how long the recipient of a new transaction needs to wait before being
sufficiently certain the sender can't change the transaction.  We assume the sender is an attacker
who wants to make the recipient believe he paid him for a while, then switch it to pay back to
himself after some time has passed.  The receiver will be alerted when that happens, but the
sender hopes it will be too late.
The receiver generates a new key pair and gives the public key to the sender shortly before
signing.  This prevents the sender from preparing a chain of blocks ahead of time by working on
it continuously until he is lucky enough to get far enough ahead, then executing the transaction at
that moment.  Once the transaction is sent, the dishonest sender starts working in secret on a
parallel chain containing an alternate version of his transaction.
The recipient waits until the transaction has been added to a block and  z blocks have been
linked  after  it.   He  doesn't  know  the  exact  amount  of  progress  the  attacker  has  made, but
assuming the honest blocks took the average expected time per block, the attacker's potential
progress will be a Poisson distribution with expected value:
=z q
p
To get the probability the attacker could still catch up now, we multiply the Poisson density for
each amount of progress he could have made by the probability he could catch up from that point:
∑k=0
∞
k
e−
k! ⋅{
q/ pz−k
if k≤ z
1 if k z}
Rearranging to avoid summing the infinite tail of the distribution...
1−∑k=0
z
k
e−
k! 1−q/ pz−k

Converting to C code...
#include <math.h>
double AttackerSuccessProbability(double q, int z)
{
    double p = 1.0 - q;
    double lambda = z * (q / p);
    double sum = 1.0;
    int i, k;
    for (k = 0; k <= z; k++)
    {
        double poisson = exp(-lambda);
        for (i = 1; i <= k; i++)
            poisson *= lambda / i;
        sum -= poisson * (1 - pow(q / p, z - k));
    }
    return sum;
}
7

Running some results, we can see the probability drop off exponentially with z.
q=0.1
z=0    P=1.0000000
z=1    P=0.2045873
z=2    P=0.0509779
z=3    P=0.0131722
z=4    P=0.0034552
z=5    P=0.0009137
z=6    P=0.0002428
z=7    P=0.0000647
z=8    P=0.0000173
z=9    P=0.0000046
z=10   P=0.0000012
q=0.3
z=0    P=1.0000000
z=5    P=0.1773523
z=10   P=0.0416605
z=15   P=0.0101008
z=20   P=0.0024804
z=25   P=0.0006132
z=30   P=0.0001522
z=35   P=0.0000379
z=40   P=0.0000095
z=45   P=0.0000024
z=50   P=0.0000006
Solving for P less than 0.1%...
P < 0.001
q=0.10   z=5
q=0.15   z=8
q=0.20   z=11
q=0.25   z=15
q=0.30   z=24
q=0.35   z=41
q=0.40   z=89
q=0.45   z=340
12. Conclusion
We have proposed a system for electronic transactions without relying on trust.  We started with
the usual framework of coins made from digital signatures, which provides strong control of
ownership, but is incomplete without a way to prevent double-spending.  To solve this, we
proposed a peer-to-peer network using proof-of-work to record a public history of transactions
that quickly becomes computationally impractical  for  an attacker to change if honest nodes
control a majority of CPU power.  The network is robust in its unstructured simplicity.  Nodes
work all at once with little coordination.  They do not need to be identified, since messages are
not routed to any particular place and only need to be delivered on a best effort basis.  Nodes can
leave  and  rejoin  the  network  at  will,  accepting  the  proof-of-work  chain  as  proof  of  what
happened while they were gone.  They vote with their CPU power, expressing their acceptance of
valid blocks by working on extending them and rejecting invalid blocks by refusing to work on
them.  Any needed rules and incentives can be enforced with this consensus mechanism.
8

References
[1] W. Dai, "b-money," http://www.weidai.com/bmoney.txt, 1998.
[2] H. Massias, X.S. Avila, and J.-J. Quisquater, "Design of a secure timestamping service with minimal
trust requirements," In 20th Symposium on Information Theory in the Benelux, May 1999.
[3] S. Haber, W.S. Stornetta, "How to time-stamp a digital document," In Journal of Cryptology, vol 3, no
2, pages 99-111, 1991.
[4] D. Bayer, S. Haber, W.S. Stornetta, "Improving the efficiency and reliability of digital time-stamping,"
In Sequences II: Methods in Communication, Security and Computer Science, pages 329-334, 1993.
[5] S. Haber, W.S. Stornetta, "Secure names for bit-strings," In Proceedings of the 4th ACM Conference
on Computer and Communications Security, pages 28-35, April 1997.
[6] A. Back, "Hashcash - a denial of service counter-measure,"
http://www.hashcash.org/papers/hashcash.pdf, 2002.
[7] R.C. Merkle, "Protocols for public key cryptosystems," In Proc. 1980 Symposium on Security and
Privacy, IEEE Computer Society, pages 122-133, April 1980.
[8] W. Feller, "An introduction to probability theory and its applications," 1957.
9

Appears in the Proceedings of the Third Symposium on Operating Systems Design and Implementation, New Orleans, USA, February 1999
Practical Byzantine Fault Tolerance
Miguel Castro and Barbara Liskov
Laboratory for Computer Science,
Massachusetts Institute of Technology,
545 Technology Square, Cambridge, MA 02139
castro,liskov
 @lcs.mit.edu
Abstract
This paper describes a new replication algorithm that is able
to tolerate Byzantine faults. We believe that Byzantine-
fault-tolerant algorithms will be increasingly important in
the future because malicious attacks and software errors are
increasingly common and can cause faulty nodes to exhibit
arbitrary behavior. Whereas previous algorithms assumed a
synchronous system or were too slow to be used in practice,
the algorithm described in this paper is practical: it works in
asynchronous environments like the Internet and incorporates
several important optimizations that improve the response time
of previous algorithms by more than an order of magnitude. We
implemented a Byzantine-fault-tolerant NFS service using our
algorithm and measured its performance. The results show that
our service is only 3% slower than a standard unreplicated NFS.
1 Introduction
Malicious attacks and software errors are increasingly
common. The growing reliance of industry and gov-
ernment on online information services makes malicious
attacks more attractive and makes the consequences of
successful attacks more serious. In addition, the number
of software errors is increasing due to the growth in size
and complexity of software. Since malicious attacks and
software errors can cause faulty nodes to exhibit Byzan-
tine (i.e., arbitrary) behavior, Byzantine-fault-tolerant al-
gorithms are increasingly important.
This paper presents a new, practical algorithm for
state machine replication [17, 34] that tolerates Byzantine
faults. The algorithm offers both liveness and safety
provided at most
 1
3
 out of a total of
 replicas are
simultaneously faulty. This means that clients eventually
receive replies to their requests and those replies are
correct according to linearizability [14, 4]. The algorithm
works in asynchronous systems like the Internet and it
incorporates important optimizations that enable it to
perform ef®ciently.
There is a signi®cant body of work on agreement
This researchwas supportedin part by DARPA under contract DABT63-
95-C-005, monitored by Army Fort Huachuca, and under contract
F30602-98-1-0237, monitored by the Air Force Research Laboratory,
and in part by NEC. Miguel Castro was partially supported by a PRAXIS
XXI fellowship.
and replication techniques that tolerate Byzantine faults
(starting with [19]). However, most earlier work (e.g.,
[3, 24, 10]) either concerns techniques designed to
demonstrate theoretical feasibility that are too inef®cient
to be used in practice, or assumes synchrony, i.e.,
relies on known bounds on message delays and process
speeds. The systems closest to ours, Rampart [30] and
SecureRing [16], were designed to be practical, but they
rely on the synchrony assumption for correctness, which
is dangerous in the presence of malicious attacks. An
attacker may compromise the safety of a service by
delaying non-faulty nodes or the communication between
them until they are tagged as faulty and excluded from the
replica group. Such a denial-of-service attack is generally
easier than gaining control over a non-faulty node.
Our algorithm is not vulnerable to this type of
attack because it does not rely on synchrony for
safety. In addition, it improves the performance of
Rampart and SecureRing by more than an order of
magnitude as explained in Section 7. It uses only one
message round trip to execute read-only operations and
two to execute read-write operations. Also, it uses
an ef®cient authentication scheme based on message
authentication codes during normal operation; public-key
cryptography, which was cited as the major latency [29]
and throughput [22] bottleneck in Rampart, is used only
when there are faults.
To evaluate our approach, we implemented a replica-
tion library and used it to implement a real service: a
Byzantine-fault-tolerant distributed ®lesystem that sup-
ports the NFS protocol. We used the Andrew bench-
mark [15] to evaluate the performance of our system. The
results show that our system is only 3% slower than the
standard NFS daemon in the Digital Unix kernel during
normal-case operation.
Thus, the paper makes the following contributions:
It describes the ®rststate-machine replication proto-
col that correctly survives Byzantine faults in asyn-
chronous networks.
It describes a number of important optimizations that
allow the algorithm to perform well so that it can be
used in real systems.
1

It describes the implementation of a Byzantine-fault-
tolerant distributed ®lesystem.
It provides experimental results that quantify the cost
of the replication technique.
The remainder of the paper is organized as follows.
We begin by describing our system model, including our
failure assumptions. Section 3 describes the problem
solved by the algorithm and states correctness conditions.
The algorithm is described in Section 4 and some
important optimizations are described in Section 5.
Section 6 describes our replication library and how
we used it to implement a Byzantine-fault-tolerant
NFS. Section 7 presents the results of our experiments.
Section 8 discusses related work. We conclude with a
summary of what we have accomplished and a discussion
of future research directions.
2 System Model
We assume an asynchronous distributed system where
nodes are connected by a network. The network may
fail to deliver messages, delay them, duplicate them, or
deliver them out of order.
We use a Byzantine failure model, i.e., faulty nodes
may behave arbitrarily, subject only to the restriction
mentioned below. We assume independent node failures.
For this assumption to be true in the presence of malicious
attacks, some steps need to be taken, e.g., each node
should run different implementations of the service code
and operating system and should have a different root
password and a different administrator. It is possible
to obtain different implementations from the same code
base [28] and for low degrees of replication one can buy
operating systems from different vendors. N-version
programming, i.e., different teams of programmers
produce different implementations, is another option for
some services.
We use cryptographic techniques to prevent spoo®ng
and replays and to detect corrupted messages. Our
messages contain public-key signatures [33], message
authentication codes [36], and message digests produced
by collision-resistant hash functions [32]. We denote a
message
 signed by node
 as
 and the digest of
message
 by
 . We follow the common practice
of signing a digest of a message and appending it to
the plaintext of the message rather than signing the full
message (
 should be interpreted in this way). All
replicas know the others' public keys to verify signatures.
We allow for a very strong adversary that can
coordinate faulty nodes, delay communication, or delay
correct nodes in order to cause the most damage to the
replicated service. We do assume that the adversary
cannot delay correct nodes inde®nitely. We also assume
that the adversary (and the faulty nodes it controls)
are computationally bound so that (with very high
probability) it is unable to subvert the cryptographic
techniques mentioned above. For example, the adversary
cannot produce a valid signature of a non-faulty node,
compute the information summarized by a digest from
the digest, or ®nd two messages with the same digest.
The cryptographic techniques we use are thought to have
these properties [33, 36, 32].
3 Service Properties
Our algorithm can be used to implement any deterministic
replicated service with a state and some operations. The
operations are not restricted to simple reads or writes of
portions of the service state; they can perform arbitrary
deterministic computations using the state and operation
arguments. Clients issue requests to the replicated service
to invoke operations and block waiting for a reply. The
replicated service is implemented by
 replicas. Clients
and replicas are non-faulty if they follow the algorithm
in Section 4 and if no attacker can forge their signature.
The algorithm provides both safety and liveness assum-
ing no more than
 1
3
 replicas are faulty. Safety means
that the replicated service satis®es linearizability [14]
(modi®edto account for Byzantine-faulty clients [4]): it
behaves like a centralized implementation that executes
operations atomically one at a time. Safety requires the
bound on the number of faulty replicas because a faulty
replica can behave arbitrarily, e.g., it can destroy its state.
Safety is provided regardless of how many faulty
clients are using the service (even if they collude with
faulty replicas): all operations performed by faulty clients
are observed in a consistent way by non-faulty clients.
In particular, if the service operations are designed to
preserve some invariants on the service state, faulty
clients cannot break those invariants.
The safety property is insuf®cient to guard against
faulty clients, e.g., in a ®le system a faulty client can
write garbage data to some shared ®le. However, we
limit the amount of damage a faulty client can do by
providing access control: we authenticate clients and
deny access if the client issuing a request does not have
the right to invoke the operation. Also, services may
provide operations to change the access permissions for
a client. Since the algorithm ensures that the effects of
access revocation operations are observed consistently by
all clients, this provides a powerful mechanism to recover
from attacks by faulty clients.
The algorithm does not rely on synchrony to provide
safety. Therefore, it must rely on synchrony to provide
liveness; otherwise it could be used to implement
consensus in an asynchronous system, which is not
possible [9]. We guarantee liveness, i.e., clients
eventually receive replies to their requests, provided at
most
 1
3
 replicas are faulty and delay
 does not
2

grow faster than
 inde®nitely. Here, delay
 is the
time between the moment
 when a message is sent for
the ®rsttime and the moment when it is received by its
destination (assuming the sender keeps retransmitting the
message until it is received). (A more precise de®nition
can be found in [4].) This is a rather weak synchrony
assumption that is likely to be true in any real system
provided network faults are eventually repaired, yet it
enables us to circumvent the impossibility result in [9].
The resiliency of our algorithm is optimal: 3
 1 is the
minimum number of replicas that allow an asynchronous
system to provide the safety and liveness properties when
up to
 replicas are faulty (see [2] for a proof). This
many replicas are needed because it must be possible to
proceed after communicating with
 replicas, since
replicas might be faulty and not responding. However,
it is possible that the
 replicas that did not respond are
not faulty and, therefore,
 of those that responded might
be faulty. Even so, there must still be enough responses
that those from non-faulty replicas outnumber those from
faulty ones, i.e.,
 2
 . Therefore
 3
 .
The algorithm does not address the problem of fault-
tolerant privacy: a faulty replica may leak information to
an attacker. It is not feasible to offer fault-tolerant privacy
in the general case because service operations may
perform arbitrary computations using their arguments and
the service state; replicas need this information in the
clear to execute such operations ef®ciently. It is possible
to use secret sharing schemes [35] to obtain privacy even
in the presence of a threshold of malicious replicas [13]
for the arguments and portions of the state that are opaque
to the service operations. We plan to investigate these
techniques in the future.
4 The Algorithm
Our algorithm is a form of state machine replication [17,
34]: the service is modeled as a state machine that is
replicated across different nodes in a distributed system.
Each state machine replica maintains the service state
and implements the service operations. We denote the
set of replicas by
 and identify each replica using an
integer in
 0
 1
 . For simplicity, we assume
3
 1 where
 is the maximum number of
replicas that may be faulty; although there could be
more than 3
 1 replicas, the additional replicas degrade
performance (since more and bigger messages are being
exchanged) without providing improved resiliency.
The replicas move through a succession of con®gura-
tions called views. In a view one replica is the primary
and the others are backups. Views are numbered con-
secutively. The primary of a view is replica
 such that
mod
 , where
 is the view number. View
changes are carried out when it appears that the primary
has failed. Viewstamped Replication [26] and Paxos [18]
used a similar approach to tolerate benign faults (as dis-
cussed in Section 8.)
The algorithm works roughly as follows:
1. A client sends a request to invoke a service operation
to the primary
2. The primary multicasts the request to the backups
3. Replicas execute the request and send a reply to the
client
4. The client waits for
 1 replies from different
replicas with the same result; this is the result of
the operation.
Like all state machine replication techniques [34],
we impose two requirements on replicas: they must
be deterministic (i.e., the execution of an operation in
a given state and with a given set of arguments must
always produce the same result) and they must start in the
same state. Given these two requirements, the algorithm
ensures the safety property by guaranteeing that all non-
faulty replicas agree on a total order for the execution of
requests despite failures.
The remainder of this section describes a simpli®ed
version of the algorithm. We omit discussion of how
nodes recover from faults due to lack of space. We
also omit details related to message retransmissions.
Furthermore, we assume that message authentication is
achieved using digital signatures rather than the more
ef®cientscheme based on message authentication codes;
Section 5 discusses this issue further. A detailed
formalization of the algorithm using the I/O automaton
model [21] is presented in [4].
4.1 The Client
A client
 requests the execution of state machine
operation
 by sending a
 REQUEST
 message
to the primary. Timestamp
 is used to ensure exactly-
once semantics for the execution of client requests.
Timestamps for
 's requests are totally ordered such that
later requests have higher timestamps than earlier ones;
for example, the timestamp could be the value of the
client's local clock when the request is issued.
Each message sent by the replicas to the client includes
the current view number, allowing the client to track the
view and hence the current primary. A client sends
a request to what it believes is the current primary
using a point-to-point message. The primary atomically
multicasts the request to all the backups using the protocol
described in the next section.
A replica sends the reply to the request directly to
the client. The reply has the form
 REPLY
where
 is the current view number,
 is the timestamp of
the corresponding request,
 is the replica number, and
is the result of executing the requested operation.
The client waits for
 1 replies with valid signatures
from different replicas, and with the same
 and
 , before
3

accepting the result
 . This ensures that the result is valid,
since at most
 replicas can be faulty.
If the client does not receive replies soon enough, it
broadcasts the request to all replicas. If the request has
already been processed, the replicas simply re-send the
reply; replicas remember the last reply message they sent
to each client. Otherwise, if the replica is not the primary,
it relays the request to the primary. If the primary does
not multicast the request to the group, it will eventually
be suspected to be faulty by enough replicas to cause a
view change.
In this paper we assume that the client waits for one
request to complete before sending the next one. But we
can allow a client to make asynchronous requests, yet
preserve ordering constraints on them.
4.2 Normal-Case Operation
The state of each replica includes the state of the
service, a message log containing messages the replica
has accepted, and an integer denoting the replica's current
view. We describe how to truncate the log in Section 4.3.
When the primary,
 , receives a client request,
 ,
it starts a three-phase protocol to atomically multicast
the request to the replicas. The primary starts the
protocol immediately unless the number of messages
for which the protocol is in progress exceeds a given
maximum. In this case, it buffers the request. Buffered
requests are multicast later as a group to cut down on
message traf®cand CPU overheads under heavy load;this
optimization is similar to a group commit in transactional
systems [11]. For simplicity, we ignore this optimization
in the description below.
The three phases are pre-prepare, prepare, and commit.
The pre-prepare and prepare phases are used to totally
order requests sent in the same view even when the
primary, which proposes the ordering of requests, is
faulty. The prepare and commit phases are used to ensure
that requests that commit are totally ordered across views.
In the pre-prepare phase, the primary assigns a
sequence number,
 , to the request, multicasts a pre-
prepare message with
 piggybacked to all the backups,
and appends the message to its log. The message has the
form
 PRE-PREPARE
 , where
 indicates
the view in which the message is being sent,
 is the
client's request message, and
 is
 's digest.
Requests are not included in pre-prepare messages
to keep them small. This is important because pre-
prepare messages are used as a proof that the request was
assigned sequence number
 in view
 in view changes.
Additionally, it decouples the protocol to totally order
requests from the protocol to transmit the request to the
replicas; allowing us to use a transport optimized for
small messages for protocol messages and a transport
optimized for large messages for large requests.
A backup accepts a pre-prepare message provided:
the signatures in the request and the pre-prepare
message are correct and
 is the digest for
 ;
it is in view
 ;
it has not accepted a pre-prepare message for view
and sequence number
 containing a different digest;
the sequence number in the pre-prepare message is
between a low water mark,
 , and a high water mark,
.
The last condition prevents a faulty primary from
exhausting the space of sequence numbers by selecting
a very large one. We discuss how
 and
 advance in
Section 4.3.
If backup
 accepts the
 PRE-PREPARE
message, it enters the prepare phase by multicasting a
PREPARE
 message to all other replicas and
adds both messages to its log. Otherwise, it does nothing.
A replica (including the primary) accepts prepare
messages and adds them to its log provided their
signatures are correct, their view number equals the
replica's current view, and their sequence number is
between
 and
 .
We de®nethe predicate prepared
 to be true
if and only if replica
 has inserted in its log: the request
, a pre-prepare for
 in view
 with sequence number
, and 2
 prepares from different backups that match
the pre-prepare. The replicas verify whether the prepares
match the pre-prepare by checking that they have the
same view, sequence number, and digest.
The pre-prepare and prepare phases of the algorithm
guarantee that non-faulty replicas agree on a total order
for the requests within a view. More precisely, they
ensure the following invariant: if prepared
 is
true then prepared
 is false for any non-faulty
replica
 (including
 ) and any
 such that
. This is true because prepared
 and
3
 1 imply that at least
 1 non-faulty replicas
have sent a pre-prepare or prepare for
 in view
 with
sequence number
 . Thus, for prepared
to be true at least one of these replicas needs to have
sent two con¯icting prepares (or pre-prepares if it is the
primary for
 ), i.e., two prepares with the same view
and sequence number and a different digest. But this is
not possible because the replica is not faulty. Finally, our
assumption about the strength of message digests ensures
that the probability that
 and
 is
negligible.
Replica
 multicasts a
 COMMIT
 to the
other replicas when prepared
 becomes true.
This starts the commit phase. Replicas accept commit
messages and insert them in their log provided they are
properly signed, the view number in the message is equal
to the replica's current view, and the sequence number is
between
 and
4

We de®ne the committed and committed-local predi-
cates as follows: committed
 is true if and only
if prepared
 is true for all
 in some set of
1 non-faulty replicas; and committed-local
is true if and only if prepared
 is true and
 has
accepted 2
 1 commits (possibly including its own)
from different replicas that match the pre-prepare for
 ;
a commit matches a pre-prepare if they have the same
view, sequence number, and digest.
The commit phase ensures the following invariant: if
committed-local
 is true for some non-faulty
then committed
 is true. This invariant and
the view-change protocol described in Section 4.4 ensure
that non-faulty replicas agree on the sequence numbers
of requests that commit locally even if they commit in
different views at each replica. Furthermore, it ensures
that any request that commits locally at a non-faulty
replica will commit at
 1 or more non-faulty replicas
eventually.
Each replica
 executes the operation requested by
after committed-local
 is true and
 's state
re¯ects the sequential execution of all requests with
lower sequence numbers. This ensures that all non-
faulty replicas execute requests in the same order as
required to provide the safety property. After executing
the requested operation, replicas send a reply to the client.
Replicas discard requests whose timestamp is lower than
the timestamp in the last reply they sent to the client to
guarantee exactly-once semantics.
We do not rely on ordered message delivery, and
therefore it is possible for a replica to commit requests
out of order. This does not matter since it keeps the pre-
prepare, prepare, and commit messages logged until the
corresponding request can be executed.
Figure 1 shows the operation of the algorithm in the
normal case of no primary faults. Replica 0 is the primary,
replica 3 is faulty, and
 is the client.
X
request pre-prepare prepare commit reply
C
0
1
2
3
Figure 1: Normal Case Operation
4.3 Garbage Collection
This section discusses the mechanism used to discard
messages from the log. For the safety condition to hold,
messages must be kept in a replica's log until it knows that
the requests they concern have been executed by at least
1 non-faulty replicas and it can prove this to others
in view changes. In addition, if some replica misses
messages that were discarded by all non-faulty replicas,
it will need to be brought up to date by transferring all
or a portion of the service state. Therefore, replicas also
need some proof that the state is correct.
Generating these proofs after executing every opera-
tion would be expensive. Instead, they are generated
periodically, when a request with a sequence number di-
visible by some constant (e.g., 100) is executed. We will
refer to the states produced by the execution of these re-
quests as checkpoints and we will say that a checkpoint
with a proof is a stable checkpoint.
A replica maintains several logical copies of the service
state: the last stable checkpoint, zero or more checkpoints
that are not stable, and a current state. Copy-on-write
techniques can be used to reduce the space overhead
to store the extra copies of the state, as discussed in
Section 6.3.
The proof of correctness for a checkpoint is generated
as follows. When a replica
 produces a checkpoint,
it multicasts a message
 CHECKPOINT
 to the
other replicas, where
 is the sequence number of the
last request whose execution is re¯ected in the state
and
 is the digest of the state. Each replica collects
checkpoint messages in its log until it has 2
 1 of
them for sequence number
 with the same digest
signed by different replicas (including possibly its own
such message). These 2
 1 messages are the proof of
correctness for the checkpoint.
A checkpoint with a proof becomes stable and the
replica discards all pre-prepare, prepare, and commit
messages with sequence number less than or equal to
from its log; it also discards all earlier checkpoints and
checkpoint messages.
Computing the proofs is ef®cient because the digest
can be computed using incremental cryptography [1] as
discussed in Section 6.3, and proofs are generated rarely.
The checkpoint protocol is used to advance the low
and high water marks (which limit what messages will
be accepted). The low-water mark
 is equal to the
sequence number of the last stable checkpoint. The high
water mark
 , where
 is big enough so that
replicas do not stall waiting for a checkpoint to become
stable. For example, if checkpoints are taken every 100
requests,
 might be 200.
4.4 View Changes
The view-change protocol provides liveness by allowing
the system to make progress when the primary fails. View
changes are triggered by timeouts that prevent backups
from waiting inde®nitely for requests to execute. A
backup is waitingfor a request if it received a valid request
5

and has not executed it. A backup starts a timer when it
receives a request and the timer is not already running.
It stops the timer when it is no longer waiting to execute
the request, but restarts it if at that point it is waiting to
execute some other request.
If the timer of backup
 expires in view
 , the
backup starts a view change to move the system to
view
 1. It stops accepting messages (other than
checkpoint, view-change, and new-view messages) and
multicasts a
 VIEW-CHANGE
 1
 message
to all replicas. Here
 is the sequence number of the last
stable checkpoint
 known to
 ,
 is a set of 2
 1 valid
checkpoint messages proving the correctness of
 , and
is a set containing a set
 for each request
 that
prepared at
 with a sequence number higher than
 . Each
set
 contains a valid pre-prepare message (without the
corresponding client message) and 2
 matching, valid
prepare messages signed by different backups with the
same view, sequence number, and the digest of
 .
When the primary
 of view
 1 receives 2
 valid
view-change messages for view
 1 from other replicas,
it multicasts a
 NEW-VIEW
 1
 message to all
other replicas, where
 is a set containing the valid view-
change messages received by the primary plus the view-
change message for
 1 the primary sent (or would have
sent), and
 is a set of pre-prepare messages (without the
piggybacked request).
 is computed as follows:
1. The primary determines the sequence number min-s
of the latest stable checkpoint in
 and the highest
sequence number max-s in a prepare message in
 .
2. The primary creates a new pre-prepare message for
view
 1 for each sequence number
 between min-s
and max-s. There are two cases: (1) there is at least
one set in the
 component of some view-change
message in
 with sequence number
 , or (2) there
is no such set. In the ®rstcase, the primary creates
a new message
 PRE-PREPARE
 1
 , where
is the request digest in the pre-prepare message for
sequence number
 with the highest view number
in
 . In the second case, it creates a new pre-
prepare message
 PRE-PREPARE
 1
 ,
where
 is the digest of a special null request;
a null request goes through the protocol like other
requests, but its execution is a no-op. (Paxos [18]
used a similar technique to ®llin gaps.)
Next the primary appends the messages in
 to its
log. If min-s is greater than the sequence number of its
latest stable checkpoint, the primary also inserts the proof
of stability for the checkpoint with sequence number
min-s in its log, and discards information from the log
as discussed in Section 4.3. Then it enters view
 1: at
this point it is able to accept messages for view
 1.
A backup accepts a new-view message for view
 1
if it is signed properly, if the view-change messages it
contains are valid for view
 1, and if the set
 is
correct; it veri®esthe correctness of
 by performing a
computation similar to the one used by the primary to
create
 . Then it adds the new information to its log as
described for the primary, multicasts a prepare for each
message in
 to all the other replicas, adds these prepares
to its log, and enters view
 1.
Thereafter, the protocol proceeds as described in
Section 4.2. Replicas redo the protocol for messages
between min-s and max-s but they avoid re-executing
client requests (by using their stored information about
the last reply sent to each client).
A replica may be missing some request message
or a stable checkpoint (since these are not sent in new-
view messages.) It can obtain missing information from
another replica. For example, replica
 can obtain a
missing checkpoint state
 from one of the replicas
whose checkpoint messages certi®ed its correctness in
. Since
 1 of those replicas are correct, replica
 will
always obtain
 or a later certi®edstable checkpoint. We
can avoid sending the entire checkpoint by partitioning
the state and stamping each partition with the sequence
number of the last request that modi®ed it. To bring
a replica up to date, it is only necessary to send it the
partitions where it is out of date, rather than the whole
checkpoint.
4.5 Correctness
This section sketches the proof that the algorithm
provides safety and liveness; details can be found in [4].
4.5.1 Safety
As discussed earlier, the algorithm provides safety if all
non-faulty replicas agree on the sequence numbers of
requests that commit locally.
In Section 4.2, we showed that if prepared
is true, prepared
 is false for any non-faulty
replica
 (including
 ) and any
 such that
. This implies that two non-faulty
replicas agree on the sequence number of requests that
commit locally in the same view at the two replicas.
The view-change protocol ensures that non-faulty
replicas also agree on the sequence number of requests
that commit locally in different views at different replicas.
A request
 commits locally at a non-faulty replica with
sequence number
 in view
 only if committed
is true. This means that there is a set
 1 containing at least
1 non-faulty replicas such that prepared
is true for every replica
 in the set.
Non-faulty replicas will not accept a pre-prepare for
view
 without having received a new-view message
for
 (since only at that point do they enter the view). But
any correct new-view message for view
 contains
correct view-change messages from every replica
 in a
6

set
 2 of 2
 1 replicas. Since there are 3
 1 replicas,
1 and
 2 must intersect in at least one replica
 that is
not faulty.
 's view-change message will ensure that the
fact that
 prepared in a previous view is propagated to
subsequent views, unless the new-view message contains
a view-change message with a stable checkpoint with a
sequence number higher than
 . In the ®rst case, the
algorithm redoes the three phases of the atomic multicast
protocol for
 with the same sequence number
 and the
new view number. This is important because it prevents
any different request that was assigned the sequence
number
 in a previous view from ever committing. In
the second case no replica in the new view will accept any
message with sequence number lower than
 . In either
case, the replicas will agree on the request that commits
locally with sequence number
 .
4.5.2 Liveness
To provide liveness, replicas must move to a new view if
they are unable to execute a request. But it is important
to maximize the period of time when at least 2
 1
non-faulty replicas are in the same view, and to ensure
that this period of time increases exponentially until some
requested operation executes. We achieve these goals by
three means.
First, to avoid starting a view change too soon, a replica
that multicasts a view-change message for view
 1
waits for 2
 1 view-change messages for view
 1
and then starts its timer to expire after some time
 .
If the timer expires before it receives a valid new-view
message for
 1 or before it executes a request in the
new view that it had not executed previously, it starts the
view change for view
 2 but this time it will wait 2
before starting a view change for view
 3.
Second, if a replica receives a set of
 1 valid view-
change messages from other replicas for views greater
than its current view, it sends a view-change message
for the smallest view in the set, even if its timer has
not expired; this prevents it from starting the next view
change too late.
Third, faulty replicas are unable to impede progress
by forcing frequent view changes. A faulty replica
cannot cause a view change by sending a view-change
message, because a view change will happen only if at
least
 1 replicas send view-change messages, but it
can cause a view change when it is the primary (by not
sending messages or sending bad messages). However,
because the primary of view
 is the replica
 such that
mod
 , the primary cannot be faulty for more
than
 consecutive views.
These three techniques guarantee liveness unless
message delays grow faster than the timeout period
inde®nitely, which is unlikely in a real system.
4.6 Non-Determinism
State machine replicas must be deterministic but many
services involve some form of non-determinism. For
example, the time-last-modi®edin NFS is set by reading
the server's local clock; if this were done independently
at each replica, the states of non-faulty replicas would
diverge. Therefore, some mechanism to ensure that all
replicas select the same value is needed. In general, the
client cannot select the value because it does not have
enough information; for example, it does not know how
its request will be ordered relative to concurrent requests
by other clients. Instead, the primary needs to select the
value either independently or based on values provided
by the backups.
If the primary selects the non-deterministic value inde-
pendently, it concatenates the value with the associated
request and executes the three phase protocol to ensure
that non-faulty replicas agree on a sequence number for
the request and value. This prevents a faulty primary from
causing replica state to diverge by sending different val-
ues to different replicas. However, a faulty primary might
send the same, incorrect, value to all replicas. Therefore,
replicas must be able to decide deterministically whether
the value is correct (and what to do if it is not) based only
on the service state.
This protocol is adequate for most services (including
NFS) but occasionally replicas must participate in
selecting the value to satisfy a service's speci®cation.
This can be accomplished by adding an extra phase to
the protocol: the primary obtains authenticated values
proposed by the backups, concatenates 2
 1 of them
with the associated request, and starts the three phase
protocol for the concatenated message. Replicas choose
the value by a deterministic computation on the 2
 1
values and their state, e.g., taking the median. The extra
phase can be optimized away in the common case. For
example, if replicas need a value that is ªclose enoughº
to that of their local clock, the extra phase can be avoided
when their clocks are synchronized within some delta.
5 Optimizations
This section describes some optimizations that improve
the performance of the algorithm during normal-case
operation. All the optimizations preserve the liveness
and safety properties.
5.1 Reducing Communication
We use three optimizations to reduce the cost of
communication. The ®rst avoids sending most large
replies. A client request designates a replica to send
the result; all other replicas send replies containing just
the digest of the result. The digests allow the client to
check the correctness of the result while reducing network
7

bandwidth consumption and CPU overhead signi®cantly
for large replies. If the client does not receive a correct
result from the designated replica, it retransmits the
request as usual, requesting all replicas to send full
replies.
The second optimization reduces the number of
message delays for an operation invocation from 5
to 4. Replicas execute a request tentatively as soon
as the prepared predicate holds for the request, their
state re¯ects the execution of all requests with lower
sequence number, and these requests are all known to
have committed. After executing the request, the replicas
send tentative replies to the client. The client waits for
2
 1 matching tentative replies. If it receives this
many, the request is guaranteed to commit eventually.
Otherwise, the client retransmits the request and waits
for
 1 non-tentative replies.
A request that has executed tentatively may abort if
there is a view change and it is replaced by a null
request. In this case the replica reverts its state to the
last stable checkpoint in the new-view message or to its
last checkpointed state (depending on which one has the
higher sequence number).
The third optimization improves the performance of
read-only operations that do not modify the service
state. A client multicasts a read-only request to all
replicas. Replicas execute the request immediately in
their tentative state after checking that the request is
properly authenticated, that the client has access, and
that the request is in fact read-only. They send the reply
only after all requests re¯ected in the tentative state have
committed; this is necessary to prevent the client from
observing uncommitted state. The client waits for 2
 1
replies from different replicas with the same result. The
client may be unable to collect 2
 1 such replies if there
are concurrent writes to data that affect the result; in this
case, it retransmits the request as a regular read-write
request after its retransmission timer expires.
5.2 Cryptography
In Section 4, we described an algorithm that uses
digital signatures to authenticate all messages. However,
we actually use digital signatures only for view-
change and new-view messages, which are sent rarely,
and authenticate all other messages using message
authentication codes (MACs). This eliminates the main
performance bottleneck in previous systems [29, 22].
However, MACs have a fundamental limitation rela-
tive to digital signatures Ð the inability to prove that
a message is authentic to a third party. The algorithm
in Section 4 and previous Byzantine-fault-tolerant algo-
rithms [31, 16] for state machine replication rely on the
extra power of digital signatures. We modi®edour algo-
rithm to circumvent the problem by taking advantage of
speci®cinvariants, e.g, the invariant that no two different
requests prepare with the same view and sequence num-
ber at two non-faulty replicas. The modi®edalgorithm is
described in [5]. Here we sketch the main implications
of using MACs.
MACs can be computed three orders of magnitude
faster than digital signatures. For example, a 200MHz
Pentium Pro takes 43ms to generate a 1024-bit modulus
RSA signature of an MD5 digest and 0.6ms to verify
the signature [37], whereas it takes only 10.3
 s to
compute the MAC of a 64-byte message on the same
hardware in our implementation. There are other public-
key cryptosystems that generate signatures faster, e.g.,
elliptic curve public-key cryptosystems, but signature
veri®cation is slower [37] and in our algorithm each
signature is veri®edmany times.
Each node (including active clients) shares a 16-byte
secret session key with each replica. We compute
message authentication codes by applying MD5 to the
concatenation of the message with the secret key. Rather
than using the 16 bytes of the ®nalMD5 digest, we use
only the 10 least signi®cantbytes. This truncation has
the obvious advantage of reducing the size of MACs and
it also improves their resilience to certain attacks [27].
This is a variant of the secret suf®xmethod [36], which
is secure as long as MD5 is collision resistant [27, 8].
The digital signature in a reply message is replaced by a
single MAC, which is suf®cientbecause these messages
have a single intended recipient. The signatures in all
other messages (including client requests but excluding
view changes) are replaced by vectors of MACs that we
call authenticators. An authenticator has an entry for
every replica other than the sender; each entry is the
MAC computed with the key shared by the sender and
the replica corresponding to the entry.
The time to verify an authenticator is constant but the
time to generate one grows linearly with the number of
replicas. This is not a problem because we do not expect
to have a large number of replicas and there is a huge
performance gap between MAC and digital signature
computation. Furthermore, we compute authenticators
ef®ciently;MD5 is applied to the message once and the
resulting context is used to compute each vector entry
by applying MD5 to the corresponding session key. For
example, in a system with 37 replicas (i.e., a system
that can tolerate 12 simultaneous faults) an authenticator
can still be computed much more than two orders of
magnitude faster than a 1024-bit modulus RSA signature.
The size of authenticators grows linearly with the
number of replicas but it grows slowly: it is equal to
30
 1
3
 bytes. An authenticator is smaller than an
RSA signature with a 1024-bit modulus for
 13 (i.e.,
systems that can tolerate up to 4 simultaneous faults),
which we expect to be true in most con®gurations.
8

6 Implementation
This section describes our implementation. First we
discuss the replication library, which can be used as
a basis for any replicated service. In Section 6.2 we
describe how we implemented a replicated NFS on top
of the replication library. Then we describe how we
maintain checkpoints and compute checkpoint digests
ef®ciently.
6.1 The Replication Library
The client interface to the replication library consists of
a single procedure, invoke, with one argument, an input
buffer containing a request to invoke a state machine
operation. The invoke procedure uses our protocol to
execute the requested operation at the replicas and select
the correct reply from among the replies of the individual
replicas. It returns a pointer to a buffer containing the
operation result.
On the server side, the replication code makes a
number of upcalls to procedures that the server part of
the application must implement. There are procedures
to execute requests (execute), to maintain checkpoints of
the service state (make checkpoint, delete checkpoint), to
obtain the digest of a speci®edcheckpoint (get digest),
and to obtain missing information (get checkpoint,
set checkpoint). The execute procedure receives as input
a buffer containing the requested operation, executes the
operation, and places the result in an output buffer. The
other procedures are discussed further in Sections 6.3
and 6.4.
Point-to-point communication between nodes is imple-
mented using UDP, and multicast to the group of replicas
is implemented using UDP over IP multicast [7]. There
is a single IP multicast group for each service, which con-
tains all the replicas. These communication protocols are
unreliable; they may duplicate or lose messages or deliver
them out of order.
The algorithm tolerates out-of-order delivery and
rejects duplicates. View changes can be used to recover
from lost messages, but this is expensive and therefore it
is important to perform retransmissions. During normal
operation recovery from lost messages is driven by
the receiver: backups send negative acknowledgments
to the primary when they are out of date and the
primary retransmits pre-prepare messages after a long
timeout. A reply to a negative acknowledgment may
include both a portion of a stable checkpoint and missing
messages. During view changes, replicas retransmit
view-change messages until they receive a matching new-
view message or they move on to a later view.
The replication library does not implement view
changes or retransmissions at present. This does
not compromise the accuracy of the results given
in Section 7 because the rest of the algorithm is
completely implemented (including the manipulation of
the timers that trigger view changes) and because we
have formalized the complete algorithm and proved its
correctness [4].
6.2 BFS: A Byzantine-Fault-tolerant File System
We implemented BFS, a Byzantine-fault-tolerant NFS
service, using the replication library. Figure 2 shows the
architecture of BFS. We opted not to modify the kernel
NFS client and server because we did not have the sources
for the Digital Unix kernel.
A ®lesystem exported by the fault-tolerant NFS service
is mounted on the client machine like any regular NFS
®le system. Application processes run unmodi®edand
interact with the mounted ®le system through the NFS
client in the kernel. We rely on user level relay processes
to mediate communication between the standard NFS
client and the replicas. A relay receives NFS protocol
requests, calls the invoke procedure of our replication
library, and sends the result back to the NFS client.
Andrew
benchmark
kernel NFS client
replication
library
relay
client
replica 0
replication
library
snfsd
kernel VM
replica n
replication
library
snfsd
kernel VM
Figure 2: Replicated File System Architecture.
Each replica runs a user-level process with the
replication library and our NFS V2 daemon, which we
will refer to as snfsd (for simple nfsd). The replication
library receives requests from the relay, interacts with
snfsd by making upcalls, and packages NFS replies into
replication protocol replies that it sends to the relay.
We implemented snfsd using a ®xed-size memory-
mapped ®le. All the ®le system data structures, e.g.,
inodes, blocks and their free lists, are in the mapped ®le.
We rely on the operating system to manage the cache of
memory-mapped ®lepages and to write modi®edpages
to disk asynchronously. The current implementation
uses 8KB blocks and inodes contain the NFS status
information plus 256 bytes of data, which is used to store
directory entries in directories, pointers to blocks in ®les,
and text in symbolic links. Directories and ®lesmay also
use indirect blocks in a way similar to Unix.
Our implementation ensures that all state machine
9

replicas start in the same initial state and are deterministic,
which are necessary conditions for the correctness of a
service implemented using our protocol. The primary
proposes the values for time-last-modi®ed and time-
last-accessed, and replicas select the larger of the
proposed value and one greater than the maximum of all
values selected for earlier requests. We do not require
synchronous writes to implement NFS V2 protocol
semantics because BFS achieves stability of modi®ed
data and meta-data through replication [20].
6.3 Maintaining Checkpoints
This section describes how snfsd maintains checkpoints
of the ®lesystem state. Recall that each replica maintains
several logical copies of the state: the current state, some
number of checkpoints that are not yet stable, and the last
stable checkpoint.
snfsd executes ®le system operations directly in the
memory mapped ®leto preserve locality,and it uses copy-
on-write to reduce the space and time overhead associated
with maintaining checkpoints. snfsd maintains a copy-
on-write bit for every 512-byte block in the memory
mapped ®le. When the replication code invokes the
make checkpoint upcall, snfsd sets all the copy-on-write
bits and creates a (volatile) checkpoint record, containing
the current sequence number, which it receives as an
argument to the upcall, and a list of blocks. This list
contains the copies of the blocks that were modi®ed
since the checkpoint was taken, and therefore, it is
initially empty. The record also contains the digest of
the current state; we discuss how the digest is computed
in Section 6.4.
When a block of the memory mapped ®leis modi®ed
while executing a client request, snfsd checks the copy-
on-write bit for the block and, if it is set, stores the block's
current contents and its identi®erin the checkpoint record
for the last checkpoint. Then, it overwrites the block
with its new value and resets its copy-on-write bit.
snfsd retains a checkpoint record until told to discard
it via a delete checkpoint upcall, which is made by the
replication code when a later checkpoint becomes stable.
If the replication code requires a checkpoint to send
to another replica, it calls the get checkpoint upcall. To
obtain the value for a block, snfsd ®rstsearches for the
block in the checkpoint record of the stable checkpoint,
and then searches the checkpoint records of any later
checkpoints. If the block is not in any checkpoint record,
it returns the value from the current state.
The use of the copy-on-write technique and the fact
that we keep at most 2 checkpoints ensure that the space
and time overheads of keeping several logical copies
of the state are low. For example, in the Andrew
benchmark experiments described in Section 7, the
average checkpoint record size is only 182 blocks with a
maximum of 500.
6.4 Computing Checkpoint Digests
snfsd computes a digest of a checkpoint state as part
of a make checkpoint upcall. Although checkpoints
are only taken occasionally, it is important to compute
the state digest incrementally because the state may be
large. snfsd uses an incremental collision-resistant one-
way hash function called AdHash [1]. This function
divides the state into ®xed-size blocks and uses some
other hash function (e.g., MD5) to compute the digest
of the string obtained by concatenating the block index
with the block value for each block. The digest of the
state is the sum of the digests of the blocks modulo some
large integer. In our current implementation, we use the
512-byte blocks from the copy-on-write technique and
compute their digest using MD5.
To compute the digest for the state incrementally, snfsd
maintains a table with a hash value for each 512-byte
block. This hash value is obtained by applying MD5
to the block index concatenated with the block value at
the time of the last checkpoint. When make checkpoint
is called, snfsd obtains the digest
 for the previous
checkpoint state (from the associated checkpoint record).
It computes new hash values for each block whose copy-
on-write bit is reset by applying MD5 to the block index
concatenated with the current block value. Then, it adds
the new hash value to
 , subtracts the old hash value
from
 , and updates the table to contain the new hash
value. This process is ef®cientprovided the number of
modi®edblocks is small; as mentioned above, on average
182 blocks are modi®edper checkpoint for the Andrew
benchmark.
7 Performance Evaluation
This section evaluates the performance of our system
using two benchmarks: a micro-benchmark and the
Andrew benchmark [15]. The micro-benchmark provides
a service-independent evaluation of the performance of
the replication library; it measures the latency to invoke
a null operation, i.e., an operation that does nothing.
The Andrew benchmark is used to compare BFS with
two other ®lesystems: one is the NFS V2 implementation
in Digital Unix, and the other is identical to BFS except
without replication. The ®rstcomparison demonstrates
that our system is practical by showing that its latency is
similar to the latency of a commercial system that is used
daily by many users. The second comparison allows us to
evaluate the overhead of our algorithm accurately within
an implementation of a real service.
7.1 Experimental Setup
The experiments measure normal-case behavior (i.e.,
there are no view changes), because this is the behavior
10

that determines the performance of the system. All
experiments ran with one client running two relay
processes, and four replicas. Four replicas can tolerate
one Byzantine fault; we expect this reliability level to
suf®ce for most applications. The replicas and the
client ran on identical DEC 3000/400 Alpha workstations.
These workstations have a 133 MHz Alpha 21064
processor, 128 MB of memory, and run Digital Unix
version 4.0. The ®lesystem was stored by each replica
on a DEC RZ26 disk. All the workstations were
connected by a 10Mbit/s switched Ethernet and had DEC
LANCE Ethernet interfaces. The switch was a DEC
EtherWORKS 8T/TX. The experiments were run on an
isolated network.
The interval between checkpoints was 128 requests,
which causes garbage collection to occur several times in
any of the experiments. The maximum sequence number
accepted by replicas in pre-prepare messages was 256
plus the sequence number of the last stable checkpoint.
7.2 Micro-Benchmark
The micro-benchmark measures the latency to invoke
a null operation. It evaluates the performance of two
implementations of a simple service with no state that
implements null operations with arguments and results
of different sizes. The ®rstimplementation is replicated
using our library and the second is unreplicated and
uses UDP directly. Table 1 reports the response times
measured at the client for both read-only and read-
write operations. They were obtained by timing 10,000
operation invocations in three separate runs and we report
the median value of the three runs. The maximum
deviation from the median was always below 0.3% of
the reported value. We denote each operation by a/b,
where a and b are the sizes of the operation argument and
result in KBytes.
arg./res. replicated without
(KB) read-write read-only replication
0/0 3.35 (309%) 1.62 (98%) 0.82
4/0 14.19 (207%) 6.98 (51%) 4.62
0/4 8.01 (72%) 5.94 (27%) 4.66
Table 1: Micro-benchmark results (in milliseconds); the
percentage overhead is relative to the unreplicated case.
The overhead introduced by the replication library is
due to extra computation and communication. For exam-
ple, the computation overhead for the read-write 0/0 op-
eration is approximately 1.06ms, which includes 0.55ms
spent executing cryptographic operations. The remain-
ing 1.47ms of overhead are due to extra communication;
the replication library introduces an extra message round-
trip, it sends larger messages, and it increases the number
of messages received by each node relative to the service
without replication.
The overhead for read-only operations is signi®cantly
lower because the optimization discussed in Section 5.1
reduces both computation and communication overheads.
For example, the computation overhead for the read-only
0/0 operation is approximately 0.43ms, which includes
0.23ms spent executing cryptographic operations, and
the communication overhead is only 0.37ms because the
protocol to execute read-only operations uses a single
round-trip.
Table 1 shows that the relative overhead is lower for
the 4/0 and 0/4 operations. This is because a signi®cant
fraction of the overhead introduced by the replication
library is independent of the size of operation arguments
and results. For example, in the read-write 0/4 operation,
the large message (the reply) goes over the network
only once (as discussed in Section 5.1) and only the
cryptographic overhead to process the reply message is
increased. The overhead is higher for the read-write 4/0
operation because the large message (the request) goes
over the network twice and increases the cryptographic
overhead for processing both request and pre-prepare
messages.
It is important to note that this micro-benchmark
represents the worst case overhead for our algorithm
because the operations perform no work and the
unreplicated server provides very weak guarantees.
Most services will require stronger guarantees, e.g.,
authenticated connections, and the overhead introduced
by our algorithm relative to a server that implements these
guarantees will be lower. For example, the overhead
of the replication library relative to a version of the
unreplicated service that uses MACs for authentication
is only 243% for the read-write 0/0 operation and 4% for
the read-only 4/0 operation.
We can estimate a rough lower bound on the
performance gain afforded by our algorithm relative to
Rampart [30]. Reiter reports that Rampart has a latency
of 45ms for a multi-RPC of a null message in a 10 Mbit/s
Ethernet network of 4 SparcStation 10s [30]. The multi-
RPC is suf®cientfor the primary to invoke a state machine
operation but for an arbitrary client to invoke an operation
it would be necessary to add an extra message delay and
an extra RSA signature and veri®cationto authenticate
the client; this would lead to a latency of at least 65ms
(using the RSA timings reported in [29].) Even if we
divide this latency by 1.7, the ratio of the SPECint92
ratings of the DEC 3000/400 and the SparcStation 10, our
algorithm still reduces the latency to invoke the read-write
and read-only 0/0 operations by factors of more than 10
and 20, respectively. Note that this scaling is conservative
because the network accounts for a signi®cant fraction
of Rampart's latency [29] and Rampart's results were
obtained using 300-bit modulus RSA signatures, which
are not considered secure today unless the keys used to
11

generate them are refreshed very frequently.
There are no published performance numbers for
SecureRing [16] but it would be slower than Rampart
because its algorithm has more message delays and
signature operations in the critical path.
7.3 Andrew Benchmark
The Andrew benchmark [15] emulates a software
development workload. It has ®ve phases: (1) creates
subdirectories recursively; (2) copies a source tree; (3)
examines the status of all the ®les in the tree without
examining their data; (4) examines every byte of data in
all the ®les;and (5) compiles and links the ®les.
We use the Andrew benchmark to compare BFS with
two other ®lesystem con®gurations: NFS-std, which is
the NFS V2 implementation in Digital Unix, and BFS-nr,
which is identical to BFS but with no replication. BFS-nr
ran two simple UDP relays on the client, and on the server
it ran a thin veneer linked with a version of snfsd from
which all the checkpoint management code was removed.
This con®guration does not write modi®ed ®le system
state to disk before replying to the client. Therefore, it
does not implement NFS V2 protocol semantics, whereas
both BFS and NFS-std do.
Out of the 18 operations in the NFS V2 protocol only
getattr is read-only because the time-last-accessed
attribute of ®les and directories is set by operations
that would otherwise be read-only, e.g., read and
lookup. The result is that our optimization for read-
only operations can rarely be used. To show the impact
of this optimization, we also ran the Andrew benchmark
on a second version of BFS that modi®esthe lookup
operation to be read-only. This modi®cation violates
strict Unix ®lesystem semantics but is unlikely to have
adverse effects in practice.
For all con®gurations,the actual benchmark code ran
at the client workstation using the standard NFS client
implementation in the Digital Unix kernel with the same
mount options. The most relevant of these options for
the benchmark are: UDP transport, 4096-byte read and
write buffers, allowing asynchronous client writes, and
allowing attribute caching.
We report the mean of 10 runs of the benchmark for
each con®guration. The sample standard deviation for
the total time to run the benchmark was always below
2.6% of the reported value but it was as high as 14% for
the individual times of the ®rst four phases. This high
variance was also present in the NFS-std con®guration.
The estimated error for the reported mean was below
4.5% for the individual phases and 0.8% for the total.
Table 2 shows the results for BFS and BFS-nr. The
comparison between BFS-strict and BFS-nr shows that
the overhead of Byzantine fault tolerance for this service
is low Ð BFS-strict takes only 26% more time to run
BFS
phase strict r/o lookup BFS-nr
1 0.55 (57%) 0.47 (34%) 0.35
2 9.24 (82%) 7.91 (56%) 5.08
3 7.24 (18%) 6.45 (6%) 6.11
4 8.77 (18%) 7.87 (6%) 7.41
5 38.68 (20%) 38.38 (19%) 32.12
total 64.48 (26%) 61.07 (20%) 51.07
Table 2: Andrew benchmark: BFS vs BFS-nr. The times
are in seconds.
the complete benchmark. The overhead is lower than
what was observed for the micro-benchmarks because
the client spends a signi®cantfraction of the elapsed time
computing between operations, i.e., between receiving
the reply to an operation and issuing the next request,
and operations at the server perform some computation.
But the overhead is not uniform across the benchmark
phases. The main reason for this is a variation in the
amount of time the client spends computing between
operations; the ®rst two phases have a higher relative
overhead because the client spends approximately 40%
of the total time computing between operations, whereas
it spends approximately 70% during the last three phases.
The table shows that applying the read-only optimiza-
tion to lookup improves the performance of BFS sig-
ni®cantly and reduces the overhead relative to BFS-nr
to 20%. This optimization has a signi®cant impact in
the ®rstfour phases because the time spent waiting for
lookup operations to complete in BFS-strict is at least
20% of the elapsed time for these phases, whereas it is
less than 5% of the elapsed time for the last phase.
BFS
phase strict r/o lookup NFS-std
1 0.55 (-69%) 0.47 (-73%) 1.75
2 9.24 (-2%) 7.91 (-16%) 9.46
3 7.24 (35%) 6.45 (20%) 5.36
4 8.77 (32%) 7.87 (19%) 6.60
5 38.68 (-2%) 38.38 (-2%) 39.35
total 64.48 (3%) 61.07 (-2%) 62.52
Table 3: Andrew benchmark: BFS vs NFS-std. The
times are in seconds.
Table 3 shows the results for BFS vs NFS-std. These
results show that BFS can be used in practice Ð BFS-
strict takes only 3% more time to run the complete
benchmark. Thus, one could replace the NFS V2
implementation in Digital Unix, which is used daily
by many users, by BFS without affecting the latency
perceived by those users. Furthermore, BFS with the
read-only optimization for the lookup operation is
actually 2% faster than NFS-std.
The overhead of BFS relative to NFS-std is not the
12

same for all phases. Both versions of BFS are faster
than NFS-std for phases 1, 2, and 5 but slower for the
other phases. This is because during phases 1, 2, and 5 a
large fraction (between 21% and 40%) of the operations
issued by the client are synchronous, i.e., operations that
require the NFS implementation to ensure stability of
modi®ed ®le system state before replying to the client.
NFS-std achieves stability by writing modi®ed state to
disk whereas BFS achieves stability with lower latency
using replication (as in Harp [20]). NFS-std is faster than
BFS (and BFS-nr) in phases 3 and 4 because the client
issues no synchronous operations during these phases.
8 Related Work
Most previous work on replication techniques ignored
Byzantine faults or assumed a synchronous system
model (e.g., [17, 26, 18, 34, 6, 10]). Viewstamped
replication [26] and Paxos [18] use views with a primary
and backups to tolerate benign faults in an asynchronous
system. Tolerating Byzantine faults requires a much more
complex protocol with cryptographic authentication, an
extra pre-prepare phase, and a different technique to
trigger view changes and select primaries. Furthermore,
our system uses view changes only to select a new primary
but never to select a different set of replicas to form the
new view as in [26, 18].
Some agreement and consensus algorithms tolerate
Byzantine faults in asynchronous systems (e.g,[2, 3, 24]).
However, they do not provide a complete solution for
state machine replication, and furthermore, most of them
were designed to demonstrate theoretical feasibility and
are too slow to be used in practice. Our algorithm
during normal-case operation is similar to the Byzantine
agreement algorithm in [2] but that algorithm is unable
to survive primary failures.
The two systems that are most closely related to our
work are Rampart [29, 30, 31, 22] and SecureRing [16].
They implement state machine replication but are more
than an order of magnitude slower than our system and,
most importantly, they rely on synchrony assumptions.
Both Rampart and SecureRing must exclude faulty
replicas from the group to make progress (e.g., to remove
a faulty primary and elect a new one), and to perform
garbage collection. They rely on failure detectors
to determine which replicas are faulty. However,
failure detectors cannot be accurate in an asynchronous
system [21], i.e., they may misclassify a replica as faulty.
Since correctness requires that fewer than 1
 3 of group
members be faulty, a misclassi®cationcan compromise
correctness by removing a non-faulty replica from the
group. This opens an avenue of attack: an attacker
gains control over a single replica but does not change
its behavior in any detectable way; then it slows correct
replicas or the communication between them until enough
are excluded from the group.
To reduce the probability of misclassi®cation,failure
detectors can be calibrated to delay classifying a replica
as faulty. However, for the probability to be negligible
the delay must be very large, which is undesirable. For
example, if the primary has actually failed, the group will
be unable to process client requests until the delay has
expired. Our algorithm is not vulnerable to this problem
because it never needs to exclude replicas from the group.
Phalanx [23, 25] applies quorum replication tech-
niques [12] to achieve Byzantine fault-tolerance in asyn-
chronous systems. This work does not provide generic
state machine replication; instead, it offers a data reposi-
tory with operations to read and write individual variables
and to acquire locks. The semantics it provides for read
and write operations are weaker than those offered by our
algorithm; we can implement arbitrary operations that ac-
cess any number of variables,whereas in Phalanx it would
be necessary to acquire and release locks to execute such
operations. There are no published performance num-
bers for Phalanx but we believe our algorithm is faster
because it has fewer message delays in the critical path
and because of our use of MACs rather than public key
cryptography. The approach in Phalanx offers the poten-
tial for improved scalability; each operation is processed
by only a subset of replicas. But this approach to scala-
bility is expensive: it requires
 4
 1 to tolerate
faults; each replica needs a copy of the state; and the load
on each replica decreases slowly with
 (it is O
 1
 ).
9 Conclusions
This paper has described a new state-machine replication
algorithm that is able to tolerate Byzantine faults and can
be used in practice: it is the ®rst to work correctly in
an asynchronous system like the Internet and it improves
the performance of previous algorithms by more than an
order of magnitude.
The paper also described BFS, a Byzantine-fault-
tolerant implementation of NFS. BFS demonstrates that
it is possible to use our algorithm to implement real
services with performance close to that of an unreplicated
service Ð the performance of BFS is only 3% worse than
that of the standard NFS implementation in Digital Unix.
This good performance is due to a number of important
optimizations, including replacing public-key signatures
by vectors of message authentication codes, reducing
the size and number of messages, and the incremental
checkpoint-management techniques.
One reason why Byzantine-fault-tolerant algorithms
will be important in the future is that they can allow
systems to continue to work correctly even when there
are software errors. Not all errors are survivable;
our approach cannot mask a software error that occurs
13

at all replicas. However, it can mask errors that
occur independently at different replicas, including
nondeterministic software errors, which are the most
problematic and persistent errors since they are the
hardest to detect. In fact, we encountered such a software
bug while running our system, and our algorithm was able
to continue running correctly in spite of it.
There is still much work to do on improving our system.
One problem of special interest is reducing the amount
of resources required to implement our algorithm. The
number of replicas can be reduced by using
 replicas
as witnesses that are involved in the protocol only when
some full replica fails. We also believe that it is possible
to reduce the number of copies of the state to
 1 but
the details remain to be worked out.
Acknowledgments
We would like to thank Atul Adya, Chandrasekhar
Boyapati, Nancy Lynch, Sape Mullender, Andrew Myers,
Liuba Shrira, and the anonymous referees for their helpful
comments on drafts of this paper.
References
[1] M. Bellare and D. Micciancio. A New Paradigm for Collision-
free Hashing: Incrementality at Reduced Cost. In Advances in
Cryptology ±Eurocrypt 97, 1997.
[2] G. Bracha and S. Toueg. Asynchronous Consensus and Broadcast
Protocols. Journal of the ACM, 32(4), 1995.
[3] R. Canneti and T. Rabin. Optimal Asynchronous Byzantine
Agreement. Technical Report #92-15, Computer Science
Department, Hebrew University, 1992.
[4] M. Castro and B. Liskov. A Correctness Proof for a Practi-
cal Byzantine-Fault-Tolerant Replication Algorithm. Technical
Memo MIT/LCS/TM-590, MIT Laboratory for Computer Sci-
ence, 1999.
[5] M. Castro and B. Liskov. Authenticated Byzantine Fault
Tolerance Without Public-Key Cryptography. Technical Memo
MIT/LCS/TM-589, MIT Laboratory for Computer Science, 1999.
[6] F. Cristian, H. Aghili, H. Strong, and D. Dolev. Atomic Broadcast:
From Simple Message Diffusion to Byzantine Agreement. In
International Conference on Fault Tolerant Computing, 1985.
[7] S. Deering and D. Cheriton. Multicast Routing in Datagram
Internetworks and Extended LANs. ACM Transactions on
Computer Systems, 8(2), 1990.
[8] H. Dobbertin. The Status of MD5 After a Recent Attack. RSA
Laboratories' CryptoBytes, 2(2), 1996.
[9] M. Fischer, N. Lynch, and M. Paterson. Impossibility of
Distributed Consensus With One Faulty Process. Journal of the
ACM, 32(2), 1985.
[10] J. Garay and Y. Moses. Fully Polynomial Byzantine Agreement
for n
 3t Processors in t+1 Rounds. SIAM Journal of Computing,
27(1), 1998.
[11] D. Gawlick and D. Kinkade. Varieties of Concurrency Control in
IMS/VS Fast Path. Database Engineering, 8(2), 1985.
[12] D. Gifford. Weighted Voting for Replicated Data. In Symposium
on Operating Systems Principles, 1979.
[13] M. Herlihy and J. Tygar. How to make replicated data secure.
Advances in Cryptology (LNCS 293), 1988.
[14] M. Herlihy and J. Wing. Axioms for Concurrent Objects. In ACM
Symposium on Principles of Programming Languages, 1987.
[15] J. Howard et al. Scale and performance in a distributed ®lesystem.
ACM Transactions on Computer Systems, 6(1), 1988.
[16] K. Kihlstrom, L. Moser, and P. Melliar-Smith. The SecureRing
Protocols for Securing Group Communication. In Hawaii
International Conference on System Sciences, 1998.
[17] L. Lamport. Time, Clocks, and the Ordering of Events in a
Distributed System. Commun. ACM, 21(7), 1978.
[18] L. Lamport. The Part-Time Parliament. Technical Report 49,
DEC Systems Research Center, 1989.
[19] L. Lamport, R. Shostak, and M. Pease. The Byzantine Generals
Problem. ACM Transactions on Programming Languages and
Systems, 4(3), 1982.
[20] B. Liskov et al. Replication in the Harp File System. In ACM
Symposium on Operating System Principles, 1991.
[21] N. Lynch. Distributed Algorithms. Morgan Kaufmann Publishers,
1996.
[22] D. Malkhi and M. Reiter. A High-Throughput Secure Reliable
Multicast Protocol. In Computer Security Foundations Workshop,
1996.
[23] D. Malkhi and M. Reiter. Byzantine Quorum Systems. In ACM
Symposium on Theory of Computing, 1997.
[24] D. Malkhi and M. Reiter. Unreliable Intrusion Detection in
Distributed Computations. In Computer Security Foundations
Workshop, 1997.
[25] D. Malkhi and M. Reiter. Secure and Scalable Replication in
Phalanx. In IEEE Symposium on Reliable Distributed Systems,
1998.
[26] B. Oki and B. Liskov. Viewstamped Replication: A New Primary
Copy Method to Support Highly-Available Distributed Systems.
In ACM Symposium on Principles of Distributed Computing,
1988.
[27] B. Preneel and P. Oorschot. MDx-MAC and Building Fast MACs
from Hash Functions. In Crypto 95, 1995.
[28] C. Pu, A. Black, C. Cowan, and J. Walpole. A Specialization
Toolkit to Increase the Diversity of Operating Systems. In ICMAS
Workshop on Immunity-Based Systems, 1996.
[29] M. Reiter. Secure Agreement Protocols. In ACM Conference on
Computer and Communication Security, 1994.
[30] M. Reiter. The Rampart Toolkit for Building High-Integrity
Services. Theory and Practice in Distributed Systems (LNCS
938), 1995.
[31] M. Reiter. A Secure Group Membership Protocol. IEEE
Transactions on Software Engineering, 22(1), 1996.
[32] R. Rivest. The MD5 Message-Digest Algorithm. Internet RFC-
1321, 1992.
[33] R. Rivest, A. Shamir, and L. Adleman. A Method for
Obtaining Digital Signatures and Public-Key Cryptosystems.
Communications of the ACM, 21(2), 1978.
[34] F. Schneider. Implementing Fault-Tolerant Services Using The
State Machine Approach: A Tutorial. ACM Computing Surveys,
22(4), 1990.
[35] A. Shamir. How to share a secret. Communications of the ACM,
22(11), 1979.
[36] G. Tsudik. Message Authentication with One-Way Hash
Functions. ACM Computer Communications Review, 22(5), 1992.
[37] M. Wiener. Performance Comparison of Public-Key Cryptosys-
tems. RSA Laboratories' CryptoBytes, 4(1), 1998.
14

Distributed Transactions FAQ

Q: How does this material fit into 6.5840?

A: When data is distributed over many computers, it's common for a
single operation to need to read and/or modify multiple data items,
perhaps stored on different computers. How such multi-step operations
interact with concurrent operations on the same data, and what happens
if a crash occurs in the middle of such an operation, are usually
critical questions for the system's robustness and ease of
programming. The gold standard for good behavior is transactions,
often provided by database systems. Transactions are usually
implemented with two-phase locking and logging; distributed
transactions usually add two-phase commit. Today's reading from the
6.1800 (6.033) textbook explains those ideas.

Most of the storage systems we've looked at so far provide operations
like put() and get() that involve only single records. In constrast,
transactions usually refer to multi-record operations (e.g., bank
transfers from one account to another). When the records in involved
in a transaction are stored in different places (e.g. in a sharded
storage system), then we're talking about distributed transactions,
for which two-phase commit is helpful. These ideas will show up in
some of the upcoming papers we read (e.g. Spanner and FaRM).

Q: Why is it so important for transactions to be atomic?

A: What "transaction" means is that the entire sequence of steps
inside the transaction occurs atomically with respect to failures and
other transactions. Atomic here means "all or none". Transactions are
a feature provided by some storage systems to make programming easier.
A situation where transactions are helpful is bank transfers. If the
bank wants to transfer $100 from Alice's account to Bob's account, it
would be awkward if a crash left Alice debited by $100 but Bob *not*
credited by $100. If your storage system supports transactions, you
can write something like

BEGIN TRANSACTION
  decrease Alice's balance by 100;
  increase Bob's balance by 100;
END TRANSACTION

and the transaction system will make sure the transaction is atomic.
Either both happen, or neither, even if there's a failure; and no
other transaction will observe the intermediate situation where only
one balance has been modified.

Q: Could one use Raft instead of two-phase commit?

A: No: two-phase commit and Raft solve different problems.

Two-phase commit causes different computers to do *different* things
(e.g. Alice's bank debits Alice, Bob's bank credits Bob), and causes
them *all* to do their thing, or none of them. Two-phase commit cannot
make progress if any participant can't be reached: it has to wait for
all participating computers to perform their part of the transaction.

Raft causes a majority of the peers to all do the *same* thing (so
they remain replicas). It's OK for Raft to wait only for a majority,
since the peers are replicas, and therefor Raft-based systems can be
available in the face of failures.

Q: What is the difference between two-phase locking and two-phase commit?

A: 2PL is a scheme for acquiring locks for records in a transaction,
to ensure that different transactions that use the same records don't
interfere with each other; it is useful in both non-distributed and
distributed settings.

2PC is a scheme to execute a transaction across multiple machines,
where each machine has some of the records used in the transaction;
2PC ensures that each machine does its part of the transaction.

2PC systems often use 2PL.

Q: In two-phase commit, why would a worker send an abort message,
rather than a PREPARED message?

A: Perhaps the participant crashed and rebooted after it did some of
its work for the transaction but before it received the prepare
message; during the crash it will have lost the record of tentative
updates it made and locks it acquired, so it cannot complete the
transaction. Another possibility (depending on how the database works)
is if the worker detected a violated constraint on the data (e.g. the
transaction tried to write a record with a duplicate key in a table
that requires unique keys). Another possibility is that the worker is
involved in a deadlock, and must abort to break the deadlock.

Q: Can two-phase locking generate deadlock?

A: Yes. If two simultaneous transactions both use records R1 and R2,
but in opposite orders, they may each acquire one of the locks, and
then deadlock trying to get the other lock. Databases detect these
deadlocks and break them. A database can detect deadlock by timing out
lock acquisition, or by finding cycles in the waits-for graph among
transactions. A deadlock can be broken by aborting one of the
participating transactions.

Q: What is serializability?

A: It's a common criterion for correctness of concurrent transactions
in databases. An execution of a bunch of transactions is serializable
if it yields the same results as some serial (one-at-a-time) execution
of those transactions. Results include both the content of records in
the database and any outputs from the transactions. So if transactions
T1 and T2 execute concurrently, the results are serializable if they
are the same as executing T1, waiting for T1 to complete, and then
executing T2; or executing T2, waiting for it to complete, and then
executing T1.

Q: Why does it matter whether locks are held until after a transaction
commits or aborts?

A: If transactions release locks before they commit, it can be hard to
avoid certain non-serializable (incorrect) executions due to aborts or
crashes. In this example, suppose T1 releases the lock on x after it
updates x, but before it commits or aborts:

  T1:           T2:
  BEGIN
  x = x + 1
                BEGIN
                y = x
                END

  END

It can't be legal for y to end up greater than x. Yet if T1 releases
its lock on x, then T2 acquires the lock, writes y, and commits, but
then T1 aborts or the system crashes and cannot complete T1, we will
end up with y greater than x.

It's to avoid having to cope with the above that people use the
"strong strict" variant of 2PL, which only releases locks after a
commit or abort.

Q: What is the point of the two-phase locking rule that says a
transaction isn't allowed to acquire any locks after the first time
that it releases a lock?

A: Acquiring after releasing can lead to non-serializable executions.

  T1:         T2:
  x = x + 1
              z = x + y
  y = y + 1

Suppose x and y start out as zero, and both transactions execute, and
successfully commit. The only final values of z that are allowed by
serializability are zero and 2 (corresponding to the orders T2;T1 and
T1;T2). But if T1 releases its lock on x before acquiring the lock on
y and modifying y, T2 could completely execute and commit while T1 is
between its two statements, giving z a value of 1, which is not legal.
If T1 keeps its lock on x while using y, as two-phase locking demands,
this problem is avoided.

Q: How does two-phase commit solve the dilemma of the two generals (or
the Byzantine Generals Problem)?

A: It doesn't. Two-phase commit doesn't encounter the two generals
problem, and thus doesn't need to solve it.

One difference is that, in two-phase commit, there's just one entity
making the decision (the TC), and it can't disagree with itself.
Whereas in the two generals problem, there are two independent
deciders who have trouble communicating, so there's room for
disagreement. Another difference is that two-phase commit has no
real-time requirement (nothing like the requirement that the generals
agree by dawn); it's OK for workers to wait for as long as needed to
receive the TC's decision.

Q: Are the locks exclusive, or can they allow multiple readers to have
simultaneous access?

A: By default, "lock" in 6.5840 refers to an exclusive lock. But there
are databases that can grant locking access to a record to either
multiple readers, or a single writer. Some care has to be taken when a
transaction reads a record and then writes it, since the lock will
initially be a read lock and then must be upgraded to a write lock.
There's also increased opportunity for deadlock in some situations; if
two transactions simultaneously want to increment the same record
(i.e. read, add one, write), they might deadlock when upgrading a read
lock to a write lock on that record, whereas if locks are always
exclusive, they won't deadlock.

Q: How should one decide between pessimistic and optimistic
concurrency control?

A: If your transactions conflict a lot (use the same records, and one
or more transactions writes), then locking is better. Locking causes
conflicting transactions to wait, whereas most OCC systems deal with
conflict by aborting; aborts (really the consequent retries) are
expensive.

If your transactions rarely conflict, then OCC is preferable to
locking. OCC doesn't spend CPU time acquiring/releasing locks and, as
long as conflicts are rare, OCC rarely aborts. The "validation" phase
of OCC systems often uses locks, but they are usually held for shorter
periods of time than the locks in pessimistic designs.

Q: What should two-phase commit workers do if the transaction
coordinator crashes?

A: If a worker has told the coordinator that it is ready to commit,
then the worker cannot later change its mind. The reason is that the
coordinator may (before it crashed) have told other workers to commit.
So the worker has to wait (with locks held) for the coordinator to
reboot and send (or re-send) its decision.

Waiting indefinitely with locks held is a real problem, since the
locks can force a growing set of other transactions to block as well.
So people tend to avoid two-phase commit, or they try to make
coodinators reliable. For example, Google's Spanner replicates
coordinators (and all other servers) using Paxos.

Q: Why don't people use three-phase commit, which allows workers to
commit or abort even if the coordinator crashes?

A: Three-phase commit only works if workers can reliably distinguish
between the coordinator being dead and the network not delivering
packets. For example, three-phase commit won't work correctly if
there's a network partition. In most practical networks, it's not
possible to distinguish a dead computer from a network failure, so
three-phase commit can't safely be used.

Q: Can there be more than one transaction active? How do participants
know which transaction a message refers to?

A: There can be many concurrent transactions, managed by many TCs. A
TC assigns a unique transaction ID (TID) to each transaction. Every
message includes the TID of the relevant transaction. TCs and
participants tag entries in their tables with the TID, so that (for
example) when a COMMIT message arrives at a participant, it knows what
tentative records to make permanent, and what locks to release.

Q: How does a two-phase commit system undo modifications if a
transaction has to abort?

A: Each participant performs modifications to temporary copies of the
records. If the participant answers "yes" to the TC's prepare message,
the participant must first save the temporary record values to its log
on disk, so it can find them if it crashes and restarts. If the TC
decides to commit, the participant must copy the temporary values to
the real database records; if the TC decides to abort, the participant
must discard the temporary records.

Q: How does serializability relate to linearizability?

A: They are similar notions, arising from different communities. Both
require the final outcome to be the same as some serial execution.
Serializability refers to entire transactions, each involving multiple
operations and multiple records. Linearizability usually refers to
single operations, each on a single record. It's also the case that
linearizability requires that the equivalent serial execution be
consistent with the real-time order of the actual execution, while
serializability usually does not.

Q: Why do logs appear so often in the designs we look at?

A: A log is a good way to capture the serial order that the system has
chosen for operations, so that e.g. all replicas perform the
operations in the same order, or a server considers operations in the
same order after a crash+reboot as it did before the crash.

Many distributed systems keep multiple operations in flight (often
called a window or pipeline of operations). Often the fate of
operations is not known until some time after they are received. A log
is a good way to keep track of such pending operations. Raft is an
example of this arrangement: a follower receives a stream of commands
from the leader, but doesn't hear that they are committed until later,
and must have a place to store commands until they are committed.

A log is an efficient way to write data to hard disk or SSD, since
both media are faster at sequential writes (i.e. appends to the log)
than at random writes.

A log is a convenient way for crash-recovery software to see how far
the system got before it crashed, and whether the last transactions
have complete records in the log and thus can safely be replayed. That
is, a log is a convenient way to implement crash-recoverable atomic
transactions, via write-ahead logging.

Q: Are there structures other than logs that would work as well?

A: There's nothing as general-purpose as logs.

You can record order by storing data in some other way (e.g. a b-tree)
and storing sequence numbers with the data (Frangipani does this for
meta-data, in addition to using logs).

You wouldn't have to worry about performance if you used a persistent
storage system that was as fast for random updates as for sequential,
for example battery-backed RAM. However, such systems are often more
expensive and less robust than hard drives or SSDs.

For the write-ahead property, you could store a mini-log for each data
record. However, it might then be time-consuming for the
crash-recovery software to find the full set of incomplete mini-logs.

A different way to get crash-recoverable atomic operations is to
prepare an entire new data structure in fresh storage, and then use a
single commiting write to substitute it for the original data
structure. This makes the most sense with tree- shaped data
structures. The NetApp WAFL file system uses that idea:

https://atg.netapp.com/wp-content/uploads/2000/01/file-system-design.pdf

This arrangement may make it hard to support concurrent transactions.

Q: What is the Lock Manager?

A: The software module that implements acquire() and release(). It may
also implement deadlock detection by constructing a waits-for-graph as
users of the module acquire locks using acquire().  In some cases the
module is a separate service running on a machine (e.g., the lock
service in Franginpani).

Q: Why does Section 9.5.3 of the reading say that two-phase locking
forbids this sequence?

T1: READ X
T2: WRITE Y
T1: WRITE Y

A: Perhaps the text means that T2 goes on to do other unrelated things
before committing. In that case two-phase locking would force T1 to
wait until T2 completely finished. However, in fact it would be
correct (serializable) for T1 to commit before T2 finished. That is,
there are correct executions that two-phase locking forbids.

6.824 FAQ for Chain replication for supporting high throughput and
availability (OSDI 2004) by Renesse and Schneider

Q: Is chain replication used in practice over other things like Raft
or Paxos?

A: Systems often use both. A common way of building distributed
systems is to use a configuration server (called the master in the
paper) for maintaining configuration info (e.g., who is primary?) and
a replication system for replicating data.  Paxos/Raft are
commonly-used to build the configuration server while the replication
system often uses primary-backup or chain replication.  The reason to
use Raft/Paxos for configuration server is it must handle split-brain
syndrome.  The reason to use primary/backup for data replication is
that it is simpler than Raft/Paxos and Raft, for example, is not good
at for replicating large amounts of data.  The replication system can
rely on the configuration server to avoid split-brain syndrome.

Q: How does CR cope with network partition and prevent split brain?

A: At a high level, a chain will pause operation if one of its servers
or network links fails, and wait for the configuration server to notice the problem
and reconfigure the chain. Let's consider separately the situation
before the configuration server notices a problem, and after it notices.

Before the configuration server notices (or if the configuration server doesn't notice any
problem), if the partition prevents communication between successive
chain servers, updates will stop completing because they can no
longer travel down the chain all the way from head to tail. Read
queries will continue to work for clients that can talk to the tail.
The system is safe (linearizable), but not very live since updates
can't complete.

At some point the configuration server may see that it can't communicate with one or
more of the chain servers, and will consider those servers to have
failed (though they may actually be alive).

If the configuration server can't talk to any of the chain's servers, then the
master will do nothing, and the existing chain may continue to provide
correct service (perhaps without completing updates) to the clients
that can talk to it.

If the configuration server thinks just the head is dead, it will direct clients to
send updates to the 2nd server in the chain, and tell the 2nd server
that it is now the head. But perhaps the old head is not dead, and
merely partitioned from the configuration server. In that case, the paper does not
explain how to avoid split brain: now there may be two servers
operating as head and forwarding conflicting updates to the next
server in the chain. You can imagine solutions -- for example the new
head, which is the old 2nd server in the chain, could reject updates
sent to it by the old head.

If the configuration server thinks the tail is dead, it will tell clients to send
read queries to the N-1'th server in the chain, and tell that server
that it is now the tail. But perhaps the old tail is not dead, and
merely partitioned from the configuration server. In that case, some clients may
still send read queries to the old tail, which will now return stale
values because it is no longer receiving updates from the chain.
Again, this is split brain, but the paper doesn't explain how to avoid
it. A possible solution is for the configuration server to grant a lease to the
tail, and to delay designating any new tail until the previous tail's
lease has expired.

Q: What are the tradeoffs of Chain Replication vs Raft or Paxos?

A: Both CR and Raft/Paxos are replicated state machines. They can be
used to replicate any service that can fit into a state machine mold
(basically, processes a stream of requests one at a time). One
application for Raft/Paxos is object storage -- you'll build object
storage on top of Raft in Lab 3. Similarly, the underlying machinery
of CR could be used for services other than storage, for example to
implement a lock server.

CR is likely to be faster than Raft because the CR head does less work
than the Raft leader: the CR head sends writes to just one replica,
while the Raft leader must send all operations to all followers. CR
has a performance advantage for reads as well, since it serves them
from the tail (not the head), while the Raft leader must serve all
client requests.

However, Raft/Paxos and CR differ significantly in their failure
properties. If there are N replicas, a CR system can recover if even a
single replica survives. Raft and Paxos require a majority of the N
replicas to be available in order to operate, so in that sense they
are less tolerant of failure. But a CR chain has to pause updates if
there's even a single failure, and must wait for the configuration
server to notice and reconfigure the chain; in the paper's setup this
takes ten seconds. Raft/Paxos, in contrast, can continue operating
without interruption as long as a majority is available, so they
handle a slow or flaky or briefly unavailable replica more smoothly
than CR.

Q: Would Chain Replication be significantly faster or slower than the
kind of primary/backup used in GFS?

A: If there are just two replicas, there's probably not much
difference. Though maybe CR would be faster for writes since the tail
can send responses directly to the client; in a classic primary/backup
scheme, the primary has to wait for the backup to acknowledge a write
before the primary responds to the client.

If there are three or more replicas, the primary in a classic
primary/backup system has to send each write to each of the replicas. If
the write data is big, these network sends could put a significant load
on the primary. Chain Replication spreads this networking load over all
the replicas, so CR's head node might be less of a performance
bottleneck than a classic primary. On the other hand maybe the
client-observed latency for writes would be higher in CR.

Q: Section 5.2 evaluates multiple chains. What does this mean?

A: In Chain Replication, only the head and tail directly serve client
requests; the other replicas help fault tolerance but not performance.
Since the load on the head and tail is thus likely to be higher than
the load on intermediate nodes, you could get into a situation where
performance is bottlenecked by head/tail, yet there is plenty of idle
CPU available in the intermediate nodes.

Section 5.2 uses CR in a way that avoids this limitation. A data
center will probably have lots of distinct CR chains, each serving a
fraction (shard) of the objects. Suppose you have three servers (S1,
S2, and S3) and three chains (C1, C2, C3). Then you can have the three
chains be:

  C1: S1 S2 S3
  C2: S2 S3 S1
  C3: S3 S1 S2

Now, assuming activity on the three chains is roughly equal, the load on
the three servers will also be roughly equal. In particular the load of
serving client requests (head and tail) will be roughly equally divided
among the three servers.

Q: Section 3 says servers are assumed to be fail-stop. Are servers
typically fail-stop? How does that work?

A: Servers, networks, disks, &c are not actually fail-stop. CPU
hardware sometimes produces incorrect answers, disks and networks
sometimes corrupt data, software sometimes has bugs that cause it to
malfunction without warning, human operators sometimes mis-configure
systems. Server hardware is designed to be fail-stop for some errors
(checksums catch most corrupted network packets, ECC catches most RAM
errors, &c), but not all errors.

What the paper means by "we assume servers to be fail-stop" is "if all
failures are fail-stop, then the claims we make in this paper will
hold. If you encounter a non-fail-stop failure, then the claims in the
paper may not hold."

Most of the systems we'll look at assume fail-stop failures, and may
silently malfunction if there are non-fail-stop ("Byzantine")
failures. But there are designs that have good behavior in many
non-fail-stop situations. First, systems derived from a paper titled
Practical Byzantine Fault Tolerance (PBFT) by Castro and Liskov; PBFT
is like Raft but the servers check each others' actions with extra
rounds of cryptographically authenticated communication. Second,
systems in which clients can check the correctness of results that
servers return, typically by use of cryptographic hashes or
signatures. This can be tricky because clients need to defend against
a server that returns data whose signature or hash is correct, but is
not the latest value. Systems like this include SUNDR and Bitcoin.

Q: Is Chain Replication used by other systems?

A: Some examples: Amazon's EBS, Ceph's Rados, Google's Parameter
Server, COPS, and FAWN.

Q: I'm getting confused with all the symbols (particularly the circle
with a plus sign inside) and the invariants given.

A: The confusion perhaps arises because the XOR symbol doesn't mean
XOR on boolean values.  It is more like a union, but not set union
because the left and right are not sets, but sequences.  My guess is
that the analogy with boolean XOR symbol is that a CR operation can
appear only in the left or only in the right side of the XOR on
sequences (but not both).

Q: This scheme would be bad if we care about latency right?

A: Yes and No.  Yes, the latency of update operations is proportional
to the length of the chain.  No, the latency of read operation is low:
only the tail is involved.

Q: The paper reports that chain replication has the highest MTBU when
using the `rndpar` volume placement strategy. Are there any competing
advantages offered by `ring` or is `rndpar` objectively better?

A: Figure 7 suggests that for a small number of servers, ring has a
slight advantage over rndpar. The reason is that, for a chain length
of N, if N random servers fail, there is more likely to be some chain
that uses just those N servers with rndpar than with ring. That's
important because the chain on those servers then cannot be repaired.
With more servers, rndpar has an advantage because it spreads the work
of repair over more servers, so that repair is more likely to complete
before the next failure. If failures occur faster than repair, that
increases the chance that all of a chain's servers will fail before
any can be repaired.

Q: For the failure of other servers as described on page 5, what
happens if the ack from the tail gets lost some time along the
chain, and r never gets deleted from sent_i? Updates are not
idempotent, so it can't just be resent down the chain, correct?

A: Updates are pushed down the chain, one server at the time.  Thus
only the one-but-last server is waiting for the tail's ack.  It will
keep retrying until it gets the ack.  All updates are performed in
order that the head made them; it is fine to do the same update a few
times.

Q: Question: In real systems, how important is the strong consistency
guarantee (does it cause people to use this system over GFS-like
systems?) If not, why do people prefer other systems?

A: Programming with weak consistency can be difficult. I have seen
no/few studies that quantify this. But, there is anecdotal evidence
that this matters; for example, when Amazon changed to strong
consistency for S3, Dropbox (a user of S3) was able to simplify their
code significantly.

Similarly, Google over the years has been offering services with
stronger consistency for their programmers; for example, Spanner
provides stronger consistency than GFS, partially to make programmers'
lives easier.

Q: To reduce latency, why not have an arrangement where we have the
head, t - 2 concurrent middle servers, and a tail?

A: You could but it complicates the design. Now the recovery is more
complicated; who should take over if the head fails? New logic is
required to ensure that the most up-to-date of the t-2 servers is the
one chosen as head. In CR unmodified it is clear: the next server in
the chain.

Q: What's a "reliable FIFO link"?

A: By "reliable FIFO link" the paper means "TCP over an ordinary
network". I suspect also that each update includes the sequence number
assigned to it by the head, to make it easier to ensure that the
updates are sent and processed in order. This "reliable FIFO link" may
fail, if the network stops delivering packets.

Chain Replication for Supporting
High Throughput and Availability
Robbert van Renesse
rvr@cs.cornell.edu
Fred B. Schneider
fbs@cs.cornell.edu
F AST Search & Transfer ASA
Tromsø, Norway
and
Department of Computer Science
Cornell University
Ithaca, New York 14853
Abstract
Chain replication is a new approach to coordinating
clusters of fail-stop storage servers. The approach is
intended for supporting large-scale storage services
that exhibit high throughput and availability with-
out sacrificing strong consistency guarantees. Be-
sides outlining the chain replication protocols them-
selves, simulation experiments explore the perfor-
mance characteristics of a prototype implementa-
tion. Throughput, availability, and several object-
placement strategies (including schemes based on
distributed hash table routing) are discussed.
1 Introduction
A storage system typically implements operations
so that clients can store, retrieve, and/or change
data. File systems and database systems are per-
haps the best known examples. With a file system,
operations (read and write) access a single file and
are idempotent; with a database system, operations
(transactions) may each access multiple objects and
are serializable.
This paper is concerned with storage systems that
sit somewhere between file systems and database
systems. In particular, w e are concerned with stor-
age systems, henceforth called storage services,t h a t
• store objects (of an unspecified nature),
• support query operations to return a value de-
rived from a single object, and
• support update operations to atomically change
the state of a single object according to some
pre-programmed, possibly non-deterministic,
computation involving the prior state of that
object.
A file system write is thus a special case of our stor-
age service update which, in turn, is a special case
of a database transaction.
Increasingly, we see on-line vendors (like Ama-
zon.com), search engines (like Google’s and
FAST’s), and a host of other information-intensive
services provide value by connecting large-scale stor-
age systems to networks. A storage service is the
appropriate compromise for such applications, when
a database system would be too expensive and a file
system lacks rich enough semantics.
One challenge when building a large-scale stor-
age service is maintaining high availability and
high throughput despite failures and concomitant
changes to the storage service’s configuration, as
faulty components are detected and replaced.
Consistency guarantees also can be crucial. But
even when they are not, the construction of an appli-
cation that fronts a storage service is often simpli-
fied given strong consistency guarantees ,w h i c ha s -
sert that (i) operations to query and update indi-
vidual objects are executed in some sequential order
and (ii) the effects of update operations are necessar-
ily reflected in results returned by subsequent query
operations.
Strong consistency guarantees are often thought
to be in tension with achieving high throughput
and high availability. So system designers, reluctant
to sacrifice system throughput or availability, regu-
larly decline to support strong consistency guaran-
tees. The Google File System (GFS) illustrates this
thinking [11]. In fact, strong consistency guarantees

in a large-scale storage service are not incompatible
with high throughput and availability. And the new
chain replication approach to coordinating fail-stop
servers, which is the subject of this paper, simulta-
neously supports high throughput, availability, and
strong consistency.
We proceed as follows. The interface to a generic
storage service is specified in §2. In §3, we explain
how query and update operations are implemented
using chain replication. Chain replication can be
viewed as an instance of the primary/backup ap-
proach, so §4 compares them. Then, §5 summarizes
experiments to analyze throughput and availability
using our prototype implementation of chain replica-
tion and a simulated network. Some of these simula-
tions compare chain replication with storage systems
(like CFS [7] and PAST [19]) based on distributed
hash table (DHT) routing; other simulations reveal
surprising behaviors when a system employing chain
replication recovers from server failures. Chain repli-
cation is compared in §6 to other work on scalable
storage systems, trading consistency for availability,
and replica placement. Concluding remarks appear
in §7, followed by endnotes.
2 A Storage Service Interface
Clients of a storage service issue requests for query
and update operations. While it would be possible
to ensure that each request reaching the storage ser-
vice is guaranteed to be performed, the end-to-end
argument [20] suggests there is little point in doing
so. Clients are better off if the storage service sim-
ply generates a reply for each request it receives and
completes, because this allows lost requests and lost
replies to be handled as well: a client re-issues a re-
quest if too much time has elapsed without receiving
ar e p l y .
• The reply for query(objId, opts) is derived from
the value of object objId; options opts charac-
terizes what parts of objId are returned. The
value of objId remains unchanged.
• The reply for update(objId, newVal, opts)d e -
pends on options opts and, in the general case,
can be a value V produced in some nondeter-
ministic pre-programmed way involving the cur-
rent value ofobjId and/or valuenewVal; V then
becomes the new value of objId.
1
Query operations are idempotent, but update op-
erations need not be. A client that re-issues a non-
idempotent update request must therefore take pre-
cautions to ensure the update has not already been
State is:
Hist
objID : update request sequence
PendingobjID : request set
Tr a n s i t i o n s a r e :
T1: Client request r arrives:
PendingobjID := PendingobjID ∪{ r}
T2: Client request r ∈ PendingobjID ignored:
PendingobjID := PendingobjID −{ r}
T3: Client request r ∈ PendingobjID processed:
PendingobjID := PendingobjID −{ r}
if r = query(objId, opts) then
reply according options opts based
on HistobjID
else if r = update(objId, newVal, opts) then
HistobjID := Hist objID · r
reply according options opts based
on HistobjID
Figure 1: Client’s View of an Object.
performed. The client might, for example, first issue
a query to determine whether the current value of
the object already reflects the update.
A client request that is lost before reaching the
storage service is indistinguishable to that client
from one that is ignored by the storage service. This
means that clients would not be exposed to a new
failure mode when a storage server exhibits transient
outages during which client requests are ignored. Of
course, acceptable client pe rformance likely would
depend on limiting the frequency and duration of
transient outages.
With chain replication, the duration of each tran-
sient outage is far shorter than the time required to
remove a faulty host or to add a new host. So, client
request processing proceeds with minimal disruption
in the face of failure, recovery, and other reconfig-
uration. Most other replica-management protocols
either block some operations or sacrifice consistency
guarantees following failures and during reconfigu-
rations.
We specify the functionality of our storage service
by giving the client view of an object’s state and of
that object’s state transitions in response to query
and update requests. Figure 1 uses pseudo-code to
give such a specification for an object objID.
The figure defines the state of objID in terms
of two variables: the sequence
2 Hist objID of up-
dates that have been performed on objID and a set
PendingobjID of unprocessed requests.

replies
TAIL
queries
HEAD
updates
Figure 2: A chain.
Then, the figure lists possible state transitions.
Transition T1 asserts that an arriving client re-
quest is added to PendingobjID . That some pend-
ing requests are ignored is specified by transition
T2—this transition is presumably not taken too fre-
quently. Transition T3 gives a high-level view of
request processing: the request r is first removed
from Pending
objID; query then causes a suitable re-
ply to be produced whereas update also appends r
(denoted by ·)t o HistobjID.3
3 Chain Replication Protocol
Servers are assumed to be fail-stop [21]:
• each server halts in response to a failure rather
than making erroneous state transitions, and
• a server’s halted state can be detected by the
environment.
With an object replicated on t servers, as many as
t− 1 of the servers can fail without compromising the
object’s availability. The object’s availability is thus
increased to the probability that all servers hosting
that object have failed; simulations in §5.4 explore
this probability for typical storage systems. Hence-
forth, we assume that at most t − 1o ft h es e r v e r s
replicating an object fail concurrently.
In chain replication, the servers replicating a given
object objID are linearly ordered to form a chain.
(See Figure 2.) The first server in the chain is called
the head, the last server is called thetail,a n dr e q u e s t
processing is implemented by the servers roughly as
follows:
Reply Generation. The reply for every request is
generated and sent by the tail.
Query Processing. Each query request is directed
to the tail of the chain and processed there
atomically using the replica of objID stored at
the tail.
Update Processing. Each update request is di-
rected to the head of the chain. The request
is processed there atomically using replica of
objID at the head, then state changes are for-
warded along a reliable FIFO link to the next
element of the chain (where it is handled and
forwarded), and so on until the request is han-
dled by the tail.
Strong consistency thus follows because query re-
quests and update requests are all processed serially
at a single server (the tail).
Processing a query request involves only a single
server, and that means query is a relatively cheap
operation. But when an update request is processed,
computation done at t − 1o ft h et servers does not
contribute to producing the reply and, arguably, is
redundant. The redundant servers do increase the
fault-tolerance, though.
Note that some redundant computation associ-
ated with the t − 1 servers is avoided in chain repli-
cation because the new value is computed once by
the head and then forwarded down the chain, so
each replica has only to perform a write. This for-
warding of state changes also means update can be a
non-deterministic operation—the non-deterministic
choice is made once, by the head.
3.1 Protocol Details
Clients do not directly read or write variables
Hist
objID and PendingobjID of Figure 1, so we are
free to implement them in any way that is conve-
nient. When chain replication is used to implement
the specification of Figure 1:
• Hist
objID is defined to be HistT
objID ,t h ev a l u e
of HistobjID stored by tail T of the chain, and
• PendingobjID is defined to be the set of client
requests received by any server in the chain and
not yet processed by the tail.
The chain replication protocols for query processing
and update processing are then shown to satisfy the
specification of Figure 1 by demonstrating how each
state transition made by any server in the chain is
equivalent either to a no-op or to allowed transitions
T1, T2, or T3.
Given the descriptions above for how Hist
objID
and PendingobjID are implemented by a chain (and
assuming for the moment that failures do not occur),
we observe that the only server transitions affecting
Hist
objID and PendingobjID are: (i) a server in the
chain receiving a request from a client (which affects
Pending
objID), and (ii) the tail processing a client

request (which affects Hist objID). Since other server
transitions are equivalent to no-ops, it suffices to
show that transitions (i) and (ii) are consistent with
T1 through T3.
Client Request Arrives at Chain. Clients send
requests to either the head (update) or the tail
(query). Receipt of a request r by either adds
r to the set of requests received by a server but
not yet processed by the tail. Thus, receipt of
r by either adds r to Pending
objID (as defined
above for a chain), and this is consistent with
T1.
Request Processed by T ail. Execution causes
the request to be removed from the set of
requests received by any replica that have not
yet been processed by the tail, and therefore
it deletes the request from Pending
objID (as
defined above for a chain)—the first step of
T3. Moreover, the processing of that request
by tail T uses replica Hist
T
objID which, as
defined above, implements HistobjID—and this
is exactly what the remaining steps of T3
specify.
Coping with Server Failures
In response to detecting the failure of a server that is
part of a chain (and, by the fail-stop assumption, all
such failures are detected), the chain is reconfigured
to eliminate the failed server. For this purpose, we
employ a service, called the master,t h a t
• detects failures of servers,
• informs each server in the chain of its new pre-
decessor or new successor in the new chain ob-
tained by deleting the failed server,
• informs clients which server is the head and
which is the tail of the chain.
In what follows, we assume the master is a single
process that never fails. This simplifies the expo-
sition but is not a realistic assumption; our pro-
totype implementation of chain replication actually
replicates a master process on multiple hosts, using
Paxos [16] to coordinate those replicas so they be-
have in aggregate like a single process that does not
fail.
The master distinguishes three cases: (i) failure
of the head, (ii) failure of the tail, and (iii) failure
of some other server in the chain. The handling
of each, however, depends on the following insight
about how updates are propagated in a chain.
Let the server at the head of the chain be labeled
H, the next server be labeled H +1 , etc., through
the tail, which is given label T. Define
Hist
i
objID ⪯ Hist j
objID
to hold if sequence 4 of requests Histi
objID at the
server with label i is a prefix of sequence Histj
objID
at the server with label j. Because updates are sent
between elements of a chain over reliable FIFO links,
the sequence of updates received by each server is a
prefix of those received by its successor. So we have:
Update Propagation Invariant. For servers
labeled i and j such that i ≤ j holds (i.e., i is
a predecessor of j in the chain) then:
Histj
objID ⪯ Histi
objID .
F ailure of the Head. This case is handled by the
master removing H from the chain and making the
successor to H the new head of the chain. Such a
successor must exist if our assumption holds that at
most t − 1 servers are faulty.
Changing the chain by deleting H is a transition
and, as such, must be shown to be either a no-
op or consistent with T1, T2, and/or T3 of Fig-
ure 1. This is easily done. Altering the set of
servers in the chain could change the contents of
Pending
objID—recall, PendingobjID is defined as the
set of requests received by any server in the chain
and not yet processed by the tail, so deleting server
H from the chain has the effect of removing from
Pending
objID those requests received by H but not
yet forwarded to a successor. Removing a request
from PendingobjID is consistent with transition T2,
so deleting H from the chain is consistent with the
specification in Figure 1.
F ailure of the T ail. This case is handled by re-
moving tail T from the chain and making predeces-
sor T − of T the new tail of the chain. As before,
such a predecessor must exist given our assumption
that at most t − 1 server replicas are faulty.
This change to the chain alters the values of
both PendingobjID and Hist objID, but does so in
a manner consistent with repeated T3 transitions:
PendingobjID decreases in size because HistT
objID ⪯
HistT −
objID (due to the Update Propagation Invariant,
since T − <T holds), so changing the tail from T
to T − potentially increases the set of requests com-
pleted by the tail which, by definition, decreases
the set of requests in PendingobjID. Moreover, as
required by T3, those update requests completed

by T − but not completed by T do now appear in
Hist objID because with T − now the tail, Hist objID is
defined as HistT −
objID .
Failure of Other Servers. Failure of a server S
internal to the chain is handled by deleting S from
the chain. The master first informs S’s successor S+
of the new chain configuration and then informs S’s
predecessor S− . This, however, could cause the Up-
date Propagation Invariant to be invalidated unless
some means is employed to ensure update requests
that S received before failing will still be forwarded
along the chain (since those update requests already
do appear in Hist i
objID for any predecessor i of S).
The obvious candidate to perform this forwarding
is S
− , but some bookkeeping and coordination are
now required.
Let U be a set of requests and let <U be a total
ordering on requests in that set. Define a request
sequence
r to be consistent with (U, <U )i f( i )a l lr e -
quests in
 r appear in U and (ii) requests are arranged
in
 r in ascending order according to<U . Finally, for
request sequences
 r and
 r′ consistent with (U, <U ),
define
 r ⊕
 r′ to be a sequence of all requests appear-
ing in
 r or in
 r′ such that
 r ⊕
 r′ is consistent with
(U, <U ) (and therefore requests in sequence
 r ⊕
 r′
are ordered according to <U ).
The Update Propagation Invariant is preserved by
requiring that the first thing a replicaS− connecting
to a new successor S+ does is: send to S+ (using
the FIFO link that connects them) those requests in
Hist S−
objID that might not have reachedS+;o n l ya f t e r
those have been sent may S− process and forward
requests that it receives subsequent to assuming its
new chain position.
To this end, each server i maintains a list Senti
of update requests that i has forwarded to some
successor but that might not have been processed
by the tail. The rules for adding and deleting el-
ements on this list are straightforward: Whenever
server i forwards an update request r to its succes-
sor, server i also appends r to Sent
i. The tail sends
an acknowledgement ack(r) to its predecessor when
it completes the processing of update requestr.A n d
upon receipt ack(r), a server i deletes r from Senti
and forwards ack(r) to its predecessor.
A request received by the tail must have been re-
ceived by all of its predecessors in the chain, so we
can conclude:
Inprocess Requests Invariant. If i ≤ j then
Histi
objID = Hist j
objID ⊕ Senti.
3
master S
1
2
4
S+S−
Figure 3: Space-time diagram for deletion of internal
replica.
Thus, the Update Propagation Invariant will be
maintained if S− , upon receiving notification from
the master that S+ is its new successor, first for-
wards the sequence of requests in SentS− to S+.
Moreover, there is no need for S− to forward the
prefix of SentS− that already appears in HistS+
objID .
The protocol whose execution is depicted in Fig-
ure 3 embodies this approach (including the opti-
mization of not sending more of the prefix than nec-
essary). Message 1 informs S+ of its new role; mes-
sage 2 acknowledges and informs the master what
is the sequence number sn of the last update re-
quest S+ has received; message 3 informs S− of its
new role and of sn so S− can compute the suffix of
SentS− to send to S+; and message 4 carries that
suffix.
Extending a Chain. Failed servers are removed
from chains. But shorter chains tolerate fewer fail-
ures, and object availability ultimately could be
compromised if ever there are too many server fail-
ures. The solution is to add new servers when chains
get short. Provided the rate at which servers fail is
not too high and adding a new server does not take
too long, then chain length can be kept close to the
desired t servers (so t − 1 further failures are needed
to compromise object availability).
A new server could, in theory, be added anywhere
in a chain. In practice, adding a server T
+ to the
very end of a chain seems simplist. For a tail T +,
the value of SentT + is always the empty list, so ini-
tializing SentT + is trivial. All that remains is to
initialize local object replica HistT +
objID in a way that
satisfies the Update Propagation Invariant.
The initialization of HistT +
objID can be accom-

plished by having the chain’s current tailT forward
the object replica HistT
objID it stores to T +.T h e
forwarding (which may take some time if the ob-
ject is large) can be concurrent with T’s processing
query requests from clients and processing updates
from its predecessor, provided each update is also
appended to Sent
T .S i n c eHistT +
objID ⪯ Hist T
objID
holds throughout this forwarding, Update Propaga-
tion Invariant holds. Therefore, once
HistT
objID = Hist T +
objID ⊕ SentT
holds, Inprocess Requests Invariant is established
and T + can begin serving as the chain’s tail:
• T is notified that it no longer is the tail. T
is thereafter free to discard query requests it
receives from clients, but a more sensible policy
is for T to forward such requests to new tailT +.
• Requests in SentT are sent (in sequence) toT +.
• The master is notified that T + is the new tail.
• Clients are notified that query requests should
be directed to T +.
4 Primary/Backup Protocols
Chain replication is a form of primary/backup ap-
proach [3], which itself is an instance of the state ma-
chine approach [22] to replica management. In the
primary/backup approach, one server, designated
the primary
• imposes a sequencing on client requests (and
thereby ensures strong consistency holds),
• distributes (in sequence) to other servers,
known as backups, the client requests or result-
ing updates,
• awaits acknowledgements from all non-faulty
backups, and
• after receiving those acknowledgements then
sends a reply to the client.
If the primary fails, one of the back-ups is promoted
into that role.
With chain replication, the primary’s role in se-
quencing requests is shared by two replicas. The
head sequences update requests; the tail extends
that sequence by interleaving query requests. This
sharing of responsibility not only partitions the se-
quencing task but also enab les lower-latency and
lower-overhead processing for query requests, be-
cause only a single server (the tail) is involved in
processing a query and that processing is never de-
layed by activity elsewhere in the chain. Compare
that to the primary backup approach, where the pri-
mary, before responding to a query, must await ac-
knowledgements from backups for prior updates.
In both chain replication and in the pri-
mary/backup approach, update requests must be
disseminated to all servers replicating an object or
else the replicas will diverge. Chain replication does
this dissemination serially, resulting in higher la-
tency than the primary/backup approach where re-
quests were distributed to backups in parallel. With
parallel dissemination, the time needed to generate
a reply is proportional to the maximum latency of
any non-faulty backup; with serial dissemination, it
is proportional to the sum of those latencies.
Simulations reported in §5 quantify all of these
performance differences, including variants of chain
replication and the primary/backup approach in
which query requests are sent to any server (with ex-
pectations of trading increased performance for the
strong consistency guarantee).
Simulations are not necessary for understanding
the differences in how server failures are handled by
the two approaches, though. The central concern
here is the duration of any transient outage expe-
rienced by clients when the service reconfigures in
response to a server failure; a second concern is the
added latency that server failures introduce.
The delay to detect a server failure is by far the
dominant cost, and this co st is identical for both
chain replication and the primary/backup approach.
What follows, then, is an analysis of the recovery
costs for each approach assuming that a server fail-
ure has been detected; message delays are presumed
to be the dominant source of protocol latency.
For chain replication, there are three cases to con-
sider: failure of the head, failure of a middle server,
and failure of the tail.
• Head F ailure. Query processing continues un-
interrupted. Update processing is unavailable
for 2 message delivery delays while the master
broadcasts a message to the new head and its
successor, and then it notifies all clients of the
new head using a broadcast.
• Middle Server F ailure. Query processing
continues uninterrupted. Update processing
can be delayed but update requests are not
lost, hence no transien t outage is experienced,
provided some server in a prefix of the chain
that has received the request remains operating.

Failure of a middle server can lead to a delay in
processing an update request—the protocol of
Figure 3 involves 4 message delivery delays.
• T ail F ailure.Query and update processing are
both unavailable for 2 message delivery delays
while the master sends a message to the new
tail and then notifies all clients of the new tail
u s i n gab r o a d c a s t .
With the primary/backup approach, there are two
cases to consider: failure of the primary and failure
of a backup. Query and update requests are affected
the same way for each.
• Primary F ailure. A transient outage of 5
message delays is experienced, as follows. The
master detects the failure and broadcasts a mes-
sage to all backups, requesting the number of
updates each has processed and telling them
to suspend processing requests. Each backup
replies to the master. The master then broad-
casts the identity of the new primary to all
backups. The new primary is the one having
processed the largest number of updates, and
it must then forward to the backups any up-
dates that they are missing. Finally, the master
broadcasts a message notifying all clients of the
new primary.
• Backup F ailure. Query processing continues
uninterrupted provided no update requests are
in progress. If an update request is in progress
then a transient outage of at most 1 message de-
lay is experienced while the master sends a mes-
sage to the primary indicating that acknowl-
edgements will not be forthcoming from the
faulty backup and requests should not subse-
quently be sent there.
So the worst case outage for chain replication
(tail failure) is never as long as the worst case out-
age for primary/backup (primary failure); and the
best case for chain replication (middle server fail-
ure) is shorter than the best case outage for pri-
mary/backup (backup failure). Still, if duration of
transient outage is the dominant consideration in
designing a storage service then choosing between
chain replication and the primary/backup approach
requires information about the mix of request types
and about the chances of various servers failing.
5 Simulation Experiments
To better understand throughput and availability
for chain replication, we performed a series of ex-
periments in a simulated network. These involve
prototype implementations of chain replication as
well as some of the alternatives. Because we are
mostly interested in delays intrinsic to the processing
and communications that chain replication entails,
we simulated a network with infinite bandwidth but
with latencies of 1 ms per message.
5.1 Single Chain, No Failures
First, we consider the simple case when there is only
one chain, no failures, and replication factor t is 2,
3, and 10. We compare throughput for four different
replication management alternatives:
• chain: Chain replication.
• p/b: Primary/backup.
• weak-chain: Chain replication modified so
query requests go to any random server.
• weak-p/b: Primary/backup modified so query
requests go to any random server.
Note, weak-chain and weak-p/b do not imple-
ment the strong consistency guarantees that chain
and p/b do.
We fix the query latency at a server to be 5 ms and
fix the update latency to be 50 ms. (These numbers
are based on actual values for querying or updating
a web search index.) We assume each update en-
tails some initial processing involving a disk read,
and that it is cheaper to forward object-differences
for storage than to repeat the update processing
anew at each replica; we expect that the latency
for a replica to process an object-difference message
would be 20 ms (corresponding to a couple of disk
accesses and a modest computation).
So, for example, if a chaincomprises three servers,
the total latency to perform an update is 94 ms: 1
ms for the message from the client to the head, 50
ms for an update latency at the head, 20 ms to pro-
cess the object difference message at each of the two
other servers, and three additional 1 ms forwarding
latencies. Query latency is only 7 ms, however.
In Figure 4 we graph total throughput as a func-
tion of the percentage of requests that are updates
for t =2 , t =3a n d t = 10. There are 25 clients,
each doing a mix of requests split between queries
and updates consistent with the given percentage.
Each client submits one request at a time, delaying
between requests only long enough to receive the
response for the previous request. So the clients
together can have as many as 25 concurrent re-
quests outstanding. Throughput for weak-chain

0
100
200
300
400
500
600
0 5 10 15 20 25 30 35 40 45 50
total throughput
percentage updates
weak
chain
p/b
0
100
200
300
400
500
600
0 5 10 15 20 25 30 35 40 45 50
total throughput
percentage updates
weak
chain
p/b
0
100
200
300
400
500
600
0 5 10 15 20 25 30 35 40 45 50
total throughput
percentage updates
weak
chain
p/b
(a) t =2 ( b ) t =3 ( c ) t =1 0
Figure 4: Request throughput as a function of the percentage of updates for various replication management
alternatives chain, p/b,a n dweak (denoting weak-chain,a n dweak-p/b) and for replication factors t.
and weak-p/b was found to be virtually identical,
so Figure 4 has only a single curve—labeled weak—
rather than separate curves for weak-chain and
weak-p/b.
Observe that chain replication (chain)h a se q u a l
or superior performance to primary-backup ( p/b)
for all percentages of updates and each replica-
tion factor investigated. This is consistent with
our expectations, because the head and the tail in
chain replication share a load that, with the pri-
mary/backup approach, is handled solely by the pri-
mary.
The curves for the weak variant of chain replica-
tion are perhaps surprising, as these weak variants
are seen to perform worse than chain replication
(with its strong consistency) when there are more
than 15% update requests. Two factors are involved:
• The weak variants of chain replication and pri-
mary/backup outperform pure chain replica-
tion for query-heavy loads by distributing the
query load over all servers, an advantage that
increases with replication factor.
• Once the percentage of update requests in-
creases, ordinary chain replication outperforms
its weak variant—since all updates are done at
the head. In particular, under pure chain repli-
cation (i) queries are not delayed at the head
awaiting completion of update requests (which
are relatively time consuming) and (ii) there is
more capacity available at the head for update
request processing if query requests are not also
being handled there.
Since weak-chainand weak-p/b do not implement
strong consistency guarantees, there would seem to
be surprisingly few settings where these replication
management schemes would be preferred.
Finally, note that the throughput of both chain
replication and primary backup is not affected by
replication factor provided there are sufficient con-
current requests so that multiple requests can be
pipelined.
5.2 Multiple Chains, No Failures
If each object is managed by a separate chain and
objects are large, then adding a new replica could
involve considerable delay because of the time re-
quired for transferring an object’s state to that new
replica. If, on the other hand, objects are small,
then a large storage service will involve many ob-
jects. Each processor in the system is now likely to
host servers from multiple chains—the costs of mul-
tiplexing the processors and communications chan-
nels may become prohibitive. Moreover, the failure
of a single processor now affects multiple chains.
A set of objects can always be grouped into a sin-
gle volume, itself something that could be considered
an object for purposes of chain replication, so a de-
signer has considerable latitude in deciding object
size.
For the next set of experiments, we assume
• a constant number of volumes,
• a hash function maps each object to a volume,
hence to a unique chain, and
• each chain comprises servers hosted by proces-
sors selected from amon g those implementing
the storage service.

0
20
40
60
80
100
120
0 20 40 60 80 100 120 140
average throughput
#servers
queries only
5%
10%
25%
50%
updates only
Figure 5: Average request throughput per client as
a function of the number of servers for various per-
centages of updates.
Clients are assumed to send their requests to a
dispatcher which (i) computes the hash to deter-
mine the volume, hence chain, storing the object
of concern and then (ii) forwards that request to the
corresponding chain. (The master sends configura-
tion information for each volume to the dispatcher,
avoiding the need for the master to communicate di-
rectly with clients. Interposing a dispatcher adds a
1ms delay to updates and queries, but doesn’t af-
fect throughput.) The reply produced by the chain
is sent directly to the client and not by way of the
dispatcher.
There are 25 clients in our experiments, each sub-
mitting queries and updates at random, uniformly
distributed over the chains. The clients send re-
quests as fast as they can, subject to the restriction
that each client can have only one request outstand-
ing at a time.
To facilitate comparisons with the GFS experi-
ments [11], we assume 5000 volumes each replicated
three times, and we vary the number of servers.
We found little or no difference among chain, p/b,
weak chain ,a n d weak p/b alternatives, so Fig-
ure 5 shows the average request throughput per
client for one—chain replication—as a function of
the number of servers, for varying percentages of
update requests.
5.3 Effects of Failures on Throughput
With chain replication, each server failure causes a
three-stage process to start:
1. Some time (we conservatively assume 10 sec-
onds in our experiments) elapses before the
master detects the server failure.
2. The offending server is then deleted from the
chain.
3. The master ultimately adds a new server to
that chain and initiates adata recovery process,
which takes time proportional to (i) how much
data was being stored on the faulty server and
(ii) the available network bandwidth.
Delays in detecting a failure or in deleting a faulty
server from a chain can increase request processing
latency and can increase transient outage duration.
The experiments in this section explore this.
We assume a storage service characterized by the
parameters in Table 1; these values are inspired by
what is reported for GFS [11]. The assumption
about network bandwidth is based on reserving for
data recovery at most half the bandwidth in a 100
Mbit/second network; the time to copy the 150 Gi-
gabytes stored on one server is now 6 hours and 40
minutes.
In order to measure the effects of a failures on the
storage service, we apply a load. The exact details
of the load do not matter greatly. Our experiments
use eleven clients. Each client repeatedly chooses a
random object, performs an operation, and awaits
a reply; a watchdog timer causes the client to start
the next loop iteration if 3 seconds elapse and no
reply has been received. Ten of the clients exclu-
sively submit query operations; the eleventh client
exclusively submits update operations.
parameter
 value
number of servers (N)
 24
number of volumes
 5000
chain length (t)
 3
data stored per server
 150 Gigabytes
maximum network band-
width devoted to data
recovery to/from any
server
6.25 Megabytes/sec
server reboot time after a
failure
10 minutes
Table 1: Simulated Storage Service Characteristics.

90
95
100
105
110
00:30 01:00 01:30 02:00
query thruput
time
90
95
100
105
110
00:30 01:00 01:30 02:00
query thruput
time
9
10
11
00:30 01:00 01:30 02:00
upd. thruput
time
9
10
11
00:30 01:00 01:30 02:00
upd. thruput
time
(a) one failure (b) two failures
Figure 6: Query and update throughput with one or two failures at time 00:30.
Each experiment described executes for 2 simu-
lated hours. Thirty minutes into the experiment,
the failure of one or two servers is simulated (as
in the GFS experiments). The master detects that
failure and deletes the failed server from all of the
chains involving that server. For each chain that was
shortened by the failure, the master then selects a
new server to add. Data recovery to those servers is
started.
Figure 6(a) shows aggregate query and update
throughputs as a function of time in the case a single
server F fails. Note the sudden drop in throughput
when the simulated failure occurs 30 minutes into
the experiment. The resolution of the x- a x i si st o o
coarse to see that the throughput is actually zero for
about 10 seconds after the failure, since the master
requires a bit more than 10 seconds to detect the
server failure and then delete the failed server from
all chains.
With the failed server deleted from all chains, pro-
cessing now can proceed, albeit at a somewhat lower
rate because fewer servers are operational (and the
same request processing load must be shared among
them) and because data recovery is consuming re-
sources at various servers. Lower curves on the
graph reflect this. After 10 minutes, failed server
F becomes operational again, and it becomes a pos-
sible target for data recovery. Every time data re-
covery of some volume successfully completes at F,
query throughput improves (as seen on the graph).
This is because F, now the tail for another chain, is
handling a growing proportion of the query load.
One might expect that after all data recovery con-
cludes, the query throughput would be what it was
at the start of the experiment. The reality is more
subtle, because volumes are no longer uniformly dis-
tributed among the servers. In particular, server
F will now participate in fewer chains than other
servers but will be the tail of every chain in which it
does participate. So the load is no longer well bal-
anced over the servers, and aggregate query through-
put is lower.
Update throughput decreases to 0 at the time of
the server failure and then, once the master deletes
the failed server from all chains, throughput is actu-
ally better than it was initially. This throughput im-
provement occurs because the server failure causes
some chains to be length 2 (rather than 3), reduc-
ing the amount of work involved in performing an
update.
The GFS experiments [11] consider the case where
two servers fail, too, so Figure 6(b) depicts this
for our chain replication protocol. Recovery is still
smooth, although it takes additional time.
5.4 Large Scale Replication of Criti-
cal Data
As the number of servers increases, so should the
aggregate rate of server failures. If too many servers
fail, then a volume might become unavailable. The

0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
MTBU (days)
# servers
t = 4
t = 3
t = 2
t = 1
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
MTBU (days)
# servers
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
# servers
t = 4
t = 3
t = 2
t = 1
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
# servers
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
# servers
t = 4
t = 3
t = 2
t = 1
0.0001
0.001
0.01
0.1
1
10
100
1000
10000
1 10 100
# servers
(a) ring (b) rndseq (c) rndpar
Figure 7: The MTBU and 99% confidence intervals as a function of the number of servers and replication
factor for three different placement strategies: (a) DHT-based placement with maximum possible parallel
recovery; (b) random placement, but with parallel recovery limited to the same degree as is possible with
DHTs; (c) random placement with maximum possible parallel recovery.
probability of this depends on how volumes are
placed on servers and, in particular, the extent to
which parallelism is possible during data recovery.
We have investigated three volume placement
strategies:
• ring: Replicas of a volume are placed at con-
secutive servers on a ring, determined by a con-
sistent hash of the volume identifier. This is
the strategy used in CFS [7] and PAST [19].
The number of parallel data recoveries possible
is limited by the chain length t.
• rndpar: Replicas of a volume are placed ran-
domly on servers. This is essentially the strat-
e g yu s e di nG F S .
5 Notice that, given enough
servers, there is no limit on the number of par-
allel data recoveries possible.
• rndseq: Replicas of a volume are placed ran-
domly on servers (as in rndpar), but the max-
imum number of parallel data recoveries is lim-
ited by t (as in ring). This strategy is not used
in any system known to us but is a useful bench-
mark for quantifying the impacts of placement
and parallel recovery.
To understand the advantages of parallel data re-
covery, consider a serverF that fails and was partic-
ipating in chains C
1,C 2,...,C n. For each chain Ci,
data recovery requires a source from which the vol-
ume data is fetched and a host that will become the
new element of chain Ci. Given enough processors
and no constraints on the placement of volumes, it
is easy to ensure that the new elements are all dis-
joint. And with random placement of volumes, it is
likely that the sources will be disjoint as well. With
disjoint sources and new elements, data recovery for
chains C
1,C 2,...,C n can occur in parallel. And a
shorter interval for data recovery of C1,C 2,...,C n,
implies that there is a shorter window of vulnera-
bility during which a small number of concurrent
failures would render some volume unavailable.
We seek to quantify the mean time between un-
availability (MTBU) of any object as a function of
the number of servers and the placement strategy.
Each server is assumed to exhibit exponentially dis-
tributed failures with a MTBF (Mean Time Between
Failures) of 24 hours.
6 As the number of servers in
a storage system increases, so would the number of
volumes (otherwise, why add servers). In our exper-
iments, the number of volumes is defined to be 100
times the initial number of servers, with each server
storing 100 volumes at time 0.
We postulate that the time it takes to copy all
the data from one server to another is four hours,
which corresponds to copying 100 Gigabytes across
a 100 Mbit/sec network restricted so that only half
bandwidth can be used for data recovery. As in the
GFS experiments, the maximum number of parallel
data recoveries on the network is limited to 40% of
the servers, and the minimum transfer time is set to
10 seconds (the time it takes to copy an individual
GFS object, which is 64 KBytes).
Figure 7(a) shows that the MTBU for the ring
strategy appears to have an approximately Zipfian
distribution as a function of the number of servers.

Thus, in order to maintain a particular MTBU, it
is necessary to grow chain length t when increasing
the number of servers. From the graph, it seems
as though chain length needs to be increased as the
logarithm of the number of servers.
Figure 7(b) shows the MTBU forrndseq.F o rt>
1, rndseq has lower MTBU than ring.C o m p a r e d
to ring, random placement is inferior because with
random placement there are more sets of t servers
that together store a copy of a chain, and therefore
there is a higher probability of a chain getting lost
due to failures.
However, random placement makes additional op-
portunities for parallel recovery possible if there are
enough servers. Figure 7(c) shows the MTBU for
rndpar.F o r f e w s e r v e r s ,rndpar performs the same
as rndseq, but the increasing opportunity for paral-
lel recovery with the number of servers improves the
MTBU, and eventually rndpar outperforms rnd-
seq, and more importantly, it outperforms ring.
6 Related Work
Scalability . Chain replication is an example of
what Jimen´ez-Peris and Pati˜no-Mart´ınez [14] call a
ROWAA (r
ead o
ne, w
 rite a
ll a
vailable) approach.
They report that ROWAA approaches provide su-
perior scaling of availability to quorum techniques,
claiming that availability of ROWAA approaches
improves exponentially with the number of repli-
cas. They also argue that non-ROWAA approaches
to replication will necessarily be inferior. Because
ROWAA approaches also exhibit better throughout
than the best known quorum systems (except for
nearly write-only applications) [14], ROWAA would
seem to be the better choice for replication in most
real settings.
Many file services trade c onsistency for perfor-
mance and scalability. Examples include Bayou [17],
Ficus [13], Coda [15], and Sprite [5]. Typically,
these systems allow continued operation when a net-
work partitions by offering tools to fix inconsisten-
cies semi-automatically. Our chain replication does
not offer graceful handling of partitioned operation,
trading that instead for supporting all three of: high
performance, scalability, and strong consistency.
Large-scale peer-to-peer reliable file systems are a
relatively recent avenue of inquiry. OceanStore [6],
FARSITE [2], and PAST [19] are examples. Of
these, only OceanStore provides strong (in fact,
transactional) consistency guarantees.
Google’s File System (GFS) [11] is a large-scale
cluster-based reliable file system intended for ap-
plications similar to those motivating the invention
of chain replication. But in GFS, concurrent over-
writes are not serialized and read operations are not
synchronized with write operations. Consequently,
different replicas can be left in different states, and
content returned by read operations may appear to
vanish spontaneously from GFS. Such weak seman-
tics imposes a burden on programmers of applica-
tions that use GFS.
Availability versus Consistency . Yu and Vah-
dat [25] explore the trade-off between consistency
and availability. They argue that even in relaxed
consistency models, it is important to stay as close
to strong consistency as possible if availability is to
be maintained in the long run. On the other hand,
Gray et al. [12] argue that systems with strong con-
sistency have unstable behavior when scaled-up, and
they propose thetentative update transaction for cir-
cumventing these scalability problems.
Amza et al. [4] present a one-copy serializable
transaction protocol tha t is optimized for replica-
tion. As in chain replication, updates are sent to
all replicas whereas queries are processed only by
replicas known to store all completed updates. (In
chain replication, the tail is the one replica known
to store all completed updates.) The protocol of [4]
performs as well as replication protocols that provide
weak consistency, and it scales well in the number
of replicas. No analysis is given for behavior in the
face of failures.
Replica Placement. Previous work on replica
placement has focussed on achieving high through-
put and/or low latency rather than on supporting
high availability. Acharya and Zdonik [1] advocate
locating replicas according to predictions of future
accesses (basing those predictions on past accesses).
In the Mariposa project [23], a set of rules allows
users to specify where to create replicas, whether
to move data to the query or the query to the data,
where to cache data, and more. Consistency is trans-
actional, but no consideration is given to availabil-
ity. Wolfson et al. consider strategies to optimize
database replica placement in order to optimize per-
formance [24]. The OceanStore project also con-
siders replica placement [10, 6] but from the CDN
(Content Distribution Network, such as Akamai)
perspective of creating as few replicas as possible
while supporting certain quality of service guaran-
tees. There is a significant body of work (e.g., [18])
concerned with placemen t of web page replicas as
well, all from the perspective of reducing latency
and network load.

Douceur and Wattenhofer investigate how to max-
imize the worst-case availability of files in FAR-
SITE [2], while spreading the storage load evenly
across all servers [8, 9]. Servers are assumed to have
varying availabilities. The algorithms they consider
repeatedly swap files between machines if doing so
improves file availability. The results are of a theo-
retical nature for simple scenarios; it is unclear how
well these algorithms will work in a realistic storage
system.
7 Concluding Remarks
Chain replication supports high throughput for
query and update requests, high availability of data
objects, and strong consistency guarantees. This is
possible, in part, because storage services built us-
ing chain replication can and do exhibit transient
outages but clients cannot distinguish such outages
from lost messages. Thus, the transient outages that
chain replication introduces do not expose clients to
new failure modes—chain replication represents an
interesting balance between what failures it hides
from clients and what failures it doesn’t.
When chain replication is employed, high avail-
ability of data objects comes from carefully se-
lecting a strategy for pl acement of volume repli-
cas on servers. Our experiments demonstrated that
with DHT-based placement strategies, availability
is unlikely to scale with in creases in the numbers
of servers; but we also demonstrated that random
placement of volumes does permit availability to
scale with the number of servers if this placement
strategy is used in concert with parallel data recov-
ery, as introduced for GFS.
Our current prototype is intended primarily for
use in relatively homogeneous LAN clusters. Were
our prototype to be deployed in a heterogeneous
wide-area setting, then uniform random placement
of volume replicas would no longer make sense. In-
stead, replica placement would have to depend on
access patterns, network proximity, and observed
host reliability. Protocols to re-order the elements
of a chain would likely b ecome crucial in order to
control load imbalances.
Our prototype chain replication implementation
consists of 1500 lines of Java code, plus another 2300
lines of Java code for a Paxos library. The chain
replication protocols are structured as a library that
makes upcalls to a storage service (or other appli-
cation). The experiments in this paper assumed a
“null service” on a simulated network. But the li-
brary also runs over the Java socket library, so it
could be used to support a variety of storage service-
like applications.
Acknowledgements.
Thanks to our colleagues
H˚akon Brug˚ard, Kjetil Jacobsen, and Knut Omang at
FAST who first brought this problem to our attention.
Discussion with Mark Linderman and Sarah Chung were
helpful in revising an earlier version of this paper. We
are also grateful for the comments of the OSDI reviewers
and shepherd Margo Seltzer. A grant from the Research
Council of Norway to FAST ASA is noted and acknowl-
edged.
Van Renesse and Schneider are supported, in part, by
AFOSR grant F49620–03–1–0156 and DARPA/AFRL-
IFGA grant F30602–99–1–0532, although the views and
conclusions contained herein are those of the authors and
should not be interpreted as necessarily representing the
official policies or endorsements, either expressed or im-
plied, of these organizations or the U.S. Government.
Notes
1The case where V = newVal yields a semantics
for update that is simply a file system write opera-
tion; the case whereV = F(newVal, objID)a m o u n t s
to support for atomic read-modify-write operations
on objects. Though powerful, this semantics falls
short of supporting transactions, which would allow
a request to query and/or update multiple objects
indivisibly.
2An actual implementation would probably store
the current value of the object rather than storing
the sequence of updates that produces this current
value. We employ a sequence of updates represen-
tation here because it simplifies the task of arguing
that strong consistency guarantees hold.
3If HistobjID s t o r e st h ec u r r e n tv a l u eo fobjID
rather than its entire history then “ HistobjID · r”
should be interpreted to denote applying the update
to the object.
4If Histi
objID is the current state rather than a
sequence of updates, then ⪯ is defined to be the
“prior value” relation rather than the “prefix of”
relation.
5Actually, the placemen t strategy is not dis-
cussed in [11]. GFS does some load balancing that
results in an approximately even load across the
servers, and in our simulations we expect that ran-
dom placement is a good approximation of this strat-
egy.

6An unrealistically shor t MTBF was selected
here to facilitate running long-duration simulations.
References
[1] S. Acharya and S.B. Zdonik. An efficient scheme
for dynamic data replication. Technical Report CS-
93-43, Brown University, September 1993.
[2] A. Adya, W.J. Bolosky, M. Castro, G. Cermak,
R. Chaiken, J.R. Douceur, J. Howell, J.R. Lorch,
M. Theimer, and R.P. Wattenhofer. FARSITE: Fed-
erated, Available, and Reliable Storage for an In-
completely Trusted Environment. In P r o c .o ft h e
5th Symp. on Operating Systems Design and Imple-
mentation, Boston, MA, December 2002. USENIX.
[3] P.A. Alsberg and J.D. Day. A principle for resilient
sharing of distributed resources. In Proc. of the 2nd
Int. Conf. on Software Engineering , pages 627–644,
October 1976.
[ 4 ]C .A m z a ,A . L .C o x ,a n dW .Z w a e n e p o e l . D i s -
tributed Versioning: Consistent replication for scal-
ing back-end databases of dynamic content web
sites. In Proc. of Middleware’03 , pages 282–304,
Rio de Janeiro, Brazil, June 2003.
[5] M.G. Baker and J.K. Ousterhout. Availability in
the Sprite distributed file system. Operating Sys-
tems Review , 25(2):95–98, April 1991. Also ap-
peared in the 4th ACM SIGOPS European Work-
shop – Fault Tolerance Support in Distributed Sys-
tems.
[6] Y. Chen, R.H. Katz, and J. Kubiatowicz. Dynamic
replica placement for scalable content delivery. In
Proc. of the 1st Int. Workshop on Peer-To-Peer
Systems, Cambridge, MA, March 2002.
[7] F. Dabek, M.F. Kaashoek, D. Karger, R. Morris,
and I. Stoica. Wide-area cooperative storage with
CFS. In Proc. of the 18th ACM Symp. on Operating
Systems Principles, Banff, Canada, October 2001.
[8] J.R. Douceur and R.P. Wattenhofer. Competitive
hill-climbing strategies for replica placement in a
distributed file system. In Proc. of the 15th In-
ternational Symposium on DIStributed Computing ,
Lisbon, Portugal, October 2001.
[9] J.R. Douceur and R.P. Wattenhofer. Optimizing
file availability in a secure serverless distributed file
system. In Proc. of the 20th Symp. on Reliable Dis-
tributed Systems. IEEE, 2001.
[10] D. Geels and J. Kubiatowicz. Replica manage-
ment should be a game. In Proc. of the 10th Eu-
ropean SIGOPS Workshop , Saint-Emilion, France,
September 2002. ACM.
[11] S. Ghermawat, H. Gobioff, and S.-T. Leung. The
Google file system. In Proc. of the 19th ACM Symp.
on Operating Systems Principles , Bolton Landing,
NY, October 2003.
[12] J. Gray, P. Helland, P. O’Neil, and D. Shasha. The
dangers of replication and a solution. InP r o c .o ft h e
International Conference on Management of Data
(SIGMOD), pages 173–182. ACM, June 1996.
[13] J.S. Heidemann and G.J. Popek. File system devel-
opment with stackable layers. ACM Transactions
on Computer Systems , 12(1):58–89, February 1994.
[14] R. Jimen´ez-Peris and M. Pati˜no-Mart´ınez. Are quo-
rums an alternative for data replication? ACM
Transactions on Database Systems , 28(3):257–294,
September 2003.
[15] J. Kistler and M. Satyanarayanann. Disconnected
operation in the Coda file system (preliminary ver-
sion). ACM Transactions on Computer Systems ,
10(1):3–25, February 1992.
[16] L. Lamport. The part-time parliament. ACM
Transactions on Computer Systems , 16(2):133–169,
1998.
[17] K. Petersen, M.J. Spreitzer, D.B. Terry, M.M.
Theimer, and A.J. Demers. Flexible update propa-
gation for weakly consistent replication. In Proc. of
the 16th ACM Symp. on Operating Systems Prin-
ciples, pages 288–301, Saint-Malo, France, October
1997.
[18] L. Qiu, V.N. Padmanabhan, and G.M. Voelker. On
the placement of web server replicas. In Proc. of
the 20th INFOCOM , Anchorage, AK, March 2001.
IEEE.
[19] A. Rowstron and P. Druschel. Storage manage-
ment and caching in PAST, a large scale, persis-
tent peer-to-peer storage utility. In Proc. of the
18th ACM Symp. on Operating Systems Principles ,
Banff, Canada, October 2001.
[20] J. Saltzer, D. Reed, and D. Clark. End-to-end ar-
guments in system design. ACM Transactions on
Computer Systems, 2(4):277–288, November 1984.
[21] F.B. Schneider. Byzantine generals in action: Im-
plementing fail-stop processors. ACM Transactions
on Computer Systems , 2(2):145–154, May 1984.
[22] F.B. Schneider. Implementing fault-tolerant ser-
vices using the state machine approach: A tutorial.
ACM Computing Surveys , 22(4):299–319, Decem-
ber 1990.
[23] M. Stonebraker, P.M. Aoki, R. Devine, W. Litwin,
and M. Olson. Mariposa: A new architecture for
distributed data. In Proc. of the 10th Int. Conf. on
Data Engineering, Houston, TX, 1994.
[24] O. Wolfson, S. Jajodia, and Y. Huang. An adaptive
data replication algorithm. ACM Transactions on
Computer Systems, 22(2):255–314, June 1997.
[25] H. Yu and A. Vahdat. The cost and limits of
availability for replicated services. In Proc. of the
18th ACM Symp. on Operating Systems Principles ,
Banff, Canada, October 2001.

No compromises: distributed transactions with
consistency, availability, and performance
Aleksandar Dragojevi´c, Dushyanth Narayanan, Edmund B. Nightingale,
Matthew Renzelmann, Alex Shamis, Anirudh Badam, Miguel Castro
Microsoft Research
Abstract
Transactions with strong consistency and high availability
simplify building and reasoning about distributed systems.
However, previous implementations performed poorly. This
forced system designers to avoid transactions completely,
to weaken consistency guarantees, or to provide single-
machine transactions that require programmers to partition
their data. In this paper, we show that there is no need to
compromise in modern data centers. We show that a main
memory distributed computing platform called FaRM can
provide distributed transactions with strict serializability,
high performance, durability, and high availability. FaRM
achieves a peak throughput of 140 million TATP transac-
tions per second on 90 machines with a 4.9 TB database, and
it recovers from a failure in less than 50 ms. Key to achiev-
ing these results was the design of new transaction, replica-
tion, and recovery protocols from first principles to leverage
commodity networks with RDMA and a new, inexpensive
approach to providing non-volatile DRAM.
1. Introduction
Transactions with high availability and strict serializabil-
ity [35] simplify programming and reasoning about dis-
tributed systems by providing a simple, powerful abstrac-
tion: a single machine that never fails and that executes one
transaction at a time in an order consistent with real time.
However, prior attempts to implement this abstraction in a
distributed system resulted in poor performance. Therefore,
systems such as Dynamo [13] or Memcached [1] improve
performance by either not supporting transactions or by im-
plementing weak consistency guarantees. Others (e.g., [3–
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
SOSP’15, October 4–7, 2015, Monterey, CA.
Copyright is held by the owner/author(s).
ACM 978-1-4503-3834-9/15/10.
http://dx.doi.org/10.1145/2815400.2815425
6, 9, 28]), provide transactions only when all the data resides
within a single machine, forcing programmers to partition
their data and complicating reasoning about correctness.
This paper demonstrates that new software in modern
data centers can eliminate the need to compromise. It de-
scribes the transaction, replication, and recovery protocols in
FaRM [16], a main memory distributed computing platform.
FaRM provides distributed ACID transactions with strict se-
rializability, high availability, high throughput and low la-
tency. These protocols were designed from first principles
to leverage two hardware trends appearing in data centers:
fast commodity networks with RDMA and an inexpensive
approach to providing non-volatile DRAM. Non-volatility
is achieved by attaching batteries to power supply units and
writing the contents of DRAM to SSD when the power fails.
These trends eliminate storage and network bottlenecks, but
they also expose CPU bottlenecks that limit their perfor-
mance benefit. FaRM’s protocols follow three principles to
address these CPU bottlenecks: reducing message counts,
using one-sided RDMA reads and writes instead of mes-
sages, and exploiting parallelism effectively.
FaRM scales out by distributing objects across the ma-
chines in a data center while allowing transactions to span
any number of machines. Rather than replicate coordi-
nators and data partitions using Paxos (e.g., as in [11]),
FaRM reduces message counts by using vertical Paxos [25]
with primary-backup replication, and unreplicated coordina-
tors that communicate directly with primaries and backups.
FaRM uses optimistic concurrency control with a four phase
commit protocol (lock, validation, commit backup, and com-
mit primary) [16] but we improved the original protocol by
eliminating the messages to backups in the lock phase.
FaRM further reduces CPU overhead by using one-sided
RDMA operations. One-sided RDMA uses no remote CPU
and it avoids most local CPU overhead. FaRM transactions
use one-sided RDMA reads during transaction execution
and validation. Therefore, they use no CPU at remote read-
only participants. Additionally, coordinators use one-sided
RDMA when logging records to non-volatile write-ahead
logs at the replicas of objects modified in a transaction. For
54

example, the coordinator uses a single one-sided RDMA to
write a commit record to a remote backup. Hence, transac-
tions use no foreground CPU at backups. CPU is used later
in the background when lazily truncating logs to update ob-
jects in-place.
Using one-sided RDMA requires new failure-recovery
protocols. For example, FaRM cannot rely on servers to
reject incoming requests when their leases [18] expire be-
cause requests are served by the NICs, which do not sup-
port leases. We solve this problem by usingprecise member-
ship [10] to ensure that machines agree on the current con-
figuration membership and send one-sided operations only
to machines that are members. FaRM also cannot rely on
traditional mechanisms that ensure participants have the re-
sources necessary to commit a transaction during the prepare
phase because transaction records are written to participant
logs without involving the remote CPU. Instead, FaRM uses
reservations to ensure there is space in the logs for all the
records needed to commit and truncate a transaction before
starting the commit.
The failure recovery protocol in FaRM is fast because it
leverages parallelism effectively. It distributes recovery of
every bit of state evenly across the cluster and it parallelizes
recovery across cores in each machine. In addition, it uses
two optimizations to allow transaction execution to proceed
in parallel with recovery. First, transactions begin accessing
data affected by a failure after a lock recovery phase that
takes only tens of milliseconds to complete rather than wait
several seconds for the rest of recovery. Second, transactions
that are unaffected by a failure continue executing without
blocking. FaRM also provides fast failure detection by lever-
aging the fast network to exchange frequent heart-beats, and
it uses priorities and pre-allocation to avoid false positives.
Our experimental results show that you can have it all:
consistency, high availability, and performance. FaRM re-
covers from single machine failures in less than 50 ms and
it outperforms state-of-the-art single-machine in-memory
transactional systems with just a few machines. For exam-
ple, it achieves better throughput than Hekaton [14, 26]
when running on just three machines and it has both better
throughput and latency than Silo [39, 40].
2. Hardware trends
FaRM’s design is motivated by the availability of plentiful,
cheap DRAM in data center machines. A typical data center
configuration has 128–512 GB of DRAM per 2-socket ma-
chine [29], and DRAM costs less than $12/GB1. This means
that a petabyte of DRAM requires only 2000 machines, and
this is sufficient to hold the data sets of many interesting ap-
plications. In addition, FaRM exploits two hardware trends
to eliminate storage and network bottlenecks: non-volatile
DRAM, and fast commodity networks with RDMA.
1 16 GB DDR4 DIMMs on newegg.com, 21 March 2015.
1 SSD 2 SSDs 3 SSDs 4 SSDs0
20
40
60
80
100
120Energy required (J/GB)
Figure 1. Energy to copy one GB from DRAM to SSD
2.1 Non-volatile DRAM
A “distributed uninterruptible power supply (UPS)” exploits
the wide availability of Lithium-ion batteries to lower the
cost of a data center UPS over a traditional, centralized
approach that uses lead-acid batteries. For example, Mi-
crosoft’s Open CloudServer (OCS) specification includes
Local Energy Storage (LES) [30, 36], which integrates Li-
ion batteries with the power supply units in each 24-machine
chassis within a rack. The estimated LES UPS cost is less
than $0.005 per Joule. 2 This approach is more reliable than
a traditional UPS: Li-ion batteries are overprovisioned with
multiple independent cells, and any battery failure impacts
only a portion of a rack.
A distributed UPS effectively makes DRAM durable.
When a power failure occurs, the distributed UPS saves the
contents of memory to a commodity SSD using the energy
from the battery. This not only improves common-case per-
formance by avoiding synchronous writes to SSD, it also
preserves the lifetime of the SSD by writing to it only when
failures occur. An alternative approach is to use non-volatile
DIMMs (NVDIMMs), which contain their own private flash,
controller and supercapacitor (e.g., [2]). Unfortunately, these
devices are specialized, expensive, and bulky. In contrast,
a distributed UPS uses commodity DIMMs and leverages
commodity SSDs. The only additional cost is the reserved
capacity on the SSD and the UPS batteries themselves.
Battery provisioning costs depend on the energy required
to save memory to SSDs. We measured an unoptimized pro-
totype on a standard 2-socket machine. On failure, it turns
off the HDDs and NIC and saves in-memory data to a sin-
gle M.2 (PCIe) SSD, and it consumes 110 Joules per GB of
data saved. Roughly 90 Joules is used to power the two CPU
sockets on the machine during the save. Additional SSDs
reduce the time to save data and therefore the energy con-
sumed (Figure 1). Optimizations, like putting the CPUs into
a low-power state, will further reduce energy consumption.
In the worst-case configuration, (single SSD, no opti-
mization) at $0.005 per Joule, the energy cost of non-
2 Li-ion is 5x cheaper than traditional lead-acid based UPS, which costs $31
million per 25 MW data center. A 25 MW data center can house 100,000
machines, and hence the Li-ion UPS cost per machine is $62. A 24-machine
chassis has 6 PSUs, each with an LES that is provisioned for at least 1600 W
for 5 seconds and 1425 W for a further 30 seconds, i.e. a total of 50 kJ per
PSU or 12.5 kJ per machine, giving a cost per Joule of $0.0048.
55

8 16 32 64 128 256 512 1024 2048
Transfer size (bytes)
0
5
10
15
20Operations / µs / machine
RDMA
RPC
Figure 2. Per-machine RDMA and RPC read performance
volatility is $0.55/GB and the storage cost of reserving SSD
capacity is $0.90/GB3. The combined additional cost is less
than 15% of the base DRAM cost, which is a significant
improvement over NVDIMMs that cost 3–5x as much as
DRAM. Therefore, it is feasible and cost-effective to treat all
machine memory as non-volatile RAM (NVRAM). FaRM
stores all data in memory, and considers it durable when it
has been written to NVRAM on multiple replicas.
2.2 RDMA networking
FaRM uses one-sided RDMA operations where possible be-
cause they do not use the remote CPU. We based this deci-
sion both on our prior work and on additional measurements.
In [16], we showed that on a 20-machine RoCE [22] clus-
ter, RDMA reads performed 2x better than a reliable RPC
over RDMA when all machines read randomly chosen small
objects from the other machines in the cluster. The bottle-
neck was the NIC message rate and our implementation of
RPC requires twice as many messages as one-sided reads.
We replicated this experiment on a 90-machine cluster where
each machine has two Infiniband FDR (56 Gbps) NICs. This
more than doubles the message rate per machine when com-
pared with [16] and eliminates the NIC message rate bottle-
neck. Both RDMA and RPC are now CPU bound and the
performance gap increases to 4x, as seen in Figure 2.This il-
lustrates the importance of reducing CPU overhead to realize
the potential of the new hardware.
3. Programming model and architecture
FaRM provides applications with the abstraction of a global
address space that spans machines in a cluster. Each machine
runs application threads and stores objects in the address
space. The FaRM API [16] provides transparent access to
local and remote objects within transactions. An application
thread can start a transaction at any time and it becomes the
transaction’s coordinator. During a transaction’s execution,
the thread can execute arbitrary logic as well as read, write,
allocate, and free objects. At the end of the execution, the
thread invokes FaRM to commit the transaction.
FaRM transactions use optimistic concurrency control.
Updates are buffered locally during execution and only made
3 Samsung M.2 256 GB MLC, newegg.com on 25 March 2015
FARM
Application
Region Tx log Tx log
Machine C Machine B
Machine D
(CM)
Machine A CPU
NVRAM
Tx records Tx recordsRemote
reads
Lease
renewals
Local reads
Co-ordination
service
(Zookeeper)
Msg queue
Messages
Figure 3. FaRM architecture
visible to other transactions on a successful commit. Com-
mits can fail due to conflicts with concurrent transactions
or failures. FaRM provides strict serializability [35] of all
successfully committed transactions. During transaction ex-
ecution, FaRM guarantees that individual object reads are
atomic, that they read only committed data, that successive
reads of the same object return the same data, and that reads
of objects written by the transaction return the latest value
written. It does not guarantee atomicity across reads of dif-
ferent objects but, in this case, it guarantees that the trans-
action does not commit ensuring committed transactions
are strictly serializable. This allows us to defer consistency
checks until commit time instead of re-checking consistency
on each object read. However, it adds some programming
complexity: FaRM applications must handle these tempo-
rary inconsistencies during execution [20]. It is possible to
deal with these inconsistencies automatically [12].
The FaRM API also provides lock-free reads, which are
optimized single-object read only transactions, and locality
hints, which enable programmers to co-locate related objects
on the same set of machines. These can be used by applica-
tions to improve performance as described in [16].
Figure 3 shows a FaRM instance with four machines. The
figure also shows the internal components of machine A.
Each machine runs FaRM in a user process with a kernel
thread pinned to each hardware thread. Each kernel thread
runs an event loop that executes application code and polls
the RDMA completion queues.
A FaRM instance moves through a sequence of config-
urations over time as machines fail or new machines are
added. A configuration is a tuple⟨i,S,F, CM⟩ wherei is a
unique, monotonically increasing 64-bit configuration iden-
tifier,S is the set of machines in the configuration, F is a
mapping from machines to failure domains that are expected
to fail independently (e.g., different racks), and CM ∈ S
is the configuration manager. FaRM uses a Zookeeper [21]
coordination service to ensure machines agree on the cur-
rent configuration and to store it, as in Vertical Paxos [25].
But it does not rely on Zookeeper to manage leases, detect
failures, or coordinate recovery, as is usually done. The CM
does these using an efficient implementation that leverages
RDMA to recover fast. Zookeeper is invoked by the CM
once per configuration change to update the configuration.
56

The global address space in FaRM consists of 2 GB re-
gions, each replicated on one primary andf backups, where
f is the desired fault tolerance. Each machine stores sev-
eral regions in non-volatile DRAM that can be read by other
machines using RDMA. Objects are always read from the
primary copy of the containing region, using local mem-
ory accesses if the region is on the local machine and using
one-sided RDMA reads if remote. Each object has a 64-bit
version that is used for concurrency control and replication.
The mapping of a region identifier to its primary and back-
ups is maintained by the CM and replicated with the region.
These mappings are fetched on demand by other machines
and cached by threads together with the RDMA references
needed to issue one-sided RDMA reads to the primary.
Machines contact the CM to allocate a new region. The
CM assigns a region identifier from a monotonically increas-
ing counter and selects replicas for the region. Replica selec-
tion balances the number of regions stored on each machine
subject to the constraints that there is enough capacity, each
replica is in a different failure domain, and the region is co-
located with a target region when the application specifies
a locality constraint. It then sends a prepare message to the
selected replicas with the region identifier. If all replicas re-
port success in allocating the region, the CM sends a com-
mit message to all of them. This two-phase protocol ensures
a mapping is valid and replicated at all the region replicas
before it is used.
This centralized approach provides more flexibility to
satisfy failure independence and locality constraints than
our previous approach based on consistent hashing [16]. It
also makes it easier to balance load across machines and to
operate close to capacity. With 2 GB regions, we expect up to
250 regions on a typical machine and hence that a single CM
could handle region allocation for thousands of machines.
Each machine also stores ring buffers that implement
FIFO queues [16]. They are used either as transaction logs
or message queues. Each sender-receiver pair has its own
log and message queue, which are physically located on the
receiver. The sender appends records to the log using one-
sided RDMA writes to its tail. These writes are acknowl-
edged by the NIC without involving the receiver’s CPU. The
receiver periodically polls the head of the log to process
records. It lazily updates the sender when it truncates the
log, allowing the sender to reuse space in the ring buffer.
4. Distributed transactions and replication
FaRM integrates the transaction and replication protocols
to improve performance. It uses fewer messages than tradi-
tional protocols, and exploits one-sided RDMA reads and
writes for CPU efficiency and low latency. FaRM uses
primary-backup replication in non-volatile DRAM for both
data and transaction logs, and uses unreplicated transaction
coordinators that communicate directly with primaries and
backups. It uses optimistic concurrency control with read
Serialization
point
C
P1
P2
B1
B2
Execute phase
P3
B3
Report committed to app
Commit phase
1. LOCK 2. VALIDATE 3. COMMIT
BACKUP
4. COMMIT
PRIMARY
Decision
5. TRUNCATE
Figure 4. FaRM commit protocol with a coordinator C,
primaries onP1,P 2,P 3, and backups onB1,B 2,B 3.P1 and
P2 are read and written.P3 is only read. We use dashed lines
for RDMA reads, solid ones for RDMA writes, dotted ones
for hardware acks, and rectangles for object data.
validation, as in some software transactional memory sys-
tems (e.g., TL2 [15]).
Figure 4 shows the timeline for a FaRM transaction and
tables 1 and 2 list all log record and message types used in
the transaction protocol. During the execution phase, trans-
actions use one-sided RDMA to read objects and they buffer
writes locally. The coordinator also records the addresses
and versions of all objects accessed. For primaries and back-
ups on the same machine as the coordinator, object reads
and writes to the log use local memory accesses rather than
RDMA. At the end of the execution, FaRM attempts to com-
mit the transaction by executing the following steps:
1. Lock. The coordinator writes a LOCK record to the log
on each machine that is a primary for any written object. This
contains the versions and new values of all written objects on
that primary, as well as the list of all regions with written
objects. Primaries process these records by attempting to
lock the objects at the specified versions using compare-
and-swap, and send back a message reporting whether all
locks were successfully taken. Locking can fail if any object
version changed since it was read by the transaction, or if
the object is currently locked by another transaction. In this
case, the coordinator aborts the transaction. It writes an abort
record to all primaries and returns an error to the application.
2. Validate. The coordinator performs read validation by
reading, from their primaries, the versions of all objects
that were read but not written by the transaction. If any
object has changed, validation fails and the transaction is
aborted. Validation uses one-sided RDMA reads by default.
For primaries that hold more than tr objects, validation is
done over RPC. The threshold tr (currently 4) reflects the
CPU cost of an RPC relative to an RDMA read.
3. Commit backups. The coordinator writes a COMMIT -
BACKUP record to the non-volatile logs at each backup and
then waits for an ack from the NIC hardware without inter-
57

Log record type Contents
LOCK transaction ID, IDs of all regions with objects written by the transaction, and addresses, versions, and values of
all objects written by the transaction that the destination is primary for
COMMIT -BACKUP contents are the same as lock record
COMMIT -PRIMARY transaction ID to commit
ABORT transaction ID to abort
TRUNCATE low bound transaction ID for non-truncated transactions and transaction IDs to truncate
Table 1. Log record types used in the transaction protocol. The low bound on transaction identifiers that have not been
truncated and a transaction identifier for truncation are piggybacked on each record.
Message type Contents
LOCK -REPLY transaction ID, result indicating whether locking succeeded
VALIDATE addresses and versions of objects read from destination (not sent when validation is done over RDMA reads)
NEED -RECOVERY configuration ID, region ID, and transaction IDs to be recovered (sent by backup to primary)
FETCH -TX-STATE configuration ID, region ID, and transaction IDs whose state is requested (sent by primary to backup)
SEND -TX-STATE configuration ID, region ID, transaction ID, and contents of lock record for transaction requested by fetch
REPLICATE -TX-STATE configuration ID, region ID, transaction ID, and contents of lock record (sent by primary to backup)
RECOVERY -VOTE configuration ID, region ID, transaction ID, region IDs for regions modified by the transaction, and vote
REQUEST -VOTE configuration ID, transaction ID, and region ID
COMMIT -RECOVERY configuration ID, and transaction ID
ABORT-RECOVERY configuration ID, and transaction ID
TRUNCATE -RECOVERY configuration ID, and transaction ID
Table 2. Message types used in the transaction protocol. All but the first two are used only during recovery.
rupting the backup’s CPU. TheCOMMIT -BACKUP log record
has the same payload as a LOCK record.
4. Commit primaries. After all COMMIT -BACKUP writes
have been acked, the coordinator writes a COMMIT -
PRIMARY record to the logs at each primary. It reports com-
pletion to the application on receiving at least one hardware
ack for such a record, or if it wrote one locally. Primaries
process these records by updating the objects in place, incre-
menting their versions, and unlocking them, which exposes
the writes committed by the transaction.
5. Truncate. Backups and primaries keep the records in
their logs until they are truncated. The coordinator truncates
logs at primaries and backups lazily after receiving acks
from all primaries. It does this by piggybacking identifiers
of truncated transactions in other log records. Backups apply
the updates to their copies of the objects at truncation time.
Correctness. Committed read-write transactions are seri-
alizable at the point where all the write locks were acquired,
and committed read-only transactions at the point of their
last read. This is because the versions of all read and written
objects at the serialization point are the same as the versions
seen during execution. Locking ensures this for objects that
were written and validation ensures this for objects that were
only read. In the absence of failures this is equivalent to ex-
ecuting and committing the entire transaction atomically at
the serialization point. Serializability in FaRM is also strict:
the serialization point is always between the start of execu-
tion and the completion being reported to the application.
To ensure serializability across failures, it is necessary
to wait for hardware acks from all backups before writing
COMMIT -PRIMARY . Assume that the coordinator does not
receive an ack from some backup b for a region r. Then
a primary could expose transaction modifications and later
fail together with the coordinator and the other replicas of r
withoutb ever receiving the COMMIT -BACKUP record. This
would result in losing the updates tor.
Since the read set is stored only at the coordinator, a
transaction is aborted if the coordinator fails and no com-
mit record survives to attest to the success of validation. So
it is necessary for the coordinator to wait for a successful
commit at one of the primaries before reporting a success-
ful commit to the application. This ensures that at least one
commit record survives any f failures for transactions re-
ported committed to the application. Otherwise, such a trans-
action could still abort if the coordinator and all the backups
failed before any COMMIT -PRIMARY record was written, be-
cause only LOCK records would survive and there would be
no record that validation had succeeded.
In traditional two-phase commit protocols, participants
can reserve resources to commit the transaction when they
process the prepare message, or refuse to prepare the trans-
action if they do not have enough resources. However, as
our protocol avoids involving the backups’ CPUs during the
commit, the coordinator must reserve log space at all par-
ticipants to guarantee progress. Coordinators reserve space
for all commit protocol records including truncate records in
primary and backup logs before starting the commit proto-
col. Log reservations are a local operation at the coordinator
58

because the coordinator writes records to the log it owns at
each participant. The reservation is released when the corre-
sponding record is written. Truncation record reservations
are also released if the truncation is piggybacked on an-
other message. If the log becomes full, the coordinator uses
the reservations to write explicit truncate records to free up
space in the log. This is rare but needed to ensure liveness.
Performance. For our target hardware, this protocol has
several advantages over traditional distributed commit proto-
cols. Consider a two-phase commit protocol with replication
such as Spanner’s [11]. Spanner uses Paxos [24] to replicate
the transaction coordinator and its participants, which are the
machines that store data read or written by the transaction.
Each Paxos state machine takes the role of an individual ma-
chine in a traditional two-phase commit protocol [19]. This
requires 2f + 1 replicas to toleratef failures and, since each
state machine operation requires at least 2f + 1 round trip
messages, it requires 4P (2f + 1) messages (where P is the
number of participants in the transaction).
FaRM uses primary-backup replication instead of Paxos
state machine replication. This reduces the number of copies
of data to f + 1, and also reduces the number of messages
transmitted during a transaction. Coordinator state is not
replicated and coordinators communicate directly with pri-
maries and backups, further reducing latency and message
counts. FaRM’s overhead due to replication is minimal: a
single RDMA write to each remote machine having a backup
of any written object. Backups of read-only participants are
not involved in the protocol at all. Additionally, read valida-
tion over RDMA ensures that primaries of read-only partici-
pants do no CPU work, and using one-way RDMA writes for
COMMIT -PRIMARY and COMMIT -BACKUP records reduces
waiting for remote CPUs and also allows the remote CPU
work to be lazy and batched.
The FaRM commit phase uses Pw(f + 3) one-sided
RDMA writes wherePw is the number of machines that are
primaries for objects written by the transaction, andPr one-
sided RDMA reads where Pr is the number of objects read
from remote primaries but not written. Read validation adds
two one-sided RDMA latencies to the critical path but this is
a good trade-off: the added latency is only a few microsec-
onds without load and the reduction in CPU overhead results
in higher throughput and lower latency under load.
5. Failure recovery
FaRM provides durability and high availability using repli-
cation. We assume that machines can fail by crashing but can
recover without losing the contents of non-volatile DRAM.
We rely on bounded clock drift for safety and on eventually
bounded message delays for liveness.
We provide durability for all committed transactions even
if the entire cluster fails or loses power: all committed state
can be recovered from regions and logs stored in non-volatile
DRAM. We ensure durability even if at most f replicas per
object lose the contents of non-volatile DRAM. FaRM can
also maintain availability with failures and network parti-
tions provided a partition exists that contains a majority of
the machines which remain connected to each other and to a
majority of replicas in the Zookeeper service, and the parti-
tion contains at least one replica of each object.
Failure recovery in FaRM has five phases described be-
low: failure detection, reconfiguration, transaction state re-
covery, bulk data recovery, and allocator state recovery.
5.1 Failure detection
FaRM uses leases [18] to detect failures. Every machine
(other than the CM) holds a lease at the CM and the CM
holds a lease at every other machine. Expiry of any lease
triggers failure recovery. Leases are granted using a 3-way
handshake. Each machine sends a lease request to the CM
and it responds with a message that acts as both a lease grant
to the machine and a lease request from the CM. Then, the
machine replies with a lease grant to the CM.
FaRM leases are extremely short, which is key to high
availability. Under heavy load, FaRM can use 5 ms leases for
a 90-machine cluster with no false positives. Significantly
larger clusters may require a two-level hierarchy, which in
the worst case would double failure detection time.
Achieving short leases under load required careful imple-
mentation. FaRM uses dedicated queue pairs for leases to
avoid having lease messages delayed in a shared queue be-
hind other message types. Using a reliable transport would
require an additional queue pair at the CM for each ma-
chine. This would result in poor performance due to capac-
ity misses in the NIC’s queue pair cache [16]. Instead the
lease manager uses Infiniband send and receive verbs with
the connectionless unreliable datagram transport, which re-
quires space for only one additional queue pair on the NIC.
By default, lease renewal is attempted every1/5 of the lease
expiry period to account for potential message loss.
Lease renewal must also be scheduled on the CPU in a
timely way. FaRM uses a dedicated lease manager thread
that runs at the highest user-space priority (31 on Windows).
The lease manager thread is not pinned to any hardware
thread and it uses interrupts instead of polling to avoid starv-
ing critical OS tasks that must run periodically on every
hardware thread. This increases message latency by a few
microseconds, which is not problematic for leases.
In addition, we do not assign FaRM threads to two hard-
ware threads on each machine, leaving them for the lease
manager. Our measurements show that the lease manager
usually runs on these hardware threads without impacting
other FaRM threads, but sometimes it is preempted by higher
priority tasks that cause it to run on other hardware threads.
So pinning the lease manager to a hardware thread would
likely result in false positives when using short leases.
Finally, we preallocate all memory used by the lease
manager during initialization and we page in and pin all the
code it uses to avoid delays due to memory management.
59

CM=S1
S2
S3
suspect S3
S4
1. SUSPECT
Zookeeper
stop RDMA reads to S3
Update <9, …> to
<10, {S1,S2,S4}, F, CM=S1>
2. PROBE 3. UPDATE
CONFIGURATION
5. SEND NEW
CONFIGURATION
4. REMAP
REGIONS
6. APPLY NEW
CONFIGURATION
7. COMMIT NEW
CONFIGURATION
REMAP
Figure 5. Reconfiguration
5.2 Reconfiguration
The reconfiguration protocol moves a FaRM instance from
one configuration to the next. Using one-sided RDMA op-
erations is important to achieve good performance but it im-
poses new requirements on the reconfiguration protocol. For
example, a common technique to achieve consistency is to
use leases [18]: servers check if they hold a lease for an
object before replying to requests to access the object. If a
server is evicted from the configuration, the system guaran-
tees that the objects it stores cannot be mutated until after its
lease expires (e.g., [7]). FaRM uses this technique when ser-
vicing requests from external clients that communicate with
the system using messages. But since machines in the FaRM
configuration read objects using RDMA reads without in-
volving the remote CPU, the server’s CPU cannot check if
it holds the lease. Current NIC hardware does not support
leases and it is unclear if it will in the future.
We solve this problem by implementingprecise member-
ship [10]. After a failure, all machines in a new configuration
must agree on its membership before allowing object muta-
tions. This allows FaRM to perform the check at the client
rather than at the server. Machines in the configuration do
not issue RDMA requests to machines that are not in it, and
replies to RDMA reads and acks for RDMA writes from ma-
chines no longer in the configuration are ignored.
Figure 5 shows an example reconfiguration timeline that
consists of the following steps:
1. Suspect. When a lease for a machine expires at the
CM, it suspects that machine of failure and initiates recon-
figuration. At this point it starts blocking all external client
requests. If a non-CM machine suspects the CM of failure
due to a lease expiry, it first asks one of a small number of
“backup CMs” to initiate reconfiguration (the k successors
of the CM using consistent hashing). If the configuration is
unchanged after a timeout period then it attempts the recon-
figuration itself. This design avoids a large number of simul-
taneous reconfiguration attempts if the CM fails. In all cases,
the machine initiating the reconfiguration will try to become
the new CM as part of the reconfiguration.
2. Probe. The new CM issues an RDMA read to all the
machines in the configuration except the machine that is
suspected. Any machine for which the read fails is also
suspected. These read probes allow handling of correlated
failures that affect several machines, e.g., power and switch
failures, by a single reconfiguration. The new CM proceeds
with the reconfiguration only if it obtains responses for a
majority of the probes. This ensures that if the network is
partitioned, the CM will not be in the smaller partition.
3. Update configuration. After receiving replies to the
probes, the new CM attempts to update the configuration
data stored in Zookeeper to⟨c + 1,S,F, CMid⟩, where c is
the current configuration identifier,S is the set of machines
that replied to the probes, F is the mapping of machines
to failure domains, and CM id is its own identifier. We use
Zookeeper znode sequence numbers to implement an atomic
compare-and-swap that succeeds only if the current configu-
ration is stillc. This ensures that only one machine can suc-
cessfully move the system to the configuration with identifier
c+1 (and become CM) even if multiple machines simultane-
ously attempt a configuration change from the configuration
with identifierc.
4. Remap regions. The new CM then reassigns regions
previously mapped to failed machines to restore the num-
ber of replicas to f + 1. It tries to balance load and satisfy
application-specified locality hints subject to capacity and
failure independence constraints. For failed primaries, it al-
ways promotes a surviving backup to be the new primary to
reduce the time to recover. If it detects regions that lost all
their replicas or there is no space to re-replicate regions, it
signals an error.
5. Send new configuration. After remapping regions, the
CM sends a NEW-CONFIG message to all the machines in
the configuration with the configuration identifier, its own
identifier, the identifiers of the other machines in the config-
uration, and all the new mappings of regions to machines.
NEW-CONFIG also resets the lease protocol if the CM has
changed: it acts as a lease request from the new CM to each
machine. If the CM is unchanged, lease exchange continues
during reconfiguration to detect additional failures quickly.
6. Apply new configuration. When a machine receives a
NEW-CONFIG with a configuration identifier that is greater
than its own, it updates its current configuration identifier
and its cached copy of the region mappings, and allocates
space to hold any new region replicas assigned to it. From
this point, it does not issue new requests to machines that
are not in the configuration and it rejects read responses
and write acks from those machines. It also starts blocking
requests from external clients. Machines reply to the CM
with a NEW-CONFIG -ACK message. If the CM has changed,
this both grants a lease to the CM and requests a lease.
7. Commit new configuration. Once the CM receives
NEW-CONFIG -ACK messages from all machines in the con-
figuration, it waits to ensure that any leases granted in pre-
vious configurations to machines no longer in the config-
uration have expired. The CM then sends a NEW-CONFIG -
COMMIT to all the configuration members that also acts as
60

NEW-
CONFIG
NEW-
CONFIG-
COMMIT
P
B2
B1
C
2. DRAIN
2. DRAIN
2. DRAIN
3. FIND
RECOVERING
TXs
4.
ACQUIRE
LOCKS
5.
REPLICATE
LOGS
6.
VOTE
7. DECIDE
region is active
1. BLOCK
1. BLOCK
1. BLOCK
fetch missing
transactions
Figure 6. Transaction state recovery showing a coordinator
C, primaryP , and two backupsB1 andB2
a lease grant. All members now unblock previously blocked
external client requests and initiate transaction recovery.
5.3 Transaction state recovery
FaRM recovers transaction state after a configuration change
using the logs distributed across the replicas of objects mod-
ified by a transaction. This involves recovering the state both
at the replicas of objects modified by the transaction and at
the coordinator to decide on the outcome of the transaction.
Figure 6 shows an example transaction recovery timeline.
FaRM achieves fast recovery by distributing work across
threads and machines in the cluster. Draining (step2) is done
for all message logs in parallel. Step1 and steps 3–5 are done
for all regions in parallel. Steps 6–7 are done for all recover-
ing transactions in parallel.
1. Block access to recovering regions. When the primary
of a region fails, one of the backups is promoted to be
the new primary during reconfiguration. We cannot allow
access to the region until all transactions that updated it have
been reflected at the new primary. We do this by blocking
requests for local pointers and RDMA references to the
region until step 4 when all write locks have been acquired
for all recovering transactions that updated the region.
2. Drain logs.One-sided RDMA writes also impact trans-
action recovery. A general approach to consistency across
configurations is to reject messages from old configurations.
FaRM cannot use this approach because NICs acknowledge
COMMIT -BACKUP and COMMIT -PRIMARY records written
to transaction logs regardless of the configuration in which
they were issued. Since coordinators only wait for these acks
before exposing the updates and reporting success to the ap-
plication, machines cannot always reject records from pre-
vious configurations when they process them. We solve this
problem by draining logs to ensure that all relevant records
are processed during recovery: all machines process all the
records in their logs when they receive a NEW-CONFIG -
COMMIT message. They record the configuration identifier
in a variable LastDrained when they are done.
FaRM transactions have unique identifiers⟨c,m,t,l ⟩ as-
signed at the start of commit that encode the configuration
c in which the commit started, the machine identifier m of
the coordinator, the thread identifier t of the coordinator,
and a thread-local unique identifierl. Log records for trans-
actions with configuration identifiers less than or equal to
LastDrained are rejected.
3. Find recovering transactions.A recovering transaction
is one whose commit phase spans configuration changes, and
for which some replica of a written object, some primary of
a read object, or the coordinator has changed due to recon-
figuration. During log draining, the transaction identifier and
list of updated region identifiers in each log record in each
log is examined to determine the set of recovering transac-
tions. Only recovering transactions go through transaction
recovery at primaries and backups, and coordinators reject
hardware acks only for recovering transactions.
All machines must agree on whether a given transaction
is a recovering transaction or not. We achieve this by piggy-
backing some extra metadata on the communication during
the reconfiguration phase. The CM reads the LastDrained
variable at each machine as part of the probe read. For each
regionr whose mapping has changed sinceLastDrained, the
CM sends two configuration identifiers in theNEW-CONFIG
message to that machine. These are LastPrimaryChange[r],
the last configuration identifier when the primary of r
changed, and LastReplicaChange[r], the last configuration
identifier when any replica ofr changed. A transaction that
started committing in configuration c− 1 is recovering in
configurationc unless: for all regions r containing objects
modified by the transaction LastReplicaChange[r] < c, for
all regions r′ containing objects read by the transaction
LastPrimaryChange[r′] < c, and the coordinator has not
been removed from configurationc.
Records for a recovering transaction may be distributed
over the logs of different primaries and backups updated
by the transaction. Each backup of a region sends a NEED -
RECOVERY message to the primary with the configuration
identifier, the region identifier, and the identifiers of recover-
ing transactions that updated the region.
4. Lock recovery. The primary of each region waits un-
til the local machine logs have been drained and NEED -
RECOVERY messages have been received from each backup,
to build the complete set of recovering transactions that af-
fect the region. It then shards the transactions by identifier
across its threads such that each threadt recovers the state of
transactions with coordinator thread identifiert. In parallel,
the threads in the primary fetch any transaction log records
from backups that are not already stored locally and then
lock any objects modified by recovering transactions.
When lock recovery is complete for a region, the region
is active and local and remote coordinators can obtain local
pointers and RDMA references, which allows them to read
objects and commit updates to this region in parallel with
subsequent recovery steps.
5. Replicate log records.The threads in the primary repli-
cate log records by sending backups the REPLICATE -TX-
STATE message for any transactions that they are missing.
61

The message contains the region identifier, the current con-
figuration identifier, and the same data as theLOCK record.
6. Vote. The coordinator for a recovering transaction de-
cides whether to commit or abort the transaction based on
votes from each region updated by the transaction. These
votes are sent by the primaries of each region. FaRM uses
consistent hashing to determine the coordinator for a trans-
action, ensuring that all the primaries independently agree
on the identity of the coordinator for a recovering transac-
tion. The coordinator does not change if the machine it is
running on is still in the configuration, but when a coordi-
nator fails the responsibility for coordinating its recovering
transactions is spread across the machines in the cluster.
The threads in the primary send RECOVERY -VOTE mes-
sages to their peer threads in the coordinator for each re-
covering transaction that modified the region. The vote is
commit-primary if any replica saw COMMIT -PRIMARY or
COMMIT -RECOVERY . Otherwise, it votes commit-backup if
any replica saw COMMIT -BACKUP and did not see ABORT-
RECOVERY . Otherwise, it votes lock if any replica saw a
LOCK record and no ABORT-RECOVERY . Otherwise, it votes
abort. V ote messages include the configuration identifier, the
region identifier, the transaction identifier, and the list of re-
gion identifiers modified by the transaction.
Some primaries may not initiate voting for a transaction
because either they never received a log record for the trans-
action or they already truncated the log records for the trans-
action. The coordinator sends explicit vote requests to pri-
maries that have not already voted within a timeout period
(set to 250 µs). The REQUEST -VOTE message includes the
configuration identifier, the region identifier, and the trans-
action identifier. Primaries that do have log records for the
transaction vote as before after first waiting for log replica-
tion for that transaction to complete.
Primaries that do not have any log records for the transac-
tion vote truncated if the transaction has already been trun-
cated and unknown if it has not. To determine if a transac-
tion has already been truncated, each thread maintains the set
of identifiers of transactions whose records have been trun-
cated from its logs. This set is kept compact by using a lower
bound on non-truncated transaction identifiers. The lower
bound is updated based on the lower bounds at each coordi-
nator, which are piggybacked on coordinator messages and
during reconfiguration.
7. Decide. The coordinator decides to commit a trans-
action if it receives a commit-primary vote from any re-
gion. Otherwise, it waits for all regions to vote and com-
mits if at least one region votedcommit-backup and all other
regions modified by the transaction voted lock, commit-
backup, or truncated. Otherwise it decides to abort. It then
sends COMMIT -RECOVERY or ABORT-RECOVERY to all par-
ticipant replicas. Both messages include the configuration
identifier and the transaction identifier.COMMIT -RECOVERY
is processed similarly to COMMIT -PRIMARY if received at a
primary and to COMMIT -BACKUP if received at a backup.
ABORT-RECOVERY is processed similarly to ABORT . After
the coordinator receives back acks from all primaries and
backups, it sends a TRUNCATE -RECOVERY message.
Correctness. Next we provide some intuition on how the
different steps of transaction recovery ensure strict serializ-
ability. The key idea is that recovery preserves the outcome
for transactions that were previously committed or aborted.
We say that a transaction iscommitted when either a primary
exposes transaction modifications, or the coordinator notifies
the application that the transaction committed. A transaction
is aborted when the coordinator sends an abort message or
notifies the application that the transaction has aborted. For
transactions whose outcome has not yet been decided, recov-
ery may commit or abort the transaction but it ensures that
any recovery from additional failures preserves the outcome.
The outcome of transactions that are not recovering
(step 3) is decided using the normal case protocol (Sec-
tion 4). So we will not discuss them further.
A log record for a recovering transaction that commit-
ted is guaranteed to be processed and accepted before or
during log draining (step 2). This is true because primaries
expose modifications only after processing the COMMIT -
PRIMARY record. If the coordinator notified the applica-
tion, it must have received hardware acks for all COMMIT -
BACKUP records and for at least one COMMIT -PRIMARY
record before receiving NEW-CONFIG (because it ignores
the acks after changing configuration). Therefore, since the
new configuration includes at least one replica for each re-
gion, at least one replica for at least one region will process
COMMIT -PRIMARY or COMMIT -BACKUP records, and at
least one replica for each other region will processCOMMIT -
PRIMARY , COMMIT -BACKUP , or LOCK records.
Steps 3 and 4 ensure that the primaries for the regions
modified by the transaction see these records (unless they
have been truncated). They replicate these records to the
backups (step 5) to guarantee that voting will produce the
same results even if there are subsequent failures. Then the
primaries send votes to the coordinator based on the records
they have seen (step 6).
The decision step guarantees that the coordinator decides
to commit any transaction that has previously committed. If
any replica truncated the transaction records, all primaries
will vote commit-primary, commit-backup, or truncated. At
least one primary will send a vote other than truncated be-
cause otherwise the transaction would not be recovering. If
no replicas truncated the transaction records, at least one pri-
mary will vote commit-primary or commit-backup and the
others will vote commit-primary, commit-backup or lock.
Similarly, the coordinator will decide to abort if the trans-
action was previously aborted because in this case there will
either be no commit-primary or commit-backup records or
all replicas will have received ABORT-RECOVERY .
62

Blocking access to recovering regions (step 1) and lock
recovery (step 4) guarantee that until a recovering transac-
tion has committed or aborted, no other operation can access
objects it modified.
Performance. FaRM uses several optimizations to achieve
fast failure recovery. Identifying recovering transactions lim-
its recovery work to only those transactions and regions that
were affected by the reconfiguration, which could be a small
subset of the total when a single machine in a large cluster
fails. Our results indicate that this can reduce the number of
transactions to recover by an order of magnitude. The recov-
ery work itself is parallelized across regions, machines, and
threads. Making regions available immediately after lock re-
covery improves foreground performance as new transac-
tions that access these regions do not block for long. Specif-
ically, they need not wait while new replicas of these regions
are brought up to date which requires bulk movement of data
over the network.
5.4 Recovering data
FaRM must recover (re-replicate) data at new backups for
a region to ensure that it can tolerate f replica failures in
the future. Data recovery is not necessary to resume nor-
mal case operation, so we delay it until all regions become
active to minimize impact on latency-critical lock recovery.
Each machine sends aREGIONS -ACTIVE message to the CM
when all regions for which it is primary become active. After
receiving all REGIONS -ACTIVE messages, the CM sends a
message ALL -REGIONS -ACTIVE to all machines in the con-
figuration. At this point, FaRM begins data recovery for new
backups in parallel with foreground operations.
A new backup for a region initially has a freshly allo-
cated and zeroed local region replica. It divides the region
across worker threads that recover it in parallel. Each thread
issues one-sided RDMA operations to read a block at a time
from the primary. We currently use 8 KB blocks, which is
large enough to use the network efficiently but small enough
not to impact normal case operation. To reduce impact on
foreground performance, recovery is paced by scheduling
the next read to start at a random point within an interval
after the start of the previous read (set to 4ms).
Each recovered object must be examined before being
copied to the backup. If the object has a version greater than
the local version, the backup locks the local version with
a compare-and-swap, updates the object state, and unlocks
it. Otherwise, the object has been or is being updated by a
transaction that created a version greater than or equal to the
one recovered, and the recovered state is not applied.
5.5 Recovering allocator state
The FaRM allocator splits regions into blocks (1 MB) that
are used as slabs for allocating small objects. It keeps two
pieces of meta-data: block headers, which contain the ob-
ject size, and slab free lists. Block headers are replicated
to backups when a new block is allocated. This ensures
they are available on the new primary after a failure. Since
block headers are used in data recovery, the new primary
sends them to all backups immediately after receiving NEW-
CONFIG -COMMIT . This avoids any inconsistencies when the
old primary fails while replicating the block header.
The slab free lists are kept only at the primary to reduce
the overheads of object allocation. Each object has a bit in
its header that is set by an allocation and cleared by a free
during transaction execution. This change to the object state
is replicated during transaction commit as described in Sec-
tion 4. After a failure, the free lists are recovered on the
new primary by scanning the objects in the region, which is
parallelized across all threads on the machine. To minimize
the impact on transaction lock recovery, allocation recovery
starts after ALL -REGIONS -ACTIVE is received and to mini-
mize the impact on the foreground work it is paced by scan-
ning 100 objects at a time every100 µs. Object deallocations
are queued until a slab’s free list is recovered.
6. Evaluation
6.1 Setup
Our experimental testbed consists of 90 machines used for
a FaRM cluster and 5 machines for a replicated Zookeeper
instance. Each machine has 256 GB of DRAM and two 8-
core Intel E5-2650 CPUs running Windows Server 2012 R2.
We enabled hyper-threading and used the first 30 threads
for the foreground work and the remaining 2 threads for the
lease manager. Machines have two Mellanox ConnectX-3 56
Gbps Infiniband NICs, each used by threads on a different
socket, and are connected by a single Mellanox SX6512
switch with full bisection bandwidth. FaRM was configured
to use 3-way replication (one primary and two backups) with
a lease time of 10 ms.
6.2 Benchmarks
We use two transactional benchmarks to measure FaRM’s
performance. We implemented both benchmarks in C++
against the FaRM API. Since FaRM uses a symmetric model
to exploit locality, each machine both runs the benchmark
code and stores data. Each machine runs the benchmark code
linked with FaRM’s code on the same process. In the future,
we will compile the application from a safe language like
SQL to prevent application bugs from corrupting data.
Telecommunication Application Transaction Processing
(TATP) [32] is a benchmark for high-performance main-
memory databases. Each database table is implemented as a
FaRM hash table [16]. TATP is read dominated. 70% of the
operations are single-row lookups which use FaRM’s lock
free reads [16]. They can usually be performed with a single
RDMA read and do not require a commit phase. 10% of
the operations read 2–4 rows and require validation during
the commit phase. The remaining 20% of the operations are
updates and require the full commit protocol. Since 70% of
63

0 30 60 90 120 150
Operations / µs
0
200
400
600
800
1000Latency (us)
Median
99th
Figure 7. TATP performance
the updates only modify a single object field, we function
ship these to the primary of the object as an optimization. We
used a database with 9.2 billion subscribers (except where
noted). TATP is partitionable but we have not partitioned it,
so most operations access data on remote machines.
TPC-C [38] is a well-known database benchmark with
complex transactions that access hundreds of rows. Our im-
plementation uses a schema with 16 indexes. Twelve of
these only require unordered (point) queries and updates and
are implemented as FaRM hash tables. Four of the indexes
also require range queries. These are implemented using the
FaRM B-tree. The B-Tree caches internal nodes at each ma-
chine and hence lookups require a single FaRM RDMA read
in the common case. We reserve 8 GB per machine for the
cache. We use fence keys [17, 27] to ensure traversal con-
sistency, similar to Minuet [37]. We omit a more detailed
description of the B-tree for space reasons.
We use a database with 21,600 warehouses. We co-
partition most of the hash table indexes as well as the clients
by warehouse, which means that around 10% of all trans-
actions access remote data. As specified by the benchmark,
“new order” transactions are 45% of the transaction mix. We
run the full mix but we report performance as the number of
successfully committed “new orders”.
6.3 Normal-case performance
We present the normal case (failure-free) performance of
FaRM as throughput-latency curves. For each benchmark,
we varied the load by first increasing the number of active
threads per machine from 2 to 30 and then increasing the
concurrency per thread, until the throughput saturated. Note
that the left end of each graph still shows significant concur-
rency and hence throughput. It does not show the minimum
latency that can be achieved by FaRM.
TATP . Figure 7 shows that FaRM performs 140 million
TATP transactions per second with58 µs median latency and
0 1 2 3 4 5
Operations / µs
0
1000
2000
3000
4000
5000
6000Latency (us)
Median
99th
Figure 8. TPC-C performance
645 µs 99th percentile latency. On the left hand side of the
graph, the median latency is only 9 µs, the 99 th percentile
latency drops to 112 µs, and FaRM performs 2 million oper-
ations per second. The multi-object distributed transactions
used by TATP commit in tens of microseconds, with a mean
commit latency of 19 µs at the lowest throughput and 138 µs
at the highest.
FaRM outperforms published TATP results for Heka-
ton [14, 26], a single-machine in-memory transactional en-
gine, by a factor of 33. The Hekaton results were obtained
using different hardware but we expect a factor of 20 im-
provement when running Hekaton on one of our testbed ma-
chines. In a smaller-scale experiment, FaRM outperformed
Hekaton with just three machines. In addition, FaRM sup-
ports much larger data sets because it scales out and it pro-
vides high availability unlike single machine systems.
TPC-C. We ran TPC-C for 60 s and we report latency and
average throughput over that period in Figure 8. FaRM per-
forms up to 4.5 million TPC-C “new order” transactions per
second with median latency of 808 µs and 99 th percentile
latency of 1.9 ms. The latency can be halved with a small
10% impact in throughput. The best published TPC-C per-
formance we know of is from Silo [39,40] which is a single-
machine in-memory system with logging to FusionIO SSDs.
FaRM’s throughput is 17x higher than Silo without logging,
and its latency at this throughput level is 128x better than
Silo with logging.4
Read performance. Although the focus of this paper is on
transactional performance and failure recovery, we were also
able to improve read-only performance relative to [16]. We
ran a key-value lookup-only workload with 16-byte keys and
32-byte values and a uniform access pattern. We achieved
a throughput of 790 million lookups/s with median latency
4 Silo reports total transaction counts which we multiplied by 45% to get
the “new order” count.
64

of 23 µs and 99th percentile latency of 73 µs. This improves
on previously reported per-machine throughput for the same
benchmark by 20% [16]. We do not double performance
despite doubling the number of NICs because the benchmark
becomes CPU bound.
6.4 Failures
To evaluate performance with failures, we ran the same
benchmarks and we killed the FaRM process on one of the
machines 35 s into the experiment. We show timelines with
the throughput of the 89 surviving machines aggregated at
1 ms intervals. The timelines are synchronized at experiment
start using RDMA messaging.
Figures 9 and 10 show a typical run of each benchmark on
different time scales. Both show throughput as a solid line.
The “time to full throughput” is a zoomed-in view around
the failure. It shows the time at which the failed machine’s
lease expired on the CM (“suspect”); the time at which all
read probes completed (“probe”); the time at which the CM
successfully updated Zookeeper (“zookeeper”); the time at
which the new configuration was committed at all surviving
machines (“config-commit”); the time at which all regions
are active (“all-active”); and the time at which background
data recovery begins (“data-rec-start”). The “time to full
data recovery” shows a zoomed-out view that includes the
time when all data is recovered at backups (”done”). A
dashed line shows the cumulative number of backup regions
recovered over time by data recovery.
TATP . The timelines for a typical TATP run are shown in
Figure 9. We configured it for maximum throughput: each
machine runs 30 threads with 8 concurrent transactions per
thread. Figure 9(a) shows that throughput drops sharply at
the failure but recovers rapidly. The system is back to peak
throughput in less than 40 ms. All regions become active in
39 ms. Figure 9(b) shows that data recovery, which is paced,
does not impact foreground throughput. The failed machine
hosted 84 2 GB regions. Each thread fetches 8 KB blocks
every 2 ms, which means that it takes around 17 s to recover
a 2 GB region on a single machine. Machines recover one
region at a time in parallel with each other and at roughly the
same pace, hence the number of regions recovered moves in
large steps. The recovery load (i.e., the number of regions
per-machine that had a replica on the failed machine) is
well balanced across the cluster: 64 machines recover one
region and 10 machines recover two. This explains why re-
replication of most regions completes in around 17 s and
why all regions are fully re-replicated in less than35 s. Some
regions are not fully allocated, so their recovery takes less
time. This is why re-replication of some regions completes
in less than 17 s.
The figure also shows that TATP has some dips in
throughput even when there are no failures. We believe that
this is because of skewed access in the benchmark; the
Figure 11. TATP performance timeline with CM failure
throughput drops when many transactions conflict and back
off on hot keys at the same time.
TPC-C. Figure 10 shows the timelines for TPC-C. Fig-
ure 10(a) shows that the system regains most of the through-
put in less than 50 ms and that all regions become active
shortly after that. It takes the system slightly more time to re-
cover transaction locks than with TATP because TPC-C has
more complex transactions. The main difference is that re-
covery of data takes longer (Figure10(b)) even though TPC-
C recovers only 63 regions in the experiment. This is because
TPC-C co-partitions its hash tables to exploit locality and
improve performance, which results in reduced recovery par-
allelism because multiple regions are replicated on the same
set of machines to satisfy the locality constraints specified
by the application. In the experiment, two machines recover
17 regions each, which leads to data recovery taking over
4 minutes. Note that TPC-C throughput degrades gradually
over time in Figure 10(b) because the size of the database
increases very quickly.
F ailing the CM. Figure 11 shows TATP throughput over
time when the CM process fails. Recovery is slower than
when a non-CM process fails. It takes about 110 ms for
throughput to get back to the same level as before the failure.
The main reason for the increase in recovery time is an in-
crease in the reconfiguration time: from 20 ms in Figure9(a)
to 97 ms. Most of this time is spent by the new CM building
data structures that are only maintained at the CM. It should
be possible to eliminate this delay by having all the machines
maintain these data structures incrementally as they learn re-
gion mappings from the CM.
Distribution of recovery times. We repeated the TATP re-
covery experiment (without CM failures) 40 times to obtain
a distribution of recovery times. The experiments were run
with a smaller data set (3.5 billion subscribers) to shorten
experiment times, but we confirmed that the time to regain
65

(a) Time to full throughput
 (b) Time to full data recovery
Figure 9. TATP performance timeline with failure
(a) Time to full throughput
 (b) Time to full data recovery
Figure 10. TPC-C performance timeline with failure
throughput after a failure was the same as for the larger data
sets. This is because this time is dominated by recovering
transaction state, and the number of concurrently executing
transactions is the same for both data set sizes. Figure 12
shows the distribution of recovery times. We measured re-
covery time from the point where the failed machine is sus-
pected by the CM until throughput recovers to 80% of the
average throughput before the failure. The median recovery
time is around 50 ms and in more than 70% of the execu-
tions the recovery time is less than 100 ms. In the remaining
cases, the recovery took more than 100 ms, but always less
than 200 ms.
Correlated failures. Some failures affect more than one
machine at the same time, e.g., power or switch failures. To
deal with such coordinated failures, FaRM allows specifying
a failure domain for each machine and the CM places each
replica of a region in a different failure domain. We group
machines in our cluster into five failure domains with 18
machines each. This corresponds to the number of ports in
each leaf module in our switch. We fail all the processes in
one of these failure domains at the same time to simulate the
failure of a top-of-rack switch.
Figure 13 shows TATP throughput over time for the
72 machines that do not fail. TATP was configured to use
around 55 regions on each machine (6.9 billion subscribers
66

30 60 90 120 150 180
Recovery (ms)
10
20
30
40
50
60
70
80
90
100Percentile
Figure 12. Distribution of recovery times for TATP
Figure 13. TATP throughput when failing 18 out of 90
machines at the same time
across the cluster) to allow enough space to re-replicate
failed regions after the failure. FaRM regains peak through-
put less than 400 ms after the failure. We repeated the exper-
iment 20 times and this time was the median of all experi-
ments. Most of this time is spent recovering transactions. We
need to recover all in-flight transactions that modified any
region with a replica in a failed machine, that read a region
with the primary in a failed machine, or that had the coordi-
nator on one of the failed machines. This results in roughly
130,000 transactions that need to be recovered, compared
to 7500 with a single failure. Re-replication of data takes 4
minutes because there are 1025 regions to re-replicate. As in
previous experiments, this does not impact throughput dur-
ing recovery because of pacing. Note that during this time
each region still has two available replicas, so there is no
need to re-replicate more aggressively.
Figure 14. TATP throughput when optimizing for re-
replication delay
Figure 15. TPC-C throughput with more aggressive data
recovery
Data recovery pacing. FaRM paces data recovery to re-
duce its impact on throughput. This increases the time to
complete re-replication of regions at new backups. Figure 14
shows throughput over time for TATP with very aggressive
data recovery: each thread fetches four 32 KB blocks con-
currently. The system only recovers peak throughput after
the majority of regions are re-replicated800 ms after the fail-
ure. However, data recovery completes much faster: recov-
ering 83 region replicas (166 GB) takes just 1.1 s. We use
this aggressive recovery setting only when regions lose all
but one replica. The aggressive recovery rate compares fa-
vorably with RAMCloud [33] which recovers 35 GB on 80
machines in 1.6 s.
TPC-C is less sensitive to interference from background
recovery traffic than TATP because only a small fraction
of accesses are to objects on remote machines. This means
that, in settings in which application-specific tuning is pos-
sible, we could re-replicate data more aggressively without
67

1 2 3 5 10 100 1000
Lease duration (ms)
0
20000
40000
60000
80000
100000Expiry count
RPC
UD
UD+thread
UD+thread+pri
Figure 16. False positives with different lease managers
impacting performance. Figure 15 shows TPC-C throughput
over time during recovery when threads fetch 32 KB blocks
every 2 ms. Re-replication completes in 65 s, which is four
times faster than with the default settings, without any im-
pact on throughput.
6.5 Lease times
To evaluate our lease manager optimizations (Section 5.1),
we ran an experiment where all threads in all machines
repeatedly issue RDMA reads to the CM for 10 min. We
disabled recovery and counted the number of (false posi-
tive) lease expiry events across the cluster for different lease
manager implementations and different lease durations. This
benchmark is a good stress test because it generates more
traffic at the CM than any of the benchmarks we described.
Figure 16 compares four lease manager implementations.
The first uses FaRM’s RPC (RPC). The others use unreliable
datagrams: on a shared thread (UD), on a dedicated thread
at normal priority (UD+thread), and with high-priority, in-
terrupts and no pinning (UD+thread+pri).
The results show that all the optimizations are necessary
to enable using lease times of 10 ms or less without false
positives. With shared queue pairs, even 100 ms leases ex-
pire very often. The number of false positives is reduced by
using unreliable datagrams but it is not eliminated due to
contention for the CPU. Using a dedicated thread allows us
to use 100 ms leases with no false positives, but10 ms leases
still expire due to CPU contention from background pro-
cesses running on the FaRM machines. With the interrupt-
driven lease manager running at high priority, we can use
5 ms leases for 10 min with no false positives. With shorter
leases, we still sometimes have false positives. We are lim-
ited by the network round trip time, which was up to 1 ms
with load, and by the resolution of the system timer, which
is 0.5 ms. The limited resolution of the system timer explains
why the interrupt-driven lease manager has more false posi-
tives than the polling-based one with 1 ms leases.
We conservatively set the leases to 10 ms in all our ex-
periments and have not observed any false positives during
their execution.
7. Related work
To our knowledge, FaRM is the first system to simultane-
ously provide high availability, high throughput, low latency,
and strict serializability. In prior work [16], we provided an
overview of an early version of FaRM that logged to SSDs
for durability and availability but we did not describe recov-
ery from failures. This paper describes a new fast recovery
protocol and an optimized transaction and replication pro-
tocol that sends significantly fewer messages and leverages
NVRAM to avoid logging to SSDs. The optimized protocol
sends up to 44% fewer messages than the transaction pro-
tocol described in [16] and also replaces messages by one-
sided RDMA reads during the validation phase. The work
in [16] only evaluated the performance of single-key transac-
tions in the absence of failures using the YCSB benchmark.
Here we evaluate the performance of transactions with and
without failures using the TATP and TPC-C benchmarks.
RAMCloud [33, 34] is a key-value store that stores a
single copy of data in memory and uses a distributed log
for durability. It does not support multi-object transactions.
On a failure, it recovers in parallel on multiple machines, and
during this period, which can take seconds, the data on failed
machines is unavailable. FaRM supports transactions, makes
data available within tens of milliseconds of a failure, and
has an order of magnitude higher throughput per machine.
Spanner [11] was discussed in Section 4. It provides
strict serializability but is not optimized for performance
over RDMA. It uses 2f + 1 replicas compared to FaRM’s
f + 1, and sends more messages to commit than FaRM.
Sinfonia [8] offers a shared address space with serializable
transactions implemented using 2-phase commit and piggy-
backing reads into the 2-phase commit in specialized cases.
FaRM offers general distributed transactions optimized to
take advantage of RDMA.
HERD [23] is an in-memory RDMA-based key-value
store that delivers high performance per server in an asym-
metric setting where clients run on different machines from
servers. It uses RDMA writes and send/receive verbs for
messaging but does not use RDMA reads. The authors
of [23] show that one-sided RDMA reads perform worse
than a specialized RPC implementation without reliability
in an asymmetric setting. Our results use reliable communi-
cation in a symmetric setting where every machine is both a
client and a server. This allows us to exploit locality, which
is important because accessing local DRAM is significantly
faster than using RDMA to access remote DRAM [16]. Pi-
laf [31] is a key-value store that uses RDMA reads. Nei-
ther Pilaf nor HERD support transactions. HERD is not fault
tolerant whereas Pilaf gets durability but not availability by
logging to a local disk.
68

Silo [39, 40] is a single-machine main-memory database
that achieves durability by logging to persistent storage.
It writes committed transactions to storage in batches to
achieve high throughput. Failure recovery involves reading
checkpoints and log records from storage. The storage in
Silo is local and thus availability is lost when the machine
fails. In contrast, FaRM is distributed and uses replication
in NVRAM for durability and high availability. FaRM can
regain peak throughput after a failure more than two orders
of magnitude faster than Silo for a much larger database.
By scaling out and using replication in NVRAM, FaRM
also achieves higher throughput and lower latency than Silo.
Hekaton [14, 26] is also a single-machine main-memory
database without support for scale-out or distributed trans-
actions. FaRM with 3 machines matches Hekaton’s perfor-
mance and with 90 machines has 33x the throughput.
8. Conclusion
Transactions make it easier to program distributed systems
but many systems avoid them or weaken their consistency to
improve availability and performance. FaRM is a distributed
main memory computing platform for modern data cen-
ters that provides strictly serializable transactions with high
throughput, low latency, and high availability. Key to achiev-
ing this are new transaction, replication, and recovery pro-
tocols designed from first principles to leverage commod-
ity networks with RDMA and a new, inexpensive approach
to providing non-volatile DRAM. The experimental results
show that FaRM provides significantly higher throughput
and lower latency than state of the art in-memory databases.
FaRM can also recover from a machine failure back to pro-
viding peak throughput in less than 50 ms, making failures
transparent to applications.
Acknowledgments
We would like to thank Jason Nieh, our shepherd, and the
anonymous reviewers for their comments. We would also
like to thank Richard Black for his help in performance
debugging, Andy Slowey and Oleg Losinets for keeping
the test cluster running, and Chiranjeeb Buragohain, Sam
Chandrashekar, Arlie Davis, Orion Hodson, Flavio Jun-
queira, Richie Khanna, James Lingard, Samantha L ¨uber,
Knut Magne Risvik, Tim Tan, Ming Wu, Ming-Chuan Wu,
Fan Yang, and Lidong Zhou for innumerous discussions and
for letting us use the whole cluster for extended periods of
time to run the final experiments.
References
[1] Memcached. http://memcached.org.
[2] Viking Technology. http://www.
vikingtechnology.com/.
[3] Apache Cassandra. http://cassandra.apache.
org/, 2015.
[4] MySQL. http://www.mysql.com/, 2015.
[5] neo4j. http://neo4j.com/, 2015.
[6] redis. http://redis.io/, 2015.
[7] A DYA, A., D UNAGAN , J., AND WOLMAN , A. Centrifuge:
Integrated lease management and partitioning for cloud ser-
vices. In Proceedings of the 7th USENIX Symposium
on Networked Systems Design and Implementation (2010),
NSDI’10.
[8] A GUILERA , M. K., M ERCHANT , A., S HAH , M., V EITCH ,
A., AND KARAMANOLIS , C. Sinfonia: A new paradigm for
building scalable distributed systems. In Proceedings of 21st
ACM SIGOPS Symposium on Operating Systems Principles
(2007), SOSP’07.
[9] C HANG , F., D EAN , J., G HEMAWAT, S., H SIEH , W. C.,
WALLACH , D. A., B URROWS , M., C HANDRA , T., F IKES ,
A., AND GRUBER , R. E. Bigtable: A distributed storage sys-
tem for structured data. In Proceedings of the 6th USENIX
Symposium on Operating Systems Design and Implementation
(2006), OSDI’06.
[10] C HOCKLER , G. V., KEIDAR , I., AND VITENBERG , R. Group
communication specifications: a comprehensive study. ACM
Computing Surveys (CSUR) 33, 4 (2001).
[11] C ORBETT , J. C., D EAN , J., E PSTEIN , M., F IKES , A.,
FROST, C., F URMAN , J. J., G HEMAWAT, S., G UBAREV , A.,
HEISER , C., H OCHSCHILD , P., H SIEH , W. C., K ANTHAK ,
S., K OGAN , E., L I, H., L LOYD , A., M ELNIK , S., M WAURA,
D., N AGLE , D., Q UINLAN , S., R AO, R., R OLIG , L., S AITO ,
Y., SZYMANIAK , M., TAYLOR , C., WANG , R., AND WOOD -
FORD , D. Spanner: Google’s globally-distributed database.
In Proceedings of the 10th USENIX Symposium on Operating
Systems Design and Implementation (2012), OSDI’12.
[12] D ALESSANDRO , L., AND SCOTT, M. L. Sandboxing transac-
tional memory. In Proceedings of the 21st ACM International
Conference on Parallel Architectures and Compilation Tech-
niques (2012), PACT’12.
[13] D ECANDIA , G., H ASTORUN , D., J AMPANI , M., K AKULA -
PATI, G., L AKSHMAN , A., P ILCHIN , A., S IVASUBRAMA -
NIAN , S., V OSSHALL , P., AND VOGELS , W. Dynamo: Ama-
zon’s highly available key-value store. In Proceedings of the
the 21st ACM Symposium on Operating Systems Principles
(2007), SOSP’07.
[14] D IACONU , C., F REEDMAN , C., I SMERT , E., L ARSON , P.-
˚A., M ITTAL , P., S TONECIPHER , R., V ERMA , N., AND
ZWILLING , M. Hekaton: SQL Server’s memory-optimized
OLTP engine. In Proceedings of the ACM SIGMOD Inter-
national Conference on Management of Data (2013), SIG-
MOD’13.
[15] D ICE , D., S HALEV , O., AND SHAVIT, N. Transactional lock-
ing II. In Proceedings of the 20th International Symposium on
Distributed Computing (2006), DISC’06.
[16] D RAGOJEVI ´C, A., N ARAYANAN , D., H ODSON , O., AND
CASTRO , M. FaRM: Fast remote memory. In Proceedings of
the 11th USENIX Conference on Networked Systems Design
and Implementation (2014), NSDI’14.
[17] G RAEFE , G. Write-optimized B-trees. In Proceedings of
the 30th International Conference on Very Large Data Bases
(2004), VLDB’04.
69

[18] G RAY, C., AND CHERITON , D. Leases: An efficient fault-
tolerant mechanism for distributed file cache consistency.
SIGOPS Operating Systems Review (OSR) 23, 5 (1989).
[19] G RAY, J., AND REUTER , A. Transaction Processing: Con-
cepts and Techniques. 1992.
[20] G UERRAOUI , R., AND KAPALKA , M. On the correctness
of transactional memory. In Proceedings of the 13th ACM
SIGPLAN Symposium on Principles and Practice of Parallel
Programming (2008), PPoPP’08.
[21] H UNT, P., K ONAR , M., J UNQUEIRA , F. P., AND REED , B.
Zookeeper: wait-free coordination for internet-scale systems.
In Proceedings of the 2010 USENIX Annual Technical Con-
ference (2010), USENIX ATC’10.
[22] I NFINI BAND TRADE ASSOCIATION . Supplement to Infini-
Band Architecture Specification V olume 1 Release 1.2.2 An-
nex A16: RDMA over Converged Ethernet (RoCE), 2010.
[23] K ALIA , A., K AMINSKY , M., AND ANDERSEN , D. G. Using
RDMA efficiently for key-value services. In Proceedings of
the 2014 Conference on Applications, Technologies, Architec-
tures, and Protocols for Computer Communications (2014),
SIGCOMM’14.
[24] L AMPORT , L. The part-time parliament. ACM Transactions
on Computer Systems 16, 2.
[25] L AMPORT , L., M ALKHI , D., AND ZHOU , L. Vertical Paxos
and primary-backup replication. In Proceedings of the 28th
ACM Symposium on Principles of Distributed Computing
(2009), PODC’09.
[26] L ARSON , P.- ˚A., B LANAS , S., D IACONU , C., F REEDMAN ,
C., P ATEL, J. M., AND ZWILLING , M. High-performance
concurrency control mechanisms for main-memory databases.
PVLDB 5, 4 (2011).
[27] L EHMAN , P. L., AND YAO, S. B. Efficient locking for con-
current operations on B-trees.ACM Transactions on Database
Systems 6, 4 (Dec. 1981).
[28] M ICROSOFT . Scaling out SQL Server. http:
//www.microsoft.com/en-us/server-
cloud/solutions/high-availability.aspx.
[29] M ICROSOFT . Open CloudServer OCS V2 specification:
Blade, 2014.
[30] M ICROSOFT . OCS Open CloudServer power sup-
ply v2.0. http://www.opencompute.org/wiki/
Server/SpecsAndDesigns, 2015.
[31] M ITCHELL , C., Y IFENG , G., AND JINYANG , L. Using one-
sided RDMA reads to build a fast, CPU-efficient key-value
store. In Proceedings of the 2013 USENIX Annual Technical
Conference (2013), USENIX ATC’13.
[32] N EUVONEN , S., W OLSKI , A., MANNER , M., AND
RAATIKKA , V. Telecom Application Transaction Pro-
cessing benchmark. http://tatpbenchmark.
sourceforge.net/.
[33] O NGARO , D., R UMBLE , S. M., S TUTSMAN , R., O USTER -
HOUT , J., AND ROSENBLUM , M. Fast crash recovery in
RAMCloud. In Proceedings of the 23rd ACM Symposium on
Operating Systems Principles (2011), SOSP’11.
[34] R UMBLE , S. M., K EJRIWAL , A., AND OUSTERHOUT , J.
Log-structured Memory for DRAM-based Storage. In Pro-
ceedings of the 12th USENIX Conference on File and Storage
Technologies (2014), FAST’14.
[35] S ETHI , R. Useless actions make a difference: Strict serializ-
ability of database updates. JACM 29, 2 (1982).
[36] S HAUN HARRIS . Microsoft reinvents datacenter power
backup with new Open Compute project specification.
http://blogs.msdn.com/b/windowsazure/
archive/2012/11/13/windows-azure-
benchmarks-show-top-performance-for-
big-compute.aspx, 2015.
[37] S OWELL , B., G OLAB , W. M., AND SHAH , M. A. Minuet: A
scalable distributed multiversion B-tree. PVLDB 5, 9 (2012).
[38] T RANSACTION PROCESSING PERFORMANCE COUNCIL
(TPC). TPC benchmark C: Standard specification. http:
//www.tpc.org.
[39] T U, S., Z HENG , W., K OHLER , E., L ISKOV, B., AND
MADDEN , S. Speedy transactions in multicore in-memory
databases. In Proceedings of the 24th Symposium on Operat-
ing Systems Principles (2013), SOSP’13.
[40] Z HENG , W., T U, S., K OHLER , E., AND LISKOV, B. Fast
databases with fast durability and recovery through multicore
parallelism. In Proceedings of the 11th USENIX Symposium
on Operating Systems Design and Implementation (2014),
OSDI’14.
70

FAQ FaRM

Q: What are some systems that currently uses FaRM?

A: FaRM seems to be a research system, and not in production use. I
suspect it will influence future designs, and perhaps itself be
developed into a production system.

Q: Why do companies (Microsoft, Google, Facebook, Yahoo, etc) publish
papers about their software, rather than keeping their designs secret?

A: These companies only publish papers about a tiny fraction of the
software they write. One reason they publish is that these systems are
partially developed by people with an academic background (i.e. who
have PhDs), who feel that part of their mission in life is to help the
world understand the new ideas they invent. They are proud of their
work and want people to appreciate it. Another reason is that such
papers may help the companies attract top talent, because the papers
show that intellectually interesting work is going on there.

Q: Does FaRM really signal the end of necessary compromises in
consistency/availability in distributed systems?

A: This part of the paper seems more like advertising than science.
History suggests that no level of performance is so high that no-one
will want more, and those people will likely be willing to compromise
in other areas to get the performance they need.

Q: What are some limitations of FaRM?

A: The data has to fit in RAM. OCC will produce lots of aborts if
transactions conflict a lot. The transaction API (described in their
NSDI 2014 paper) looks awkward to use because replies return in
callbacks. Application code has to tightly interleave executing
application transactions and polling RDMA NIC queues and logs for
messages from other computers. Application code can see
inconsistencies while executing transactions that will eventually
abort. Applications may not be able to make free use of threads for
their own purposes because FaRM pins threads to cores, and uses all
cores. FaRM requires special network hardware that's not widely
deployed. The design only makes sense if all the computers are close
to each other; it's not a recipe for geographical distribution (and
thus can have only limited fault tolerance). Of course, FaRM is a
research prototype intended to explore new ideas. It is not a finished
product intended for general use. If people continue this line of
work, we might eventually see descendants of FaRM with fewer rough
edges.

Q: What's a NIC?

A: A Network Interface Card -- the hardware that connects a computer
to the network.

Q: What is RDMA?

A: RDMA is a special feature implemented in some modern NICs. The NIC
looks for special command packets that arrive over the network, and
executes the commands itself (and does not give the packets to the
CPU). The commands specify memory operations such as write a value to
an address or read from an address and send the value back over the
network. In addition, RDMA NICs allow application code to directly
talk to the NIC hardware to send the special RDMA command packets, and
to be notified when the "hardware ACK" packet arrives indicating that
the receiving NIC has executed the command.

Q: What is one-sided RDMA?

A: "One-sided" refers to a situation where application code in one
computer uses these RDMA NICs to directly read or write memory in
another computer without involving the other computer's CPU. FaRM's
"Validate" phase in Section 4 / Figure 4 uses only a one-sided read.

FaRM sometimes uses RDMA as a fast way to implement an RPC-like scheme
to talk to software running on the receiving computer. The sender uses
RDMA to write the request message to an area of memory that the
receiver's FaRM software is polling (checking periodically); the
receiver sends its reply in the same way. The FaRM "Lock" phase uses
RDMA in this way.

The benefit of RDMA is speed. A one-sided RDMA read or write takes as
little as 1/18 of a microsecond (Figure 2), while a traditional RPC
might take 10 microseconds. Even FaRM's use of RDMA for messaging is a
lot faster than traditional RPC: user-space code in the receiver
frequently polls the incoming NIC queues in order to see new messages
quickly, rather than involving interrupts and user/kernel transitions.

Q: Why is FaRM's RDMA-based RPC faster than traditional RPC?

A: Traditional RPC requires the application to make a system call to
the local kernel, which asks the local NIC to send a packet. At the
receiving computer, the NIC writes the packet to a queue in memory and
interrupts the receving computer's kernel. The kernel copies the
packet to user space and context-switches to the receiving
application. The receving application does the reverse to send the
reply (system call to kernel, kernel talks to NIC, NIC on the other
side interrupts its kernel, &c). This point is that a huge amount of
code is executed for each RPC, and it's not very fast.

In contrast, FaRM arranges that the application code can directly read
and write memory to communicate with the NIC, and dedicates CPU cores
(which the paper calls hardware threads) to polling for incoming
messages. This eliminates costs from interrupts, system calls, copying
data between user and kernel, and context switches.

Q: Much of FaRM's performance comes from the hardware. In what ways
does the software design contribute to performance?

A: It's true that one reason FaRM is fast is that the hardware is
fast. But the hardware has been around for many years now, yet no-one
has figured out how to put all the parts together in a way that really
exploits the hardware's potential. One reason FaRM does so well is
that they simultaneously put a lot of effort into optimizing the
network, the persistent storage, and the use of CPU; many previous
systems have optimized one but not all. A specific design point is the
way FaRM uses fast one-sided RDMA (rather than slower full RPC) for
many of the interactions.

Q: Do other systems use UPS (uninterruptable power supplies, with
batteries) to implement fast but persistent storage?

A: The idea is old; for example the Harp replicated file service used
it in the early 1990s. Many storage systems use batteries in related
ways (e.g. in RAID controllers) to write persistently without waiting
for the disk. However, the kind of battery setup that FaRM uses isn't
particularly common, so software that has to be general purpose can't
rely on it. If you configure your own hardware to have batteries, then
it would make sense to modify your Raft (or k/v server) to exploit
your batteries.

Q: Would the FaRM design still make sense without the battery-backed RAM?

A: I'm not sure FaRM would make sense without non-volatile RAM,
because then the one-sided log writes (e.g. COMMIT-BACKUP in Figure 4)
would not persist across power failures. You could modify FaRM so that
all log updates were written to SSD before returning, but then it
would have much lower performance. An SSD write takes about 100
microseconds, while FaRM's one-sided RDMA writes to non-volatile RAM
take only a few microseconds.

Q: Isn't DRAM inherently volatile?

A: The authors make RAM "non-volatile" by using a UPS to allow FaRM to
write the content of RAM to an SSD on a power failure. But, this is
indeed not completely non-volatile, because if the computer crashes
for any other reason than a power failure, the content of the memory
of the failed machine is lost. This is the reason why they replicate
each region across several machines and have a fast recovery protocol.

Q: A FaRM server copies RAM to SSD if the power is about to fail.
Could they use mechanical hard drives instead of SSDs?

A: They use SSDs because they are fast. They could have used hard
drives without changing the design. However, it would then take about
10x longer to write the data to disk during a power outage, and 10x
longer to read it back in after power is restored. That would require
bigger batteries and more patience.

Q: What is the distinction between primaries, backups, and
configuration managers in FaRM? Why are there three roles?

A: The data is sharded among many primary/backup sets. The point of
the backups is to store a copy of the shard's data and logs in case
the primary fails. The primary performs all reads and writes to data
in the shard, while the backups perform only the writes (in order to
keep their copies of the data identical to the primary's copy). There's
just one configuration manager. It keeps track of which primaries and
backups are alive, and keeps track of how the data is sharded among
them. At a high level this arrangement is similar to GFS, which also
sharded data among many primary/backup sets, and also had a master
that kept track of where data is stored.

Q: Would FaRM make sense at small scale?

A: I think FaRM is only interesting if you need to support a huge
number of transactions per second. If you only need a few thousand
transactions per second, you can use off-the-shelf mature technology
like MySQL. You could probably set up a considerably smaller FaRM
system than the authors' 90-machine system. But FaRM doesn't make
sense unless you are sharding and replicating data, which means you
need at least four data servers (two shards, two servers per shard)
plus a few machines for ZooKeeper (though probably you could run
ZooKeeper on the four machines). Then maybe you have a system that
costs on the order of $10,000 dollars and can execute a few million
simple transactions per second, which is pretty good.

Q: Section 3 seems to say that a single transaction's reads may see
inconsistent data. That doesn't seem like it would be serializable!

A: Farm only guarantees serializability for transactions that commit.
If a transaction sees the kind of inconsistency Section 3 is talking
about, FaRM will abort the transaction. Applications must handle
inconsistency in the sense that they should not crash, so that they
can get as far as asking to commit, so that FaRM can abort them.

Q: How does FaRM ensure that a transaction's reads are consistent?
What happens if a transaction reads an object that is being modified
by a different transaction?

A: There are two dangers here. First, for a big object, the reader may
read the first half of the object before a concurrent transaction has
written it, and the second half after the concurrent transaction has
written it, and this might cause the reading program to crash. Second,
the reading transaction can't be allowed to commit if it might not be
serializable with a concurrent writing transaction.

Based on my reading of the authors' previous NSDI 2014 paper, the
solution to the first problem is that every cache line of every object
has a version number, and single-cache-line RDMA reads and writes are
atomic. The reading transaction's FaRM library fetches all of the
object's cache lines, and then checks whether they all have the same
version number. If yes, the library gives the copy of the object to
the application; if no, the library reads it again over RDMA. The
second problem is solved by FaRM's validation scheme described in
Section 4. In the VALIDATE step, if another transaction has written an
object read by our transaction since our transaction started, our
transaction will be aborted.

Q: How does log truncation work? When can a log entry be removed?
If one entry is removed by a truncate call, are all previous entries
also removed?

A: The TC tells the primaries and backups to delete the log entries
for a transaction after the TC sees that all of them have a
COMMIT-PRIMARY or COMMIT-BACKUP in their log. In order that recovery
will know that a transaction is done despite truncation, page 62
mentions that primaries remember completed transaction IDs even after
truncation. Truncation implies that all log entries before the
truncation point are deleted; this works because each primary/backup
has a separate log per TC.

Q: Is it possible for an abort to occur during COMMIT-BACKUP, perhaps
due to hardware failure?

A: I believe so. If one of the backups doesn't respond, and the TC
crashes, then there's a possibility that the transaction might be
aborted during recovery.

Q: Does FaRM performance suffer when many transactions need to modify
the same object?

A: When multiple transactions modify the same object at the same time,
some of them will see during Figure 4's LOCK phase that the lock is
already held. Readers may see a changed version, or a lock flag,
during the VALIDATE phase. Each such transaction will abort and
restart from the beginning. If that happens a lot, performance will
indeed suffer. The "optimistic" in "optimistic concurrency control"
refers to the hope that that such conflicts will be rare, and that the
ability to do lock-free reads will yield high performance. And indeed,
for the applications the authors measure, FaRM gets fantastic
performance. Very likely one reason is that their applications have
relatively few conflicting transactions, and thus not many aborts.

Q: Figure 7 shows significant increase in latency when the number of
operations exceeds 120 per microsecond. Why is that?

A: I suspect the limit is that the servers can only process about 140
million operations per second in total. If clients send operations
faster than that, some of them will have to wait; this waiting causes
increased latency.

Q: What is vertical Paxos?

A: It is a style of Paxos protocols where an external master performs
reconfiguration while the Paxos group can continue performing
operations while reconfiguration is in progress (see
https://lamport.azurewebsites.net/pubs/vertical-paxos.pdf for the
details).  In the FaRM paper, the authors use the term "vertical
Paxos" loosely to mean that the configuration management is done by an
external service (Zookeeper and CM) and processing writes of a
transaction is done with standard primary/backup protocol.

Q: Where does 3 come from in Pw(f + 3), where where Pw is the number
of machines that are primaries for objects written by the
transaction?

A: If f is 1, as in the paper, and only 1 primary P is involved then
the 4 (1 + 3) messages are: (1) lock request from P (2) lock reply to
P (it is depicted in figure 4 as an one-sided write RDMA); (3)
commit-backup; and (4) commit-primary.

Q: Why is it called FaRM?

A: Fast Remote Memory

GFS FAQ

Q: Did having a single master turn out to be a good idea?

A: That idea simplified initial deployment but was not so great in the
long run. This article (GFS: Evolution on Fast Forward,
https://queue.acm.org/detail.cfm?id=1594206) says that as the years
went by and GFS use grew, a few things went wrong. The number of files
grew enough that it wasn't reasonable to store all files' metadata in
the RAM of a single master. The number of clients grew enough that a
single master didn't have enough CPU power to serve them. The fact
that switching from a failed master to one of its secondaries required
human intervention made recovery slow. Apparently Google's replacement
for GFS, Colossus, splits the master over multiple servers, and has
more automated master failure recovery.

Q: Why is atomic record append at-least-once, rather than exactly
once?

Section 3.1, Step 7, says that if a write fails at one of the
secondaries, the client re-tries the write. That will cause the data
to be appended more than once at the non-failed replicas. A different
design could detect duplicate client requests despite arbitrary
failures (e.g. a primary failure between the original request and the
client's retry). You'll implement such a design in the labs, at
considerable expense in complexity and performance.

Q: How does an application know what sections of a chunk consist of
padding and duplicate records?

A: To detect padding, applications can put a predictable magic number
at the start of a valid record, or include a checksum that will likely
only be valid if the record is valid. The application can detect
duplicates by including unique IDs in records. Then, if it reads a
record that has the same ID as an earlier record, it knows that they
are duplicates of each other. GFS provides a library for applications
that handles these cases. This aspect of the GFS design effectively
moves complexity from GFS to applications, which is perhaps not ideal.

Q: How can clients find their data given that atomic record append
writes it at an unpredictable offset in the file?

A: Append (and GFS in general) is mostly intended for applications
that sequentially read entire files. Such applications will scan the
file looking for valid records (see the previous question), so they
don't need to know the record locations in advance. For example, the
file might contain URLs encountered by a set of concurrent web
crawlers. The file offset of any given URL doesn't matter much;
readers just want to be able to read the entire set of URLs.

Q: What's a checksum?

A: A checksum algorithm takes a sequence of bytes as input and returns
a single number that's a function of that sequence. For example, a
simple checksum might be the sum of all the bytes in the input. GFS
stores the checksum of each 64 kilobyte "block" in each chunk. When a
chunkserver writes a block of data to its disk, it first computes the
checksum of the block, and saves the checksum on disk. When a
chunkserver reads a block from its disk, it also reads the relevant
previously-saved checksum, re-computes a checksum from the data read
from disk, and checks that the two checksums match. If the data was
corrupted by the disk, the checksums won't match, and the chunkserver
will know to return an error. Separately, some GFS applications store
their own checksums, over application-defined records, inside GFS
files, to distinguish between correct records and padding. CRC32 is an
example of a checksum algorithm.

Q: The paper mentions reference counts -- what are they?

A: They are part of the implementation of copy-on-write for snapshots.
When GFS creates a snapshot, it doesn't copy the chunks, but instead
increases the reference counter of each chunk. This makes creating a
snapshot inexpensive. If a client writes a chunk and the master
notices the reference count is greater than one, the master first
makes a copy so that the client can update the copy (instead of the
chunk that is part of the snapshot). You can view this as delaying the
copy until it is absolutely necessary. The hope is that not all chunks
will be modified and one can avoid making some copies.

Q: If an application uses the standard POSIX file APIs, would it need
to be modified in order to use GFS?

A: Yes, but GFS isn't intended for existing applications. It is
designed for newly-written applications, such as MapReduce programs.

Q: How does GFS determine the location of the nearest replica?

A: The paper hints that GFS does this based on the IP addresses of the
servers storing the available replicas. In 2003, Google must have
assigned IP addresses in such a way that if two IP addresses are close
to each other in IP address space, then they are also close to each
other in machine-room network topology (perhaps plugged into the same
Ethernet switch, or into Ethernet switches that are themselves
directly connected).

Q: What's a lease?

A: For GFS, a lease is a period of time for which the master grants a
chunkserver the ability to act as the primary for a particular chunk.
The master guarantees not to assign a different primary for the
duration of the lease, and the primary agrees to stop acting as
primary before the lease expires (unless the primary first asks the
master to extend the lease). Leases are a way to avoid having the
primary have to repeatedly ask the master if it is still primary -- it
knows it can act as primary for the next minute (or whatever the lease
interval is) without talking to the master again.

Q: Suppose S1 is the primary for a chunk, and the network between the
master and S1 fails. The master will notice and designate some other
server as primary, say S2. Since S1 didn't actually fail, are there
now two primaries for the same chunk?

A: That would be a disaster, since both primaries might apply
different updates to the same chunk. Luckily GFS's lease mechanism
prevents this scenario. The master granted S1 a 60-second lease to be
primary. S1 knows to stop being primary before its lease expires. The
master won't grant a lease to S2 until after the lease to S1 expires.
So S2 won't start acting as primary until after S1 stops.

Q: 64 megabytes sounds awkwardly large for the chunk size!

A: The 64 MB chunk size is the unit of book-keeping in the master, and
the granularity at which files are sharded over chunkservers. Clients
can issue smaller reads and writes -- they are not forced to deal
in whole 64 MB chunks. The point of using such a big chunk size is to
reduce the size of the meta-data tables in the master, and to avoid
limiting clients that want to do huge transfers to reduce overhead. On
the other hand, files less than 64 MB in size do not get much
parallelism.

Q: Does Google still use GFS?

A: GFS has been replaced by something called
Colossus, with the same overall goals, but improvements in master
performance and fault-tolerance. In addition, many applications within
Google have switched to more database-like storage systems such as
BigTable and Spanner. However, much of the GFS design lives on in
HDFS, the storage system for the Hadoop open-source MapReduce.

https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system

Q: How acceptable is it that GFS trades correctness for performance
and simplicity?

A: This a recurring theme in distributed systems. Strong consistency
usually requires protocols that are complex and require communication
and waiting for replies (as we will see in the next few lectures). By
exploiting ways that specific application classes can tolerate relaxed
consistency, one can design systems that have good performance and
sufficient consistency. For example, GFS optimizes for MapReduce
applications, which need high read performance for large files and are
OK with having holes in files, records showing up several times, and
inconsistent reads. On the other hand, GFS would not be good for
storing account balances at a bank.

Q: What if the master fails?

A: There are replica masters with a full copy of the master state; the
paper's design requires some outside entity (a human?) to decide to
switch to one of the replicas after a master failure (Section 5.1.3).
We will see later how to build replicated services that automatically
switch to a backup server if the main server fails, and you'll build
such a thing in Lab 2.

Q: Why 3 replicas?

A: Perhaps this was the line of reasoning: two replicas are not enough
because, after one fails, there may not be enough time to re-replicate
before the remaining replica fails; three makes that scenario much
less likely. With 1000s of disks, low-probabilty events like multiple
replicas failing in short order occur uncomfortably often. Here is a
study of disk reliability from that era:
https://research.google.com/archive/disk_failures.pdf. You need to
factor in the time it takes to make new copies of all the chunks that
were stored on a failed disk; and perhaps also the frequency of power,
server, network, and software failures. The cost of disks (and
associated power, air conditioning, and rent), and the value of the
data being protected, are also relevant.

Q: What is internal fragmentation? Why does lazy allocation help?

A: Internal fragmentation is the space wasted when a system uses an
allocation unit larger than needed for the requested allocation. If
GFS allocated disk space in 64MB units, then a one-byte file would
waste almost 64MB of disk. GFS avoids this problem by allocating disk
space lazily. Every chunk is a Linux file, and Linux file systems use
block sizes of a few tens of kilobytes; so when an application creates
a one-byte GFS file, the file's chunk consumes only one Linux disk
block, not 64 MB.

Q: What benefit does GFS obtain from the weakness of its consistency?

A: It's easier to think about the additional work GFS would have to do
to achieve stronger consistency.

The primary should not let secondaries apply a write unless all the
secondaries will be able to do it. This likely requires two rounds of
communication -- one to ask all secondaries if they are alive and are
able to promise to do the write if asked, and (if all answer yes) a
second round to tell the secondaries to commit the write.

If the primary dies, some secondaries may have missed the last few
update messages the primary sent. This will cause the remaining secondaries
to have slightly differing copies of the data. Before resuming
operation, a new primary should ensure that all the secondaries have
identical copies.

Since clients re-send requests if they suspect something has gone
wrong, primaries would need to filter out operations that have already
been executed.

Clients cache chunk locations, and may send reads to a chunkserver
that holds a stale version of a chunk. GFS would need a way to
guarantee that this cannot succeed.

The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-T ak Leung
Google∗
ABSTRACT
We have designed and implemented the Go ogle File Sys-
tem, a scalable distributed file system for large distributed
data-intensive applications. It provides fault tolerance while
running on inexpensive commodity hardware, and it delivers
high aggregate performance to a large number of clients.
While sharing many of the same goals as previous dis-
tributed file systems, our design has been driven by obser-
vations of our application workloads and technological envi-
ronment, both current and anticipated, that reflect a marked
departure from some earlier file system assumptions. This
has led us to reexamine traditional choices and explore rad-
ically different design points.
The file system has successfully met our storage needs.
It is widely deployed within Google as the storage platform
for the generation and processing of data used by our ser-
vice as well as research and development efforts that require
large data sets. The largest cluster to date provides hun-
dreds of terabytes of storage across thousands of disks on
over a thousand machines, and it is concurrently accessed
by hundreds of clients.
In this paper, we present file system interface extensions
designed to support distributed applications, discuss many
aspects of our design, and report measurements from both
micro-benchmarks and real world use.
Categories and Subject Descriptors
D[ 4]: 3— Distributed file systems
General Terms
Design, reliability, performance, measurement
Keywords
Fault tolerance, scalability, data storage, clustered storage
∗ The authors can be reached at the following addresses:
{sanjay,hgobioff,shuntak}@google.com.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
SOSP’03, October 19–22, 2003, Bolton Landing, New York, USA.
Copyright 2003 ACM 1-58113-757-5/03/0010 ...$5.00.
1. INTRODUCTION
We have designed and implemented the Go ogle File Sys-
tem (GFS) to meet the rapidly growing demands of Google’s
data processing needs. GFS shares many of the same goals
as previous distributed file systems such as performance,
scalability, reliability, and availability. However, its design
has been driven by key observations of our application work-
loads and technological environment, both current and an-
ticipated, that reflect a marked departure from some earlier
file system design assumptions. We have reexamined tradi-
tional choices and explored radically different points in the
design space.
First, component failures are the norm rather than the
exception. The file system consists of hundreds or even
thousands of storage machines built from inexpensive com-
modity parts and is accessed by a comparable number of
client machines. The quantity and quality of the compo-
nents virtually guarantee that some are not functional at
any given time and some will not recover from their cur-
rent failures. We have seen problems caused by application
bugs, operating system bugs, human errors, and the failures
of disks, memory, connectors, networking, and power sup-
plies. Therefore, constant monitoring, error detection, fault
tolerance, and automatic recovery must be integral to the
system.
Second, files are huge by traditional standards. Multi-GB
files are common. Each file typically contains many applica-
tion objects such as web documents. When we are regularly
working with fast growing data sets of many TBs comprising
billions of objects, it is unwieldy to manage billions of ap-
proximately KB-sized files even when the file system could
support it. As a result, design assumptions and parameters
such as I/O operation and block sizes have to be revisited.
Third, most files are mutated by appending new data
rather than overwriting existing data. Random writes within
afi l ea r ep r a c t i c a l l yn o n - e x i s t e n t . O n c ew r i t t e n ,t h efi l e s
are only read, and often only sequentially. A variety of
data share these characteristics. Some may constitute large
repositories that data analysis programs scan through. Some
may be data streams continuously generated by running ap-
plications. Some may be archival data. Some may be in-
termediate results produced on one machine and processed
on another, whether simultaneously or later in time. Given
this access pattern on huge files, appending becomes the fo-
cus of performance optimization and atomicity guarantees,
while caching data blocks in the client loses its appeal.
Fourth, co-designing the applications and the file system
API benefits the overall system by increasing our flexibility.

For example, we have relaxed GFS’s consistency mo del to
vastly simplify the file system without imposing an onerous
burden on the applications. We have also introduced an
atomic append operation so that multiple clients can append
concurrently to a file without extra synchronization between
them. These will be discussed in more details later in the
paper.
Multiple GFS clusters are currently deployed for different
purposes. The largest ones have over 1000 storage nodes,
over 300 TB of disk storage, and are heavily accessed by
hundreds of clients on distinct machines on a continuous
basis.
2. DESIGN OVERVIEW
2.1 Assumptions
In designing a file system for our needs, we have been
guided by assumptions that offer both challenges and op-
portunities. W e alluded to some key observations earlier
and now lay out our assumptions in more details.
• The system is built from many inexpensive commodity
components that often fail. It must constantly monitor
itself and detect, tolerate, and recover promptly from
component failures on a routine basis.
• The system stores a modest number of large files. We
expect a few million files, each typically 100 MB or
larger in size. Multi-GB files are the common case
and should be managed efficiently. Small files must be
supported, but we need not optimize for them.
• The workloads primarily consist of two kinds of reads:
large streaming reads and small random reads. In
large streaming reads, individual operations typically
read hundreds of KBs, more commonly 1 MB or more.
Successive operations from the same client often read
through a contiguous region of a file. A small ran-
dom read typically reads a few KBs at some arbitrary
offset. Performance-conscious applications often batch
and sort their small reads to advance steadily through
the file rather than go back and forth.
• The workloads also have many large, sequential writes
that append data to files. Typical operation sizes are
similar to those for reads. Once written, files are sel-
dom modified again. Small writes at arbitrary posi-
tions in a file are supported but do not have to be
efficient.
• The system must efficiently implement well-defined se-
mantics for multiple clients that concurrently append
to the same file. Our files are often used as producer-
consumer queues or for many-way merging. Hundreds
of producers, running one per machine, will concur-
rently append to a file. Atomicity with minimal syn-
chronization overhead is essential. The file may be
read later, or a consumer may be reading through the
file simultaneously.
• High sustained bandwidth is more important than low
latency. Most of our target applications place a pre-
mium on processing data in bulk at a high rate, while
few have stringent response time requirements for an
individual read or write.
2.2 Interface
GFS provides a familiar file system interface, though it
does not implement a standard API such as POSIX. Files are
organized hierarchically in directories and identified by path-
names. We support the usual operations to create, delete,
open, close, read,a n d write files.
Moreover, GFS has snapshot and record append opera-
tions. Snapshot creates a copy of a file or a directory tree
at low cost. Record append allows multiple clients to ap-
pend data to the same file concurrently while guaranteeing
the atomicity of each individual client’s append. It is use-
ful for implementing multi-way merge results and producer-
consumer queues that many clients can simultaneously ap-
pend to without additional locking. W e have found these
types of files to be invaluable in building large distributed
applications. Snapshot and record append are discussed fur-
ther in Sections 3.4 and 3.3 respectively.
2.3 Architecture
AG F Sc l u s t e rc o n s i s t so fas i n g l emaster and multiple
chunkservers and is accessed by multiple clients,a ss h o w n
in Figure 1. Each of these is typically a commodity Linux
machine running a user-level server process. It is easy to run
both a chunkserver and a client on the same machine, as long
as machine resources permit and the lower reliability caused
by running possibly flaky application code is acceptable.
Files are divided into fixed-size chunks.E a c h c h u n k i s
identified by an immutable and globally unique 64 bit chunk
handle assigned by the master at the time of chunk creation.
Chunkservers store chunks on local disks as Linux files and
read or write chunk data specified by a chunk handle and
byte range. For reliability, each chunk is replicated on multi-
ple chunkservers. By default, we store three replicas, though
users can designate different replication levels for different
regions of the file namespace.
The master maintains all file system metadata. This in-
cludes the namespace, access control information, the map-
ping from files to chunks, and the current locations of chunks.
It also controls system-wide activities such as chunk lease
management, garbage collection of orphaned chunks, and
chunk migration between chunkservers. The master peri-
odically communicates with each chunkserver in HeartBeat
messages to give it instructions and collect its state.
GFS client code linked into each application implements
the file system API and communicates with the master and
chunkservers to read or write data on behalf of the applica-
tion. Clients interact with the master for metadata opera-
tions, but all data-bearing communication goes directly to
the chunkservers. We do not provide the POSIX API and
therefore need not hook into the Linux vnode layer.
Neither the client nor the chunkserver caches file data.
Client caches offer little benefit because most applications
stream through huge files or have working sets too large
to be cached. Not having them simplifies the client and
the overall system by eliminating cache coherence issues.
(Clients do cache metadata, however.) Chunkservers need
not cache file data because chunks are stored as local files
and so Linux’s buffer cache already keeps frequently accessed
data in memory.
2.4 Single Master
Having a single master vastly simplifies our design and
enables the master to make sophisticated chunk placement

Legend:
Data messages
Control messages
Application (file name, chunk index)
(chunk handle,
chunk locations)
GFS master
File namespace
/foo/bar
Instructions to chunkserver
Chunkserver state
GFS chunkserverGFS chunkserver
(chunk handle, byte range)
chunk data
chunk 2ef0
Linux file system Linux file system
GFS client
Figure 1: GFS Architecture
and replication decisions using global knowledge. However,
we must minimize its involvement in reads and writes so
that it does not become a bottleneck. Clients never read
and write file data through the master. Instead, a client asks
the master which chunkservers it should contact. It caches
this information for a limited time and interacts with the
chunkservers directly for many subsequent operations.
Let us explain the interactions for a simple read with refer-
ence to Figure 1. First, using the fixed chunk size, the client
translates the file name and byte offset specified by the ap-
plication into a chunk index within the file. Then, it sends
the master a request containing the file name and chunk
index. The master replies with the corresponding chunk
handle and locations of the replicas. The client caches this
information using the file name and chunk index as the key.
The client then sends a request to one of the replicas,
most likely the closest one. The request specifies the chunk
handle and a byte range within that chunk. Further reads
of the same chunk require no more client-master interaction
until the cached information expires or the file is reopened.
In fact, the client typically asks for multiple chunks in the
same request and the master can also include the informa-
tion for chunks immediately following those requested. This
extra information sidesteps several future client-master in-
teractions at practically no extra cost.
2.5 Chunk Size
Chunk size is one of the key design parameters. We have
chosen 64 MB, which is much larger than typical file sys-
tem block sizes. Each chunk replica is stored as a plain
Linux file on a chunkserver and is extended only as needed.
Lazy space allocation avoids wasting space due to internal
fragmentation, perhaps the greatest objection against such
al a r g ec h u n ks i z e .
Al a r g ec h u n ks i z eo ff e r ss e v e r a li m p o r t a n ta d v a n t a g e s .
First, it reduces clients’ need to interact with the master
because reads and writes on the same chunk require only
one initial request to the master for chunk location informa-
tion. The reduction is especially significant for our work-
loads because applications mostly read and write large files
sequentially. Even for small random reads, the client can
comfortably cache all the chunk location information for a
multi-TB working set. Second, since on a large chunk, a
client is more likely to perform many operations on a given
chunk, it can reduce network overhead by keeping a persis-
tent TCP connection to the chunkserver over an extended
period of time. Third, it reduces the size of the metadata
stored on the master. This allows us to keep the metadata
in memory, which in turn brings other advantages that we
will discuss in Section 2.6.1.
On the other hand, a large chunk size, even with lazy space
allocation, has its disadvantages. A small file consists of a
small number of chunks, perhaps just one. The chunkservers
storing those chunks may become hot spots if many clients
are accessing the same file. In practice, hot spots have not
been a major issue because our applications mostly read
large multi-chunk files sequentially.
However, hot spots did develop when GFS was first used
by a batch-queue system: an executable was written to GFS
as a single-chunk file and then started on hundreds of ma-
chines at the same time. The few chunkservers storing this
executable were overloaded by hundreds of simultaneous re-
quests. We fixed this problem by storing such executables
with a higher replication factor and by making the batch-
queue system stagger application start times. A potential
long-term solution is to allow clients to read data from other
clients in such situations.
2.6 Metadata
The master stores three major types of metadata: the file
and chunk namespaces, the mapping from files to chunks,
and the locations of each chunk’s replicas. All metadata is
kept in the master’s memory. The first two types (names-
paces and file-to-chunk mapping) are also kept persistent by
logging mutations to an operation log stored on the mas-
ter’s local disk and replicated on remote machines. Using
al o ga l l o w su st ou p d a t et h em a s t e rs t a t es i m p l y ,r e l i a b l y ,
and without risking inconsistencies in the event of a master
crash. The master does not store chunk location informa-
tion persistently. Instead, it asks each chunkserver about its
chunks at master startup and whenever a chunkserver joins
the cluster.
2.6.1 In-Memory Data Structures
Since metadata is stored in memory, master operations are
fast. Furthermore, it is easy and efficient for the master to
periodically scan through its entire state in the background.
This periodic scanning is used to implement chunk garbage
collection, re-replication in the presence of chunkserver fail-
ures, and chunk migration to balance load and disk space

usage across chunkservers. Sections 4.3 and 4.4 will discuss
these activities further.
One potential concern for this memory-only approach is
that the number of chunks and hence the capacity of the
whole system is limited by how much memory the master
has. This is not a serious limitation in practice. The mas-
ter maintains less than 64 bytes of metadata for each 64 MB
chunk. Most chunks are full because most files contain many
chunks, only the last of which may be partially filled. Sim-
ilarly, the file namespace data typically requires less then
64 bytes per file because it stores file names compactly us-
ing prefix compression.
If necessary to support even larger file systems, the cost
of adding extra memory to the master is a small price to pay
for the simplicity, reliability, performance, and flexibility we
gain by storing the metadata in memory.
2.6.2 Chunk Locations
The master does not keep a persistent record of which
chunkservers have a replica of a given chunk. It simply polls
chunkservers for that information at startup. The master
can keep itself up-to-date thereafter because it controls all
chunk placement and monitors chunkserver status with reg-
ular HeartBeat messages.
We initially attempted to keep chunk lo cation information
persistently at the master, but we decided that it was much
simpler to request the data from chunkservers at startup,
and periodically thereafter. This eliminated the problem of
keeping the master and chunkservers in sync as chunkservers
join and leave the cluster, change names, fail, restart, and
so on. In a cluster with hundreds of servers, these events
happen all too often.
Another way to understand this design decision is to real-
ize that a chunkserver has the final word over what chunks
it does or does not have on its own disks. There is no point
in trying to maintain a consistent view of this information
on the master because errors on a chunkserver may cause
chunks to vanish spontaneously (e.g., a disk may go bad
and be disabled) or an operator may rename a chunkserver.
2.6.3 Operation Log
The operation log contains a historical record of critical
metadata changes. It is central to GFS. Not only is it the
only persistent record of metadata, but it also serves as a
logical time line that defines the order of concurrent op-
erations. Files and chunks, as well as their versions (see
Section 4.5), are all uniquely and eternally identified by the
logical times at which they were created.
Since the operation log is critical, we must store it reli-
ably and not make changes visible to clients until metadata
changes are made persistent. Otherwise, we effectively lose
the whole file system or recent client operations even if the
chunks themselves survive. Therefore, we replicate it on
multiple remote machines and respond to a client opera-
tion only after flushing the corresponding log record to disk
both locally and remotely . The master batches several log
records together before flushing thereby reducing the impact
of flushing and replication on overall system throughput.
The master recovers its file system state by replaying the
operation log. To minimize startup time, we must keep the
log small. The master checkpoints its state whenever the log
grows beyond a certain size so that it can recover by loading
the latest checkpoint from local disk and replaying only the
Wr i te
 Record Append
Serial
 defined
 defined
success
 interspersed with
Concurrent
 consistent
 inconsistent
successes
 but undefined
Fai l ur e
 inconsistent
Table 1: File Region State After Mutation
limited number of log records after that. The checkpoint is
in a compact B-tree like form that can be directly mapped
into memory and used for namespace lookup without ex-
tra parsing. This further speeds up recovery and improves
availability.
Because building a checkpoint can take a while, the mas-
ter’s internal state is structured in such a way that a new
checkpoint can be created without delaying incoming muta-
tions. The master switches to a new log file and creates the
new checkpoint in a separate thread. The new checkpoint
includes all mutations before the switch. It can be created
in a minute or so for a cluster with a few million files. When
completed, it is written to disk both locally and remotely.
Recovery needs only the latest complete checkpoint and
subsequent log files. Older checkpoints and log files can
be freely deleted, though we keep a few around to guard
against catastrophes. A failure during checkpointing does
not affect correctness because the recovery code detects and
skips incomplete checkpoints.
2.7 Consistency Model
GFS has a relaxed consistency model that supports our
highly distributed applications well but remains relatively
simple and efficient to implement. We now discuss GFS’s
guarantees and what they mean to applications. We also
highlight how GFS maintains these guarantees but leave the
details to other parts of the paper.
2.7.1 Guarantees by GFS
File namespace mutations (e.g., file creation) are atomic.
They are handled exclusively by the master: namespace
locking guarantees atomicity and correctness (Section 4.1);
the master’s operation log defines a global total order of
these operations (Section 2.6.3).
The state of a file region after a data mutation depends
on the type of mutation, whether it succeeds or fails, and
whether there are concurrent mutations. Table 1 summa-
rizes the result. A file region is consistent if all clients will
always see the same data, regardless of which replicas they
read from. A region is defined after a file data mutation if it
is consistent and clients will see what the mutation writes in
its entirety. When a mutation succeeds without interference
from concurrent writers, the affected region is defined (and
by implication consistent): all clients will always see what
the mutation has written. Concurrent successful mutations
leave the region undefined but consistent: all clients see the
same data, but it may not reflect what any one mutation
has written. Typically, it consists of mingled fragments from
multiple mutations. A failed mutation makes the region in-
consistent (hence also undefined): different clients may see
different data at different times. We describe below how our
applications can distinguish defined regions from undefined

regions. The applications do not need to further distinguish
between different kinds of undefined regions.
Data mutations may be writes or record appends.A w r i t e
causes data to be written at an application-specified file
offset. A record append causes data (the “record”) to be
appended atomically at least once even in the presence of
concurrent mutations, but at an offset of GFS’s choosing
(Section 3.3). (In contrast, a “regular” append is merely a
write at an offset that the client believes to be the current
end of file.) The offset is returned to the client and marks
the beginning of a defined region that contains the record.
In addition, GFS may insert padding or record duplicates in
between. They occupy regions considered to be inconsistent
and are typically dwarfed by the amount of user data.
After a sequence of successful mutations, the mutated file
region is guaranteed to be defined and contain the data writ-
ten by the last mutation. GFS achieves this by (a) applying
mutations to a chunk in the same order on all its replicas
(Section 3.1), and (b) using chunk version numbers to detect
any replica that has become stale because it has missed mu-
tations while its chunkserver was down (Section 4.5). Stale
replicas will never be involved in a mutation or given to
clients asking the master for chunk locations. They are
garbage collected at the earliest opportunity.
Since clients cache chunk locations, they may read from a
stale replica before that information is refreshed. This win-
dow is limited by the cache entry’s timeout and the next
open of the file, which purges from the cache all chunk in-
formation for that file. Moreover, as most of our files are
append-only, a stale replica usually returns a premature
end of chunk rather than outdated data. When a reader
retries and contacts the master, it will immediately get cur-
rent chunk locations.
Long after a successful mutation, component failures can
of course still corrupt or destroy data. GFS identifies failed
chunkservers by regular handshakes between master and all
chunkservers and detects data corruption by checksumming
(Section 5.2). Once a problem surfaces, the data is restored
from valid replicas as soon as possible (Section 4.3). A chunk
is lost irreversibly only if all its replicas are lost before GFS
can react, typically within minutes. Even in this case, it be-
comes unavailable, not corrupted: applications receive clear
errors rather than corrupt data.
2.7.2 Implications for Applications
GFS applications can accommodate the relaxed consis-
tency model with a few simple techniques already needed for
other purposes: relying on appends rather than overwrites,
checkpointing, and writing self-validating, self-identifying
records.
Practically all our applications mutate files by appending
rather than overwriting. In one typical use, a writer gener-
ates a file from beginning to end. It atomically renames the
file to a permanent name after writing all the data, or pe-
riodically checkpoints how much has been successfully writ-
ten. Checkpoints may also include application-level check-
sums. Readers verify and process only the file region up
to the last checkpoint, which is known to be in the defined
state. Regardless of consistency and concurrency issues, this
approach has served us well. Appending is far more effi-
cient and more resilient to application failures than random
writes. Checkpointing allows writers to restart incremen-
tally and keeps readers from processing successfully written
file data that is still incomplete from the application’s per-
spective.
In the other typical use, many writers concurrently ap-
pend to a file for merged results or as a producer-consumer
queue. Record append’s append-at-least-once semantics pre-
serves each writer’s output. Readers deal with the occa-
sional padding and duplicates as follows. Each record pre-
pared by the writer contains extra information like check-
sums so that its validity can be verified. A reader can
identify and discard extra padding and record fragments
using the checksums. If it cannot tolerate the occasional
duplicates (e.g., if they would trigger non-idempotent op-
erations), it can filter them out using unique identifiers in
the records, which are often needed anyway to name corre-
sponding application entities such as web documents. These
functionalities for record I/O (except duplicate removal) are
in library code shared by our applications and applicable to
other file interface implementations at Google. With that,
the same sequence of records, plus rare duplicates, is always
delivered to the record reader.
3. SYSTEM INTERACTIONS
We designed the system to minimize the master’s involve-
ment in all operations. With that background, we now de-
scribe how the client, master, and chunkservers interact to
implement data mutations, atomic record append, and snap-
shot.
3.1 Leases and Mutation Order
Am u t a t i o ni sa no p e r a t i o nt h a tc h a n g e st h ec o n t e n t so r
metadata of a chunk such as a write or an append opera-
tion. Each mutation is performed at all the chunk’s replicas.
We use leases to maintain a consistent mutation order across
replicas. The master grants a chunk lease to one of the repli-
cas, which we call the primary.T h e p r i m a r y p i c k s a s e r i a l
order for all mutations to the chunk. All replicas follow this
order when applying mutations. Thus, the global mutation
order is defined first by the lease grant order chosen by the
master, and within a lease by the serial numbers assigned
by the primary.
The lease mechanism is designed to minimize manage-
ment overhead at the master. A lease has an initial timeout
of 60 seconds. However, as long as the chunk is being mu-
tated, the primary can request and typically receive exten-
sions from the master indefinitely. These extension requests
and grants are piggybacked on the HeartBeat messages reg-
ularly exchanged between the master and all chunkservers.
The master may sometimes try to revoke a lease before it
expires (e.g., when the master wants to disable mutations
on a file that is being renamed). Even if the master loses
communication with a primary, it can safely grant a new
lease to another replica after the old lease expires.
In Figure 2, we illustrate this process by following the
control flow of a write through these numbered steps.
1. The client asks the master which chunkserver holds
the current lease for the chunk and the locations of
the other replicas. If no one has a lease, the master
grants one to a replica it chooses (not shown).
2. The master replies with the identity of the primary and
the locations of the other ( secondary)r e p l i c a s . T h e
client caches this data for future mutations. It needs
to contact the master again only when the primary

Primary
Replica
Secondary
Replica B
Secondary
Replica A
Master
Legend:
Control
Data
3
Client
2
step 14
5
6
6
7
Figure 2: W rite Control and Data Flow
becomes unreachable or replies that it no longer holds
al e a s e .
3. The client pushes the data to all the replicas. A client
can do so in any order. Each chunkserver will store
the data in an internal LRU buffer cache until the
data is used or aged out. By decoupling the data flow
from the control flow, we can improve performance by
scheduling the expensive data flow based on the net-
work topology regardless of which chunkserver is the
primary. Section 3.2 discusses this further.
4. Once all the replicas have acknowledged receiving the
data, the client sends a write request to the primary.
The request identifies the data pushed earlier to all of
the replicas. The primary assigns consecutive serial
numbers to all the mutations it receives, possibly from
multiple clients, which provides the necessary serial-
ization. It applies the mutation to its own local state
in serial number order.
5. The primary forwards the write request to all sec-
ondary replicas. Each secondary replica applies mu-
tations in the same serial number order assigned by
the primary.
6. The secondaries all reply to the primary indicating
that they have completed the operation.
7. The primary replies to the client. Any errors encoun-
tered at any of the replicas are reported to the client.
In case of errors, the write may have succeeded at the
primary and an arbitrary subset of the secondary repli-
cas. (If it had failed at the primary, it would not
have been assigned a serial number and forwarded.)
The client request is considered to have failed, and the
modified region is left in an inconsistent state. Our
client code handles such errors by retrying the failed
mutation. It will make a few attempts at steps (3)
through (7) before falling back to a retry from the be-
ginning of the write.
If a write by the application is large or straddles a chunk
boundary , GFS client code breaks it down into multiple
write operations. They all follow the control flow described
above but may be interleaved with and overwritten by con-
current operations from other clients. Therefore, the shared
file region may end up containing fragments from different
clients, although the replicas will be identical because the in-
dividual operations are completed successfully in the same
order on all replicas. This leaves the file region in consistent
but undefined state as noted in Section 2.7.
3.2 Data Flow
We decouple the flow of data from the flow of control to
use the network efficiently. While control flows from the
client to the primary and then to all secondaries, data is
pushed linearly along a carefully picked chain of chunkservers
in a pipelined fashion. Our goals are to fully utilize each
machine’s network bandwidth, avoid network bottlenecks
and high-latency links, and minimize the latency to push
through all the data.
To fully utilize each machine’s network bandwidth, the
data is pushed linearly along a chain of chunkservers rather
than distributed in some other topology (e.g., tree). Thus,
each machine’s full outbound bandwidth is used to trans-
fer the data as fast as possible rather than divided among
multiple recipients.
To avoid network b ottlenecks and high-latency links (e.g.,
inter-switch links are often both) as much as possible, each
machine forwards the data to the “closest” machine in the
network topology that has not received it. Suppose the
client is pushing data to chunkservers S1 through S4. It
sends the data to the closest chunkserver, say S1. S1 for-
wards it to the closest chunkserver S2 through S4 closest to
S1, say S2. Similarly, S2 forwards it to S3 or S4, whichever
is closer to S2, and so on. Our network topology is simple
enough that “distances” can be accurately estimated from
IP addresses.
Finally, we minimize latency by pipelining the data trans-
fer over TCP connections. Once a chunkserver receives some
data, it starts forwarding immediately. Pipelining is espe-
cially helpful to us because we use a switched network with
full-duplex links. Sending the data immediately does not
reduce the receive rate. Without network congestion, the
ideal elapsed time for transferring B bytes to R replicas is
B/T + RL where T is the network throughput and L is la-
tency to transfer bytes between two machines. Our network
links are typically 100 Mbps ( T ), and L is far below 1 ms.
Therefore, 1 MB can ideally be distributed in about 80 ms.
3.3 Atomic Record Appends
GFS provides an atomic append operation called record
append.I n a t r a d i t i o n a l w r i t e , t h e c l i e n t s p e c i fi e s t h e o ff -
set at which data is to be written. Concurrent writes to
the same region are not serializable: the region may end up
containing data fragments from multiple clients. In a record
append, however, the client specifies only the data. GFS
appends it to the file at least once atomically (i.e., as one
continuous sequence of bytes) at an offset of GFS’s choosing
and returns that offset to the client. This is similar to writ-
ing to a file opened in O
 APPEND mode in Unix without the
race conditions when multiple writers do so concurrently.
Record append is heavily used by our distributed applica-
tions in which many clients on different machines append
to the same file concurrently. Clients would need addi-
tional complicated and expensive synchronization, for ex-
ample through a distributed lock manager, if they do so
with traditional writes. In our workloads, such files often

serve as multiple-producer/single-consumer queues or con-
tain merged results from many different clients.
Record append is a kind of mutation and follows the con-
trol flow in Section 3.1 with only a little extra logic at the
primary. The client pushes the data to all replicas of the
last chunk of the file Then, it sends its request to the pri-
mary. The primary checks to see if appending the record
to the current chunk would cause the chunk to exceed the
maximum size (64 MB). If so, it pads the chunk to the max-
imum size, tells secondaries to do the same, and replies to
the client indicating that the operation should be retried
on the next chunk. (Record append is restricted to be at
most one-fourth of the maximum chunk size to keep worst-
case fragmentation at an acceptable level.) If the record
fits within the maximum size, which is the common case,
the primary appends the data to its replica, tells the secon-
daries to write the data at the exact offset where it has, and
finally replies success to the client.
If a record append fails at any replica, the client retries the
operation. As a result, replicas of the same chunk may con-
tain different data possibly including duplicates of the same
record in whole or in part. GFS does not guarantee that all
replicas are bytewise identical. It only guarantees that the
data is written at least once as an atomic unit. This prop-
erty follows readily from the simple observation that for the
operation to report success, the data must have been written
at the same offset on all replicas of some chunk. Further-
more, after this, all replicas are at least as long as the end
of record and therefore any future record will be assigned a
higher offset or a different chunk even if a different replica
later becomes the primary. In terms of our consistency guar-
antees, the regions in which successful record append opera-
tions have written their data are defined (hence consistent),
whereas intervening regions are inconsistent (hence unde-
fined). Our applications can deal with inconsistent regions
as we discussed in Section 2.7.2.
3.4 Snapshot
The snapshot operation makes a copy of a file or a direc-
tory tree (the “source”) almost instantaneously, while min-
imizing any interruptions of ongoing mutations. Our users
use it to quickly create branch copies of huge data sets (and
often copies of those copies, recursively), or to checkpoint
the current state before experimenting with changes that
can later be committed or rolled back easily.
Like AFS [5], we use standard copy-on-write techniques to
implement snapshots. When the master receives a snapshot
request, it first revokes any outstanding leases on the chunks
in the files it is about to snapshot. This ensures that any
subsequent writes to these chunks will require an interaction
with the master to find the lease holder. This will give the
master an opportunity to create a new copy of the chunk
first.
After the leases have been revoked or have expired, the
master logs the operation to disk. It then applies this log
record to its in-memory state by duplicating the metadata
for the source file or directory tree. The newly created snap-
shot files point to the same chunks as the source files.
The first time a client wants to write to a chunk C after
the snapshot operation, it sends a request to the master to
find the current lease holder. The master notices that the
reference count for chunk C is greater than one. It defers
replying to the client request and instead picks a new chunk
handle C’. It then asks each chunkserver that has a current
replica of C to create a new chunk called C’. By creating
the new chunk on the same chunkservers as the original, we
ensure that the data can be copied locally, not over the net-
work (our disks are about three times as fast as our 100 Mb
Ethernet links). From this point, request handling is no dif-
ferent from that for any chunk: the master grants one of the
replicas a lease on the new chunk C’ and replies to the client,
which can write the chunk normally, not knowing that it has
just been created from an existing chunk.
4. MASTER OPERATION
The master executes all namespace operations. In addi-
tion, it manages chunk replicas throughout the system: it
makes placement decisions, creates new chunks and hence
replicas, and coordinates various system-wide activities to
keep chunks fully replicated, to balance load across all the
chunkservers, and to reclaim unused storage. We now dis-
cuss each of these topics.
4.1 Namespace Management and Locking
Many master operations can take a long time: for exam-
ple, a snapshot operation has to revoke chunkserver leases on
all chunks covered by the snapshot. We do not want to delay
other master operations while they are running. Therefore,
we allow multiple operations to be active and use locks over
regions of the namespace to ensure proper serialization.
Unlike many traditional file systems, GFS does not have
ap e r - d i r e c t o r yd a t as t r u c t u r et h a tl i s t sa l lt h efi l e si nt h a t
directory. Nor does it support aliases for the same file or
directory (i.e, hard or symbolic links in Unix terms). GFS
logically represents its namespace as a lookup table mapping
full pathnames to metadata. With prefix compression, this
table can be efficiently represented in memory. Each node
in the namespace tree (either an absolute file name or an
absolute directory name) has an associated read-write lock.
Each master operation acquires a set of locks before it
runs. Typically, if it involves /d1/d2/.../dn/leaf,i tw i l l
acquire read-locks on the directory names /d1, /d1/d2,. . . ,
/d1/d2/.../dn,a n de i t h e rar e a dl o c ko raw r i t el o c ko nt h e
full pathname /d1/d2/.../dn/leaf.N o t e t h a tleaf may be
afi l eo rd i r e c t o r yd e p e n d i n go nt h eo p e r a t i o n .
We now illustrate how this lo cking mechanism can prevent
afi l e /home/user/foo from being created while /home/user
is being snapshotted to /save/user.T h e s n a p s h o t o p e r -
ation acquires read locks on /home and /save,a n dw r i t e
locks on /home/user and /save/user.T h e fi l e c r e a t i o n a c -
quires read locks on /home and /home/user,a n daw r i t e
lock on /home/user/foo.T h e t w o o p e r a t i o n s w i l l b e s e r i -
alized properly because they try to obtain conflicting locks
on /home/user.F i l e c r e a t i o n d o e s n o t r e q u i r e a w r i t e l o c k
on the parent directory because there is no “directory”, or
inode-like, data structure to be protected from modification.
The read lock on the name is sufficient to protect the parent
directory from deletion.
One nice property of this locking scheme is that it allows
concurrent mutations in the same directory. For example,
multiple file creations can be executed concurrently in the
same directory: each acquires a read lock on the directory
name and a write lock on the file name. The read lock on
the directory name suffices to prevent the directory from
being deleted, renamed, or snapshotted. The write locks on

file names serialize attempts to create a file with the same
name twice.
Since the namespace can have many nodes, read-write lock
objects are allocated lazily and deleted once they are not in
use. Also, locks are acquired in a consistent total order
to prevent deadlock: they are first ordered by level in the
namespace tree and lexicographically within the same level.
4.2 Replica Placement
AG F Sc l u s t e ri sh i g h l yd i s t r i b u t e da tm o r el e v e l st h a n
one. It typically has hundreds of chunkservers spread across
many machine racks. These chunkservers in turn may be
accessed from hundreds of clients from the same or different
racks. Communication between two machines on different
racks may cross one or more network switches. Addition-
ally, bandwidth into or out of a rack may be less than the
aggregate bandwidth of all the machines within the rack.
Multi-level distribution presents a unique challenge to dis-
tribute data for scalability, reliability, and availability.
The chunk replica placement policy serves two purposes:
maximize data reliability and availability, and maximize net-
work bandwidth utilization. For both, it is not enough to
spread replicas across machines, which only guards against
disk or machine failures and fully utilizes each machine’s net-
work bandwidth. We must also spread chunk replicas across
racks. This ensures that some replicas of a chunk will sur-
vive and remain available even if an entire rack is damaged
or offline (for example, due to failure of a shared resource
like a network switch or power circuit). It also means that
traffic, especially reads, for a chunk can exploit the aggre-
gate bandwidth of multiple racks. On the other hand, write
traffic has to flow through multiple racks, a tradeoff we make
willingly.
4.3 Creation, Re-replication, Rebalancing
Chunk replicas are created for three reasons: chunk cre-
ation, re-replication, and rebalancing.
When the master creates ac h u n k ,i tc h o o s e sw h e r et o
place the initially empty replicas. It considers several fac-
tors. (1) We want to place new replicas on chunkservers with
below-average disk space utilization. Over time this will
equalize disk utilization across chunkservers. (2) We want to
limit the number of “recent” creations on each chunkserver.
Although creation itself is cheap, it reliably predicts immi-
nent heavy write traffic because chunks are created when de-
manded by writes, and in our append-once-read-many work-
load they typically become practically read-only once they
have been completely written. (3) As discussed above, we
want to spread replicas of a chunk across racks.
The master re-replicates ac h u n ka ss o o na st h en u m b e r
of available replicas falls below a user-specified goal. This
could happen for various reasons: a chunkserver becomes
unavailable, it reports that its replica may be corrupted, one
of its disks is disabled because of errors, or the replication
goal is increased. Each chunk that needs to be re-replicated
is prioritized based on several factors. One is how far it is
from its replication goal. For example, we give higher prior-
ity to a chunk that has lost two replicas than to a chunk that
has lost only one. In addition, we prefer to first re-replicate
chunks for live files as opposed to chunks that belong to re-
cently deleted files (see Section 4.4). Finally, to minimize
the impact of failures on running applications, we boost the
priority of any chunk that is blocking client progress.
The master picks the highest priority chunk and “clones”
it by instructing some chunkserver to copy the chunk data
directly from an existing valid replica. The new replica is
placed with goals similar to those for creation: equalizing
disk space utilization, limiting active clone operations on
any single chunkserver, and spreading replicas across racks.
To keep cloning traffic from overwhelming client traffic, the
master limits the numbers of active clone operations both
for the cluster and for each chunkserver. Additionally, each
chunkserver limits the amount of bandwidth it spends on
each clone operation by throttling its read requests to the
source chunkserver.
Finally, the master rebalances replicas periodically: it ex-
amines the current replica distribution and moves replicas
for better disk space and load balancing. Also through this
process, the master gradually fills up a new chunkserver
rather than instantly swamps it with new chunks and the
heavy write traffic that comes with them. The placement
criteria for the new replica are similar to those discussed
above. In addition, the master must also choose which ex-
isting replica to remove. In general, it prefers to remove
those on chunkservers with below-average free space so as
to equalize disk space usage.
4.4 Garbage Collection
After a file is deleted, GFS does not immediately reclaim
the available physical storage. It does so only lazily during
regular garbage collection at both the file and chunk levels.
We find that this approach makes the system much simpler
and more reliable.
4.4.1 Mechanism
When a file is deleted by the application, the master logs
the deletion immediately just like other changes. However
instead of reclaiming resources immediately, the file is just
renamed to a hidden name that includes the deletion times-
tamp. During the master’s regular scan of the file system
namespace, it removes any such hidden files if they have ex-
isted for more than three days (the interval is configurable).
Until then, the file can still be read under the new, special
name and can be undeleted by renaming it back to normal.
When the hidden file is removed from the namespace, its in-
memory metadata is erased. This effectively severs its links
to all its chunks.
In a similar regular scan of the chunk namespace, the
master identifies orphaned chunks (i.e., those not reachable
from any file) and erases the metadata for those chunks. In
a HeartBeat message regularly exchanged with the master,
each chunkserver reports a subset of the chunks it has, and
the master replies with the identity of all chunks that are no
longer present in the master’s metadata. The chunkserver
is free to delete its replicas of such chunks.
4.4.2 Discussion
Although distributed garbage collection is a hard problem
that demands complicated solutions in the context of pro-
gramming languages, it is quite simple in our case. We can
easily identify all references to chunks: they are in the file-
to-chunk mappings maintained exclusively by the master.
We can also easily identify all the chunk replicas: they are
Linux files under designated directories on each chunkserver.
Any such replica not known to the master is “garbage.”

The garbage collection approach to storage reclamation
offers several advantages over eager deletion. First, it is
simple and reliable in a large-scale distributed system where
component failures are common. Chunk creation may suc-
ceed on some chunkservers but not others, leaving replicas
that the master does not know exist. Replica deletion mes-
sages may be lost, and the master has to remember to resend
them across failures, both its own and the chunkserver’s.
Garbage collection provides a uniform and dependable way
to clean up any replicas not known to be useful. Second,
it merges storage reclamation into the regular background
activities of the master, such as the regular scans of names-
paces and handshakes with chunkservers. Thus, it is done
in batches and the cost is amortized. Moreover, it is done
only when the master is relatively free. The master can re-
spond more promptly to client requests that demand timely
attention. Third, the delay in reclaiming storage provides a
safety net against accidental, irreversible deletion.
In our experience, the main disadvantage is that the delay
sometimes hinders user effort to fine tune usage when stor-
age is tight. Applications that repeatedly create and delete
temporary files may not be able to reuse the storage right
away. We address these issues by expediting storage recla-
mation if a deleted file is explicitly deleted again. We also
allow users to apply different replication and reclamation
policies to different parts of the namespace. F or example,
users can specify that all the chunks in the files within some
directory tree are to be stored without replication, and any
deleted files are immediately and irrevocably removed from
the file system state.
4.5 Stale Replica Detection
Chunk replicas may become stale if a chunkserver fails
and misses mutations to the chunk while it is down. For
each chunk, the master maintains a chunk version number
to distinguish between up-to-date and stale replicas.
Whenever the master grants a new lease on a chunk, it
increases the chunk version number and informs the up-to-
date replicas. The master and these replicas all record the
new version number in their persistent state. This occurs
before any client is notified and therefore before it can start
writing to the chunk. If another replica is currently unavail-
able, its chunk version number will not be advanced. The
master will detect that this chunkserver has a stale replica
when the chunkserver restarts and reports its set of chunks
and their associated version numbers. If the master sees a
version number greater than the one in its records, the mas-
ter assumes that it failed when granting the lease and so
takes the higher version to be up-to-date.
The master removes stale replicas in its regular garbage
collection. Before that, it effectively considers a stale replica
not to exist at all when it replies to client requests for chunk
information. As another safeguard, the master includes
the chunk version number when it informs clients which
chunkserver holds a lease on a chunk or when it instructs
ac h u n k s e r v e rt or e a dt h ec h u n kf r o ma n o t h e rc h u n k s e r v e r
in a cloning operation. The client or the chunkserver verifies
the version number when it performs the operation so that
it is always accessing up-to-date data.
5. FAULT TOLERANCE AND DIAGNOSIS
One of our greatest challenges in designing the system is
dealing with frequent component failures. The quality and
quantity of components together make these problems more
the norm than the exception: we cannot completely trust
the machines, nor can we completely trust the disks. Com-
ponent failures can result in an unavailable system or, worse,
corrupted data. We discuss how we meet these challenges
and the tools we have built into the system to diagnose prob-
lems when they inevitably occur.
5.1 High Availability
Among hundreds of servers in a GFS cluster, some are
bound to be unavailable at any given time. W e keep the
overall system highly available with two simple yet effective
strategies: fast recovery and replication.
5.1.1 Fast Recovery
Both the master and the chunkserver are designed to re-
store their state and start in seconds no matter how they
terminated. In fact, we do not distinguish between normal
and abnormal termination; servers are routinely shut down
just by killing the process. Clients and other servers experi-
ence a minor hiccup as they time out on their outstanding
requests, reconnect to the restarted server, and retry. Sec-
tion 6.2.2 reports observed startup times.
5.1.2 Chunk Replication
As discussed earlier, each chunk is replicated on multiple
chunkservers on different racks. Users can specify different
replication levels for different parts of the file namespace.
The default is three. The master clones existing replicas as
needed to keep each chunk fully replicated as chunkservers
go offline or detect corrupted replicas through checksum ver-
ification (see Section 5.2). Although replication has served
us well, we are exploring other forms of cross-server redun-
dancy such as parity or erasure codes for our increasing read-
only storage requirements. We expect that it is challenging
but manageable to implement these more complicated re-
dundancy schemes in our very loosely coupled system be-
cause our traffic is dominated by appends and reads rather
than small random writes.
5.1.3 Master Replication
The master state is replicated for reliability. Its operation
log and checkpoints are replicated on multiple machines. A
mutation to the state is considered committed only after
its log record has been flushed to disk locally and on all
master replicas. For simplicity, one master process remains
in charge of all mutations as well as background activities
such as garbage collection that change the system internally.
When it fails, it can restart almost instantly. If its machine
or disk fails, monitoring infrastructure outside GFS starts a
new master process elsewhere with the replicated operation
log. Clients use only the canonical name of the master (e.g.
gfs-test), which is a DNS alias that can be changed if the
master is relocated to another machine.
Moreover, “shadow” masters provide read-only access to
the file system even when the primary master is down. They
are shadows, not mirrors, in that they may lag the primary
slightly, typically fractions of a second. They enhance read
availability for files that are not being actively mutated or
applications that do not mind getting slightly stale results.
In fact, since file content is read from chunkservers, appli-
cations do not observe stale file content. What could be

stale within short windows is file metadata, like directory
contents or access control information.
To keep itself informed, a shadow master reads a replica of
the growing operation log and applies the same sequence of
changes to its data structures exactly as the primary does.
Like the primary, it polls chunkservers at startup (and infre-
quently thereafter) to locate chunk replicas and exchanges
frequent handshake messages with them to monitor their
status. It depends on the primary master only for replica
location updates resulting from the primary’s decisions to
create and delete replicas.
5.2 Data Integrity
Each chunkserver uses checksumming to detect corruption
of stored data. Given that a GFS cluster often has thousands
of disks on hundreds of machines, it regularly experiences
disk failures that cause data corruption or loss on both the
read and write paths. (See Section 7 for one cause.) We
can recover from corruption using other chunk replicas, but
it would be impractical to detect corruption by comparing
replicas across chunkservers. Moreover, divergent replicas
may be legal: the semantics of GFS mutations, in particular
atomic record append as discussed earlier, does not guar-
antee identical replicas. Therefore, each chunkserver must
independently verify the integrity of its own copy by main-
taining checksums.
Ac h u n ki sb r o k e nu pi n t o6 4K Bb l o c k s . E a c hh a sac o r r e -
sponding 32 bit checksum. Like other metadata, checksums
are kept in memory and stored persistently with logging,
separate from user data.
For reads, the chunkserver verifies the checksum of data
blocks that overlap the read range before returning any data
to the requester, whether a client or another chunkserver.
Therefore chunkservers will not propagate corruptions to
other machines. If a block does not match the recorded
checksum, the chunkserver returns an error to the requestor
and reports the mismatch to the master. In response, the
requestor will read from other replicas, while the master
will clone the chunk from another replica. After a valid new
replica is in place, the master instructs the chunkserver that
reported the mismatch to delete its replica.
Checksumming has little effect on read performance for
several reasons. Since most of our reads span at least a
few blocks, we need to read and checksum only a relatively
small amount of extra data for verification. GFS client code
further reduces this overhead by trying to align reads at
checksum block boundaries. Moreover, checksum lookups
and comparison on the chunkserver are done without any
I/O, and checksum calculation can often be overlapped with
I/Os.
Checksum computation is heavily optimized for writes
that append to the end of a chunk (as opposed to writes
that overwrite existing data) because they are dominant in
our workloads. We just incrementally update the check-
sum for the last partial checksum block, and compute new
checksums for any brand new checksum blocks filled by the
append. Even if the last partial checksum block is already
corrupted and we fail to detect it now, the new checksum
value will not match the stored data, and the corruption will
be detected as usual when the block is next read.
In contrast, if a write overwrites an existing range of the
chunk, we must read and verify the first and last blocks of
the range being overwritten, then perform the write, and
finally compute and record the new checksums. If we do
not verify the first and last blocks before overwriting them
partially, the new checksums may hide corruption that exists
in the regions not being overwritten.
During idle periods, chunkservers can scan and verify the
contents of inactive chunks. This allows us to detect corrup-
tion in chunks that are rarely read. Once the corruption is
detected, the master can create a new uncorrupted replica
and delete the corrupted replica. This prevents an inactive
but corrupted chunk replica from fooling the master into
thinking that it has enough valid replicas of a chunk.
5.3 Diagnostic Tools
Extensive and detailed diagnostic logging has helped im-
measurably in problem isolation, debugging, and perfor-
mance analysis, while incurring only a minimal cost. With-
out logs, it is hard to understand transient, non-repeatable
interactions between machines. GFS servers generate di-
agnostic logs that record many significant events (such as
chunkservers going up and down) and all RPC requests and
replies. These diagnostic logs can be freely deleted without
affecting the correctness of the system. However, we try to
keep these logs around as far as space permits.
The RPC logs include the exact requests and responses
sent on the wire, except for the file data being read or writ-
ten. By matching requests with replies and collating RPC
records on different machines, we can reconstruct the en-
tire interaction history to diagnose a problem. The logs also
serve as traces for load testing and performance analysis.
The performance impact of logging is minimal (and far
outweighed by the benefits) because these logs are written
sequentially and asynchronously. The most recent events
are also kept in memory and available for continuous online
monitoring.
6. MEASUREMENTS
In this section we present a few micro-benchmarks to illus-
trate the bottlenecks inherent in the GFS architecture and
implementation, and also some numbers from real clusters
in use at Google.
6.1 Micro-benchmarks
We measured p erformance on a GFS cluster consisting
of one master, two master replicas, 16 chunkservers, and
16 clients. Note that this configuration was set up for ease
of testing. Typical clusters have hundreds of chunkservers
and hundreds of clients.
All the machines are configured with dual 1.4 GHz PIII
processors, 2 GB of memory, two 80 GB 5400 rpm disks, and
a1 0 0M b p sf u l l - d u p l e xE t h e r n e tc o n n e c t i o nt oa nH P2 5 2 4
switch. All 19 GFS server machines are connected to one
switch, and all 16 client machines to the other. The two
switches are connected with a 1 Gbps link.
6.1.1 Reads
N clients read simultaneously from the file system. Each
client reads a randomly selected 4 MB region from a 320 GB
file set. This is repeated 256 times so that each client ends
up reading 1 GB of data. The chunkservers taken together
have only 32 GB of memory, so we expect at most a 10% hit
rate in the Linux buffer cache. Our results should be close
to cold cache results.

Figure 3(a) shows the aggregate read rate for N clients
and its theoretical limit. The limit peaks at an aggregate of
125 MB/s when the 1 Gbps link between the two switches
is saturated, or 12.5 MB/s per client when its 100 Mbps
network interface gets saturated, whichever applies. The
observed read rate is 10 MB/s, or 80% of the per-client
limit, when just one client is reading. The aggregate read
rate reaches 94 MB/s, about 75% of the 125 MB/s link limit,
for 16 readers, or 6 MB/s per client. The efficiency drops
from 80% to 75% because as the number of readers increases,
so does the probability that multiple readers simultaneously
read from the same chunkserver.
6.1.2 Writes
N clients write simultaneously to N distinct files. Each
client writes 1 GB of data to a new file in a series of 1 MB
writes. The aggregate write rate and its theoretical limit are
shown in Figure 3(b). The limit plateaus at 67 MB/s be-
cause we need to write each byte to 3 of the 16 chunkservers,
each with a 12.5 MB/s input connection.
The write rate for one client is 6.3 MB/s, about half of the
limit. The main culprit for this is our network stack. It does
not interact very well with the pipelining scheme we use for
pushing data to chunk replicas. Delays in propagating data
from one replica to another reduce the overall write rate.
Aggregate write rate reaches 35 MB/s for 16 clients (or
2.2 MB/s per client), about half the theoretical limit. As in
the case of reads, it becomes more likely that multiple clients
write concurrently to the same chunkserver as the number
of clients increases. Moreover, collision is more likely for 16
writers than for 16 readers because each write involves three
different replicas.
Writes are slower than we would like. In practice this has
not been a major problem because even though it increases
the latencies as seen by individual clients, it does not sig-
nificantly affect the aggregate write bandwidth delivered by
the system to a large number of clients.
6.1.3 Record Appends
Figure 3(c) shows record append performance. N clients
append simultaneously to a single file. Performance is lim-
ited by the network bandwidth of the chunkservers that
store the last chunk of the file, independent of the num-
ber of clients. It starts at 6.0 MB/s for one client and drops
to 4.8 MB/s for 16 clients, mostly due to congestion and
variances in network transfer rates seen by different clients.
Our applications tend to produce multiple such files con-
currently. In other words, N clients append to M shared
files simultaneously where both N and M are in the dozens
or hundreds. Therefore, the chunkserver network congestion
in our experiment is not a significant issue in practice be-
cause a client can make progress on writing one file while
the chunkservers for another file are busy.
6.2 Real World Clusters
We now examine two clusters in use within Go ogle that
are representative of several others like them. Cluster A is
used regularly for research and development by over a hun-
dred engineers. A typical task is initiated by a human user
and runs up to several hours. It reads through a few MBs
to a few TBs of data, transforms or analyzes the data, and
writes the results back to the cluster. Cluster B is primarily
used for production data processing. The tasks last much
Cluster
 A
 B
Chunkservers
 342
 227
Available disk space
 72 TB
 180 TB
Used disk space
 55 TB
 155 TB
Number of Files
 735 k
 737 k
Number of Dead files
 22 k
 232 k
Number of Chunks
 992 k
 1550 k
Metadata at chunkservers
 13 GB
 21 GB
Metadata at master
 48 MB
 60 MB
Table 2: Characteristics of two GFS clusters
longer and continuously generate and process multi-TB data
sets with only occasional human intervention. In both cases,
as i n g l e“ t a s k ”c o n s i s t so fm a n yp r o c e s s e so nm a n ym a c h i n e s
reading and writing many files simultaneously.
6.2.1 Storage
As shown by the first five entries in the table, both clusters
have hundreds of chunkservers, support many TBs of disk
space, and are fairly but not completely full. “Used space”
includes all chunk replicas. Virtually all files are replicated
three times. Therefore, the clusters store 18 TB and 52 TB
of file data respectively.
The two clusters have similar numbers of files, though B
has a larger proportion of dead files, namely files which were
deleted or replaced by a new version but whose storage have
not yet been reclaimed. It also has more chunks because its
files tend to be larger.
6.2.2 Metadata
The chunkservers in aggregate store tens of GBs of meta-
data, mostly the checksums for 64 KB blocks of user data.
The only other metadata kept at the chunkservers is the
chunk version number discussed in Section 4.5.
The metadata kept at the master is much smaller, only
tens of MBs, or about 100 bytes per file on average. This
agrees with our assumption that the size of the master’s
memory does not limit the system’s capacity in practice.
Most of the per-file metadata is the file names stored in a
prefix-compressed form. Other metadata includes file own-
ership and permissions, mapping from files to chunks, and
each chunk’s current version. In addition, for each chunk we
store the current replica locations and a reference count for
implementing copy-on-write.
Each individual server, both chunkservers and the master,
has only 50 to 100 MB of metadata. Therefore recovery is
fast: it takes only a few seconds to read this metadata from
disk before the server is able to answer queries. However, the
master is somewhat hobbled for a period – typically 30 to
60 seconds – until it has fetched chunk location information
from all chunkservers.
6.2.3 Read and Write Rates
Table 3 shows read and write rates for various time p e-
riods. Both clusters had been up for about one week when
these measurements were taken. (The clusters had been
restarted recently to upgrade to a new version of GFS.)
The average write rate was less than 30 MB/s since the
restart. When we took these measurements, B was in the
middle of a burst of write activity generating about 100 MB/s
of data, which produced a 300 MB/s network load because
writes are propagated to three replicas.

0 5 10 15
Number of clients N
0
50
100Read rate (MB/s)
Network limit
Aggregate read rate
(a) Reads
0 5 10 15
Number of clients N
0
20
40
60Write rate (MB/s)
Network limit
Aggregate write rate
(b) Writes
0 5 10 15
Number of clients N
0
5
10Append rate (MB/s)
Network limit
Aggregate append rate
(c) Record appends
Figure 3: Aggregate Throughputs. Top curves show theoretical limits imp osed by our network top ology. Bottom curves
show measured throughputs. They have error bars that show 95% confidence intervals, which are illegible in some cases
because of low variance in measurements.
Cluster
 A
 B
Read rate (last minute)
 583 MB/s
 380 MB/s
Read rate (last hour)
 562 MB/s
 384 MB/s
Read rate (since restart)
 589 MB/s
 49 MB/s
Wr i te r ate (l as t m i nute)
 1M B / s
 101 MB/s
Wr i te r ate (l as t hour )
 2M B / s
 117 MB/s
Wr i te r ate (s i nce r es tar t)
 25 MB/s
 13 MB/s
Master ops (last minute)
 325 Ops/s
 533 Ops/s
Master ops (last hour)
 381 Ops/s
 518 Ops/s
Master ops (since restart)
 202 Ops/s
 347 Ops/s
Table 3: Performance Metrics for Two GFS Clusters
The read rates were much higher than the write rates.
The total workload consists of more reads than writes as we
have assumed. Both clusters were in the middle of heavy
read activity. In particular, A had been sustaining a read
rate of 580 MB/s for the preceding week. Its network con-
figuration can support 750 MB/s, so it was using its re-
sources efficiently. Cluster B can support peak read rates of
1300 MB/s, but its applications were using just 380 MB/s.
6.2.4 Master Load
Table 3 also shows that the rate of op erations sent to the
master was around 200 to 500 operations per second. The
master can easily keep up with this rate, and therefore is
not a bottleneck for these workloads.
In an earlier version of GFS, the master was occasionally
ab o t t l e n e c kf o rs o m ew o r k l o a d s . I ts p e n tm o s to fi t st i m e
sequentially scanning through large directories (which con-
tained hundreds of thousands of files) looking for particular
files. We have since changed the master data structures to
allow efficient binary searches through the namespace. It
can now easily support many thousands of file accesses per
second. If necessary, we could speed it up further by placing
name lookup caches in front of the namespace data struc-
tures.
6.2.5 Recovery Time
After a chunkserver fails, some chunks will become under-
replicated and must be cloned to restore their replication
levels. The time it takes to restore all such chunks depends
on the amount of resources. In one experiment, we killed a
single chunkserver in cluster B. The chunkserver had about
15,000 chunks containing 600 GB of data. To limit the im-
pact on running applications and provide leeway for schedul-
ing decisions, our default parameters limit this cluster to
91 concurrent clonings (40% of the number of chunkservers)
where each clone operation is allowed to consume at most
6.25 MB/s (50 Mbps). All chunks were restored in 23.2 min-
utes, at an effective replication rate of 440 MB/s.
In another experiment, we killed two chunkservers each
with roughly 16,000 chunks and 660 GB of data. This double
failure reduced 266 chunks to having a single replica. These
266 chunks were cloned at a higher priority, and were all
restored to at least 2x replication within 2 minutes, thus
putting the cluster in a state where it could tolerate another
chunkserver failure without data loss.
6.3 Workload Breakdown
In this section, we present a detailed breakdown of the
workloads on two GFS clusters comparable but not identi-
cal to those in Section 6.2. Cluster X is for research and
development while cluster Y is for production data process-
ing.
6.3.1 Methodology and Caveats
These results include only client originated requests so
that they reflect the workload generated by our applications
for the file system as a whole. They do not include inter-
server requests to carry out client requests or internal back-
ground activities, such as forwarded writes or rebalancing.
Statistics on I/O operations are based on information
heuristically reconstructed from actual RPC requests logged
by GFS servers. For example, GFS client code may break a
read into multiple RPCs to increase parallelism, from which
we infer the original read. Since our access patterns are
highly stylized, we expect any error to be in the noise. Ex-
plicit logging by applications might have provided slightly
more accurate data, but it is logistically impossible to re-
compile and restart thousands of running clients to do so
and cumbersome to collect the results from as many ma-
chines.
One should be careful not to overly generalize from our
workload. Since Google completely controls both GFS and
its applications, the applications tend to be tuned for GFS,
and conversely GFS is designed for these applications. Such
mutual influence may also exist between general applications

Operation
 Read
 Wr i te
 Record Append
Cluster
 XY
 XY
 XY
0K
 0.4 2.6
 00
 00
1B..1K
 0.1 4.1
 6.6 4.9
 0.2 9.2
1K..8K
 65.2 38.5
 0.4 1.0
 18.9 15.2
8K..64K
 29.9 45.1
 17.8 43.0
 78.0 2.8
64K..128K
 0.1 0.7
 2.3 1.9
 < .1 4.3
128K..256K
 0.2 0.3
 31.6 0.4
 < .1 10.6
256K..512K
 0.1 0.1
 4.2 7.7
 < .1 31.2
512K..1M
 3.9 6.9
 35.5 28.7
 2.2 25.5
1M..inf
 0.1 1.8
 1.5 12.3
 0.7 2.2
Table 4: Op erations Breakdown by Size (%). For
reads, the size is the amount of data actually read and trans-
ferred, rather than the amount requested.
and file systems, but the effect is likely more pronounced in
our case.
6.3.2 Chunkserver Workload
Table 4 shows the distribution of op erations by size. Read
sizes exhibit a bimodal distribution. The small reads (un-
der 64 KB) come from seek-intensive clients that look up
small pieces of data within huge files. The large reads (over
512 KB) come from long sequential reads through entire
files.
As i g n i fi c a n tn u m b e ro fr e a d sr e t u r nn od a t aa ta l li nc l u s -
ter Y. Our applications, especially those in the production
systems, often use files as producer-consumer queues. Pro-
ducers append concurrently to a file while a consumer reads
the end of file. Occasionally, no data is returned when the
consumer outpaces the producers. Cluster X shows this less
often because it is usually used for short-lived data analysis
tasks rather than long-lived distributed applications.
Write sizes also exhibit a bimo dal distribution. The large
writes (over 256 KB) typically result from significant buffer-
ing within the writers. Writers that buffer less data, check-
point or synchronize more often, or simply generate less data
account for the smaller writes (under 64 KB).
As for record appends, cluster Y sees a much higher per-
centage of large record appends than cluster X does because
our production systems, which use cluster Y, are more ag-
gressively tuned for GFS.
Table 5 shows the total amount of data transferred in op-
erations of various sizes. For all kinds of operations, the
larger operations (over 256 KB) generally account for most
of the bytes transferred. Small reads (under 64 KB) do
transfer a small but significant portion of the read data be-
cause of the random seek workload.
6.3.3 Appends versus Writes
Record appends are heavily used especially in our pro-
duction systems. For cluster X, the ratio of writes to record
appends is 108:1 by bytes transferred and 8:1 by operation
counts. For cluster Y, used by the production systems, the
ratios are 3.7:1 and 2.5:1 respectively. Moreover, these ra-
tios suggest that for both clusters record appends tend to
be larger than writes. F or cluster X, however, the overall
usage of record append during the measured period is fairly
low and so the results are likely skewed by one or two appli-
cations with particular buffer size choices.
As expected, our data mutation workload is dominated
by appending rather than overwriting. We measured the
amount of data overwritten on primary replicas. This ap-
Operation
 Read
 Wr i te
 Record Append
Cluster
 XY
 XY
 XY
1B..1K
 < .1 < .1
 < .1 < .1
 < .1 < .1
1K..8K
 13.8 3.9
 < .1 < .1
 < .1 0.1
8K..64K
 11.4 9.3
 2.4 5.9
 2.3 0.3
64K..128K
 0.3 0.7
 0.3 0.3
 22.7 1.2
128K..256K
 0.8 0.6
 16.5 0.2
 < .1 5.8
256K..512K
 1.4 0.3
 3.4 7.7
 < .1 38.4
512K..1M
 65.9 55.1
 74.1 58.0
 .1 46.8
1M..inf
 6.4 30.1
 3.3 28.0
 53.9 7.4
Table 5: Bytes Transferred Breakdown by Op era-
tion Size (%). For reads, the size is the amount of data
actually read and transferred, rather than the amount re-
quested. The two may differ if the read attempts to read
beyond end of file, which by design is not uncommon in our
workloads.
Cluster
 XY
Open
 26.1 16.3
Delete
 0.7 1.5
FindLocation
 64.3 65.8
FindLeaseHolder
 7.8 13.4
FindMatchingFiles
 0.6 2.2
All other combined
 0.5 0.8
Table 6: Master Requests Breakdown by Typ e (%)
proximates the case where a client deliberately overwrites
previous written data rather than appends new data. For
cluster X, overwriting accounts for under 0.0001% of bytes
mutated and under 0.0003% of mutation operations. For
cluster Y, the ratios are both 0.05%. Although this is minute,
it is still higher than we expected. It turns out that most
of these overwrites came from client retries due to errors or
timeouts. They are not part of the workload per se but a
consequence of the retry mechanism.
6.3.4 Master Workload
Table 6 shows the breakdown by typ e of requests to the
master. Most requests ask for chunk locations ( FindLo-
cation)f o rr e a d sa n dl e a s eh o l d e ri n f o r m a t i o n(FindLease-
Locker)f o rd a t am u t a t i o n s .
Clusters X and Y see significantly different numbers of
Delete requests because cluster Y stores production data
sets that are regularly regenerated and replaced with newer
versions. Some of this difference is further hidden in the
difference in Open requests because an old version of a file
may be implicitly deleted by being opened for write from
scratch (mode “w” in Unix open terminology).
FindMatchingFiles is a pattern matching request that sup-
ports “ls” and similar file system operations. Unlike other
requests for the master, it may process a large part of the
namespace and so may be expensive. Cluster Y sees it much
more often because automated data processing tasks tend to
examine parts of the file system to understand global appli-
cation state. In contrast, cluster X’s applications are under
more explicit user control and usually know the names of all
needed files in advance.
7. EXPERIENCES
In the process of building and deploying GFS, we have
experienced a variety of issues, some operational and some
technical.

Initially, GFS was conceived as the backend file system
for our production systems. Over time, the usage evolved
to include research and development tasks. It started with
little support for things like permissions and quotas but now
includes rudimentary forms of these. While production sys-
tems are well disciplined and controlled, users sometimes
are not. More infrastructure is required to keep users from
interfering with one another.
Some of our biggest problems were disk and Linux related.
Many of our disks claimed to the Linux driver that they
supported a range of IDE protocol versions but in fact re-
sponded reliably only to the more recent ones. Since the pro-
tocol versions are very similar, these drives mostly worked,
but occasionally the mismatches would cause the drive and
the kernel to disagree about the drive’s state. This would
corrupt data silently due to problems in the kernel. This
problem motivated our use of checksums to detect data cor-
ruption, while concurrently we modified the kernel to handle
these protocol mismatches.
Earlier we had some problems with Linux 2.2 kernels due
to the cost of fsync().I t s c o s t i s p r o p o r t i o n a l t o t h e s i z e
of the file rather than the size of the modified portion. This
was a problem for our large operation logs especially before
we implemented checkpointing. We worked around this for
at i m eb yu s i n gs y n c h r o n o u sw r i t e sa n de v e n t u a l l ym i g r a t e d
to Linux 2.4.
Another Linux problem was a single reader-writer lock
which any thread in an address space must hold when it
pages in from disk (reader lock) or modifies the address
space in an mmap() call (writer lock). We saw transient
timeouts in our system under light load and looked hard for
resource bottlenecks or sporadic hardware failures. Even-
tually, we found that this single lock blocked the primary
network thread from mapping new data into memory while
the disk threads were paging in previously mapped data.
Since we are mainly limited by the network interface rather
than by memory copy bandwidth, we worked around this by
replacing mmap() with pread() at the cost of an extra copy.
Despite occasional problems, the availability of Linux code
has helped us time and again to explore and understand
system behavior. When appropriate, we improve the kernel
and share the changes with the open source community.
8. RELATED WORK
Like other large distributed file systems such as AFS [5],
GFS provides a location independent namespace which en-
ables data to be moved transparently for load balance or
fault tolerance. Unlike AFS, GFS spreads a file’s data across
storage servers in a way more akin to xFS [1] and Swift [3] in
order to deliver aggregate performance and increased fault
tolerance.
As disks are relatively cheap and replication is simpler
than more sophisticated RAID [9] approaches, GFS cur-
rently uses only replication for redundancy and so consumes
more raw storage than xFS or Swift.
In contrast to systems like AFS, xFS, Frangipani [12], and
Intermezzo [6], GFS does not provide any caching below the
file system interface. Our target workloads have little reuse
within a single application run because they either stream
through a large data set or randomly seek within it and read
small amounts of data each time.
Some distributed file systems like Frangipani, xFS, Min-
nesota’s GFS[11] and GPFS [10] remove the centralized server
and rely on distributed algorithms for consistency and man-
agement. We opt for the centralized approach in order to
simplify the design, increase its reliability, and gain flexibil-
ity. In particular, a centralized master makes it much easier
to implement sophisticated chunk placement and replication
policies since the master already has most of the relevant
information and controls how it changes. We address fault
tolerance by keeping the master state small and fully repli-
cated on other machines. Scalability and high availability
(for reads) are currently provided by our shadow master
mechanism. Updates to the master state are made persis-
tent by appending to a write-ahead log. Therefore we could
adapt a primary-copy scheme like the one in Harp [7] to pro-
vide high availability with stronger consistency guarantees
than our current scheme.
We are addressing a problem similar to Lustre [8] in terms
of delivering aggregate performance to a large number of
clients. However, we have simplified the problem signifi-
cantly by focusing on the needs of our applications rather
than building a POSIX-compliant file system. Additionally,
GFS assumes large number of unreliable components and so
fault tolerance is central to our design.
GFS most closely resembles the NASD architecture [4].
While the NASD architecture is based on network-attached
disk drives, GFS uses commodity machines as chunkservers,
as done in the NASD prototype. Unlike the NASD work,
our chunkservers use lazily allocated fixed-size chunks rather
than variable-length objects. Additionally, GFS implements
features such as rebalancing, replication, and recovery that
are required in a production environment.
Unlike Minnesota’s GFS and NASD, we do not seek to
alter the model of the storage device. We focus on ad-
dressing day-to-day data processing needs for complicated
distributed systems with existing commodity components.
The producer-consumer queues enabled by atomic record
appends address a similar problem as the distributed queues
in River [2]. While River uses memory-based queues dis-
tributed across machines and careful data flow control, GFS
uses a persistent file that can be appended to concurrently
by many producers. The River model supports m-to-n dis-
tributed queues but lacks the fault tolerance that comes with
persistent storage, while GFS only supports m-to-1 queues
efficiently. Multiple consumers can read the same file, but
they must coordinate to partition the incoming load.
9. CONCLUSIONS
The Google File System demonstrates the qualities es-
sential for supporting large-scale data processing workloads
on commodity hardware. While some design decisions are
specific to our unique setting, many may apply to data pro-
cessing tasks of a similar magnitude and cost consciousness.
We started by reexamining traditional file system assump-
tions in light of our current and anticipated application
workloads and technological environment. Our observations
have led to radically different points in the design space.
We treat comp onent failures as the norm rather than the
exception, optimize for huge files that are mostly appended
to (perhaps concurrently) and then read (usually sequen-
tially), and both extend and relax the standard file system
interface to improve the overall system.
Our system provides fault tolerance by constant moni-
toring, replicating crucial data, and fast and automatic re-
covery. Chunk replication allows us to tolerate chunkserver

failures. The frequency of these failures motivated a novel
online repair mechanism that regularly and transparently re-
pairs the damage and compensates for lost replicas as soon
as possible. Additionally, we use checksumming to detect
data corruption at the disk or IDE subsystem level, which
becomes all too common given the number of disks in the
system.
Our design delivers high aggregate throughput to many
concurrent readers and writers performing a variety of tasks.
We achieve this by separating file system control, which
passes through the master, from data transfer, which passes
directly between chunkservers and clients. Master involve-
ment in common operations is minimized by a large chunk
size and by chunk leases, which delegates authority to pri-
mary replicas in data mutations. This makes possible a sim-
ple, centralized master that does not become a bottleneck.
We b elieve that improvements in our networking stack will
lift the current limitation on the write throughput seen by
an individual client.
GFS has successfully met our storage needs and is widely
used within Google as the storage platform for research and
development as well as production data processing. It is an
important tool that enables us to continue to innovate and
attack problems on the scale of the entire web.
ACKNOWLEDGMENTS
We wish to thank the following p eople for their contributions
to the system or the paper. Brain Bershad (our shepherd)
and the anonymous reviewers gave us valuable comments
and suggestions. Anurag Acharya, Jeff Dean, and David des-
Jardins contributed to the early design. Fay Chang worked
on comparison of replicas across chunkservers. Guy Ed-
jlali worked on storage quota. Markus Gutschke worked
on a testing framework and security enhancements. David
Kramer worked on performance enhancements. Fay Chang,
Urs Hoelzle, Max Ibel, Sharon Perl, Rob Pike, and Debby
Wallach commented on earlier drafts of the pap er. Many of
our colleagues at Google bravely trusted their data to a new
file system and gave us useful feedback. Yoshka helped with
early testing.
REFERENCES
[1] Thomas Anderson, Michael Dahlin, Jeanna Neefe,
David Patterson, Drew Roselli, and Randolph Wang.
Serverless network file systems. In Proceedings of the
15th ACM Symposium on Operating System
Principles,p a g e s1 0 9 – 1 2 6 ,C o p p e rM o u n t a i nR e s o r t ,
Colorado, December 1995.
[2] Remzi H. Arpaci-Dusseau, Eric Anderson, Noah
Treuhaft, David E. Culler, Joseph M. Hellerstein,
David Patterson, and Kathy Yelick. Cluster I/O with
River: Making the fast case common. In Proceedings
of the Sixth Workshop on Input/Output in Parallel
and Distributed Systems (IOPADS ’99) ,p a g e s1 0 – 2 2 ,
Atlanta, Georgia, May 1999.
[3] Luis-Felipe Cabrera and Darrell D. E. Long. Swift:
Using distributed disk striping to provide high I/O
data rates. Computer Systems ,4 ( 4 ) : 4 0 5 – 4 3 6 ,1 9 9 1 .
[4] Garth A. Gibson, David F. Nagle, Khalil Amiri, Jeff
Butler, Fay W. Chang, Howard Gobioff, Charles
Hardin, Erik Riedel, David Rochberg, and Jim
Zelenka. A cost-effective, high-bandwidth storage
architecture. In Proceedings of the 8th Architectural
Support for Programming Languages and Operating
Systems,p a g e s9 2 – 1 0 3 ,S a nJ o s e ,C a l i f o r n i a ,O c t o b e r
1998.
[5] John Howard, Michael Kazar, Sherri Menees, David
Nichols, Mahadev Satyanarayanan, Robert
Sidebotham, and Michael West. Scale and
performance in a distributed file system. ACM
Transactions on Computer Systems ,6 ( 1 ) : 5 1 – 8 1 ,
February 1988.
[6] InterMezzo. http://www.inter-mezzo.org, 2003.
[7] Barbara Liskov, Sanjay Ghemawat, Robert Gruber,
Paul Johnson, Liuba Shrira, and Michael Williams.
Replication in the Harp file system. In 13th
Symposium on Operating System Principles ,p a g e s
226–238, Pacific Grove, CA, October 1991.
[8] Lustre. http://www.lustreorg, 2003.
[9] David A. Patterson, Garth A. Gibson, and Randy H.
Katz. A case for redundant arrays of inexpensive disks
(RAID). In Proceedings of the 1988 ACM SIGMOD
International Conference on Management of Data ,
pages 109–116, Chicago, Illinois, September 1988.
[10] Frank Schmuck and Roger Haskin. GPFS: A
shared-disk file system for large computing clusters. In
Proceedings of the First USENIX Conference on File
and Storage Technologies,p a g e s2 3 1 – 2 4 4 ,M o n t e r e y ,
California, January 2002.
[11] Steven R. Soltis, Thomas M. Ruwart, and Matthew T.
O’Keefe. The Gobal File System. In Proceedings of the
Fifth NASA Goddard Space Flight Center Conference
on Mass Storage Systems and Technologies ,C o l l e g e
Park, Maryland, September 1996.
[12] Chandramohan A. Thekkath, Timothy Mann, and
Edward K. Lee. Frangipani: A scalable distributed file
system. In Proceedings of the 16th ACM Symposium
on Operating System Principles ,p a g e s2 2 4 – 2 3 7 ,
Saint-Malo, France, October 1997.

Q: Can I stop these complaints about my unused variable/import?

A: There's a good explanation at https://golang.org/doc/faq#unused_variables_and_imports.

Q: Is the defer keyword in other languages?

A: Defer was new in Go. We originally added it to provide a way to
recover from panics (see "recover" in the spec), but it turned out
to be very useful for idioms like "defer mu.Unlock()" as well.
Later, Swift added a defer statement too. It seems clearly inspired
by Go but I'm not sure how close the details are.

Q: Why is the type after the variable declaration, unlike C languages?

A: There's a good explanation at https://blog.golang.org/gos-declaration-syntax.

Q: Why not adopt classes and OOP like in C++ and Java?

A: We believe that Go's approach to object-oriented programming,
which is closer to Smalltalk than to Java/C++/Simula, is more
lightweight and makes it easier to adapt large programs. I talked
about this at Google I/O in 2010. See
https://github.com/golang/go/wiki/GoTalks#go-programming for links
to the video and slides.

Q: Why does struct require a trailing comma on a multiline definition?

A: Originally it didn't, but all statements were terminated by
semicolons. We made semicolons optional shortly after the public
release of Go. When we did that, we tried to avoid Javascript's
mistake of making the semicolon rules very complex and error-prone.
Instead we have a simple rule: every line ends in an implicit
semicolon unless the final token is something that cannot possibly
end a statement (for example, a plus sign, or a comma). One effect
of this is that if you don't put the trailing comma on the line,
it gets an implicit semicolon, which doesn't parse well. It's
unfortunate, and it wasn't that way before the semicolon rules, but
we're so happy about not typing semicolons all the time that we'll
live with it. The original proposal for semicolon insertion is at
https://groups.google.com/d/msg/golang-nuts/XuMrWI0Q8uk/kXcBb4W3rH8J.
See the next answer also.

Q: Why are list definitions inconsistent, where some need commas and some do not?

A: The ones that don't need commas need semicolons, but those
semicolons are being inserted automatically (see previous answer).
The rule is that statements are separated by semicolons and smaller
pieces of syntax by commas:

    import "x";
    import "y";

    var x = []int{
        1,
        2,
        3,
    }

When you factor out a group of imports, you still have semicolons:

    import (
        "x";
        "y";
    )

    var x = []int{
        1,
        2,
        3,
    }

But then when we made semicolons optional, the semicolons disappeared
from the statement blocks leaving the commas behind:

    import (
        "x"
        "y"
    )

    var x = []int{
        1,
        2,
        3,
    }

Now the distinction is between nothing and something, instead of
two different characters, and it's more pronounced. If we had known
from the start that semicolons would be optional I think we might
have used them in more syntactic forms, or maybe made some forms
accept either commas or semicolons. At this point that seems
unlikely, though.

Q: Why does Go name its while loops "for"?

A: C has both while(cond) {} and for(;cond;) {}. It didn't seem
like Go needed two keywords for the same thing.

Q: There seem to be a lot of new languages emerging these days,
including Rust, D, Swift and Nim, among probably others.  Are there
any lessons you've learned from these other languages and their
communities that you wish you'd been able to incorporate into Go?

A: I do watch those languages for developments. I think they've
learned things from Go and I hope we've also learned things from
them. Some day I'd like the Go compiler to do a better job of
inferring ownership rules, or maybe even having lightweight ownership
expressions in the type system. Javari, Midori, Pony, and Rust are
inspirations here. I wrote a bit more about this at
https://research.swtch.com/go2017.

Q: Why the focus on concurrency and goroutines?

A: We knew from past experience that good concurrency support using
channels and lightweight processes would make writing the kinds of
systems we built at Google a lot easier, as I hope the lecture
showed. There's a bit more explanation at https://golang.org/doc/faq#csp,
and some background about our earlier experiences at
https://swtch.com/~rsc/thread/.

Q: Does Go pass function arguments by value or by reference?

A: For most types (e.g. numbers, structs) Go passes a copy of the
value, so that the caller does not see any modifications that the
callee makes. strings are effectively call-by-value since their
content cannot be modified.

A few built-in types -- channels, maps, and slices -- are effectively
call-by-reference in the sense that updates to the data structure are
seen by both caller and callee.

Calling a method with a pointer receiver passes the object by
reference (by passing a pointer to it), even though the call lacks any
& and thus looks like it should be by value.

This non-uniformity can be confusing. For example, if you want to pass
a sync.WaitGroup to a function, you probably need to pass a pointer to
it with &wg, whereas you should probably pass a channel without any &.

For a complex type like a map or a slice, there's room for
disagreement about what call-by-value and call-by-reference mean. If
you have a variable of type map, and you think the map lives inside
the variable, then Go's maps look like call-by-reference. If you think
the variable contains a pointer to map data in the heap, then Go's
maps look like call-by-value (where the value is the pointer).

IronFleet: Proving Practical Distributed Systems Correct
Chris Hawblitzel, Jon Howell, Manos Kapritsos, Jacob R. Lorch,
Bryan Parno, Michael L. Roberts, Srinath Setty, Brian Zill
Microsoft Research
Abstract
Distributed systems are notorious for harboring subtle bugs.
Verification can, in principle, eliminate these bugs a priori,
but verification has historically been difficult to apply at full-
program scale, much less distributed-system scale.
We describe a methodology for building practical and
provably correct distributed systems based on a unique blend
of TLA-style state-machine refinement and Hoare-logic ver-
ification. We demonstrate the methodology on a complex
implementation of a Paxos-based replicated state machine
library and a lease-based sharded key-value store. We prove
that each obeys a concise safety specification, as well as de-
sirable liveness requirements. Each implementation achieves
performance competitive with a reference system. With our
methodology and lessons learned, we aim to raise the stan-
dard for distributed systems from “tested” to “correct.”
1. Introduction
Distributed systems are notoriously hard to get right. Protocol
designers struggle to reason about concurrent execution on
multiple machines, which leads to subtle errors. Engineers
implementing such protocols face the same subtleties and,
worse, must improvise to fill in gaps between abstract proto-
col descriptions and practical constraints, e.g., that real logs
cannot grow without bound. Thorough testing is considered
best practice, but its efficacy is limited by distributed systems’
combinatorially large state spaces.
In theory, formal verification can categorically eliminate
errors from distributed systems. However, due to the com-
plexity of these systems, previous work has primarily fo-
cused on formally specifying [ 4, 13, 27, 41, 48, 64], verify-
ing [3, 52, 53, 59, 61], or at least bug-checking [20, 31, 69]
distributed protocols, often in a simplified form, without
extending such formal reasoning to the implementations.
In principle, one can use model checking to reason about
the correctness of both protocols [ 42, 59] and implemen-
tations [46, 47, 69]. In practice, however, model checking
is incomplete—the accuracy of the results depends on the
accuracy of the model—and does not scale [4].
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
SOSP’15, October 4–7, 2015, Monterey, CA.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3834-9/15/10. . . $15.00.
http://dx.doi.org/10.1145/2815400.2815428
This paper presents IronFleet, the first methodology for
automated machine-checked verification of the safety and
liveness of non-trivial distributed system implementations.
The IronFleet methodology is practical: it supports complex,
feature-rich implementations with reasonable performance
and a tolerable proof burden.
Ultimately, IronFleet guarantees that the implementation
of a distributed system meets a high-level, centralized spec-
ification. For example, a sharded key-value store acts like
a key-value store, and a replicated state machine acts like
a state machine. This guarantee categorically rules out race
conditions, violations of global invariants, integer overflow,
disagreements between packet encoding and decoding, and
bugs in rarely exercised code paths such as failure recov-
ery [70]. Moreover, it not only rules out bad behavior, it tells
us exactly how the distributed system will behave at all times.
The IronFleet methodology supports proving both safety
and liveness properties of distributed system implementa-
tions. A safety property says that the system cannot perform
incorrect actions; e.g., replicated-state-machine linearizabil-
ity says that clients never see inconsistent results. A liveness
property says that the system eventually performs a useful
action, e.g., that it eventually responds to each client request.
In large-scale deployments, ensuring liveness is critical, since
a liveness bug may render the entire system unavailable.
IronFleet takes the verification of safety properties further
than prior work (§9), mechanically verifying two full-featured
systems. The verification applies not just to their protocols
but to actual imperative implementations that achieve good
performance. Our proofs reason all the way down to the
bytes of the UDP packets sent on the network, guaranteeing
correctness despite packet drops, reorderings, or duplications.
Regarding liveness, IronFleet breaks new ground: to our
knowledge, IronFleet is the first system to mechanically
verify liveness properties of a practical protocol, let alone an
implementation.
IronFleet achieves comprehensive verification of complex
distributed systems via a methodology for structuring and
writing proofs about them, as well as a collection of generic
verified libraries useful for implementing such systems. Struc-
turally, IronFleet’s methodology uses a concurrency contain-
ment strategy (§3) that blends two distinct verification styles
within the same automated theorem-proving framework, pre-
venting any semantic gaps between them. We use TLA-style
state-machine refinement [36] to reason about protocol-level
concurrency, ignoring implementation complexities, then use
Floyd-Hoare-style imperative verification [17, 22] to reason

about those complexities while ignoring concurrency. To
simplify reasoning about concurrency, we impose a machine-
checked reduction-enabling obligation on the implementa-
tion (§3.6). Finally, we structure our protocols using always-
enabled actions (§4.2) to greatly simplify liveness proofs.
To facilitate writing proofs about distributed systems, we
have developed techniques for writing automation-friendly in-
variant proofs (§3.3), as well as disciplines and tool improve-
ments for coping with prover limitations (§6). For liveness
proofs, we have constructed an embedding of TLA (§4.1) in
our automated verification framework that includes heuristics
for reliably unleashing the power of automated proving.
To help developers, we have built general-purpose verified
libraries for common tasks, such as packet parsing and
marshalling, relating concrete data structures to their abstract
counterparts, and reasoning about collections. We have also
written a verified library of 40 fundamental TLA rules useful
for writing liveness proofs.
To illustrate IronFleet’s applicability, we have built and
proven correct two rather different distributed systems:
IronRSL, a Paxos-based [ 35] replicated-state-machine li-
brary, and IronKV, a sharded key-value store. All IronFleet
code is publicly available [25].
IronRSL’s implementation is complex, including many de-
tails often omitted by prior work; e.g., it supports state trans-
fer, log truncation, dynamic view-change timeouts, batching,
and a reply cache. We prove complete functional correctness
and its key liveness property: if the network is eventually
synchronous for a live quorum of replicas, then a client re-
peatedly submitting a request eventually receives a reply.
Unlike IronRSL, which uses distribution for reliability,
IronKV uses it for improved throughput by moving “hot”
keys to dedicated machines. For IronKV, we prove complete
functional correctness and an important liveness property: if
the network is fair then the reliable-transmission component
eventually delivers each message.
While verification rules out a host of problems, it is not
a panacea (§8). IronFleet’s correctness is not absolute; it
relies on several assumptions (§2.5). Additionally, verification
requires more up-front development effort: the automated
tools we use fill in many low-level proof steps automatically
(§6.3.1), but still require considerable assistance from the
developer (§6.3.2). Finally, we focus on verifying newly
written code in a verification-friendly language (§2.2), rather
than verifying existing code.
In summary, this paper makes the following contributions:
• We demonstrate the feasibility of mechanically verify-
ing that practical distributed implementations, i.e., func-
tionally complete systems with reasonable performance,
match simple, logically centralized specifications.
• We describe IronFleet’s novel methodology for uniting
TLA-style refinement with Floyd-Hoare logic within a
single automated verification framework.
• We provide the first machine-verified liveness proofs of
non-trivial distributed systems.
L0 L1 L2 L3 L4
H0 H1 H2 H3 H4 H5 H6 H7
Figure 1. State Machine Refinement. The low-level state ma-
chine behavior L0. . .L4 refines the high-level one H0. . .H7 because
each low-level state corresponds to a high-level state. For each cor-
respondence, shown as a dashed line, the two states must satisfy the
spec’s refinement conditions. Low-level step L0→L1, as is typical,
maps to one high-level step H0→H1. However, low-level steps can
map to zero (L2→L3) or several (L3→L4) high-level steps.
method halve(x:int) returns (y:int)
requires x > 0;
ensures y < x;
{ y := x / 2; }
Figure 2. Simple Floyd-Hoare verification example.
• We describe engineering disciplines and lessons for veri-
fying distributed systems.
2. Background and Assumptions
We briefly describe the existing verification techniques that
IronFleet draws upon, as well as our assumptions.
2.1 State Machine Refinement
State machine refinement [1, 18, 34] is often used to rea-
son about distributed systems [ 4, 27, 41, 48, 52, 64]. The
developer describes the desired system as a simple abstract
state machine with potentially infinitely many states and non-
deterministic transition predicates. She then creates a series
of increasingly complex (but still declarative) state machines,
and proves that each one refines the one “above” it (Figure 1).
State machine L refinesH if each of L’s possible behaviors,
i.e., each (potentially infinite) sequence of states the machine
may visit, corresponds to an equivalent behavior of H. To
gain the benefits of abstraction this approach provides, the
developer must choose the layer abstractions intelligently, a
subtle choice needed for each new context.
State machine refinement in a distributed-system context
(e.g., TLA-style refinement) typically considers declarative
specifications, not imperative code. PlusCal [37] attempts to
bridge this gap, but has only been used for tiny programs.
2.2 Floyd-Hoare Verification
Many program verification tools support Floyd-Hoare style [17,
22] first-order predicate logic reasoning about imperative pro-
grams. In other words, they allow the programmer to annotate
a program with assertions about the program’s state, and the
verifier checks that the assertions hold true for all possible
program inputs. For example, the code in Figure 2 asserts
a condition about its input via a precondition and asserts a
condition about its output via a postcondition.
As in our previous work [21], we use Dafny [39], a high-
level language that automates verification via the Z3 [ 11]
SMT solver. This enables it to fill in many low-level proofs

automatically; for example, it easily verifies the program in
Figure 2 for all possible inputs x without any assistance.
However, many proposition classes are not decidable in
general, so Z3 uses heuristics. For example, propositions
involving universal quantifiers (∀) and existential quantifiers
(∃) are undecidable. Thus, it is possible to write correct
code in Dafny that the solver nevertheless cannot prove
automatically. Instead, the developer may insert annotations
to guide the verifier’s heuristics to a proof. For instance, the
developer can write a trigger to cue the verifier as to which
values to instantiate a quantified variable with [12].
Once a program verifies, Dafny compiles it to C# and has
the .NET compiler produce an executable. Other languages
(e.g., C++) are currently unsupported, but it would likely be
possible to compile Dafny to them. Our previous work [21]
shows how to compile Dafny to verifiable assembly to avoid
depending on the Dafny compiler, .NET, and Windows.
Like most verification tools, Dafny only considers one
single-threaded program, not a collection of concurrently
executing hosts. Indeed, some verification experts estimate
that the state-of-the-art in concurrent program verification
lags that of sequential verification by a decade [51].
2.3 Reduction
Given a fine-grained behavior from a real concurrent system,
we can use reduction [40] to convert it to an equivalent
behavior of coarse-grained steps, simplifying verification.
Crucially, two steps can swap places in the behavior if
swapping them has no effect on the execution’s outcome.
Reduction is typically used in the context of shared-
memory concurrent programs [9, 14, 33] and synchronization
primitives [65]. Applying reduction requires identifying all of
the steps in the system, proving commutativity relationships
among them, and applying these relationships to create an
equivalent behavior with a more useful form. We tackle these
challenges in the context of distributed systems in§3.6.
2.4 Temporal Logic of Actions (TLA)
Temporal logic [54] and its extension TLA [34] are standard
tools for reasoning about safety and liveness. Temporal logic
formulas are predicates about the system’s current and future
states. The simplest type of formula ignores the future; e.g.,
in a lock system, a formula P could be “host h holds the lock
now.” Other formulas involve the future; e.g., ♦P means P
eventually holds, and □P means P holds now and forever.
For example, the property∀h∈ Hosts : □♦P means that for
any host, it is always true that h will eventually hold the lock.
TLA typically considers abstract specifications, not im-
perative code. Furthermore, a na¨ıve embedding of TLA can
often pose problems for automated verification. After all,
each □ involves a universal quantifier and each♦ involves
an existential quantifier. Since Z3 needs heuristics to decide
propositions with quantifiers (§2.2), it can fail due to inade-
quate developer annotations. We address this in§4.1.
I0 I1 I2 I3
H0 H1 H2 H3 H4
P0 P1 P2 P3
High-level spec (§3.1)
Distributed protocol (§3.2)
Implementation (§3.4)
refinement (§3.3)
refinement (§3.5)
Figure 3. Verification Overview. IronFleet divides a distributed
system into carefully chosen layers. We use TLA style verification to
prove that any behavior of the protocol layer (e.g., P0. . .P3) refines
some behavior of the high-level spec (e.g., H0. . .H4). We then use
Floyd-Hoare style to prove that any behavior of the implementation
(e.g., I0. . .I3) refines a behavior of the protocol layer.
2.5 Assumptions
Our guarantees rely on the following assumptions.
A small amount of our code is assumed, rather than proven,
correct. Thus, to trust the system, a user must read this code.
Specifically, the spec for each system is trusted, as is the brief
main-event loop described in§3.7.
We do not assume reliable delivery of packets, so the
network may arbitrarily delay, drop, or duplicate packets. We
do assume the network does not tamper with packets, and
that the addresses in packet headers are trustworthy. These
assumptions about message integrity are easy to enforce
within, say, a datacenter or VPN, and could be relaxed by
modeling the necessary cryptographic primitives to talk about
keys instead of addresses [21].
We assume the correctness of Dafny, the .NET compiler
and runtime, and the underlying Windows OS. Previous
work [21] shows how to compile Dafny code into verifiable
assembly code to avoid these dependencies. We also rely on
the correctness of the underlying hardware.
Our liveness properties depend on further assumptions. For
IronRSL, we assume a quorum of replicas run their respective
main loops with a minimum frequency, never running out
of memory, and the network eventually delivers messages
synchronously among them; more details are in§5.1.4. For
IronKV, we assume that each host’s main loop executes
infinitely often and that the network is fair, i.e., a message
sent infinitely often is eventually delivered.
3. The IronFleet Verification Methodology
IronFleet organizes a distributed system’s implementation
and proof into layers (Figure 3) to avoid the intermingling of
subtle distributed protocols with implementation complexity.
At the top (§3.1), we write a simple spec for the system’s
behavior. We then write an abstract distributed protocol layer
(§3.2) and use TLA-style techniques to prove that it refines
the spec layer (§3.3). Then we write an imperative implemen-
tation layer to run on each host (§3.4) and prove that, despite
the complexities introduced when writing real systems code,
the implementation correctly refines the protocol layer (§3.5).
To avoid complex reasoning about interleaved execution of
low-level operations at multiple hosts, we use a concurrency

datatype SpecState = SpecState(history:seq<HostId>)
predicate SpecInit (ss:SpecState)
{ |ss.history|==1 && ss.history[0] in AllHostIds () }
predicate SpecNext (ss_old:SpecState,ss_new:SpecState)
{ exists new_holder :: new_holder in AllHostIds () &&
ss_new.history == ss_old.history + [new_holder] }
predicate SpecRelation (is:ImplState,ss:SpecState)
{ forall p :: p in is.sentPackets && p.msg.lock? ==>
p.src == ss.history[p.msg.epoch] }
Figure 4. A toy lock specification.
containment strategy: the proofs above assume that every
implementation step performs an atomic protocol step. Since
the real implementation’s execution is not atomic, we use
a reduction argument (§3.6) to show that a proof assuming
atomicity is equally valid as a proof for the real system. This
argument requires a mechanically verified property of the
implementation, as well as a small paper-only proof about
the implications of the property.
§4 extends this methodology to prove liveness properties.
3.1 The High-Level Spec Layer
What does it mean for a system to be correct? One can
informally enumerate a set of properties and hope they are
sufficient to provide correctness. A more rigorous way is
to define a spec, a succinct description of every allowable
behavior of the system, and prove that an implementation
always generates outputs consistent with the spec.
With IronFleet, the developer writes the system’s spec
as a state machine: starting with some initial state, the spec
succinctly describes how that state can be transformed. The
spec defines the state machine via threepredicates, i.e., func-
tions that return true or false. SpecInit describes accept-
able starting states, SpecNext describes acceptable ways
to move from an old to a new state, and SpecRelation
describes the required conditions on the relation between an
implementation state and its corresponding abstract state. For
instance, in Figure 3, SpecInit constrains what H0 can be,
SpecNext constrains steps like H0→H1 and H1→H2, and
SpecRelation constrains corresponding state pairs like
(I1, H1) and (I2, H4). To avoid unnecessary constraints on im-
plementations of the spec, SpecRelation should only talk
about the externally visible behavior of the implementation,
e.g., the set of messages it has sent so far.
As a toy example, the spec in Figure 4 describes a simple
distributed lock service with a single lock that passes amongst
the hosts. It defines the system’s state as a history: a sequence
of host IDs such that the nth host in the sequence held the
lock in epoch n. Initially, this history contains one valid host.
The system can step from an old to a new state by appending
a valid host to the history. An implementation is consistent
with the spec if all lock messages for epoch n come from the
nth host in the history.
By keeping the spec simple, a skeptic can study the spec
to understand the system’s properties. In our example, she
can easily conclude that the lock is never held by more
datatype Host = Host(held:bool,epoch:int)
predicate HostInit (s:Host,id:HostId,held:bool)
{ s.held==held && s.epoch==0 }
predicate HostGrant (s_old:Host,s_new:Host,
spkt:Packet) {
{ s_old.held && !s_new.held && spkt.msg.transfer?
&& spkt.msg.epoch == s_old.epoch+1 }
predicate HostAccept (s_old:Host,s_new:Host,
rpkt:Packet,spkt:Packet)
{ !s_old.held && s_new.held && rpkt.msg.transfer?
&& s_new.epoch == rpkt.msg.epoch == spkt.msg.epoch
&& spkt.msg.lock? }
predicate HostNext (s_old:Host,s_new:Host,
rpkt:Packet,spkt:Packet)
{ HostGrant(s_old,s_new,spkt) ||
HostAccept(s_old,s_new,rpkt,spkt) }
Figure 5. Simplified host state machine for a lock service.
than one host. Since the spec captures all permitted system
behaviors, she can later verify additional properties of the
implementation just by verifying they are implied by the spec.
3.2 The Distributed-Protocol Layer
At the untrusted distributed-protocol layer, the IronFleet
methodology introduces the concept of independent hosts
that communicate only via network messages. To manage
this new complexity, we keep this layer simple and abstract.
In more detail, we formally specify, in Dafny (§2.2), a dis-
tributed system state machine. This state machine consists of
N host state machines and a collection of network packets. In
each step of the distributed system state machine, one host’s
state machine takes a step, allowing it to atomically read mes-
sages from the network, update its state, and send messages
to the network;§3.6 relaxes this atomicity assumption.
The developer must specify each host’s state machine: the
structure of the host’s local state, how that state is initialized
(HostInit), and how it is updated (HostNext). IronFleet
reduces the developer’s effort in the following three ways.
First, we use a simple, abstract style for the host state and
network interface; e.g., the state uses unbounded mathemati-
cal integers (ignoring overflow issues), unbounded sequences
of values (e.g., tracking all messages ever sent or received),
and immutable types (ignoring memory management and
heap aliasing). The network allows hosts to send and receive
high-level, structured packets, hence excluding the challenges
of marshalling and parsing from this layer.
Second, we use a declarative predicate style. In other
words, HostNext merely describes how host state can
change during each step; it gives no details about how to effect
those changes, let alone how to do so with good performance.
Third, from the protocol’s perspective, each of the steps
defined above takes place atomically, greatly simplifying the
proof that the protocol refines the spec layer (§3.3). In§3.6,
we connect this proof assuming atomicity to a real execution.
Continuing our lock example, the protocol layer might
define a host state machine as in Figure 5. During the dis-
tributed system’s initialization of each host via HostInit,
exactly one host is given the lock via the held parameter.

The HostNext predicate then says that a host may step
from an old to a new state if the new state is the result of
one of two actions, each represented by its own predicate.
The two actions are giving away the lock (HostGrant) and
receiving the lock from another host (HostAccept). A host
may grant the lock if in the old state it holds the lock, and
if in the new state it no longer holds it, and if the outbound
packet (spkt) represents a transfer message to another host.
Accepting a lock is analogous.
3.3 Connecting the Protocol Layer to the Spec Layer
The first major theorem we prove about each system is that
the distributed protocol layer refines the high-level spec layer.
In other words, given a behavior of IronFleet’s distributed
system in which N hosts take atomic protocol steps defined
by the HostNext predicate, we provide a corresponding
behavior of the high-level state machine spec.
We use the standard approach to proving refinement, as
illustrated in Figure 3. First, we define arefinement function
PRef that takes a state of the distributed protocol state
machine and returns the corresponding state of the centralized
spec. We could use a relation instead of a function, but the
proof is easier with a function [ 1]. Second, we prove that
PRef of the initial state of the distributed protocol satisfies
SpecInit. Third, we prove that if a step of the protocol
takes the state from ps_old to ps_new, then there exists
a legal sequence of high-level spec steps that goes from
PRef(ps_old) to PRef(ps_new).
Unlike previous refinement-based work (§2.1), we use a
language, Dafny [39], designed for automated theorem prov-
ing. This reduces but does not eliminate the human proof
effort required (§6.3). Since we also verify our implementa-
tion in Dafny (§3.5), we avoid any semantic gaps between
the implementation’s view of the protocol and the protocol
we actually prove correct.
The challenge of proving the protocol-to-spec theorem
comes from reasoning about global properties of the dis-
tributed system. One key tool is to establish invariants: pred-
icates that should hold throughout the execution of the dis-
tributed protocol. In the lock example, we might use the
invariant that the lock is either held by exactly one host or
granted by one in-flight lock-transfer message. We can prove
this invariant inductively by showing that every protocol step
preserves it. Showing refinement of the spec is then simple.
Identifying the right invariants for a given protocol re-
quires a deep understanding of the protocol, but it is a skill
one develops with experience (§6).
Invariant quantifier hiding. Many useful invariants, like
“For every reply message sent, there exists a corresponding
request message sent,” involve quantifiers. Unfortunately,
such quantifiers pose problems for verifiers (§2.2). We have
thus adopted a style we call invariant quantifier hiding: we
prove some invariants involving quantifiers without explicitly
exposing those quantifiers to the verifier. The key is to
establish the invariant with a proof that explicitly instantiates
lemma ReplyToReq (reply:MessageReply,
behavior:map<int,HostState>, step: nat)
returns (req:MessageRequest)
requires IsValidBehaviorUpTo (behavior, step);
requires reply in behavior[step].network;
ensures req in behavior[step].network;
ensures Matches (req, reply);
{
assert step > 0; // because a packet was sent
if !(reply in behavior[step-1].network) {
req := OnlyExecReplies(behavior, step-1);
} else { // apply induction
req := ReplyToReq(behavior, step-1, reply);
}
}
Figure 6. Establishing an invariant with implicit quantifiers.
all bound variables. For each universal quantifier in the
invariant that does not succeed an existential quantifier, the
quantified variable is an input parameter of the proof. For each
existential quantifier in the invariant, the quantified variable
is an output parameter. For instance, the invariant from the
beginning of this paragraph could be proved with Figure 6.
It is easy to write this proof because we must prove it only
for a specific reply message, not all of them. As illustrated
above, there are only two cases to consider: (1) the reply
message was just generated, in which case we only have to
consider the last action taken, or (2) the reply message was
already present in the previous step, in which case we can
complete the proof by induction on step.
It is also easy to use this proof because instead of stating
a fact about the existence of a request message, it explicitly
provides a witness to that existence. Typically, a developer
only needs to prove the invariant for a specific reply message;
this form lets her establish precisely that fact. If the developer
needs the universally-quantified version, she can establish it
by invoking the invariant’s proof in a loop.
3.4 The Implementation Layer
Unlike in the declarative protocol layer, in the implementation
layer the developer writes single-threaded, imperative code
to run on each host. This code must cope with all of the
ugly practicalities we abstracted away in the protocol layer.
For instance, it must handle real-world constraints on how
hosts interact: since network packets must be bounded-sized
byte arrays, we need to prove the correctness of our routines
for marshalling high-level data structures into bytes and for
parsing those bytes. We also write the implementation with
performance in mind, e.g., using mutable arrays instead of
immutable sequences and using uint64s instead of infinite-
precision integers. The latter requires us to prove the system
correct despite the potential for integer overflow.
Dafny does not natively support networking, so we extend
the language with a trusted UDP specification that exposes
Init, Send, and Receive methods. For example, Send
expects an IP address and port for the destination and an
array of bytes for the message body. When compiled, calls to

these Dafny methods invoke the .NET UDP network stack.
Send also automatically inserts the host’s correct IP address,
satisfying our assumption about packet headers in§2.5.
The network interface maintains a ghost variable (i.e., a
variable used only for verification, not execution) that records
a “journal” of every Send and Receive that takes place,
including all of the arguments and return values. We use this
journal when proving properties of the implementation (§3.5).
3.5 Connecting the Implementation to the Protocol
The second major theorem we prove about each IronFleet
system is that the implementation layer correctly refines the
protocol. To do this, we prove that even though the imple-
mentation operates on concrete local state, which uses heap-
dependent, bounded representations, it is still a refinement
of the protocol layer, which operates on abstract types and
unbounded representations.
First, we prove that the host implementation refines the
host state machine described in the protocol layer. This refine-
ment proof is analogous to the one in§3.3, though simplified
by the fact that each step in the implementation corresponds to
exactly one step of the host state machine. We define a refine-
ment function HRef that maps a host’s implementation state
to a host protocol state. We prove that the code ImplInit
to initialize the host’s state ensures HostInit(HRef(hs)),
and that the codeImplNext to execute one host step ensures
HostNext(HRef(hs_old),HRef(hs_new)).
Then, we use this to prove that a distributed system
comprising N host implementations, i.e., what we actually
intend to run, refines the distributed protocol of N hosts.
We use a refinement function IRef that maps states of
the distributed implementation to states of the distributed
protocol. The refinement proof is largely straightforward
because each step of the distributed implementation in which
a host executes ImplNext corresponds to one step of the
distributed protocol where a host takes a HostNext step.
The difficult part is proving that the network state in the
distributed system implementation refines the network state
in the protocol layer. Specifically, we must prove that every
send or receive of a UDP packet corresponds to a send or
receive of an abstract packet. This involves proving that when
host A marshals a data structure into an array of bytes and
sends it to host B, B parses out the identical data structure.
The last major theorem we prove is that the distributed
implementation refines the abstract centralized spec. For this,
we use the refinement functions from our two major refine-
ment theorems, composing them to form our final refinement
function PRef(IRef(·)). The key part of this proof is
establishing that the specified relation conditions hold, i.e.,
that for all implementation states is, SpecRelation(is,
IRef(PRef(is))) holds.
3.6 Abstracting Non-Atomicity via Reduction
Sections 3.1–3.5 describe a mechanically verified proof struc-
ture that assumes that every implementation step performs an
AR AP BR AS BP AS AR BP AP BS AS
actual
execution
AR AP AS BR AS BP BP AR BS AP AS
AR AP AS AS BR BP BP BS AR AP AS
equivalent
execution
HostNextA HostNextAHostNextB
Figure 7. Reduction. In the real execution behavior, the send
(S), receive (R), and local processing (P) steps at hosts A and B
are fully interleaved. However, certain steps commute to yield an
equivalent behavior. Since we impose constraints on the structure
of the implementation’s event handlers (Figure 8), we can commute
steps until all of the implementation-level steps in a given host’s
event handler (circled) are contiguous. This reduced behavior then
admits a direct refinement to the distributed protocol layer.
atomic protocol step. However, the implementation’s event
handler is not atomic: while one host receives packets, com-
putes locally, and sends packets, other hosts do the same con-
currently, leading to arbitrary interleavings of these low-level
operations. To bridge this gap, we use a “reduction” argument
(§2.3). Reduction is typically used to reason about threads
and processes sharing a single machine, but we apply it to rea-
soning about distributed systems. Although Dafny does not
provide a general mechanism for reasoning about reduction,
we are still able to use Dafny to enforce an obligation on the
implementation that enables reduction. A machine-checked
proof that this obligation enables reduction is future work;
instead, we sketch an informal argument here.
Hosts are unable to see others’ state except indirectly by
observing the packets they send. Thus, it is possible to take a
behavior of the system, representing the order in which events
really occurred, and posit an alternate order in which (1) each
host receives the same packets in the same order, (2) packet
send ordering is preserved, (3) packets are never received
before they are sent, and (4) the ordering of operations on
any individual host is preserved. Any proof of correctness
assuming such an ordering implies a proof for the original
behavior, since only the externalized behavior of the system,
the content and ordering messages sent, matters.
Figure 7 shows an example of such reordering. We start
with the real behavior at the bottom and reorder until we reach
the behavior at the top. For instance, we can reorder A’s first
send before B’s first receive because we know its contents
cannot have depended on B’s receive. The top behavior has no
interleavings between different hosts’ HostNext steps, and
thus is a legal behavior in which we have proved correctness.
Thus, the correctness proof also applies to the real behavior.
As a result, we can always reduce a real execution behavior
to a sequence of atomic steps via such reorderings if we
constrain the implementation to, in any given step, perform
all of its receives before all its sends. We call this areduction-
enabling obligation, which we use Dafny to enforce (§3.7).
With this obligation, we ensure that our proof of correctness

method Main () {
var s := ImplInit();
while (true)
invariant ImplInvariant (s);
{
ghost var journal_old := get_event_journal();
ghost var ios_performed:seq<IoEvent>;
s, ios_performed := ImplNext(s);
assert get_event_journal() ==
journal_old + ios_performed;
assert ReductionObligation (ios_performed);
}
}
Figure 8. Mandatory host event-handler loop.
assuming atomicity is equally valid as a proof of correctness
for the real system.
One complication is that when a host performs a time-
dependent operation like reading its clock, it creates a causal
constraint even without communication with other hosts.
This is because the clock represents an imperfect sample
from a globally shared reality. Thus, the reduction-enabling
obligation is extended as follows: A step may perform at
most one time-dependent operation, i.e., at most one clock
read, blocking receive, or non-blocking receive that returns
no packets. The step must perform all receives before this
time-dependent operation, and all sends after it.
3.7 Trusted Code
Nearly all IronFleet code is verified using the above methodol-
ogy, so there are only a few lines of code and proof assertions
that a user must read to gain confidence in the system. First,
she must read the high-level centralized spec to understand
what is being guaranteed. Second, she must read the asser-
tion, but not the proof of the assertion, that if each host in
a distributed system runs ImplInit followed by a loop of
ImplNext, then there exists a corresponding abstract behav-
ior of the centralized spec. Third, she must read the top-level
main host routine (Figure 8) to convince herself that each
host runs ImplInit and ImplNext. This code also ensures
that each host step meets its reduction-enabling constraint by
using the journal of externally visible events from§3.4.
4. Verifying Liveness
§3 describes the high-level spec as a state machine. Such a
spec says what the implementation must not do: it must never
deviate from the state machine’s behavior. However, it is also
useful to specify what the implementationmust do; properties
of this form are called liveness properties. For example, we
might specify that the lock implementation eventually grants
the lock to each host (Figure 9). Thus, a spec will typically
include not just a state machine but also liveness properties.
Some researchers have proposed heuristics for detecting
and quashing likely sources of liveness violations [ 31, 66],
but it is better to definitivelyprove their absence. With such
a proof, we do not have to reason about, e.g., deadlock or
predicate LockBehaviorFair (b:map<int,SpecState>)
{ forall h:Host, i: int :: h in AllHostIds () && i >= 0
==> exists j :: j >= i && h == last(b[j].history) }
Figure 9. Desired liveness property for the lock service.
livelock; such conditions and any others that can prevent the
system from making progress are provably ruled out.
Liveness properties are much harder to verify than safety
properties. Safety proofs need only reason about two system
states at a time: if each step between two states preserves the
system’s safety invariants, then we can inductively conclude
that all behaviors are safe. Liveness, in contrast, requires
reasoning about infinite series of system states. Such reason-
ing creates challenges for automated theorem provers (§2.4),
often causing the prover to time out rather than return a suc-
cessful verification or a useful error message.
With IronFleet, we address these challenges via a custom
TLA embedding in Dafny that focuses the prover’s efforts
in fruitful directions. We then use our TLA embedding to
build a library of fundamental TLA proof rules verified from
first principles. This library is a useful artifact for proving
liveness properties of arbitrary distributed systems: its rules
allow both the human developer and Dafny to operate at a
high level by taking large proof steps with a single call to a
lemma from the library. Finally, by structuring our protocols
with always-enabled actions, we significantly simplify the
task of proving liveness properties.
4.1 TLA Embedding and Library
As discussed in §2.4, TLA [ 34] is a standard tool for rea-
soning about liveness. IronFleet embeds TLA in Dafny by
modeling a TLA behavior, an infinite sequence of system
states, as a mapping B from integers to states, where B[0] is
the initial state and B[i] is the ith subsequent state. A liveness
property is a constraint on the behavior of the state machine.
For example, Figure 9 says that for every host h, there is
always a later time when h will hold the lock.
Our embedding hides key definitions from the prover
except where truly needed, and instead provides verified
lemmas that relate them to one another. For example, we
represent temporal logic formulas as opaque objects (i.e.,
objects Dafny knows nothing about) of type temporal,
and TLA transformations like □ as functions that convert
temporal objects to temporal objects.
Of course, in some contexts we actually do need to reason
about the internal meaning of □ and ♦. State-of-the-art SMT
solvers like Z3 do not yet provide decision procedures for
temporal operators like □ and ♦ directly. However, we can
encode these operators using explicit quantification over
steps (□ universally quantifies over all future steps, while
♦ existentially quantifies over some future step). We can
then provide the SMT solver with heuristics to control these
quantifiers using the solver’s support for triggers [ 12], as
discussed in§2.2. One simple heuristic proved effective in
many situations: when the solver is considering a future

step j for one formula, such as ♦Q, the heuristic requests
that the solver also consider j as a candidate step for other
formulas starting with □ or ♦, such as □P and ♦(P∧ Q).
This allows the solver to automatically prove formulas like
(♦Q)∧ (□P) =⇒ ♦(P∧ Q).
This heuristic is effective enough to automatically prove
40 fundamental TLA proof rules, i.e., rules for deriving one
formula from other formulas [ 34]. The heuristic allows us
to prove complicated rules efficiently; e.g., we stated and
proved Lamport’s INV1 rule about invariants in only 27 lines
of Dafny, and his WF1 rule about fairness in only 16 lines.
Our liveness proofs use these fundamental proof-rule
lemmas to justify temporal formula transformations. For
instance, as we discuss in§4.4, a liveness proof can usually
prove most of its steps by repeatedly invoking the WF1 rule.
4.2 Always-Enabled Actions
To achieve liveness, our protocol must satisfy fairness
properties. That is, it must ensure that each action, e.g.,
HostGrant or HostAccept, occurs in a timely fashion.
Lamport [36] suggests that such properties take the form
“if action A becomes always enabled, i.e., always possible
to do, the implementation must eventually do it.” However,
having terms of this form in verified code is problematic. If
the fairness property is a complex formula, it can be difficult
to characterize the set of states from which the action is
possible. This difficulty complicates both proving that the
fairness property is sufficient to ensure liveness properties,
and proving that the protocol has the fairness property.
Thus, we instead adopt always-enabled actions; i.e., we
only use actions that are always possible to do. For instance,
we would not use HostGrant from Figure 5 since it is
impossible to perform if you do not hold the lock. Instead,
we might use “if you hold the lock, grant it to the next host;
otherwise, do nothing”, which can always be done.
Our approach deviates from Lamport’s standard fairness
formulas, which means it can admit specifications that are not
machine closed [36]. Machine closure ensures that liveness
conditions do not combine with safety conditions to create
an unimplementable spec, such as that the implementation
must both grant a lock (to be fair) and not grant a lock
(to be safe, because it does not hold the lock). Fortunately,
machine closure is no concern in IronFleet: the existence
of an implementation that meets a fairness property is itself
proof that the property does not prevent implementation!
4.3 Proving Fairness Properties
Following IronFleet’s general philosophy of having the imple-
mentation layer deal only with implementation complexities,
we put the burden of satisfying fairness properties on the
protocol layer. The implementation satisfies the properties
automatically since its main method implements HostNext.
The mandatory structure from Figure 8 ensures that
HostNext runs infinitely often. So, all we must prove
is that if HostNext runs infinitely often, then each action
occurs infinitely often. We do this by havingHostNext be
a scheduler that guarantees each action occurs regularly.
One way to do this is to use a simple round-robin scheduler.
We currently have proofs in our library that if HostNext is
a round-robin scheduler that runs infinitely often, then each
action runs infinitely often. Furthermore, if the main host
method runs with frequency F (expressed, e.g., in times per
second), then each of its n actions occurs with frequency F/n.
4.4 Liveness Proof Strategies
Most of a liveness proof involves demonstrating that if some
condition Ci holds then eventually another condition Ci+1
holds. By chaining such proofs together, we can prove that
if some assumed initial condition C0 holds then eventually
some useful condition Cn holds. For instance, in IronRSL,
we prove that if a replica receives a client’s request, it
eventually suspects its current view; if it suspects its current
view, it eventually sends a message to the potential leader
of a succeeding view; and, if the potential leader receives a
quorum of suspicions, it eventually starts the next view.
Most steps in this chain require an application of a variant
of Lamport’s WF1 rule [ 34]. This variant involves a start-
ing condition Ci, an ending condition Ci+1, and an always-
enabled action predicate Action. It states that Ci leads to Ci+1
if the following three requirements are met:
1. If Ci holds, it continues to hold as long as Ci+1 does not.
2. If a transition satisfying Action occurs when Ci holds, it
causes Ci+1 to hold.
3. Transitions satisfying Action occur infinitely often.
We use this in Dafny as follows. Suppose we need a lemma
that shows Ci leads to Ci+1. We first find the action transition
Action intended to cause this. We then establish each of
requirements 1 and 2 with an invariant proof that considers
only pairs of adjacent steps. We then invoke the proof from
§4.3 that each of the action transitions occurs infinitely often
to establish requirement 3. Finally, having established the
three preconditions for the WF1 lemma from our verified
library, we call that lemma.
In some cases, we need lemmas from our library that prove
other variants of the WF1 proof rule sound. For instance,
often we must prove that Ci leads to Ci+1 not just eventually
but within a bounded time. For this, we have a variant of
WF1 that proves Ci+1 holds within the inverse of Action’s
frequency. It uses a modified requirement 3: that Action
occurs with a minimum frequency.
Another useful variant of WF1 is delayed, bounded-time
WF1. It applies when Action only induces Ci+1 after a certain
time t; this is common in systems that rate-limit certain
actions for performance reasons. For instance, to amortize
the cost of agreement, the IronRSL action for proposing
a batch of requests has a timer preventing it from sending
an incomplete batch too soon after the last batch. Delayed,
bounded-time WF1 uses a modified requirement 2: “IfAction
occurs when Ci holds and the time is≥ t, it causes Ci+1 to

hold.” This variant proves that Ci+1 eventually holds after t
plus the inverse of the action’s frequency.
Sometimes, a liveness proof needs more than a chain of
conditions: it must prove that multiple conditions eventually
hold simultaneously. For instance, in IronRSL we must prove
that a potential leader eventually knows suspicions from every
replica in the quorum at once. For this, we use our temporal
heuristics to prove sound the proof rule: “If every condition
in a set of conditions eventually holds forever, then eventually
all the conditions in the set hold simultaneously forever.” We
also have and use a bounded-time variant of this rule.
5. System Implementation
We use the IronFleet methodology to implement two practical
distributed systems and prove them correct: a Paxos-based
replicated state machine library and a lease-based sharded
key-value store. All IronFleet code is publicly available [25].
5.1 IronRSL: A Replicated State Machine Library
IronRSL replicates a deterministic application on multiple
machines to make that application fault-tolerant. Such repli-
cation is commonly used for services, like Chubby and
Zookeeper [5, 24], on which many other services depend.
Due to these dependencies, correctness bugs in replication
can lead to cascading problems, and liveness bugs can lead
to widespread outages of all dependent services.
IronRSL guarantees safety and liveness without sacrific-
ing complex implementation features necessary to run real
workloads. For instance, it uses batching to amortize the cost
of consensus across multiple requests, log truncation to con-
strain memory usage, responsive view-change timeouts to
avoid hard-coded assumptions about timing, state transfer to
let nodes recover from extended network disconnection, and
a reply cache to avoid unnecessary work.
5.1.1 The High-Level Specification
The spec for IronRSL is simply linearizability: it must gen-
erate the same outputs as a system that runs the application
sequentially on a single node. Our implementation achieves
this in the same way typical replicated state machine libraries
do: it runs the application on multiple nodes, and uses the
MultiPaxos [35] consensus protocol to feed the same requests
in the same order to each replica.
5.1.2 The Distributed-Protocol Layer
Protocol. In the protocol layer, each host’s state consists
of four components, based on Lamport’s description of
Paxos [35]: a proposer, an acceptor, a learner, and an executor.
The host’s action predicates include, for instance, proposing a
batch of requests (Figure 10) or sending the local application
state to a host that has fallen behind.
Protocol invariants. The protocol’s key invariant, known as
agreement, is that two learners never decide on different re-
quest batches for the same slot. Establishing this invariant
requires establishing several more invariants about earlier pro-
predicate ExistsProposal (m_set:set<Msg1b>, op: Op)
{ exists p :: p in m_set && op in p.msg.votes }
predicate ProposeBatch (s:Proposer,s’:Proposer)
{ if |s.1bMsgs| < quorumSize then no_op()
else if ExistsProposal (s.1bMsgs,s.nextOp) then
var new_batches := s.proposedBatches[s.nextOp :=
BatchFromHighestBallot(s.1bMsgs, s.nextOp)];
s’ == s[nextOp := s.nextOp + 1]
[proposedBatches := new_batches]
else ... }
Figure 10. A step predicate example from IronRSL (simplified).
tocol actions. For instance, we prove that ProposeValue
(Figure 10) cannot propose a batch if a different one may
have already been learned. The action’s predicate states that
batches can only be proposed when the host has received
a 1b message from at least f + 1 acceptors. We use this to
prove that this quorum of acceptors intersects with any other
quorum that might have accepted a batch in a previous ballot.
Protocol refinement. After establishing the agreement in-
variant, we prove that executing the sequence of decided
request batches is equivalent to taking steps in the high-level
state machine. One challenge is that multiple replicas execute
the same request batches, but the corresponding high-level
steps must be taken only once. We address this by refining
the distributed system to an abstract state machine that ad-
vances not when a replica executes a request batch but when
a quorum of replicas has voted for the next request batch.
5.1.3 The Implementation Layer
Often, the most difficult part of writing a method to imple-
ment a protocol action is proving that the method has the
appropriate effect on the refined state. For this, IronRSL re-
lies on our generic refinement library (§5.3), which lightens
the programmer’s burden by proving useful properties about
the refinement of common data structures.
Another difficulty is that the protocol sometimes describes
the relationship between the host’s pre-action and post-action
state in a non-constructive way. For instance, it says that the
log truncation point should be set to the nth highest number
in a certain set. It describes how to test whether a number
is the nth highest number in a set, but not how to actually
compute such a quantity. Thus, the implementer must write a
method to do this and prove it correct.
Writing and maintaining invariants is also useful in the im-
plementation. Most IronRSL methods need some constraints
on the concrete state they start with. For instance, without
some constraint on the size of the log, we cannot prove that
the method that serializes it can fit the result into a UDP
packet. We incorporate this constraint (and many others) into
an invariant over the concrete state. Each method learns these
properties on entry and must prove them before returning.
Invariants are also a crucial part of performance optimiza-
tion. Consider, for example, theExistsProposal method
in ProposeBatch. A na¨ıve implementation would always
iterate through all votes in all 1b messages, a costly process.

Instead, we augment the host state with an additional variable,
maxOpn, and prove an invariant that no 1b message exceeds
it. Thus, in the common case that s.nextOp ≥ maxOpn,
the implementation need not scan any 1b messages.
5.1.4 IronRSL Liveness
We also prove our implementation is live: if a client repeat-
edly sends a request to all replicas, it eventually receives
a reply. No consensus protocol can be live under arbitrary
conditions [16], so this property must be qualified by as-
sumptions. We assume there exists a quorum of replicas Q, a
minimum scheduler frequency F, a maximum network delay
∆, a maximum burst sizeB, and a maximum clock errorE, all
possibly unknown to the implementation, such that (1) even-
tually, the scheduler on each replica in Q runs with frequency
at least F, never exhausting memory; (2) eventually, any mes-
sage sent between replicas in Q and/or the client arrive within
∆; (3) eventually, no replica in Q receives packets at an over-
whelming rate, i.e., each receives no more than B packets
per 10B
F + 1 time units; (4) whenever a replica in Q reads its
clock, the reading differs from true global time by at most
E; and (5) no replica in Q ever stops making progress due to
reaching an overflow-prevention limit.
Our proof strategy is as follows. First, we use our library’s
round-robin scheduler proofs to prove that our protocol fairly
schedules each action (§4.3). Next, we prove that eventually
no replica in Q has a backlog of packets in its queue, so
thereafter sending a message among replicas in Q leads to
the receiver acting on that message within a certain bound.
Next, using WF1 (§4.4), we prove that if the client’s request
is never executed, then for any time period T , eventually a
replica in Q becomes the undisputed leader for that period.
Finally, using bounded-time WF1 variants (§4.4), we prove
there exists a T such that an undisputed leader can ensure the
request gets executed and responded to within T .
5.2 IronKV: A Sharded Key-Value Store
We also apply the IronFleet methodology to build IronKV,
a system that uses distribution for a completely different
purpose: to scale its throughput by dynamically sharding a
key-value store across a set of nodes.
The high-level spec of IronKV’s state machine is concise:
it is simply a hash table, as shown in Figure 11.
5.2.1 The Distributed-Protocol Layer
Each host’s state consists of a hash table storing a subset
of the key space and a “delegation map” mapping each key
to the host responsible for it. On protocol initialization, one
designated host is responsible for the entire key space; thus,
each host’s delegation map maps every key to that host.
To gain throughput and to relieve hot spots, IronKV allows
an administrator to delegate sequential key ranges (shards) to
other hosts. When a host receives such an order, it sends the
corresponding key-value pairs to the intended recipient and
updates its delegation map to reflect the new owner.
type Hashtable = map<Key,Value>
type OptValue = ValuePresent(v:Value) | ValueAbsent
predicate SpecInit (h:Hashtable) { h == map [] }
predicate Set (h:Hashtable,h’:Hashtable,
k:Key, ov: OptValue)
{ h’ == if ov.ValuePresent? then h[k := ov.v]
else map ki | ki in h && ki!=k :: h[ki] }
predicate Get (h:Hashtable,h’:Hashtable,
k:Key, ov: OptValue)
{ h’ == h && ov == if k in h then ValuePresent (h[k])
else ValueAbsent () }
predicate SpecNext (h:Hashtable,h’:Hashtable)
{ exists k, ov :: Set(h,h’,k,ov) || Get(h,h’,k,ov) }
Figure 11. Complete high-level spec for IronKV state machine
If such a message is lost, the protocol layer cannot be
shown to refine the high-level specification, since the corre-
sponding key-value pairs vanish. To avoid this, we design
a sequence-number-based reliable-transmission component
that requires each host to acknowledge messages it receives,
track its own set of unacknowledged messages, and period-
ically resend them. The liveness property we prove is that
if the network is fair (i.e., any packet sent infinitely often
is eventually delivered), then any packet submitted to the
reliable-transmission component is eventually received.
The most important invariant for IronKV’s proof is that
every key is claimed either by exactly one host or in-flight
packet. Using this invariant and the exactly-once delivery se-
mantics we prove about our reliable-transmission component,
we show that the protocol layer refines the high-level spec.
5.2.2 The Implementation Layer
As in IronRSL, we prove that modifications to a host’s
concrete state refine changes to the protocol-layer state. The
delegation map, however, poses a challenge unique to IronKV.
The protocol layer uses an infinite map with an entry for
every possible key. However, the implementation layer must
use concrete data types with bounded size and reasonable
performance. Thus, we implement and prove correct an
efficient data structure in which each host keeps only a
compact list of key ranges, along with the identity of the host
responsible for each range. This complexity we introduce
for the sake of performance creates opportunities for bugs.
However, by establishing invariants about the data structure
(e.g., the ranges are kept in sorted order), we prove that it
refines the abstract infinite map used by the protocol layer.
This lets us introduce this complex data structure without risk
of data loss or any other error.
5.3 Common Libraries
In developing IronRSL and IronKV, we have written and ver-
ified several generic libraries useful for distributed systems.
Generic refinement. A common task is proving that an op-
eration on concrete implementation-layer objects refines the
corresponding operation on protocol-layer objects. For exam-
ple, IronRSL’s implementation uses a map from uint64s
to IP addresses where the protocol uses a map from mathe-

matical integers to abstract node identifiers. In the proof, we
must show that removing an element from the concrete map
has the same effect on the abstract version.
To simplify such tasks, we have built a generic library for
reasoning about refinement between common data structures,
such as sequences and maps. Given basic properties about
the relationship between the concrete types and the abstract
types, e.g., that the function mapping concrete map keys to
abstract maps keys is injective, the library shows that various
concrete map operations, such as element lookup, addition,
and removal, refine the corresponding abstract operations.
Marshalling and parsing. All distributed systems need to
marshal and parse network packets, a tedious task prone to
bugs. Both tasks necessarily involve significant interaction
with the heap, since packets are ultimately represented as ar-
rays of bytes. Unfortunately, even state-of-the-art verification
tools struggle to verify heap operations (§6.2). Hence, we
have written and verified a generic grammar-based parser and
marshaller to hide this pain from developers. For each dis-
tributed system, the developer specifies a high-level grammar
for her messages. To marshal or unmarshal, the developer
simply maps between her high-level structure and a generic
data structure that matches her grammar. The library handles
the conversion to and from a byte array.
As evidence for the library’s utility, we initially wrote
an IronRSL-specific library. This took a person-month, and
relatively little of this code would have been useful in other
contexts. Dissatisfied, we built the generic library. This
required several more weeks, but given the generic library,
adding the IronRSL-specific portions only required two hours;
the IronKV-specific portions required even less.
Collection Properties. Another common task for distributed
systems is reasoning about properties of sequences, sets,
maps, etc. For instance, many IronRSL operations require
reasoning about whether a set of nodes form a quorum.
Thus, we have developed a library proving many useful
relationships about such collections. For example, one lemma
proves that if two sets are related by an injective function,
then their sizes are the same.
6. Lessons Learned
We summarize additional lessons we learned, beyond us-
ing invariant quantifier hiding (§3.3) and always-enabled ac-
tions (§4.2), useful for future developers of verified systems.
6.1 Use the Set of Sent Messages in Invariants
The IronFleet network model is monotonic: once a message
is sent, it is kept in a ghost state variable forever. This is
necessary to prove that the system behaves correctly even
if the network delivers messages arbitrarily late. Since the
set of messages can only grow, it is often easy to prove
invariants about it. In contrast, an invariant that reasons over
mutable host state is harder to prove. Thus, where possible,
it is useful to have invariants be properties only of the set of
messages sent so far, as is often done in proofs of security for
cryptographic protocols [8]. Essentially, the system’s network
model provides this set as a free “history variable” [1].
6.2 Model Imperative Code Functionally
Verifying imperative code is challenging compared with
verifying purely functional code, even when using a state-
of-the-art tool like Dafny that is designed for imperative
programs (§2.2). Thus, we found it profitable to implement
the system in two stages. First, we develop an implementation
using immutable value (functional) types and show that it
refines the protocol layer. Avoiding heap reasoning simplifies
the refinement proof, but, it produces a slow implementation,
since it cannot exploit the performance of heap references.
In the second stage, we replace the value types with mutable
heap types, improving performance while solving only a
narrow verification problem.
We apply this pattern in building IronRSL and IronKV;
e.g., the functional implementation manipulates IP addresses
as value types and the performant one uses references to OS
handles. This strategy takes advantage of Dafny’s support for
mixing functional programming and imperative programming
styles: we can first run the functional code and measure its
performance, then optimize the performance-critical sections
into imperative heap-based code as needed. Using a language
without good functional programming support (such as C)
would have made it harder to pursue this strategy.
6.3 Use Automation Judiciously
Automated verification tools reduce the human effort needed
to complete a proof, but they often require additional guid-
ance from the developer in order to find a proof, or, equally
importantly, to find a proof in a reasonable amount of time.
6.3.1 Automation Successes
In many cases, Dafny’s automated reasoning allows the
developer to write little or no proof annotation. For instance,
Dafny excels at automatically proving statements about linear
arithmetic. Also, its heuristics for dealing with quantifiers,
while imperfect, often produce proofs automatically.
Dafny can also prove more complex statements auto-
matically. For instance, the lemma proving that IronRSL’s
ImplNext always meets the reduction-enabling obligation
consists of only two lines: one for the precondition and one
for the postcondition. Dafny automatically enumerates all ten
possible actions and all of their subcases, and observes that
all of them produce I/O sequences satisfying the property.
Similarly, automated reasoning allows many invariant
proofs to be quite brief, by reasoning as follows: If the invari-
ant about a host’s state holds in step i but not i + 1, the host
must have taken some action. However, none of the actions
can cause the invariant to stop holding. Typically, this last
part requires no proof annotation as the verifier can internally
enumerate all cases, even for IronRSL with its many compli-
cated actions. Sometimes the verifier cannot handle a tricky
case automatically, in which case the developer must insert

proof annotations. However, even then, the developer need
not mention, let alone enumerate, the other cases.
6.3.2 Automation Challenges
Even the fastest automated verification tools can take a long
time to explore a huge search space. By default, Dafny reveals
all predicate definitions to its SMT solver Z3, potentially
giving Z3 a large search space. For example, each distributed
protocol’s HostNext transitively includes almost every
other definition in the protocol. Similarly, message-parsing
code refers to a large tree of possible message types. Having
such big trees in scope exposes the SMT solver to a bounded
but still large search space, e.g., any mention of a state invokes
every predicate about states.
To keep verification time manageable and avoid verifier
timeouts, we use Dafny’s opaque attribute and reveal
directive to selectively hide irrelevant definitions from the
SMT solver, and reveal them only when needed to complete a
proof [21]. This leads to a more modular style of verification.
In addition to hiding large definitions, we also useopaque
to hide logic features that are hard to automate. For example,
we mark recursive predicate definitions opaque to prevent the
solver from blindly unrolling the definitions too many times.
To provide greater flexibility, we modify Dafny to also
support a fuel attribute for functions. Fuel controls how
many times the SMT solver may expand a function’s defini-
tion. Giving a function zero fuel is equivalent to marking the
function opaque, while giving a fuel of five allows the solver
to unroll a recursive function up to five times. By allowing
the programmer to specify a function’s fuel at the scope of
a statement, method, class, module, or program, we allow
different portions of the code to be more or less aggressive
about revealing function definitions.
Formulas that make heavy use of quantifiers (forall and
exists) may also lead to timeouts because the SMT solver can
instantiate the quantifiers more than it needs to, depending
on which triggers the solver chooses to control instantiation.
In many places, we adopt coding styles that avoid quantifiers
(§3.3). In other places, when we find the default triggers in
Dafny overly liberal, leading to too many instantiations, we
modify Dafny to use more cautious triggers. In some cases,
we also annotate our Dafny code with manual triggers to
reduce instantiations. In particularly problematic formulas,
such as chains of alternating quantifiers (e.g., for all X there
exists a Y such that for all Z...) and set comprehensions,
we mark the containing predicate opaque. Temporal logic
formulas can easily lead to alternating quantifiers, so we
define□ and ♦ to be opaque by default.
7. Evaluation
IronFleet’s premise is that automated verification is a viable
engineering approach, ready for developing real distributed
systems. We evaluate that hypothesis by answering the follow-
ing questions: (1) How does verification affect the develop-
Spec Impl Proof Time to Verify
(source lines of code) (minutes)
High-Level Spec:
IronRSL 85 – – –
IronKV 34 – – –
Temporal Logic 208 – – –
Distributed Protocol:
IronRSL Protocol – – 1202 4
Refinement 35 – 3379 26
Liveness 167 – 7869 115
IronKV Protocol – – 726 2
Refinement 36 – 3998 12
Liveness 98 – 2093 23
TLA Library – – 1824 2
Implementation:
IO/Native Interface 591 – – –
Common Libraries 134 833 7690 13
IronRSL 6 2941 7535 152
IronKV 6 1340 2937 42
Total 1400 5114 39253 395
Figure 12. Code sizes and verification times.
ment of distributed systems? (2) How does the performance
of a verified system compare with an unverified one?
7.1 Developer Experience
To assess practicality, we evaluate the developer experience
as well as the effort required to produce verified systems.
The experience of producing verified software shares
some similarities with that of unverified software. Dafny
provides near-real-time IDE-integrated feedback. Hence, as
the developer writes a given method or proof, she typically
sees feedback in 1–10 seconds indicating whether the verifier
is satisfied. To ensure the entire system verifies, our build
system tracks dependencies across files and outsources, in
parallel, each file’s verification to a cloud virtual machine.
Thus, while a full integration build done serially requires
approximately six hours, in practice, the developer rarely
waits more than 6–8 minutes, which is comparable to any
other large system integration build.
An IronFleet developer must write a formal trusted spec,
a distributed protocol layer, and proof annotations to help the
verifier see the refinements between them. Figure 12 quanti-
fies this effort by reporting the amount of proof annotation
required for each layer of the system. We count all non-spec,
non-executable code as proof annotation; this includes, for
example, requires and ensures clauses, loop invariants, and
all lemmas and invocations thereof. Note that the high-level
trusted specification for IronRSL is only 85 SLOC, and for
IronKV it is only 34, making them easy to inspect for correct-
ness. At the implementation layer, our ratio of proof annota-
tion to executable code is 3.6 to 1. We attribute this relatively
low ratio to our proof-writing techniques (§3.3,§4.1,§6) and
our automated tools (§6.3.1).
In total, developing the IronFleet methodology and apply-
ing it to build and verify two real systems required approxi-
mately 3.7 person-years.

 1
 10
 100
 1000
10 20 30 40 50
Latency
(ms)
Throughput (kilo reqs/s)
IronRSL
Baseline
IronRSL (Batch)
Baseline (Batch)
Figure 13. IronRSL’s performance is competitive with an unveri-
fied baseline. Results averaged over 3 trials.
In exchange for this effort, IronFleet produces a provably
correct implementation with desirable liveness properties. In-
deed, except for unverified components like our C# client,
both IronRSL (including replication, view changes, log trun-
cation, batching, etc.) as well as IronKV (including delegation
and reliable delivery) worked the first time we ran them.
7.2 Performance of Verified Distributed Systems
A reasonable criticism of any new toolchain focused on veri-
fication is that its structure might impair runtime efficiency.
While we focus most of our energy on overcoming verifica-
tion burdens, we also try to produce viable implementations.
Our IronRSL experiments run three replicas on three
separate machines, each equipped with an Intel Xeon L5630
2.13 GHz processor and 12 GB RAM, connected over a
1 Gbps network. Our IronKV experiments use two such
machines connected over a 10 Gbps network.
IronRSL. Workload is offered by 1–256 parallel client
threads, each making a serial request stream and measur-
ing latency. As an unverified baseline, we use the MultiPaxos
Go-based implementation from the EPaxos codebase [15, 45]
For both systems, we use the same application state machine:
it maintains a counter and it increments the counter for every
client request. Figure 13 summarizes our results. We find that
IronRSL’s peak throughput is within 2.4× of the baseline.
IronKV. To measure the throughput and latency of IronKV,
we preload the server with 1000 keys, then run a client with
1–256 parallel threads; each thread generates a stream of Get
(or Set) requests in a closed loop. As an unverified baseline,
we use Redis [ 57], a popular key/value store written in C
and C++, with the client-side write buffer disabled. For both
systems, we use 64-bit unsigned integers as keys and byte
arrays of varying sizes as values. Figure 14 summarizes our
results. We find that IronKV’s performance is competitive
with that of Redis.
As a final note, in all our experiments the bottleneck was
the CPU (not the memory, disk, or network).
8. Discussion and Future Work
§7.1 shows that in exchange for strong guarantees (which
depend on several assumptions, per§2.5), IronFleet requires
considerably more developer effort. Furthermore, in our
experience, there is a distinct learning curve when bringing
10
20
30
40
50
128B 1KB
Get
8KB 128B 1KB
Set
8KB
Peak throughput
(kilo reqs/sec)
 IronKV
Redis
Figure 14. IronKV’s performance is competitive with Redis, an
unverified key-value store. Results averaged over 3 trials.
aboard developers unfamiliar with writing verified code. Most
developers would prefer to use a language like C++, so
enabling that is an important topic of future work.
§7.2 shows that while our systems achieve respectable
performance, they do not yet match that of the unverified
baselines. Some of that gap stems directly from our use of
verification. Verifying mutable data structures is challenging
(§6.2), and our measurements indicate that this is a significant
bottleneck for our code. The baselines we compare against
have been highly optimized; we have also optimized our
code, but each optimization must be proven correct. Hence,
given a fixed time budget, IronFleet will likely produce fewer
optimizations. IronFleet also pays a penalty for compiling to
C#, which imposes run-time overhead to enforce type safety
on code that provably does not need it.
More fundamentally, aiming for full verification makes
it challenging to reuse existing libraries, e.g., for optimized
packet serialization. Before our previous [ 21] and current
work (§5.3), Dafny had no standard libraries, necessitating
significant work to build them; more such work lies ahead.
While our systems are more full-featured than previous
work (§9), they still lack many standard features offered by
the unverified baselines. Some features, such as reconfigu-
ration in IronRSL, only require additional developer time.
Other features require additional verification techniques; e.g.,
post-crash recovery requires reasoning about the effects of
machine crashes that wipe memory but not disk.
In future work, we aim to mechanically verify our reduc-
tion argument and prove that our implementation run its main
loop in bounded time [2], never exhausts memory, and never
reaches its overflow-prevention limit under reasonable condi-
tions, e.g., if it never performs more than 264 operations.
9. Related Work
9.1 Protocol Verification
Distributed system protocols are known to be difficult to
design correctly. Thus, a systems design is often accompanied
by a formal English proof of correctness, typically relegated
to a technical report or thesis. Examples include Paxos [55],
the BFT protocol for Byzantine fault tolerance [ 6, 7], the
reconfiguration algorithm in SMART [23, 41], Raft [49, 50],
Zookeeper’s consistent broadcast protocol Zab [ 28, 29],
Egalitarian Paxos [44, 45], and the Chord DHT [62, 63].

However, paper proofs, no matter how formal, can con-
tain errors. Zane showed that the “provably correct” Chord
protocol, when subjected to Alloy abstract model checking,
maintains none of its published invariants [71]. Thus, some re-
searchers have gone further and generated machine-checkable
proofs. Kellom¨aki created a proof of the Paxos consensus pro-
tocol checked in PVS [30]. Lamport’s TLAPS proof system
has been used to prove safety, but not liveness, properties of
the BFT protocol [38]. In all such cases, the protocols proven
correct have been much smaller and simpler than ours. For
instance, Kellom¨aki’s and Lamport’s proofs concerned single-
instance Paxos and BFT, which make only one decision total.
9.2 Model Checking
Model checking exhaustively explores a system’s state space,
testing whether a safety property holds in every reachable
state. This combinatorial exploration requires that the system
be instantiated with finite, typically tiny, parameters. As
a result, a positive result provides only confidence, not
proof of safety; furthermore, that confidence depends on the
modeler’s wisdom in parameter selection. Model checking
has been applied to myriad systems including a Python
implementation of Paxos [26]; Mace implementations of a
variety of distributed systems [31]; and, via MODIST [69],
unmodified binaries of Berkeley DB, MPS Paxos, and the
PacificA primary-backup replication system.
Model checking scales poorly to complex distributed
specs [4]. Abstract interpretation can help with such scal-
ing but does not fundamentally eliminate model checking’s
limitations. For instance, Zave’s correction to Chord uses the
Alloy model checker but only to partially automate the proof
of a single necessary invariant [72].
9.3 System Verification
The recent increase in the power of software verification has
emboldened several research groups to use it to prove the
correctness of entire systems implementations. seL4 is a mi-
crokernel written in C [32], with full functional correctness
proven using the Isabelle/HOL theorem prover. mCertiKOS-
hyp [19] is a small verified hypervisor, whose verification in
the Coq interactive proof assistant places a strong emphasis
on modularity and abstraction. ExpressOS [43] uses Dafny
to sanity-check a policy manager for a microkernel. Our Iron-
clad project [21] shows how to completely verify the security
of sensitive services all the way down to the assembly. Iron-
Fleet differs by verifying a distributed implementation rather
than code running on a single machine, and by verifying
liveness, as well as safety, properties.
Researchers have also begun to apply software verification
to distributed systems. Ridge [58] proves the correctness of
a persistent message queue written in OCaml; however, his
system is substantially smaller in scale than ours and has no
proven liveness properties.
Schiper et al. [ 60] verify the correctness of a Paxos
implementation by building it in EventML [56] and proving
correctness, but not liveness, with the NuPRL prover [ 10].
However, they do not verify the state machine replication
layer of this Paxos implementation, only the consensus
algorithm, ignoring complexities such as state transfer. They
also make unclear assumptions about network behavior. In
contrast to our methodology, which exploits multiple levels
of abstraction and refinement, the EventML approach posits
a language below which all code generation is automatic, and
above which a human can produce a one-to-one refinement.
It is unclear if this approach will scale up to more complex
and diverse distributed systems.
In concurrent work, Wilcox et al. [67, 68] propose Verdi,
a compiler-inspired approach to building verified distributed
system implementations. With Verdi, the developer writes
and proves her system correct in Coq using a simplified
environment (e.g., a single-machine system with a perfectly
reliable network). Verdi’s verified system transformers then
convert the developer’s implementation into an equivalent
implementation that is robust in a more hostile environment;
their largest system transformer is an implementation of Raft
that adds fault tolerance. Compared with IronFleet, Verdi
offers a cleaner approach to composition. Unlike IronRSL,
at present Verdi’s Raft implementation does not support
verified marshalling and parsing, state transfer, log truncation,
dynamic view-change timeouts, a reply cache, or batching.
Also, Verdi does not prove any liveness properties.
10. Conclusion
The IronFleet methodology slices a system into specific lay-
ers to make verification of practical distributed system im-
plementations feasible. The high-level spec gives the sim-
plest description of the system’s behavior. The protocol
layer deals solely with distributed protocol design; we con-
nect it to the spec using TLA+ [ 36] style verification. At
the implementation layer, the programmer reasons about a
single-host program without worrying about concurrency.
Reduction and refinement tie these individually-feasible
components into a methodology that scales to practically-
sized concrete implementations. This methodology admits
conventionally-structured implementations capable of pro-
cessing up to 18,200 requests/second (IronRSL) and 28,800
requests/second (IronKV), performance competitive with un-
verified reference implementations.
Acknowledgments
We thank Rustan Leino for not just building Dafny but also
cheerfully providing ongoing guidance and support in improv-
ing it. We thank Leslie Lamport for useful discussions about
refinement and formal proofs, particularly proofs of liveness.
We thank Shaz Qadeer for introducing us to the power of
reduction. We thank Andrew Baumann, Ernie Cohen, Galen
Hunt, Lidong Zhou, and the anonymous reviewers for useful
feedback. Finally, we thank our shepherd Jim Larus for his
interactive feedback that significantly improved the paper.

References
[1] ABADI , M., AND LAMPORT , L. The existence of refinement
mappings. Theoretical Computer Science 82, 2 (May 1991).
[2] BLACKHAM , B., S HI, Y., CHATTOPADHYAY, S., R OYCHOUD -
HURY, A., AND HEISER , G. Timing analysis of a protected
operating system kernel. In Proceedings of the IEEE Real-Time
Systems Symposium (RTSS) (2011).
[3] BOKOR , P., K INDER , J., S ERAFINI , M., AND SURI , N.
Efficient model checking of fault-tolerant distributed protocols.
In Proceedings of the Conference on Dependable Systems and
Networks (DSN) (2011).
[4] BOLOSKY , W. J., D OUCEUR , J. R., AND HOWELL , J. The
Farsite project: a retrospective. ACM SIGOPS Operating
Systems Review 41 (2) (April 2007).
[5] BURROWS , M. The Chubby lock service for loosely-coupled
distributed systems. In Proceedings of the Symposium on
Operating Systems Design and Implementation (OSDI) (2006).
[6] CASTRO , M., AND LISKOV, B. A correctness proof for a prac-
tical Byzantine-fault-tolerant replication algorithm. Tech. Rep.
MIT/LCS/TM-590, MIT Laboratory for Computer Science,
June 1999.
[7] CASTRO , M., AND LISKOV, B. Practical Byzantine fault
tolerance and proactive recovery. ACM Transactions on
Computer Systems (TOCS) 20, 4 (Nov. 2002).
[8] COHEN , E. First-order verification of cryptographic protocols.
Journal of Computer Security 11, 2 (2003).
[9] COHEN , E., AND LAMPORT , L. Reduction in TLA. In
Concurrency Theory (CONCUR) (1998).
[10] CONSTABLE , R. L., A LLEN , S. F., B ROMLEY , H. M.,
CLEAVELAND , W. R., C REMER , J. F., H ARPER , R. W.,
HOWE, D. J., K NOBLOCK , T. B., M ENDLER , N. P., P ANAN -
GADEN , P., S ASAKI , J. T., AND SMITH , S. F. Implement-
ing Mathematics with the Nuprl Proof Development System .
Prentice-Hall, Inc., 1986.
[11] DE MOURA , L. M., AND BJØRNER , N. Z3: An efficient
SMT solver. In Proceedings of the Conference on Tools
and Algorithms for the Construction and Analysis of Systems
(2008).
[12] DETLEFS , D., N ELSON , G., AND SAXE , J. B. Simplify: A
theorem prover for program checking. In J. ACM (2003).
[13] DOUCEUR , J. R., AND HOWELL , J. Distributed directory
service in the Farsite file system. In Proceedings of the
Symposium on Operating Systems Design and Implementation
(OSDI) (November 2006).
[14] ELMAS , T., Q ADEER , S., AND TASIRAN , S. A calculus of
atomic actions. In Proceedings of the ACM Symposium on
Principles of Programming Languages (POPL) (Jan. 2009).
[15] EPaxos code. https://github.com/efficient/
epaxos/, 2013.
[16] FISCHER , M. J., L YNCH , N. A., AND PATERSON , M. S.
Impossibility of distributed consensus with one faulty process.
Journal of the ACM (JACM) 32, 2 (April 1985).
[17] FLOYD , R. Assigning meanings to programs. In Proceedings
of Symposia in Applied Mathematics (1967).
[18] GARLAND , S. J., AND LYNCH , N. A. Using I/O automata for
developing distributed systems. Foundations of Component-
Based Systems 13 (2000).
[19] GU, R., K OENIG , J., R AMANANANDRO , T., S HAO, Z., W U,
X. N., W ENG , S.-C., Z HANG , H., AND GUO, Y. Deep speci-
fications and certified abstraction layers. InProceedings of the
ACM Symposium on Principles of Programming Languages
(POPL) (2015).
[20] GUO, H., W U, M., Z HOU , L., H U, G., Y ANG , J., AND
ZHANG , L. Practical software model checking via dynamic
interface reduction. In Proceedings of the ACM Symposium on
Operating Systems Principles (SOSP) (2011), ACM.
[21] HAWBLITZEL , C., H OWELL , J., L ORCH , J. R., N ARAYAN,
A., P ARNO , B., Z HANG , D., AND ZILL , B. Ironclad apps:
End-to-end security via automated full-system verification. In
Proceedings of the USENIX Symposium on Operating Systems
Design and Implementation (OSDI) (October 2014).
[22] HOARE , T. An axiomatic basis for computer programming.
Communications of the ACM 12 (1969).
[23] HOWELL , J., L ORCH , J. R., AND DOUCEUR , J. R. Cor-
rectness of Paxos with replica-set-specific views. Tech. Rep.
MSR-TR-2004-45, Microsoft Research, 2004.
[24] HUNT, P., K ONAR , M., J UNQUEIRA , F. P., AND REED , B.
ZooKeeper: Wait-free coordination for Internet-scale systems.
In Proceedings of the USENIX Annual Technical Conference
(ATC) (2010).
[25] IronFleet code. https://research.microsoft.com/
projects/ironclad/, 2015.
[26] JONES , E. Model checking a Paxos implementation. http:
//www.evanjones.ca/model-checking-paxos.
html, 2009.
[27] JOSHI , R., L AMPORT , L., M ATTHEWS , J., TASIRAN , S., T UT-
TLE , M., AND YU, Y. Checking cache coherence protocols
with TLA+. Journal of Formal Methods in System Design 22,
2 (March 2003).
[28] JUNQUEIRA , F. P., R EED , B. C., AND SERAFINI , M. Dissect-
ing Zab. Tech. Rep. YL-2010-007, Yahoo! Research, Decem-
ber 2010.
[29] JUNQUEIRA , F. P., R EED , B. C., AND SERAFINI , M. Zab:
High-performance broadcast for primary-backup systems. In
Proceedings of the IEEE/IFIP Conference on Dependable
Systems & Networks (DSN) (2011).
[30] KELLOM ¨AKI , P. An annotated specification of the consensus
protocol of Paxos using superposition in PVS. Tech. Rep. 36,
Tampere University of Technology, 2004.
[31] KILLIAN , C. E., A NDERSON , J. W., B RAUD , R., J HALA , R.,
AND VAHDAT, A. M. Mace: Language support for building
distributed systems. In Proceedings of the ACM Conference on
Programming Language Design and Implementation (PLDI)
(2007).
[32] KLEIN , G., A NDRONICK , J., E LPHINSTONE , K., M URRAY,
T., S EWELL , T., K OLANSKI , R., AND HEISER , G. Com-
prehensive formal verification of an OS microkernel. ACM
Transactions on Computer Systems 32, 1 (2014).

[33] LAMPORT , L. A theorem on atomicity in distributed algo-
rithms. Tech. Rep. SRC-28, DEC Systems Research Center,
May 1988.
[34] LAMPORT , L. The temporal logic of actions. ACM Trans-
actions on Programming Languages and Systems 16, 3 (May
1994).
[35] LAMPORT , L. The part-time parliament. ACM Transactions
on Computer Systems (TOCS) 16, 2 (May 1998).
[36] LAMPORT , L. Specifying Systems: The TLA+ Languange and
Tools for Hardware and Software Engineers. Addison-Wesley,
2002.
[37] LAMPORT , L. The PlusCal algorithm language. InProceedings
of the International Colloquium on Theoretical Aspects of
Computing (ICTAC) (Aug. 2009).
[38] LAMPORT , L. Byzantizing Paxos by refinement. In Proceed-
ings of the International Conference on Distributed Computing
(DISC) (2011).
[39] LEINO , K. R. M. Dafny: An automatic program verifier for
functional correctness. In Proceedings of the Conference on
Logic for Programming, Artificial Intelligence, and Reasoning
(LPAR) (2010).
[40] LIPTON , R. J. Reduction: A method of proving properties of
parallel programs. Communications of the ACM, 18, 12 (1975).
[41] LORCH , J. R., A DYA, A., B OLOSKY , W. J., C HAIKEN , R.,
DOUCEUR , J. R., AND HOWELL , J. The SMART way to
migrate replicated stateful services. In Proceedings of the
ACM European Conference on Computer Systems (EuroSys)
(2006).
[42] LU, T., M ERZ , S., W EIDENBACH , C., B ENDISPOSTO , J.,
LEUSCHEL , M., R OGGENBACH , M., M ARGARIA , T., PAD-
BERG , J., T AENTZER , G., L U, T., M ERZ , S., AND WEI-
DENBACH , C. Model checking the Pastry routing protocol.
In Workshop on Automated Verification of Critical Systems
(2010).
[43] MAI, H., P EK, E., X UE, H., K ING , S. T., AND MADHUSU -
DAN, P. Verifying security invariants in ExpressOS. In Pro-
ceedings of the ACM Conference on Architectural Support for
Programming Languages and Operating Systems (ASPLOS)
(March 2013).
[44] MORARU , I., A NDERSEN , D. G., AND KAMINSKY , M. A
proof of correctness of Egalitarian Paxos. Tech. Rep. CMU-
PDL-13-111, Carnegie Mellon University Parallel Data Labo-
ratory, August 2013.
[45] MORARU , I., A NDERSEN , D. G., AND KAMINSKY , M. There
is more consensus in egalitarian parliaments. In Proceedings of
the ACM Symposium on Operating System Principles (SOSP)
(2013).
[46] MUSUVATHI , M., P ARK , D., C HOU , A., E NGLER , D., AND
DILL , D. L. CMC: A pragmatic approach to model checking
real code. InProceedings of the USENIX Symposium Operating
Systems Design and Implementation (OSDI) (2002).
[47] MUSUVATHI , M., Q ADEER , S., B ALL , T., B ASLER , G.,
NAINAR , P. A., AND NEAMTIU , I. Finding and reproduc-
ing heisenbugs in concurrent programs. In Proceedings of
the USENIX Symposium on Operating Systems Design and
Implementation (OSDI) (2008).
[48] NEWCOMBE , C., R ATH, T., Z HANG , F., M UNTEANU , B.,
BROOKER , M., AND DEARDEUFF , M. How Amazon Web
Services uses formal methods. Communications of the ACM
58, 4 (Apr. 2015).
[49] ONGARO , D. Consensus: Bridging theory and practice. Tech.
Rep. Ph.D. thesis, Stanford University, August 2014.
[50] ONGARO , D., AND OUSTERHOUR , J. In search of an under-
standable consensus algorithm. In Proceedings of the USENIX
Annual Technical Conference (ATC)(June 2014).
[51] P ARKINSON , M. The next 700 separation logics. In Proceed-
ings of the IFIP Conference on Verified Software: Theories,
Tools, Experiments (VSTTE) (Aug. 2010).
[52] PARNO , B., L ORCH , J. R., D OUCEUR , J. R., M ICKENS , J.,
AND MCCUNE , J. M. Memoir: Practical state continuity for
protected modules. In Proceedings of the IEEE Symposium on
Security and Privacy (May 2011).
[53] PEK, E., AND BOGUNOVIC , N. Formal verification of com-
munication protocols in distributed systems. In Proceedings of
the Joint Conferences on Computers in Technical Systems and
Intelligent Systems (2003).
[54] PRIOR , A. N. Papers on Time and Tense. Oxford University
Press, 1968.
[55] PRISCO , R. D., AND LAMPSON , B. Revisiting the Paxos
algorithm. In Proceedings of the International Workshop on
Distributed Algorithms (WDAG) (1997).
[56] RAHLI , V. Interfacing with proof assistants for domain
specific programming using EventML. In Proceedings of
the International Workshop on User Interfaces for Theorem
Provers (UITP) (July 2012).
[57] Redis. http://redis.io/. Implementation used: ver-
sion 2.8.2101 of the MSOpenTech distribution https://
github.com/MSOpenTech/redis, 2015.
[58] RIDGE , T. Verifying distributed systems: The operational
approach. In Proceedings of the ACM Symposium on Principles
of Programming Languages (POPL) (January 2009).
[59] SAISSI , H., B OKOR , P., M UFTUOGLU , C., S URI , N., AND
SERAFINI , M. Efficient verification of distributed protocols us-
ing stateful model checking. In Proceedings of the Symposium
on Reliable Distributed Systems SRDS (Sept 2013).
[60] SCHIPER , N., R AHLI , V., VAN RENESSE , R., B ICKFORD ,
M., AND CONSTABLE , R. Developing correctly replicated
databases using formal tools. In Proceedings of the IEEE/IFIP
Conference on Dependable Systems and Networks (DSN)(June
2014).
[61] SCIASCIO , E., D ONINI , F., M ONGIELLO , M., AND
PISCITELLI , G. Automatic support for verification of secure
transactions in distributed environment using symbolic model
checking. In Conference on Information Technology Interfaces
(June 2001), vol. 1.
[62] STOICA , I., M ORRIS , R., K ARGER , D., K AASHOEK , M. F.,
AND BALAKRISHNAN , H. Chord: A scalable peer-to-peer
lookup service for Internet applications. In Proceedings of the
ACM SIGCOMM Conference on Applications, Technologies,
Architectures, and Protocols for Computer Communication
(August 2001).

[63] STOICA , I., M ORRIS , R., K ARGER , D., K AASHOEK , M. F.,
AND BALAKRISHNAN , H. Chord: A scalable peer-to-
peer lookup service for Internet applications. Tech. Rep.
MIT/LCS/TR-819, MIT Laboratory for Computer Science,
March 2001.
[64] TASIRAN , S., Y U, Y., BATSON , B., AND KREIDER , S. Using
formal specifications to monitor and guide simulation: Verify-
ing the cache coherence engine of the Alpha 21364 micropro-
cessor. In International Workshop on Microprocessor Test and
Verification (June 2002), IEEE.
[65] WANG , L., AND STOLLER , S. D. Runtime analysis of
atomicity for multithreaded programs. IEEE Transactions
on Software Engineering 32 (Feb. 2006).
[66] WANG , Y., K ELLY, T., K UDLUR , M., L AFORTUNE , S., AND
MAHLKE , S. A. Gadara: Dynamic deadlock avoidance for
multithreaded programs. In Proceedings of the USENIX
Symposium on Operating Systems Design and Implementation
(OSDI) (December 2008).
[67] WILCOX , J., W OOS , D., P ANCHEKHA , P., T ATLOCK , Z.,
WANG , X., E RNST , M., AND ANDERSON , T. UW CSE
News: UW CSE’s Verdi team completes first full formal
verification of Raft consensus protocol. https://news.
cs.washington.edu/2015/08/07/, August 2015.
[68] WILCOX , J. R., W OOS , D., P ANCHEKHA , P., T ATLOCK ,
Z., WANG , X., E RNST , M. D., AND ANDERSON , T. Verdi:
A framework for implementing and formally verifying dis-
tributed systems. In Proceedings of the ACM Conference on
Programming Language Design and Implementation (PLDI)
(June 2015).
[69] YANG , J., C HEN , T., W U, M., X U, Z., L IU, X., L IN, H.,
YANG , M., L ONG , F., Z HANG , L., AND ZHOU , L. MODIST:
Transparent model checking of unmodified distributed systems.
In Proceedings of the USENIX Symposium on Networked
Systems Design and Implementation (NSDI) (April 2009).
[70] YUAN, D., L UO, Y., Z HUANG , X., R ODRIGUES , G. R.,
ZHAO, X., Z HANG , Y., JAIN , P. U., AND STUMM , M. Sim-
ple testing can prevent most critical failures: An analysis of
production failures in distributed data-intensive systems. In
Proceedings of the USENIX Symposium on Operating Systems
Design and Implementation (OSDI) (October 2014).
[71] ZAVE, P. Using lightweight modeling to understand Chord.
ACM SIGCOMM Computer Communication Review 42 , 2
(April 2012).
[72] ZAVE, P. How to make Chord correct (using a stable base).
Tech. Rep. 1502.06461 [cs.DC], arXiv, February 2015.

Secure Untrusted Data Repository (SUNDR)
Jinyuan Li, Maxwell Krohn∗
, David Mazi`eres, and Dennis Shasha
NYU Department of Computer Science
Abstract
SUNDR is a network file system designed to store data
securely on untrusted servers. SUNDR lets clients de-
tect any attempts at unauthorized file modification by
malicious server operators or users. SUNDR’s protocol
achieves a property called fork consistency, which guar-
antees that clients can detect any integrity or consistency
failures as long as they see each other’s file modifications.
An implementation is described that performs compara-
bly with NFS (sometimes better and sometimes worse),
while offering significantly stronger security.
1 Introduction
SUNDR is a network file system that addresses a long-
standing tension between data integrity and accessibility .
Protecting data is often viewed as the problem of build-
ing a better fence around storage servers—limiting the
number of people with access, disabling unnecessary soft-
ware that might be remotely exploitable, and staying cur-
rent with security patches. This approach has two draw-
backs. First, experience shows that people frequently do
not build high enough fences (or sometimes entrust fences
to administrators who are not completely trustworthy).
Second and more important, high fences are inconvenient;
they restrict the ways in which people can access, update,
and manage data.
This tension is particularly evident for free software
source code repositories. Free software projects often
involve geographically dispersed developers committing
source changes from all around the Internet, making it
impractical to fend off attackers with firewalls. Hosting
code repositories also requires a palette of tools such as
CVS [4] and SSH [35], many of which have had remotely
exploitable bugs.
Worse yet, many projects rely on third-party host-
ing services that centralize responsibility for large
numbers of otherwise independent code repositories.
sourceforge.net, for example, hosts CVS repositories
∗now at MIT CS & AI Lab
for over 20,000 different software packages. Many of
these packages are bundled with various operating sys-
tem distributions, often without a meaningful audit. By
compromising sourceforge, an attacker can therefore in-
troduce subtle vulnerabilities in software that may even-
tually run on thousands or even millions of machines.
Such concerns are no mere academic exercise. For ex-
ample, the Debian GNU/Linux development cluster was
compromised in 2003 [2]. An unauthorized attacker used
a sniffed password and a kernel vulnerability to gain su-
peruser access to Debian’s primary CVS and Web servers.
After detecting the break-in, administrators were forced
to freeze development for several days, as they employed
manual and ad-hoc sanity checks to assess the extent of
the damage. Similar attacks have also succeeded against
Apache [1], Gnome [32], and other popular projects.
Rather than hope for invulnerable servers, we have de-
veloped SUNDR, a network file system that reduces the
need to trust storage servers in the first place. SUNDR
cryptographically protects all file system contents so that
clients can detect any unauthorized attempts to change
files. In contrast to previous Byzantine-fault-tolerant fil e
systems [6, 27] that distribute trust but assume a thresh-
old fraction of honest servers, SUNDR vests the authority
to write files entirely in users’ public keys. Even a mali-
cious user who gains complete administrative control of a
SUNDR server cannot convince clients to accept altered
contents of files he lacks permission to write.
Because of its security properties, SUNDR also creates
new options for managing data. By using SUNDR, orga-
nizations can outsource storage management without fear
of server operators tampering with data. SUNDR also en-
ables new options for data backup and recovery: after a
disaster, a SUNDR server can recover file system data
from untrusted clients’ file caches. Since clients always
cryptographically verify the file system’s state, they are
indifferent to whether data was recovered from untrusted
clients or resided on the untrusted server all along.
This paper details the SUNDR file system’s design and
implementation. We first describe SUNDR’s security pro-
tocol and then present a prototype implementation that
gives performance generally comparable to the popular

NFS file system under both an example software develop-
ment workload and microbenchmarks. Our results show
that applications like CVS can benefit from SUNDR’s
strong security guarantees while paying a digestible per-
formance penalty.
2 Setting
SUNDR provides a file system interface to remote stor-
age, like NFS [29] and other network file systems. To se-
cure a source code repository, for instance, members of a
project can mount a remote SUNDR file system on direc-
tory /sundr and use /sundr/cvsroot as a CVS reposi-
tory. All checkouts and commits then take place through
SUNDR, ensuring users will detect any attempts by the
hosting site to tamper with repository contents.
Figure 1 shows SUNDR’s basic architecture. When ap-
plications access the file system, the client software inter-
nally translates their system calls into a series offetch and
modify operations, where fetch means retrieving a file’s
contents or validating a cached local copy, and modify
means making new file system state visible to other users.
Fetch and modify, in turn, are implemented in terms of
SUNDR protocol RPCs to the server. Section 3 explains
the protocol, while Section 5 describes the server design.
To set up a SUNDR server, one runs the server software
on a networked machine with dedicated SUNDR disks
or partitions. The server can then host one or more file
systems. To create a file system, one generates a pub-
lic/private superuser signature key pair and gives the pub-
lic key to the server, while keeping the private key secret.
The private key provides exclusive write access to the root
directory of the file system. It also directly or indirectly
allows access to any file below the root. However, the
privileges are confined to that one file system. Thus, when
a SUNDR server hosts multiple file systems with different
superusers, no single person has write access to all files.
Each user of a SUNDR file system also has a signature
key. When establishing an account, users exchange public
keys with the superuser. The superuser manages accounts
with two superuser-owned file in the root directory of the
file system: .sundr.users lists users’ public keys and
numeric IDs, while.sundr.group designates groups and
their membership. To mount a file system, one must spec-
ify the superuser’s public key as a command-line argu-
ment to the client, and must furthermore give the client ac-
cess to a private key. (SUNDR could equally well manage
keys and groups with more flexible certificate schemes;
the system only requires some way for users to validate
each other’s keys and group membership.)
Throughout this paper, we use the term user to desig-
RPC
block store
fetch/
cache layer
consistency server
Server
application
Client
syscall
security layer
modify
Figure 1: Basic SUNDR architecture.
nate an entity possessing the private half of a signature
key mapped to some user ID in the .sundr.users file.
Depending on context, this can either be the person who
owns the private key, or a client using the key to act on
behalf of the user. However, SUNDR assumes a user is
aware of the last operation he or she has performed. In the
implementation, the client remembers the last operation it
has performed on behalf of each user. To move between
clients, a user needs both his or her private key and the last
operation performed on his or her behalf (concisely spec-
ified by a version number). Alternatively, one person can
employ multiple user IDs (possibly with the same public
key) for different clients, assigning all file permissions t o
a personal group.
SUNDR’s architecture draws an important distinction
between the administration of servers and the administra-
tion of file systems. To administer a server, one does not
need any private superuser keys. 1 In fact, for best secu-
rity, key pairs should be generated on separate, trusted ma-
chines, and private keys should never reside on the server,
even in memory. Important keys, such as the superuser
key, should be stored off line when not in use (for exam-
ple on a floppy disk, encrypted with a passphrase).
3 The SUNDR protocol
SUNDR’s protocol lets clients detect unauthorized at-
tempts to modify files, even by attackers in control of the
server. When the server behaves correctly, a fetch reflects
exactly the authorized modifications that happened before
it.2 We call this property fetch-modify consistency.
If the server is dishonest, clients enforce a slightly
1The server does actually have its own public key, but only to p re-
vent network attackers from “framing” honest servers; the se rver key is
irrelevant to SUNDR’s security against compromised servers.
2Formally, happens before can be any irreflexive partial order that
preserves the temporal order of non-concurrent operations ( as in Lin-
earizability [11]), orders any two operations by the same cli ent, and or-
ders a modification with respect to any other operation on the same file.
2

weaker property calledfork consistency. Intuitively, under
fork consistency, a dishonest server could cause a fetch by
a user A to miss a modify by B. However, either user will
detect the attack upon seeing a subsequent operation by
the other. Thus, to perpetuate the deception, the server
must fork the two user’s views of the file system. Put
equivalently, if A’s client accepts some modification by
B, then at least until B performed that modification, both
users had identical, fetch-modify-consistent views of the
file system.
We have formally specified fork consistency [16], and,
assuming digital signatures and a collision-resistant has h
function, proven SUNDR’s protocol achieves it [17].
Therefore, a violation of fork consistency means the un-
derlying cryptography was broken, the implementation
deviated from the protocol, or there is a flaw in our map-
ping from high-level Unix system calls to low-level fetch
and modify operations.
In order to discuss the implications of fork consistency
and to describe SUNDR, we start with a simple straw-man
file system that achieves fork consistency at the cost of
great inefficiency (Section 3.1). We then propose an im-
proved system with more reasonable bandwidth require-
ments called “Serialized SUNDR” (Section 3.3). We fi-
nally relax serialization requirements, to arrive at “con-
current SUNDR,” the system we have built (Section 3.4).
3.1 A straw-man file system
In the roughest approximation of SUNDR, the straw-man
file system, we avoid any concurrent operations and allow
the system to consume unreasonable amounts of band-
width and computation. The server maintains a single,
untrusted global lock on the file system. To fetch or mod-
ify a file, a user first acquires the lock, then performs the
desired operation, then releases the lock. So long as the
server is honest, the operations are totally ordered and
each operation completes before the next begins.
The straw-man file server stores a complete, ordered
list of every fetch or modify operation ever performed.
Each operation also contains a digital signature from the
user who performed it. The signature covers not just the
operation but also the complete history of all operations
that precede it . For example, after five operations, the
history might appear as follows:
sig
mod(f3)
user B
fetch(f3)fetch(f2)
user A
sig sig
user A
mod(f2)
sig
fetch(f2)
sig
user Buser A
To fetch or modify a file, a client acquires the global
lock, downloads the entire history of the file system, and
validates each user’s most recent signature. The client
also checks that its own user’s previous operation is in
the downloaded history (unless this is the user’s very first
operation on the file system).
The client then traverses the operation history to con-
struct a local copy of the file system. For each modify en-
countered, the client additionally checks that the operation
was actually permitted, using the user and group files to
validate the signing user against the file’s owner or group.
If all checks succeed, the client appends a new operation
to the list, signs the new history, sends it to the server, and
releases the lock. If the operation is a modification, the
appended record contains new contents for one or more
files or directories.
Now consider, informally, what a malicious server
can do. To convince a client of a file modification, the
server must send it a signed history. Assuming the server
does not know users’ keys and cannot forge signatures,
any modifications clients accept must actually have been
signed by an authorized user. The server can still trick
users into signing inappropriate histories, however, by
concealing other users’ previous operations. For instance,
consider what would happen in the last operation of the
above history if the server failed to show user B the most
recent modification to file f2. Users A and B would sign
the following histories:
user B:
user A:
sig
sig
sig
sig
fetch(f2)
sig
sig
sig
sig
user B user A user A
user Buser Auser Buser A
fetch(f2)
mod(f3) fetch( f3) mod( f2)
mod(f3) fetch( f3)
user A
fetch(f2)
Neither history is a prefix of the other. Since clients
always check for their own user’s previous operation in
the history, from this point on,A will sign only extensions
of the first history and B will sign only extensions of the
second. Thus, while before the attack the users enjoyed
fetch-modify consistency, after the attack the users have
been forked.
Suppose further that the server acts in collusion with
malicious users or otherwise comes to possess the signa-
ture keys of compromised users. If we restrict the analysis
to consider only histories signed by honest (i.e., uncom-
promised) users, we see that a similar forking property
holds. Once two honest users sign incompatible histo-
ries, they cannot see each others’ subsequent operations
without detecting the problem. Of course, since the server
can extend and sign compromised users’ histories, it can
change any files compromised users can write. The re-
3

maining files, however, can be modified only in honest
users’ histories and thus continue to be fork consistent.
3.2 Implications of fork consistency
Fork consistency is the strongest notion of integrity possi-
ble without on-line trusted parties. Suppose user A comes
on line, modifies a file, and goes off line. Later, B comes
on line and reads the file. If B doesn’t know whether A
has accessed the file system, it cannot detect an attack in
which the server simply discards A’s changes. Fork con-
sistency implies this is the only type of undetectable attack
by the server on file integrity or consistency. Moreover, if
A and B ever communicate or see each other’s future file
system operations, they can detect the attack.
Given fork consistency, one can leverage any trusted
parties that are on line to gain stronger consistency, even
fetch-modify consistency. For instance, as described later
in Section 5, the SUNDR server consists of two pro-
grams, a block store for handling data, and a consistency
server with a very small amount of state. Moving the con-
sistency server to a trusted machine trivially guarantees
fetch-modify consistency. The problem is that trusted ma-
chines may have worse connectivity or availability than
untrusted ones.
To bound the window of inconsistency without placing
a trusted machine on the critical path, one can use a “time
stamp box” with permission to write a single file. The
box could simply update that file through SUNDR every
5 seconds. All users who see the box’s updates know they
could only have been partitioned from each other in the
past 5 seconds. Such boxes could be replicated for Byzan-
tine fault tolerance, each replica updating a single file.
Alternatively, direct client-client communication can
be leveraged to increase consistency. Users can write
login and logout records with current network addresses
to files so as to find each other and continuously ex-
change information on their latest operations. If a mali-
cious server cannot disrupt network communication be-
tween clients, it will be unable to fork the file system state
once on-line clients know of each other. Those who deem
malicious network partitions serious enough to warrant
service delays in the face of client failures can conserva-
tively pause file access during communication outages.
3.3 Serialized SUNDR
The straw-man file system is impractical for two reasons.
First, it must record and ship around complete file sys-
tem operation histories, requiring enormous amounts of
bandwidth and storage. Second, the serialization of oper-
ations through a global lock is impractical for a multi-user
network file system. This subsection explains SUNDR’s
solution to the first problem; we describe a simplified file
system that still serializes operations with a global lock,
but is in other respects similar to SUNDR. Subsection 3.4
explains how SUNDR lets clients execute non-conflicting
operations concurrently.
Instead of signing operation histories, as in the straw-
man file system, SUNDR effectively takes the approach
of signing file system snapshots. Roughly speaking, users
sign messages that tie together the complete state of all
files with two mechanisms. First, all files writable by a
particular user or group are efficiently aggregated into a
single hash value called thei-handle using hash trees[18].
Second, each i-handle is tied to the latest version of every
other i-handle using version vectors [23].
3.3.1 Data structures
Before delving into the protocol’s details, we begin by de-
scribing SUNDR’s storage interface and data structures.
Like several recent file systems [9, 20], SUNDR names
all on-disk data structures by cryptographic handles. The
block store indexes most persistent data structures by their
20-byte SHA-1hashes, making the server a kind of large,
high-performance hash table. It is believed to be compu-
tationally infeasible to find any two different data blocks
with the same SHA-1 hash. Thus, when a client requests
the block with a particular hash, it can check the integrity
of the response by hashing it. An incidental benefit of
hash-based storage is that blocks common to multiple files
need be stored only once.
SUNDR also stores messages signed by users. These
are indexed by a hash of the public key and an index num-
ber (so as to distinguish multiple messages signed by the
same key).
Figure 2 shows the persistent data structures SUNDR
stores and indexes by hash, as well as the algorithm
for computing i-handles. Every file is identified by a
⟨principal, i-number⟩ pair, where principal is the user or
group allowed to write the file, and i-number is a per-
principal inode number. Directory entries map file names
onto ⟨principal, i-number⟩ pairs. A per-principal data
structure called the i-table maps each i-number in use
to the corresponding inode. User i-tables map each i-
number to a hash of the corresponding inode, which we
call the file’s i-hash. Group i-tables add a level of indi-
rection, mapping a group i-number onto a user i-number.
(The indirection allows the same user to perform multiple
successive writes to a group-owned file without updating
the group’s i-handle.) Inodes themselves contain SHA-1
hashes of file data blocks and indirect blocks.
Each i-table is stored as a B+-tree, where internal nodes
4

i-handle
group g’s
2 → H(i2)
3 → H(i3)
4 → H(i4)
5 → H(i5)
6 → H(i6)
.
.
.
0K→ H(d0)
8K→ H(d1)
.
.
.
(maps offset→ data)i-handle
group g’s i-table (tg)
H ∗ (tu2 )
(maps i#→ ⟨ user,i#⟩)
user u2’s
metadata
inode i4
(maps i#→ i-hash)
user u2’s i-table (tu2)
data block d0
.
.
. "locore.S" → ⟨ u2, 5⟩
"main.c" → ⟨ g, 4⟩
.
.
.
inode i6
2 → ⟨ u1, 7⟩
3 → ⟨ u2, 4⟩
4 → ⟨ u1, 2⟩
. . .
(maps name→ ⟨ u/g, i#⟩)
directory blockH ∗ (tg)
Figure 2: User and group i-handles. An i-handle is the root of a hash tree containing a user or group i-table. ( H
denotes SHA-1, while H ∗ denotes recursive application of SHA-1 to compute the root of a hash tree.) A group i-table
maps group inode numbers to user inode numbers. A user i-table maps a user’s inode numbers to i-hashes. An i-hash
is the hash of an inode, which in turn contains hashes of file data blocks.
contain the SHA-1 hashes of their children, thus forming
a hash tree. The hash of the B+-tree root is the i-handle.
Since the block store allows blocks to be requested by
SHA-1 hash, given a user’s i-handle, a client can fetch
and verify any block of any file in the user’s i-table by re-
cursively requesting the appropriate intermediary blocks .
The next question, of course, is how to obtain and verify
a user’s latest i-handle.
3.3.2 Protocol
i-handles are stored in digitally-signed messages known
as version structures, shown in Figure 3. Each version
structure is signed by a particular user. The structure must
always contain the user’s i-handle. In addition, it can op-
tionally contain one or more i-handles of groups to which
the user belongs. Finally, the version structure contains
a version vector consisting of a version number for every
user and group in the system.
When user u performs a file system operation,u’s client
acquires the global lock and downloads the latest version
structure for each user and group. We call this set of ver-
sion structures the version structure list, or VSL. (Much
of the VSL’s transfer can be elided if only a few users and
groups have changed version structures since the user’s
last operation.) The client then computes a new version
structure z by potentially updating i-handles and by set-
ting the version numbers in z to reflect the current state of
the file system.
More specifically, to set the i-handles in z, on a fetch,
g-5 . . .⟩
structure (yu2) u2’s i-table (tu2)
2 → H(i2)
3 → H(i3)
.
.
.
3 → ⟨ u2, 4⟩
2 → ⟨ u1, 7⟩
g’s i-table (tg)
.
.
.
H ∗ (tu2 )
u2
g : H ∗ (tg)
version vector:
u2’s signature
⟨u1-7 u2-3
u2’s version
Figure 3: A version structure containing a group i-handle.
the client simply copies u’s previous i-handle into z, as
nothing has changed. For a modify, the client computes
and includes new i-handles for u and for any groups
whose i-tables it is modifying.
The client then sets z’s version vector to reflect the ver-
sion number of each VSL entry. For any version structure
like z, and any principal (user or group) p, let z[p] denote
p’s version number inz’s version vector (or0 if z contains
no entry for p). For each principal p, if yp is p’s entry in
the VSL (i.e., the version structure containing p’s latest
i-handle), set z[p] ← yp[p].
Finally, the client bumps version numbers to reflect the
i-handles in z. It sets z[u] ← z[u] + 1 , since z always
5

hB
1.
2.
3.
4.
5.
sigA h A ⟨A-1⟩
sig
sig
sig
sig
⟨A-1 B-1⟩
⟨A-2 B-1⟩
⟨A-3 B-1⟩
⟨A-2 B-2⟩
A
A
B
B
hA
h′
A
hB
Figure 4: Signed version structures with a forking attack.
contains u’s i-handle, and for any group g whose i-handle
z contains, sets z[g] ← z[g] + 1.
The client then checks the VSL for consistency. Given
two version structures x and y, we define x ≤ y iff
∀p x[p] ≤ y[p]. To check consistency, the client verifies
that the VSL contains u’s previous version structure, and
that the set of all VSL entries combined with z is totally
ordered by ≤. If it is, the user signs the new version struc-
ture and sends it to the server with a COMMIT RPC. The
server adds the new structure to the VSL and retires the
old entries for updated i-handles, at which point the client
releases the file system lock.
Figure 4 revisits the forking attack from the end of Sec-
tion 3.1, showing how version vectors evolve in SUNDR.
With each version structure signed, a user reflects the
highest version number seen from every other user, and
also increments his own version number to reflect the
most recent i-handle. A violation of consistency causes
users to sign incompatible version structures—i.e., two
structures x and y such that x ̸≤ y and y ̸≤ x. In this
example, the server performs a forking attack after step 3.
User A updates his i-handle from hA to h′
A in 4, but in 5,
B is not aware of the change. The result is that the two
version structures signed in 4 and 5 are incompatible.
Just as in the straw-man file system, once two users
have signed incompatible version structures, they will
never again sign compatible ones, and thus cannot ever
see each other’s operations without detecting the attack
(as proven in earlier work [16]).
One optimization worth mentioning is that SUNDR
amortizes the cost of recomputing hash trees over several
operations. As shown in Figure 5, an i-handle contains
not just a hash tree root, but also a small log of changes
that have been made to the i-table. The change log further-
more avoids the need for other users to fetch i-table blocks
∆2 4 → ⟨ u1, 2⟩
5 → ⟨ u3, 4⟩
3 → ⟨ u2, 4⟩
2 → ⟨ u1, 7⟩
.
.
.
group g’s i-table (tg)
(maps i#→ ⟨ user,i#⟩)
group g’s
i-handle
H ∗ (t′
g)
change log:
∆1
.
.
.
Figure 5: i-table for group g, showing the change log. t′
g
is a recent i-table; applying the log to t′
g yields tg.
when re-validating a cached file that has not changed since
the hash tree root was last computed.
3.4 Concurrent SUNDR
While the version structures in SUNDR detect inconsis-
tency, serialized SUNDR is too conservative in what it
prohibits. Each client must wait for the previous client’s
version vector before computing and signing its own, so
as to reflect the appropriate version numbers. Instead, we
would like most operations to proceed concurrently. The
only time one client should have to wait for another is
when it reads a file the other is in the process of writing.3
3.4.1 Update certificates
SUNDR’s solution to concurrent updates is for users to
pre-declare a fetch or modify operation before receiving
the VSL from the server. They do so with signed mes-
sages called update certificates. If yu is u’s current VSL
entry, an update certificate foru’s next operation contains:
• u’s next version number
(
yu[u] + 1 , unless u is
pipelining multiple updates
)
,
• a hash of u’s VSL entry (H(yu)), and
• a (possibly empty) list of modifications to perform.
Each modification (or delta) can be one of four types:
• Set file ⟨user, i#⟩ to i-hash h.
• Set group file ⟨group, i#⟩ to ⟨user, i#⟩.
• Set/delete entry name in directory ⟨user/group, i#⟩.
3One might wish to avoid waiting for other clients even in the ev ent
of such a read-after-write conflict. However, this turns out to be impos-
sible with untrusted servers. If a single signed message could atomically
switch between two file states, the server could conceal the c hange ini-
tially, then apply it long after forking the file system, when users should
no longer see each others’ updates.
6

• Pre-allocate a range of group i-numbers (pointing
them to unallocated user i-numbers).
The client sends the update certificate to the server in an
UPDATE RPC. The server replies with both the VSL and a
list of all pending operations not yet reflected in the VSL,
which we call the pending version list or PVL.
Note that both fetch and modify operations require UP-
DATE RPCs, though fetches contain no deltas. (The RPC
name refers to updating the VSL, not file contents.) More-
over, when executing complex system calls such as re-
name, a single UPDATE RPC may contain deltas affecting
multiple files and directories, possibly in different i-tables.
An honest server totally orders operations according to
the arrival order of UPDATE RPCs. If operation O1 is
reflected in the VSL or PVL returned for O2’s UPDATE
RPC, then we say O1 happened before O2. Conversely,
if O2 is reflected in O1’s VSL or PVL, then O2 happened
before O1. If neither happened before the other, then the
server has mounted a forking attack.
When signing an update certificate, a client cannot pre-
dict the version vector of its next version structure, as
the vector may depend on concurrent operations by other
clients. The server, however, knows precisely what op-
erations the forthcoming version structure must reflect.
For each update certificate, the server therefore calculates
the forthcoming version structure, except for the i-handle.
This unsigned version structure is paired with its update
certificate in the PVL, so that the PVL is actually a list of
⟨update certificate, unsigned version structure⟩ pairs.
The algorithm for computing a new version structure,
z, begins as in serialized SUNDR: for each principal p,
set z[p] ← yp[p], where yp is p’s entry in the VSL. Then,
z’s version vector must be incremented to reflect pending
updates in the PVL, including u’s own. For user version
numbers, this is simple; for each update certificate signed
by user u, set z[u] ← z[u] + 1. For groups, the situation is
complicated by the fact that operations may commit out of
order when slow and fast clients update the same i-table.
For any PVL entry updating group g’s i-table, we wish to
increment z[g] if and only if the PVL entry happened af-
ter yg (since we already initialized z[g] with yg[g]). We
determine whether or not to increment the version num-
ber by comparing yg to the PVL entry’s unsigned version
vector, call it ℓ. If ℓ ̸≤ yg, set z[g] ← z[g] + 1. The result
is the same version vector one would obtain in serialized
SUNDR by waiting for all previous version structures.
Upon receiving the VSL and PVL, a client ensures that
the VSL, the unsigned version structures in the PVL, and
its new version structure are totally ordered. It also checks
for conflicts. If none of the operations in the PVL change
files the client is currently fetching or group i-tables it is
modifying, the client simply signs a new version structure
and sends it to the server for inclusion in the VSL.
3.4.2 Update conflicts
If a client is fetching a file and the PVL contains a modifi-
cation to that file, this signifies a read-after-write conflict.
In this case, the client still commits its version structure
as before but then waits for fetched files to be commit-
ted to the VSL before returning to the application. (A
FETCHPENDING RPC lets clients request a particular ver-
sion structure from the server as soon as it arrives.)
A trickier situation occurs when the PVL contains a
modification to a group i-handle that the client also wishes
to modify, signifying a write-after-write conflict. How
should a client, u, modifying a group g’s i-table, tg, re-
compute g’s i-handle, hg, when other operations in the
PVL also affect tg? Since any operation in the PVL hap-
pened before u’s new version structure, call it z, the han-
dle hg in z must reflect all operations on tg in the PVL.
On the other hand, if the server has behaved incorrectly,
one or more of the forthcoming version structures corre-
sponding to these PVL entries may be incompatible with
z. In this case, it is critical that z not somehow “launder”
operations that should have alerted people to the server’s
misbehavior.
Recall that clients already check the PVL for read-after-
write conflicts. When a client sees a conflicting mod-
ification in the PVL, it will wait for the corresponding
VSL entry even if u has already incorporated the change
in hg. However, the problem remains that a malicious
server might prematurely drop entries from the PVL, in
which case a client could incorrectly fetch modifications
reflected by tg but never properly committed.
The solution is for u to incorporate any modifications
of tg in the PVL not yet reflected in yg, and also to record
the current contents of the PVL in a new field of the ver-
sion structure. In this way, other clients can detect missing
PVL entries when they notice those entries referenced in
u’s version structure. Rather than include the full PVL,
which might be large, u simply records, for each PVL en-
try, the user performing the operation, that user’s version
number for the operation, and a hash of the expected ver-
sion structure with i-handles omitted.
When u applies changes from the PVL, it can often do
so by simply appending the changes to the change log of
g’s i-handle, which is far more efficient than rehashing the
i-table and often saves u from fetching uncached portions
of the i-table.
7

∆3: set ⟨g, 4⟩ → ⟨ u2, 11⟩
to directory ⟨g, 4⟩
∆2: add entry (“X” → ⟨ u1, 7⟩)
∆1: set ⟨u1, 7⟩ → hX
∆3: set ⟨g, 4⟩ → ⟨ u1, 8⟩
to directory ⟨g, 4⟩
. . .
. . .
. . .
4 → ⟨ u1, 8⟩
g’s i-table at T1
user u1
“X” → ⟨ u1, 7⟩
T2 “Y” → ⟨ u2, 10⟩
metadata
(/sundr/tmp)
u2’s inode 11
4 → ⟨ u2, 11⟩
. . .
. . .
. . .
g’s i-table at T2
“X” → ⟨ u1, 7⟩
(/sundr/tmp)
u1’s inode 8
Server
UPDATE
UPDATE
u1’s signature
H(yu1 )u1 version 7 version 3 H(yu2 )u2
u2’s update certificateu1’s update certificate
user u2
VSLPVL = (
u1, u2’s updates,
unsigned version structures)
COMMIT
COMMIT
T1
unsigned version structure)
VSL
PVL = (
u1’s update,
u2’s signature
. . .
metadata
∆1: set ⟨u2, 10⟩ → hY
∆2: add entry (“Y” → ⟨ u2, 10⟩)
Figure 6: Concurrent updates to /sundr/tmp/ by different users.
3.4.3 Example
Figure 6 shows an example of two users u1 and u2 in the
group g modifying the same directory. u1 creates file X
while u2 creates Y, both in /sundr/tmp/. The directory
is group-writable, while the files are not. (For the exam-
ple, we assume no other pending updates.)
Assume /sundr/tmp/ is mapped to group g’s i-
number 4. User u1 first calculates the i-hash of file X,
call it hX, then allocates his own i-number for X, call it 7.
u1 then allocates another i-number, 8, to hold the contents
of the modified directory. Finally, u1 sends the server an
update certificate declaring three deltas, namely the map-
ping of file ⟨u1, 7⟩ to i-hash hX, the addition of entry
(“X” → ⟨ u1, 7⟩) to the directory, and the re-mapping of
g’s i-number 4 to ⟨u1, 8⟩.
u2 similarly sends the server an update certificate for
the creation of file Y in /sundr/tmp/. If the server orders
u1’s update beforeu2’s, it will respond tou1 with the VSL
and a PVL containing only u1’s update, while it will send
u2 a PVL reflecting both updates. u2 will therefore apply
u1’s modification to the directory before computing the
i-handle for g, incorporating u1’s directory entry for X.
u2 would also ordinarily incorporate u1’s re-mapping of
the directory ⟨g, 4⟩ → ⟨ u1, 7⟩, except that u2’s own re-
mapping of the same directory supersedes u1’s.
An important subtlety of the protocol, shown in Fig-
ure 7, is that u2’s version structure contains a hash ofu1’s
forthcoming version structure (without i-handles). This
ensures that if the server surreptitiously drops u1’s update
certificate from the PVL beforeu1 commits, whoever sees
the incorrect PVL must be forked from both u1 and u2.
g-5 . . .⟩
u1
version 7
H(yu1 )
u1’s signature
∆1, ∆2, ∆3
u1’s unsigned
version
(ℓ u1)
structure
H ∗ (tu2 )
u2
g : H ∗ (tg)
u2’s version
structurecertificate
u1’s update
.
.
.
⟨u1-7 u2-2
⟨u1-7 u2-3
.
.
.
u2’s signature
⟨u1-7-H(ℓ u1 )⟩
g-4 . . .⟩
Figure 7: A pending update by user u1, reflected in user
u2’s version structure.
4 Discussion
SUNDR only detects attacks; it does not resolve them.
Following a server compromise, two users might find
themselves caching divergent copies of the same direc-
tory tree. Resolving such differences has been studied
in the context of optimistic file system replication [13,
22], though invariably some conflicts require application-
specific reconciliation. With CVS, users might employ
CVS’s own merging facilities to resolve forks.
SUNDR’s protocol leaves considerable opportunities
for compression and optimization. In particular, though
version structure signatures must cover a version vector
with all users and groups, there is no need to transmit en-
tire vectors in RPCs. By ordering entries from most- to
8

least-recently updated, the tail containing idle principa ls
can be omitted on all but a client’s first UPDATE RPC.
Moreover, by signing a hash of the version vector and
hashing from oldest to newest, clients could also pre-hash
idle principals’ version numbers to speed version vector
signatures. Finally, the contents of most unsigned version
structures in the PVL is implicit based on the order of the
PVL and could be omitted (since the server computes un-
signed version structures deterministically based on the
order in which it receives UPDATE RPCs). None of these
optimizations is currently implemented.
SUNDR’s semantics differ from those of traditional
Unix. Clients supply file modification and inode change
times when modifying files, allowing values that might be
prohibited in Unix. There is no time of last access. Di-
rectories have no “sticky bit.” A group-writable file in
SUNDR is not owned by a user (as in Unix) but rather
is owned by the group; such a file’s “owner” field indi-
cates the last user who wrote to it. In contrast to Unix
disk quotas, which charge the owner of a group-writable
file for writes by other users, if SUNDR’s block store en-
forced quotas, they would charge each user for precisely
the blocks written by that user.
One cannot change the owner of a file in SUNDR.
However, SUNDR can copy arbitrarily large files at the
cost of a few pointer manipulations, due to its hash-based
storage mechanism. Thus, SUNDR implements chown by
creating a copy of the file owned by the new user or group
and updating the directory entry to point to the new copy.
Doing so requires write permission on the directory and
changes the semantics of hard links (since chown only af-
fects a single link).
Yet another difference from Unix is that the owner of a
directory can delete any entries in the directory, including
non-empty subdirectories to which he or she does not have
write permission. Since Unix already allows users to re-
name such directories away, additionally allowing delete
permission does not appreciably affect security. In a sim-
ilar vein, users can create multiple hard links to directo-
ries, which could confuse some Unix software, or could
be useful in some situations. Other types of malformed
directory structure are interpreted as equivalent to some-
thing legal (e.g., only the first of two duplicate directory
entries counts).
SUNDR does not yet offer read protection or confiden-
tiality. Confidentiality can be achieved through encrypted
storage, a widely studied problem [5, 10, 12, 34].
In terms of network latency, SUNDR is comparable
with other polling network file systems. SUNDR waits
for an UPDATE RPC to complete before returning from an
application file system call. If the system call caused only
modifies, or if all fetched data hit in the cache, this is the
only synchronous round trip required; theCOMMIT can be
sent in the background (except forfsync). This behavior is
similar to systems such as NFS3, which makes anACCESS
RPC on each open and writes data back to the server on
each close. We note that callback- or lease-based file sys-
tems can actually achieve zero round trips when the server
has committed to notifying clients of cache invalidations.
5 File system implementation
The SUNDR client is implemented at user level, using a
modified version of the xfs device driver from the ARLA
file system [33] on top of a slightly modified FreeBSD
kernel. Server functionality is divided between two pro-
grams, a consistency server, which handles update cer-
tificates and version structures, and a block store, which
actually stores data, update certificates, and version struc-
tures on disk. For experiments in this paper, the block
server and consistency server ran on the same machine,
communicating over Unix-domain sockets. They can also
be configured to run on different machines and communi-
cate over an authenticated TCP connection.
5.1 File system client
The xfs device driver used by SUNDR is designed for
whole-file caching. When a file is opened, xfs makes an
upcall to the SUNDR client asking for the file’s data. The
client returns the identity of a local file that has a cached
copy of the data. All reads and writes are performed on
the cached copy, without further involvement of SUNDR.
When the file is closed (or flushed with fsync), if it has
been modified, xfs makes another upcall asking the client
to write the data back to the server. Several other types of
upcalls allow xfs to look up names in directories, request
file attributes, create/delete files, and change metadata.
As distributed, xfs’s interface posed two problems for
SUNDR. First, xfs caches information like local file bind-
ings to satisfy some requests without upcalls. In SUNDR,
some of these requests require interaction with the consis-
tency server for the security properties to hold. We there-
fore modified xfs to invalidate its cache tokens immedi-
ately after getting or writing back cached data, so as to
ensure that the user-level client gets control whenever the
protocol requires an UPDATE RPC. We similarly changed
xfs to defeat the kernel’s name cache.
Second, some system calls that should require only a
single interaction with the SUNDR consistency server re-
sult in multiple kernel vnode operations and xfs upcalls.
For example, the system call “ stat ("a/b/c", &sb) ”
9

results in three xfs GETNODE upcalls (for the directory
lookups) and one GETATTR . The whole system call should
require only one UPDATE RPC. Yet if the user-level client
does not know that the four upcalls are on behalf of the
same system call, it must check the freshness of its i-
handles four separate times with four UPDATE RPCs.
To eliminate unnecessary RPCs, we modified the
FreeBSD kernel to count the number of system call invo-
cations that might require an interaction with the consis-
tency server. We increment the counter at the start of every
system call that takes a pathname as an argument (e.g.,
stat, open, readlink, chdir). The SUNDR client
memory-maps this counter and records the last value it
has seen. If xfs makes an upcall that does not change the
state of the file system, and the counter has not changed,
then the client can use its cached copies of all i-handles.
5.2 Signature optimization
The cost of digital signatures on the critical path in
SUNDR is significant. Our implementation therefore uses
the ESIGN signature scheme, 4 which is over an order
of magnitude faster than more popular schemes such as
RSA. All experiments reported in this paper use 2,048-bit
public keys, which, with known techniques, would require
a much larger work factor to break than 1,024-bit RSA.
To move verification out of the critical path, the consis-
tency server also processes and replies to anUPDATE RPC
before verifying the signature on its update certificate. It
verifies the signature after replying, but before accepting
any other RPCs from other users. If the signature fails
to verify, the server removes the update certificate from
the PVL and and drops the TCP connection to the forging
client. (Such behavior is acceptable because only a faulty
client would send invalid signatures.) This optimization
allows the consistency server’s verification of one signa-
ture to overlap with the client’s computation of the next.
Clients similarly overlap computation and network la-
tency. Roughly half the cost of an ESIGN signature is at-
tributable to computations that do not depend on the mes-
sage contents. Thus, while waiting for the reply to an UP-
DATE RPC, the client precomputes its next signature.
5.3 Consistency server
The consistency server orders operations for SUNDR
clients and maintains the VSL and PVL as described in
Section 3. In addition, it polices client operations and re-
jects invalid RPCs, so that a malicious user cannot cause
4Specifically, we use the version of ESIGN shown secure in the ran-
dom oracle model by [21], with parameter e = 8.
an honest server to fail. For crash recovery, the consis-
tency server must store VSL and PVL to persistent stor-
age before responding to client RPCs. The current consis-
tency server stores these to the block server. Because the
VSLs and PVLs are small relative to the size of the file
system, it would also be feasible to use non-volatile RAM
(NVRAM).
6 Block store implementation
A block storage daemon called bstor handles all disk
storage in SUNDR. Clients interact directly with bstor
to store blocks and retrieve them by SHA-1 hash value.
The consistency server uses bstor to store signed update
and version structures. Because a SUNDR server does
not have signature keys, it lacks permission to repair the
file system after a crash. For this reason, bstor must
synchronously store all data to disk before returning to
clients, posing a performance challenge. bstor therefore
heavily optimizes synchronous write performance.
bstor’s basic idea is to write incoming data blocks to
a temporary log, then to move these blocks to Venti-like
storage in batches. Venti [24] is an archival block store
that appends variable-sized blocks to a large, append-only
IDE log disk while indexing the blocks by SHA-1 hash
on one or more fast SCSI disks. bstor’s temporary log
relaxes the archival semantics of Venti, allowing short-
lived blocks to be deleted within a small window of their
creation. bstor maintains an archival flavor, though, by
supporting periodic file system snapshots.
The temporary log allows bstor to achieve low latency
on synchronous writes, which under Venti require an in-
dex lookup to ensure the block is not a duplicate. More-
over, bstor sector-aligns all blocks in the temporary log,
temporarily wasting an average of half a sector per block
so as to avoid multiple writes to the same sector, which
would each cost at least one disk rotation. The temporary
log improves write throughput even under sustained load,
because transferring blocks to the permanent log in large
batches allows bstor to order index disk accesses.
bstor keeps a large in-memory cache of recently used
blocks. In particular, it caches all blocks in the temporary
log so as to avoid reading from the temporary log disk.
Though bstor does not currently use special hardware, in
Section 7 we describe how SUNDR’s performance would
improve if bstor had a small amount of NVRAM to store
update certificates.
6.1 Interface
bstor exposes the following RPCs to SUNDR clients:
10

STORE (header, block)
RETRIEVE (hash)
VSTORE (header, pubkey, n, block)
VRETRIEVE (pubkey, n, [time])
DECREF (hash)
SNAPSHOT ()
The STORE RPC writes a block and its header to sta-
ble storage if bstor does not already have a copy of the
block. The header has information encapsulating the
block’s owner and creation time, as well as fields use-
ful in concert with encoding or compression. The RE-
TRIEVE RPC retrieves a block from the store given its
SHA-1 hash. It also returns the first header STORE d with
the particular block.
The VSTORE and VRETRIEVE RPCs are like STORE
and RETRIEVE , but for signed blocks. Signed blocks are
indexed by the public key and a small index number, n.
VRETRIEVE , by default, fetches the most recent version
of a signed block. When supplied with a timestamp as an
optional third argument, VRETRIEVE returns the newest
block written before the given time.
DECREF (short for “decrement reference count”) in-
forms the store that a block with a particular SHA-1 hash
might be discarded. SUNDR clients use DECREF to dis-
card temporary files and short-lived metadata. bstor’s
deletion semantics are conservative. When a block is first
stored, bstor establishes a short window (one minute by
default) during which it can be deleted. If a clientSTORE s
then DECREF s a block within this window, bstor marks
the block as garbage and does not permanently store it.
If two clients store the same block during the dereference
window, the block is marked as permanent.
An administrator should issue a SNAPSHOT RPC peri-
odically to create a coherent file system image that clients
can later revert to in the case of accidental data disrup-
tion. Upon receiving this RPC, bstor simply immunizes
all newly-stored blocks from future DECREF ’s and flags
them to be stored in the permanent log. SNAPSHOT and
VRETRIEVE ’s time argument are designed to allow brows-
ing of previous file system state, though this functionality
is not yet implemented in the client.
6.2 Index
bstor’s index system locates blocks on the permanent log,
keyed by their SHA-1 hashes. An ideal index is a sim-
ple in-memory hash table mapping 20-byte SHA-1 block
hashes to 8-byte log disk offsets. If we assume that the
average block stored on the system is 8 KB, then the in-
dex must have roughly 1/ 128 the capacity of the log disk.
Although at present such a ratio of disk to memory is pos-
sible with commodity components, we are not convinced
that memory will keep up with hard disks in the future.
We instead use Venti’s strategy of striping a disk-
resident hash table over multiple high-speed SCSI
disks. bstor hashes 20-byte SHA-1 hashes down to
⟨index-disk-id,index-disk-offset⟩ pairs. The disk offsets
point to sector-sized on-disk data structures called buck-
ets, which contain 15 index-entries, sorted by SHA-1
hash. index-entries in turn map SHA-1 hashes to offsets
on the permanent data log. Whenever an index-entry is
written to or read from disk, bstor also stores it in an in-
memory LRU cache.
bstor accesses the index system as Venti does when
answering RETRIEVE RPCs that miss the block cache.
When bstor moves data from the temporary to the per-
manent log, it must access the index system sometimes
twice per block (once to check a block is not a duplicate,
and once to write a new index entry after the block is com-
mitted the permanent log). In both cases, bstor sorts these
disk accesses so that the index disks service a batch of
requests with one disk arm sweep. Despite these opti-
mizations, bstor writes blocks to the permanent log in the
order they arrived; randomly reordering blocks would hin-
der sequential read performance over large files.
6.3 Data management
To recover from a crash or an unclean shutdown, the sys-
tem first recreates an index consistent with the permanent
log, starting from its last known checkpoint. Index re-
covery is necessary because the server updates the index
lazily after storing blocks to the permanent log.bstor then
processes the temporary log, storing all fresh blocks to the
permanent log, updating the index appropriately.
Venti’s authors argue that archival storage is practical
because IDE disk capacity is growing faster than users
generate data. For users who do not fit this paradigm,
however, bstor could alternatively be modified to support
mark-and-sweep garbage collection. The general idea is
to copy all reachable blocks to a new log disk, then recycle
the old disk. With two disks, bstor could still respond to
RPCs during garbage collection.
7 Performance
The primary goal in testing SUNDR was to ensure that
its security benefits do not come at too high a price rela-
tive to existing file systems. In this section, we compare
SUNDR’s overall performance to NFS. We also perform
microbenchmarks to help explain our application-level re-
11

sults, and to support our claims that our block server out-
performs a Venti-like architecture in our setting.
7.1 Experimental setup
We carried out our experiments on a cluster of 3 GHz
Pentium IV machines running FreeBSD 4.9. All ma-
chines were connected with fast Ethernet with ping times
of 110 µs. For block server microbenchmarks, we ad-
ditionally connected the block server and client with gi-
gabit Ethernet. The machine running bstor has 3 GB of
RAM and an array of disks: four Seagate Cheetah 18 GB
SCSI drives that spin at 15,000 RPM were used for the in-
dex; two Western Digital Caviar 180 GB 7200 RPM EIDE
drives were used for the permanent and temporary logs.
7.2 Microbenchmarks
7.2.1 bstor
Our goals in evaluating bstor are to quantify its raw per-
formance and justify our design improvements relative to
Venti. In our experiments, we configured bstor’s four
SCSI disks each to use 4 GB of space for indexing. If
one hopes to maintain good index performance (and not
overflow buckets), then the index should remain less than
half full. With our configuration (8 GB of usable index
and 32-byte index entries), bstor can accommodate up to
2 TB of permanent data. For flow control and fairness,
bstor allowed clients to make up to 40 outstanding RPCs.
For the purposes of the microbenchmarks, we disabled
bstor’s block cache but enabled an index cache of up to
100,000 entries. The circular temporary log was 720 MB
and never filled up during our experiments.
We measured bstor’s performance while storing and
fetching a batch of 20,000 unique 8 KB blocks. Figure 8
shows the averaged results from 20 runs of a 20,000 block
experiment. In all cases, standard deviations were less
than 5% of the average results. The first two results show
that bstor can absorb bursts of 8 KB blocks at almost twice
fast Ethernet rates, but that sustained throughput is limited
by bstor’s ability to shuffle blocks from the temporary to
the permanent logs, which it can do at 11.9 MB/s. The
bottleneck in STORE ing blocks to the temporary log is cur-
rently CPU, and future versions of bstor might eliminate
some unnecessary memcpys to achieve better throughput.
On the other hand, bstor can process the temporary log
only as fast as it can read from its index disks, and there
is little room for improvement here unless disks become
faster or more index disks are used.
To compare with a Venti-like system, we implemented
a Venti-like store mechanism. In VENTI
STORE , bstor
Operation MB/s
STORE (burst) 18.4
STORE (sustained) 11.9
VENTI STORE 5.1
RETRIEVE (random + cold index cache) 1.2
RETRIEVE (sequential + cold index cache) 9.1
RETRIEVE (sequential + warm index cache) 25.5
Figure 8: bstor throughput measurements with the block
cache disabled.
first checks for a block’s existence in the index and stores
the block to the permanent log only if it is not found.
That is, each VENTI
STORE entails an access to the index
disks. Our results show that VENTI STORE can achieve
only 27% of STORE ’s burst throughput, and 43% of its
sustained throughput.
Figure 8 also presents read measurements for bstor. If
a client reads blocks in the same order they are written
(i.e., “sequential” reads), then bstor need not seek across
the permanent log disk. Throughput in this case is limited
by the per-block cost of locating hashes on the index disks
and therefore increases to 25.5 MB/s with a warm index
cache. Randomly-issued reads fare poorly, even with a
warm index cache, because bstor must seek across the
permanent log. In the context of SUNDR, slow random
RETRIEVE s should not affect overall system performance
if the client aggressively caches blocks and reads large
files sequentially.
Finally, the latency of bstor RPCs is largely a func-
tion of seek times. STORE RPCs do not require seeks
and therefore return in 1.6 ms. VENTI
STORE returns in
6.7 ms (after one seek across the index disk at a cost of
about 4.4 ms). Sequential RETRIEVE s that hit and miss
the index cache return in 1.9 and 6.3 ms, respectively. A
seek across the log disk takes about 6.1 ms; therefore ran-
dom RETRIEVE s that hit and miss the index cache return
in 8.0 and 12.4 ms respectively.
7.2.2 Cryptographic overhead
SUNDR clients sign and verify version structures and up-
date certificates using 2,048-bit ESIGN keys. Our im-
plementation (based on the GNU Multiprecision library
version 4.1.4) can complete signatures in approximately
150 µs and can verify them 100 µs. Precomputing a sig-
nature requires roughly 80 µs, while finalizing a precom-
puted signature is around 75 µs. We observed that these
measurements can vary on the Pentium IV by as much as a
factor of two, even in well-controlled micro-benchmarks.
By comparison, an optimized version of the Rabin sig-
nature scheme with 1,280-bit keys, running on the same
12

hardware, can compute signatures in 3.1 ms and can ver-
ify them in 27 µs.
7.3 End-to-end evaluation
In end-to-end experiments, we compare SUNDR to both
NFS2 and NFS3 servers running on the same hardware.
To show NFS in the best possible light, the NFS exper-
iments run on the fast SCSI disks SUNDR uses for in-
dexes, not the slower, larger EIDE log disks. We include
NFS2 results because NFS2’s write-through semantics are
more like SUNDR’s. Both NFS2 and SUNDR write all
modified file data to disk before returning from a close
system call, while NFS3 does not offer this guarantee.
Finally, we described in Section 5.3 that SUNDR
clients must wait for the consistency server to write small
pieces of data (VSLs and PVLs) to stable storage. The
consistency server’s storing of PVLs in particular is on
the client’s critical path. We present result sets for con-
sistency servers running with and without flushes to sec-
ondary storage. We intend the mode with flushes disabled
to simulate a consistency server with NVRAM.
All application results shown are the average of three
runs. Relative standard deviations are less than 8% unless
otherwise noted.
7.3.1 LFS small file benchmark
The LFS small file benchmark [28] tests SUNDR’s perfor-
mance on simple file system operations. This benchmark
creates 1,000 1 KB files, reads them back, then deletes
them. We have modified the benchmark slightly to write
random data to the 1 KB files; writing the same file 1,000
times would give SUNDR’s hash-based block store an un-
fair advantage.
Figure 9 details our results when only one client is ac-
cessing the file system. In the create phase of the bench-
mark, a single file creation entails system calls to open,
read and close. On SUNDR/NVRAM, the open call in-
volves two serialized rounds of the consistency protocol,
each of which costs about 2 ms; the write call is a no-
op, since file changes are buffered until close; and the
close call involves one round of the protocol and one
synchronous write of file data to the block server, which
the client can overlap. Thus, the entire sequence takes
about 6 ms. Without NVRAM, each round of the protocol
takes approximately 1-2 ms longer, because the consis-
tency server must wait for bstor to flush.
Unlike SUNDR, an NFS server must wait for at least
one disk seek when creating a new file because it syn-
chronously writes metadata. A seek costs at least 4 ms on
our fast SCSI drives, and thus NFS can do no better than
create read unlink
0
5
10
Run Time (s)
NFS2
NFS3
SUNDR
SUNDR / NVRAM
Figure 9: Single client LFS Small File Benchmark. 1000
operations on files with 1 KB of random content.
4 ms per file creation. In practice, NFS requires about
6 ms to service the three system calls in the create stage.
In the read phase of the benchmark, SUNDR performs
one round of the consistency protocol in the open system
call. The NFS3 client still accesses the server with an
ACCESS RPC, but the server is unlikely to need any data
not in its buffer cache at this point, and hence no seeking is
required. NFS2 does not contact the server in this phase.
In the unlink stage of the benchmark, clients issue a
single unlink system call per file. An unlink for SUNDR
triggers one round of the consistency protocol and an
asynchronous write to the block server to store updated
i-table and directory blocks. SUNDR and SUNDR/
NVRAM in particular can outperform NFS in this stage
of the experiment because NFS servers again require at
least one synchronous disk seek per file unlinked.
We also performed experiments with multiple clients
performing the LFS small file benchmark concurrently in
different directories. Results for the create phase are re-
ported in Figure 10 and the other phases of the benchmark
show similar trends. A somewhat surprising result is that
SUNDR actually scales better than NFS as client concur-
rency increases in our limited tests. NFS is seek-bound
even in the single client case, and the number of seeks
the NFS servers require scale linearly with the number of
concurrent clients. For SUNDR, latencies induced by the
consistency protocol limit individual client performance ,
but these latencies overlap when clients act concurrently.
SUNDR’s disk accesses are also scalable because they are
sequential, sector-aligned writes to bstor’s temporary log.
7.3.2 Group contention
The group protocol incurs additional overhead when fold-
ing other users’ changes into a group i-table or directory.
We characterized the cost of this mechanism by measur-
13

1 2 31 2 3
Concurrent Clients
0
10
20
30
Average Run Time (s)
NFS2
NFS3
SUNDR
SUNDR / NVRAM
Figure 10: Concurrent LFS Small File Benchmark, cre-
ate phase. 1000 creations of 1 KB files. (Relative stan-
dard deviation for SUNDR in 3 concurrent clients case is
13.7%)
ing a workload with a high degree of contention for a
group-owned directory. We ran a micro-benchmark that
simultaneously created 300 new files in the same, group-
writable directory on two clients. Each concurrent create
required the client to re-map the group i-number in the
group i-table and apply changes to the user’s copy of the
directory.
The clients took an average of 4.60 s and 4.26 s on
SUNDR/NVRAM and NFS3 respectively. For compari-
son, we also ran the benchmark concurrently in two sep-
arate directories, which required an average of 2.94 s
for SUNDR/NVRAM and 4.05 s for NFS3. The results
suggests that while contention incurs a noticeable cost,
SUNDR’s performance even in this case is not too far out
of line with NFS3.
7.3.3 Real workloads
Figure 11 shows SUNDR’s performance in untaring, con-
figuring, compiling, installing and cleaning an emacs 20.7
distribution. During the experiment, the SUNDR client
sent a total of 42,550 blocks to the block server, which
totaled 139.24 MB in size. Duplicate blocks, which bstor
discards, account for 29.5% of all data sent. The client
successfully DECREF ed 10,747 blocks, for a total space
savings of 11.3%. In the end, 25,740 blocks which totaled
82.21 MB went out to permanent storage.
SUNDR is faster than NFS2 and competitive with
NFS3 in most stages of the Emacs build process. We be-
lieve that SUNDR’s sluggish performance in the install
phase is an artifact of our implementation, which serial-
izes concurrent xfs upcalls for simplicity (and not correct-
untar config make install clean
0
20
40
60Run Time (s)
NFS2
NFS3
SUNDR
SUNDR / NVRAM
Figure 11: Installation procedure for emacs 20.7
1 2 31 2 3
Concurrent Clients
20
40
60
80
100
Average Run Time (s)
NFS2
SUNDR
NFS3
SUNDR / NVRAM
Figure 12: Concurrent untar of emacs 20.7.tar
ness). Concurrent xfs upcalls are prevalent in this phase of
the experiment due to theinstall command’s manipulation
of file attributes.
Figure 12 details the performance of the untar phase of
the Emacs build as client concurrency increases. We noted
similar trends for the other phases of the build process.
These experiments suggest that the scalability SUNDR
exhibited in the LFS small file benchmarks extends to real
file system workloads.
7.3.4 CVS on SUNDR
We tested CVS over SUNDR to evaluate SUNDR’s per-
formance as a source code repository. Our experiment
follows a typical progression. First, client A imports an
arbitrary source tree—in this test groff-1.17.2, which
has 717 files totaling 6.79 MB. Second, clients A and B
check out a copy to their local disks. Third, A commits
groff-1.18, which affects 549 files (6.06 MB). Lastly,
B updates its local copy. Figure 13 shows the results.
SUNDR fares badly on the commit phase because CVS
repeatedly opens, memory maps, unmaps, and closes each
14

Phase SUNDR SUNDR NFS3 SSH
NVRAM
Import 13.0 10.0 4.9 7.0
Checkout 13.5 11.5 11.6 18.2
Commit 38.9 32.8 15.7 11.5
Update 19.1 15.9 13.3 11.5
Figure 13: Run times for CVS experiments (in seconds).
repository file several times in rapid succession. Ev-
ery open requires an iteration of the consistency proto-
col in SUNDR, while FreeBSD’s NFS3 apparently elides
or asynchronously performs ACCESS RPCs after the first
of several closely-spaced open calls. CVS could feasibly
cache memory-mapped files at this point in the experi-
ment, since a single CVS client holds a lock on the di-
rectory. This small change would significantly improve
SUNDR’s performance in the benchmark.
8 Related work
A number of non-networked file systems have used cryp-
tographic storage to keep data secret [5, 34] and check
integrity [31]. Several network file systems provide
varying degrees integrity checks but reduce integrity on
read sharing [25] or are vulnerable to consistency at-
tacks [10, 12, 19]. SUNDR is the first system to pro-
vide well-defined consistency semantics for an untrusted
server. An unimplemented but previously published ver-
sion of the SUNDR protocol [16] had no groups and thus
did not address write-after-write conflicts.
The Byzantine fault-tolerant file system, BFS [6], uses
replication to ensure the integrity of a network file sys-
tem. As long as more than 2/ 3 of a server’s replicas are
uncompromised, any data read from the file system will
have been written by a legitimate user. SUNDR, in con-
trast, does not require any replication or place any trust
in machines other than a user’s client. However, SUNDR
provides weaker freshness guarantees than BFS, because
of the possibility that a malicious SUNDR server can fork
the file system state if users have no other evidence of
each other’s on-line activity.
Several projects have investigated storing file systems
on peer-to-peer storage systems comprised of potentially
untrusted nodes. Farsite [3] spreads such a file system
across people’s unreliable desktop machines. CFS [7] is
a secure read-only file P2P system. Ivy [20], a read-write
version of CFS, can be convinced to re-order operations
clients have already seen. Pond [27] relies on a trusted
“inner core” of machines for security, distributing trust in
a BFS-like way.
SUNDR uses hash trees, introduced in [18], to verify a
file block’s integrity without touching the entire file sys-
tem. Duchamp [8], BFS [6], SFSRO [9] and TDB [14]
have all made use of hash trees for comparing data or
checking the integrity of part of a larger collection of data.
SUNDR uses version vectors to detect consistency vio-
lations. Version vectors were used by Ficus [22] to detect
update conflicts between file system replicas, and have
also been used to secure partial orderings [26, 30]. Our
straw-man file system somewhat resembles timeline en-
tanglement [15], which reasons about the temporal order-
ing of system states using hash chains.
9 Conclusions
SUNDR is a general-purpose, multi-user network file sys-
tem that never presents applications with incorrect file
system state, even when the server has been compromised.
SUNDR’s protocol provably guarantees fork consistency,
which essentially ensures that the server either behaves
correctly or that its failure will be detected after commu-
nication among users. In any event, the consequences of
an undetected server compromise are limited to conceal-
ing users’ operations from each other after some forking
point; the server cannot tamper with, inject, re-order, or
suppress file writes in any other way.
Measurements of our implementation show perfor-
mance that is usually close to and sometimes better than
the popular NFS file system. Yet by reducing the amount
of trust placed in the server, SUNDR both increases peo-
ple’s options for managing data and significantly im-
proves the security of their files.
Acknowledgments
Thanks to Michael Freedman, Kevin Fu, Daniel Giffin,
Frans Kaashoek, Jinyang Li, Robert Morris, the anony-
mous reviewers, and our shepherd Jason Flinn.
This material is based upon work supported by the
National Science Foundation (NSF) under grant CCR-
0093361. Maxwell Krohn is partially supported by an
NSF Graduate Fellowship, David Mazi `eres by an Alfred
P. Sloan research fellowship, and Dennis Shasha by NSF
grants IIS-9988636, MCB-0209754, and MCB-0115586.
References
[1] Apache.org compromise report. http://www.apache.org/
info/20010519-hack.html, May 2001.
[2] Debian investigation report after server compromises. http://
www.debian.org/News/2003/20031202, December 2003.
15

[3] Atul Adya, William J. Bolosky, Miguel Castro, Gerald Cer mak,
Ronnie Chaiken, John R. Douceur, Jon Howell, Jacob R. Lorch,
Marvin Theimer, and Roger P. Wattenhofer. FARSITE: Federated,
available, and reliable storage for an incompletely trustedenviron-
ment. In Proceedings of the 5th Symposium on Operating Systems
Design and Implementation, pages 1–14, December 2002.
[4] Brian Berliner. CVS II: Parellizing software developmen t. In
Proceedings of the Winter 1990 USENIX , Colorado Springs, CO,
1990. USENIX.
[5] Matt Blaze. A cryptographic file system for unix. In 1st ACM
Conference on Communications and Computing Security , pages
9–16, November 1993.
[6] Miguel Castro and Barbara Liskov. Practical byzantine f ault toler-
ance. In Proceedings of the 3rd Symposium on Operating Systems
Design and Implementation , pages 173–186, New Orleans, LA,
February 1999.
[7] Frank Dabek, M. Frans Kaashoek, David Karger, Robert Mor ris,
and Ion Stoica. Wide-area cooperative storage with cfs. In Pro-
ceedings of the 18th ACM Symposium on Operating Systems Prin-
ciples, pages 202–215, Chateau Lake Louise, Banff, Canada, Oc-
tober 2001. ACM.
[8] Dan Duchamp. A toolkit approach to partially disconnecte d op-
eration. In Proceedings of the 1997 USENIX , pages 305–318.
USENIX, January 1997.
[9] Kevin Fu, M. Frans Kaashoek, and David Mazi `eres. Fast and se-
cure distributed read-only file system. ACM Transactions on Com-
puter Systems, 20(1):1–24, February 2002.
[10] Eu-Jin Goh, Hovav Shacham, Nagendra Modadugu, and Dan
Boneh. SiRiUS: Securing Remote Untrusted Storage. In Proceed-
ings of the Tenth Network and Distributed System Security (NDSS)
Symposium, pages 131–145. Internet Society (ISOC), February
2003.
[11] Maurice P. Herlihy and Jeannette M. Wing. Linearizabil ity: a cor-
rectness condition for concurrent objects. ACM Transactions on
Programming Languages Systems, 12(3):463–492, 1990.
[12] M. Kallahalla, E. Riedel, R. Swaminathan, Q. Wang, and K. Fu.
Plutus: Scalable secure file sharing on untrusted storage. I n 2nd
USENIX conference on File and Storage Technologies (FAST ’03),
San Francisco, CA, April 2003.
[13] James J. Kistler and M. Satyanarayanan. Disconnected op eration
in the coda file system. ACM Transactions on Computer Systems,
10(1):3–25, 1992.
[14] Umesh Maheshwari and Radek Vingralek. How to build a trus ted
database system on untrusted storage. In Proceedings of the 4th
Symposium on Operating Systems Design and Implementation ,
San Diego, October 2000.
[15] Petros Maniatis and Mary Baker. Secure history preserv ation
through timeline entanglement. In Proceedings of the 11th
USENIX Security Symposium, San Francisco, CA, August 2002.
[16] David Mazi `eres and Dennis Shasha. Building secure file systems
out of Byzantine storage. In Proceedings of the 21st Annual ACM
SIGACT-SIGOPS Symposium on Principles of Distributed Com-
puting, pages 108–117, July 2002.
[17] David Mazi `eres and Dennis Shasha. Building secure file systems
out of Byzantine storage. Technical Report TR2002–826, NYU
Department of Computer Science, May 2002.
[18] Ralph C. Merkle. A digital signature based on a conventi onal
encryption function. In Carl Pomerance, editor, Advances in
Cryptology—CRYPTO ’87, volume 293 of Lecture Notes in Com-
puter Science, pages 369–378, Berlin, 1987. Springer-Verlag.
[19] Ethan Miller, Darrell Long, William Freeman, and Benjamin Reed.
Strong security for distributed file systems. In Proceedings of the
20th IEEE International Performance, Computing, and Communi-
cations Conference, pages 34–40, Phoenix, AZ, April 2001.
[20] Athicha Muthitacharoen, Robert Morris, Thomer M. Gil, a nd Ben-
jie Chen. Ivy: A read/write peer-to-peer file system. In Proceed-
ings of the 5th Symposium on Operating Systems Design and Im-
plementation, pages 31–44, December 2002.
[21] Tatsuaki Okamoto and Jacques Stern. Almost uniform densi ty of
power residues and the provable security of ESIGN. In Advances
in Cryptology – ASIACRYPT, pages 287–301, 2003.
[22] T. W. Page, Jr., R. G. Guy, J. S. Heidemann, D. H. Ratner, P. L.
Reiher, A. Goel, G. H. Kuenning, and G. J. Popek. Perspective s
on optimistically replicated peer-to-peer filing. Software Practice
and Experience, 28(2):155–180, February 1998.
[23] D. Stott Parker, Jr., Gerald J. Popek, Gerard Rudisin, A llen
Stoughton, Bruce J. Walker, Evelyn Walton, Johanna M. Chow,
David Edwards, Stephen Kiser, and Charles Kline. Detection of
mutual inconsistency in distributed systems. IEEE Transactions
on Software Engineering, SE-9(3):240–247, May 1983.
[24] Sean Quinlan and Sean Dorward. Venti: a new approach to archival
storage. In First USENIX conference on File and Storage Tech-
nologies (FAST ’02), Monterey, CA, January 2002.
[25] David Reed and Liba Svobodova. Swallow: A distributed data stor-
age system for a local network. In A. West and P. Janson, edito rs,
Local Networks for Computer Communications , pages 355–373.
North-Holland Publ., Amsterdam, 1981.
[26] Michael Reiter and Li Gong. Securing causal relationsh ips in dis-
tributed systems. The Computer Journal, 38(8):633–642, 1995.
[27] Sean Rhea, Patrick Eaton, and Dennis Geels. Pond: The
OceanStore prototype. In 2nd USENIX conference on File and
Storage Technologies (FAST ’03), San Francisco, CA, April 2003.
[28] M. Rosenblum and J. Ousterhout. The design and implementa tion
of a log-structured file system. In Proceedings of the 13th ACM
Symposium on Operating Systems Principles , pages 1–15, Pacific
Grove, CA, October 1991. ACM.
[29] Russel Sandberg, David Goldberg, Steve Kleiman, Dan Walsh, and
Bob Lyon. Design and implementation of the Sun network filesys-
tem. In Proceedings of the Summer 1985 USENIX, pages 119–130,
Portland, OR, 1985. USENIX.
[30] Sean W. Smith and J. D. Tygar. Security and privacy for par tial
order time. In Proceedings of the ISCA International Conference
on Parallel and Distributed Computing Systems, pages 70–79, Las
Vegas, NV , October 1994.
[31] Christopher A. Stein, John H. Howard, and Margo I. Seltz er. Uni-
fying file system protection. In Proceedings of the 2001 USENIX.
USENIX, June 2001.
[32] Owen Taylor. Intrusion on www.gnome.org. http:/
/mail.gnome.org/archives/gnome-announce-list/
2004-March/msg00114.html, March 2004.
[33] Assar Westerlund and Johan Danielsson. Arla—a free AFS client.
In Proceedings of the 1998 USENIX, Freenix track, New Orleans,
LA, June 1998. USENIX.
[34] Charles P. Wright, Michael Martino, and Erez Zadok. NCryptfs: A
secure and convenient cryptographic file system. In Proceedings
of the Annual USENIX Technical Conference, pages 197–210, June
2003.
[35] Tatu Yl ¨onen. SSH – secure login connections over the Internet. In
Proceedings of the 6th USENIX Security Symposium, pages 37–42,
San Jose, CA, July 1996.
16

Q: What problem does linearizability solve?

A: The problem being solved is fulfilling the need for a consistency
model: a definition of correct client-visible behavior of a network
service in the face of concurrent requests from multiple clients, lost
and re-transmitted requests, communication delays, server replication,
server failure and recovery, and server sharding. A consistency model
helps programmers design applications (clients) so that they provide
the behavior the programmers intend. And it helps service designers
decide whether specific design decisions are OK or not.

Here's an example of the kind of question linearizability can help
answer. Suppose we have a replicated storage service (like GFS).
Client C1 sends a write RPC for a certain key and receives a "success"
reply; after that, client C2 sends a read RPC for the same key and
receives a value back; no other clients modify that key; is C2
guaranteed to see C1's write? The answer to this question is important
for the design of the storage service, since it affects whether
replicas can serve reads, whether updates must be preserved if servers
crash, whether caches must be kept strictly up to date, &c.

Linearizability's answer is yes, C2 must see C1's write. And as a
result, a linearizable storage system often involves complex and
expensive management of replication, crash recovery, caching, &c.

Q: What's the definition of linearizability?

A: Linearizability is defined on "histories": traces of client
operations, annotated by the time at which each client operation
starts (is launched by a client), and the time at which the client
sees that the operation has finished. Linearizability tells you if an
individual history is legal. We say that a service is linearizable if
every history it can generate is linearizable.

There is one event in the history for a client starting an operation,
and another for the client deciding the operation has finished. Thus
the history makes concurrency among clients, and network delays,
explicit. Typically the start and finish events correspond to a
request and a response message exchanged with the server.

A history is linearizable if you can assign a "linearization point" (a
time) to each operation, where each operation's point lies between the
times of its start and finish events, and the history's response
values are the same as you'd get if you executed the operations one at
a time in point order. If no assignment of linearization points
satisfies these two requirements, the history is not linearizable.

Q: Why is linearizability a desirable consistency model?

A: Because it is relatively strong, in the sense of forbidding many
"anomalous" behaviours that might cause problems for application
programmers. Services with stronger consistency guarantees tend to be
easier for programmers to design for than weaker ones.

For example, suppose one part of an application computes a value,
writes it to the storage system, and then sets a flag in the storage
system indicating that the computed value is ready:

  v = compute...
  put("value", v)
  put("done", true)

On a different computer a program checks "done" to see if the value is
available, and uses it if it is:

  if get("done") == true:
    v = get("value")
    print v

If the storage system that implements put() and get() is linearizable,
the above programs will work as expected.

With many weaker consistency models, the above programs will not work
as one might hope. For example, a storage system providing "eventual
consistency" might re-order the two puts (so that "done" is true even
though "value" is not available), or might yield a stale (old) value
for either of the get()s.

Q: When trying to demonstrate that a history is linearizable, how does
one decide where to place the linearization point for each operation?

A: The idea is that, in order to show that an execution is
linearizable, you (the human) need to find places to put the little
orange lines (linearization points). That is, in order to show that a
history is linearizable, you need to find an assignment of
linearization points (and thus an order of operations) that conforms
to these requirements:

  * All function calls have a linearization point at some instant
    between their invocation and their response.

  * All functions appear to occur instantly at their linearization
    point, behaving as specified by the sequential definition.

So, some placements of linearization points are invalid because they lie
outside of the time span of a request; others are invalid because they
violate the sequential definition (for a key/value store, a violation means
that a read does not observe the most recently written value, where
"recent" refers to linearization points).

For a complex history you may need to try many assignments of
linearization points in order to find one that demonstrates that the
history is linearizable. If you try them all, and none works, then the
history is not linearizable.

Q: Why not use the time at which the client sent the command as the
linearization point? I.e. have the system execute operations in the
order that clients sent them?

A: It's hard to build a system that guarantees that behavior -- the
start time is the time at which the client code issued the request,
but the service might not receive the request until much later due to
network delays. That is, requests may arrive at the service in an
order that's quite different from the order of start times. The
service could in principle delay execution of every arriving request
in case a request with an earlier issue time arrives later, but it's
hard to know how long to wait since networks can impose unbounded
delays. And it would increase delays for every request, perhaps by a
lot. That said, Spanner, which we'll look at later, uses a related
technique.

A correctness specification like linearizability needs to walk a fine
line between being lax enough to implement efficiently, but strict
enough to provide useful guarantees to application programs. "Appears to
execute operations in invocation order" is too strict to implement
efficiently, whereas linearizability's "appears to execute somewhere
between invocation and response" is implementable though not as
straightforward for application programmers.

Q: How do services implement linearizability?

A: If the service is implemented as a single server, with no
replication or caching or internal parallelism, it's nearly enough for
the service to execute client requests one a time as they arrive. One
complication comes from clients that re-send requests because they
think the network has lost messages: for requests with side-effects,
the service must take care to execute any given client request only
once. Replication, fault tolerance, and caching involve further design
complexity.

An nice consequence of linearizability is that the service has freedom
in the order in which it executes concurrent (overlapping-in-time)
operations. In particular, if operations from client C1 and C2 are
concurrent, the server could execute C2's operation first even if C1
started before C2. On the other hand, if C1 finished before C2
started, linearizability requires the service to act as if it executed
C1's operation before C2's (i.e. C2's operation is required to observe
the effects of C1's operation, if any).

Q: What are other consistency models?

A: Look for

    eventual consistency
    causal consistency
    fork consistency
    serializability
    sequential consistency
    timeline consistency

And there are others from the worlds of databases, CPU memory/cache
systems, and file systems.

In general, different models differ in how intuitive they are for
application programmers, and how much performance you can get with
them. For example, eventual consistency allows many anomalous results
(e.g. even if a write has completed, subsequent reads might not see
it), but in a distributed/replicated setting can be implemented with
higher performance than linearizability.

Q: Why is linearizability used as a consistency model versus other ones,
such as eventual consistency?

A: People do often build storage systems that provide consistency weaker
than linearizability, such as eventual and causal consistency.

Linearizability has some nice properties for application writers:

  * reads always observe fresh data.
  * if there are no concurrent writes, all readers see the
    same data.
  * on most linearizable systems you can add mini-transactions
    like test-and-set (because most linearizable designs end up
    executing operations on each data item one-at-a-time).

Weaker schemes like eventual and causal consistency can allow higher
performance, since they don't require all copies of data to be updated
right away. This higher performance is often the deciding factor. For
some applications weak consistency causes no problems, for example if
one is storing data items that are never updated, such as images or
video.

However, weak consistency introduces some complexity for application
writers:

  * reads can observe out-of-date (stale) data.
  * reads can observe writes out of order.
  * if you write, and then read, you may not see your write,
    but instead see stale data.
  * concurrent updates to the same items aren't executed
    one-at-a-time, so it's hard to to implement mini-transactions
    like test-and-set or atomic increment.

Q: Linearizability doesn't seem particularly "strong", since you can
be reading different data even when you execute two commands at the
same time; are there stronger notions?

A: True, linearizability is reminiscent of using threads in a program
without using locks. It's possible to program correctly this way but
it requires care.

An example of a stronger notion of consistency is transactions, as
found in many databases, which effectively lock any data used. For
programs that read and write multiple data items, transactions make
programming easier than linearizability. "Serializability" is the name
of one consistency model that provides transactions.

However, transaction systems are significantly more complex, slower, and
harder to make fault-tolerant than linearizable systems.

Q: Is it a problem that concurrent get()s might see different values if
there's also a concurrent put()?

A: It's often not a problem in the context of storage systems. For
example, if the value we're talking about is my profile photograph,
and two different people ask to see it at the same time that I'm
updating the photo, then it's reasonable for them to see different
photos (either the old or new one).

Another way of looking at this is that it's the same behavior that
programmers already are familiar with on multi-core computers:
concurrent loads from different cores of a memory location that's
simultaneously being written are not guaranteed to all see the same
value.

Q: What are some examples of real-world linearizable storage systems?
And of storage systems with weaker consistency guarantees?

A: Google's Spanner and Amazon's S3 are storage systems that provide
linearizability.

Google's GFS, Amazon's Dynamo, and Cassandra provide weaker
consistency; they are probably best classified as eventually
consistent.

Q: What do people do to ensure their distributed systems are correct?

A: Thorough testing is a common plan, perhaps using a linearizability
checker such as Porcupine.

Use of formal methods is also common; have a look here for some
examples:

https://arxiv.org/pdf/2210.13661.pdf

https://assets.amazon.science/67/f9/92733d574c11ba1a11bd08bfb8ae/how-amazon-web-services-uses-formal-methods.pdf

https://dl.acm.org/doi/abs/10.1145/3477132.3483540

https://www.ccs.neu.edu/~stavros/papers/2022-cpp-published.pdf

https://www.cs.purdue.edu/homes/pfonseca/papers/eurosys2017-dsbugs.pdf

https://www.andrew.cmu.edu/user/bparno/papers/ironfleet.pdf

Q: How hard is it to formally prove a service to be correct?

A: It turns out that proving significant theorems about complex
programs is difficult -- much more difficult than ordinary
programming.

You can get a feel for this by trying the labs for this course:

  https://6826.csail.mit.edu/2020/

Q: How does a team decide that they have tested a product thoroughly
enough to ship to customers?

A: It's a good idea to start shipping product, and getting revenue,
before your company runs out of money and goes bankrupt. People test
as much as they can before that point, and usually try to persuade a
few early customers to use the product (and help reveal bugs) with the
understanding that it might not work correctly. Maybe you are ready to
ship when the product is functional enough to satisfy many customers
and has no known major bugs.

Independent of this, a wise customer will also test software that they
depend on. No serious organization expects any software to be bug-free.

Q: How do linearizability checkers work?

A: A simple linearizability checker would try every possible order (or
choice of linearization points) to see if one is valid according to
the rules in the definition of linearizability. Because that would be
too slow on big histories, clever checkers avoid looking at clearly
impossible orders (e.g. if a proposed linearization point is before
the operation's start time), decompose the history into sub-histories
that can be checked separately when that's possible, and use
heuristics to try more likely orders first.

These papers describe the techniques; I believe Knossos is based on the
first paper, and Porcupine adds ideas from the second paper:

    http://www.cs.ox.ac.uk/people/gavin.lowe/LinearizabiltyTesting/paper.pdf
    https://arxiv.org/pdf/1504.00204.pdf

Q: Are there examples of real-world systems tested with Porcupine or
similar testing frameworks?

A: Such testing is common -- for example, have a look at
https://jepsen.io/analyses; Jepsen is an organization that has tested
the correctness (and linearizability, where appropriate) of many storage
systems.

For Porcupine specifically, here's an example:

  https://www.vldb.org/pvldb/vol15/p2201-zare.pdf

MapReduce: Simplified Data Processing on Large Clusters
Jeffrey Dean and Sanjay Ghemawat
jeff@google.com, sanjay@google.com
Google, Inc.
Abstract
MapReduce is a programming model and an associ-
ated implementation for processing and generating large
data sets. Users specify a map function that processes a
key/value pair to generate a set of intermediate key/value
pairs, and a reduce function that merges all intermediate
values associated with the same intermediate key. Many
real world tasks are expressible in this model, as shown
in the paper.
Programs written in this functional style are automati-
cally parallelized and executed on a large cluster of com-
modity machines. The run-time system takes care of the
details of partitioning the input data, scheduling the pro-
gram’s execution across a set of machines, handling ma-
chine failures, and managing the required inter-machine
communication. This allows programmers without any
experience with parallel and distributed systems to eas-
ily utilize the resources of a large distributed system.
Our implementation of MapReduce runs on a large
cluster of commodity machines and is highly scalable:
a typical MapReduce computation processes many ter-
abytes of data on thousands of machines. Programmers
find the system easy to use: hundreds of MapReduce pro-
grams have been implemented and upwards of one thou-
sand MapReduce jobs are executed on Google’s clusters
every day.
1 Introduction
Over the past five years, the authors and many others at
Google have implemented hundreds of special-purpose
computations that process large amounts of raw data,
such as crawled documents, web request logs, etc., to
compute various kinds of derived data, such as inverted
indices, various representations of the graph structure
of web documents, summaries of the number of pages
crawled per host, the set of most frequent queries in a
given day, etc. Most such computations are conceptu-
ally straightforward. However, the input data is usually
large and the computations have to be distributed across
hundreds or thousands of machines in order to finish in
a reasonable amount of time. The issues of how to par-
allelize the computation, distribute the data, and handle
failures conspire to obscure the original simple compu-
tation with large amounts of complex code to deal with
these issues.
As a reaction to this complexity, we designed a new
abstraction that allows us to express the simple computa-
tions we were trying to perform but hides the messy de-
tails of parallelization, fault-tolerance, data distribution
and load balancing in a library. Our abstraction is in-
spired by the map and reduce primitives present in Lisp
and many other functional languages. We realized that
most of our computations involved applying a map op-
eration to each logical “record” in our input in order to
compute a set of intermediate key/value pairs, and then
applying a reduce operation to all the values that shared
the same key, in order to combine the derived data ap-
propriately. Our use of a functional model with user-
specified map and reduce operations allows us to paral-
lelize large computations easily and to use re-execution
as the primary mechanism for fault tolerance.
The major contributions of this work are a simple and
powerful interface that enables automatic parallelization
and distribution of large-scale computations, combined
with an implementation of this interface that achieves
high performance on large clusters of commodity PCs.
Section 2 describes the basic programming model and
gives several examples. Section 3 describes an imple-
mentation of the MapReduce interface tailored towards
our cluster-based computing environment. Section 4 de-
scribes several refinements of the programming model
that we have found useful. Section 5 has performance
measurements of our implementation for a variety of
tasks. Section 6 explores the use of MapReduce within
Google including our experiences in using it as the basis
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 137

for a rewrite of our production indexing system. Sec-
tion 7 discusses related and future work.
2 Programming Model
The computation takes a set of input key/value pairs, and
produces a set of output key/value pairs. The user of
the MapReduce library expresses the computation as two
functions: Map and Reduce.
Map, written by the user, takes an input pair and pro-
duces a set of intermediate key/value pairs. The MapRe-
duce library groups together all intermediate values asso-
ciated with the same intermediate key I and passes them
to the Reduce function.
The Reduce function, also written by the user, accepts
an intermediate key I and a set of values for that key. It
merges together these values to form a possibly smaller
set of values. Typically just zero or one output value is
produced per Reduce invocation. The intermediate val-
ues are supplied to the user’s reduce function via an iter-
ator. This allows us to handle lists of values that are too
large to fit in memory.
2.1 Example
Consider the problem of counting the number of oc-
currences of each word in a large collection of docu-
ments. The user would write code similar to the follow-
ing pseudo-code:
map(String key, String value):
// key: document name
// value: document contents
for each word w in value:
EmitIntermediate(w, "1");
reduce(String key, Iterator values):
// key: a word
// values: a list of counts
int result = 0;
for each v in values:
result += ParseInt(v);
Emit(AsString(result));
The map function emits each word plus an associated
count of occurrences (just ‘1’ in this simple example).
The reduce function sums together all counts emitted
for a particular word.
In addition, the user writes code to fill in a mapreduce
specification object with the names of the input and out-
put files, and optional tuning parameters. The user then
invokes the MapReduce function, passing it the specifi-
cation object. The user’s code is linked together with the
MapReduce library (implemented in C++). Appendix A
contains the full program text for this example.
2.2 Types
Even though the previous pseudo-code is written in terms
of string inputs and outputs, conceptually the map and
reduce functions supplied by the user have associated
types:
map (k1,v1) → list(k2,v2)
reduce (k2,list(v2)) → list(v2)
I.e., the input keys and values are drawn from a different
domain than the output keys and values. Furthermore,
the intermediate keys and values are from the same do-
main as the output keys and values.
Our C++ implementation passes strings to and from
the user-defined functions and leaves it to the user code
to convert between strings and appropriate types.
2.3 More Examples
Here are a few simple examples of interesting programs
that can be easily expressed as MapReduce computa-
tions.
Distributed Grep: The map function emits a line if it
matches a supplied pattern. The reduce function is an
identity function that just copies the supplied intermedi-
ate data to the output.
Count of URL Access Frequency: The map func-
tion processes logs of web page requests and outputs
⟨URL, 1⟩. The reduce function adds together all values
for the same URL and emits a ⟨URL, total count ⟩
pair.
Reverse Web-Link Graph: The map function outputs
⟨target, source⟩ pairs for each link to a target
URL found in a page named source. The reduce
function concatenates the list of all source URLs as-
sociated with a given target URL and emits the pair:
⟨target, list(source)⟩
T erm-Vector per Host: A term vector summarizes the
most important words that occur in a document or a set
of documents as a list of ⟨word, f requency⟩ pairs. The
map function emits a ⟨hostname, term vector ⟩
pair for each input document (where the hostname is
extracted from the URL of the document). The re-
duce function is passed all per-document term vectors
for a given host. It adds these term vectors together,
throwing away infrequent terms, and then emits a final
⟨hostname, term vector ⟩ pair.
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association138

User
Program
Master
(1) fork
worker
(1) fork
worker
(1) fork
(2)
assign
map
(2)
assign
reduce
split 0
split 1
split 2
split 3
split 4
output
file 0
(6) write
worker
(3) read
worker
(4) local write
Map
phase
Intermediate files
(on local disks)
worker output
file 1
Input
files
(5) remote read
Reduce
phase
Output
files
Figure 1: Execution overview
Inverted Index: The map function parses each docu-
ment, and emits a sequence of ⟨word, document ID ⟩
pairs. The reduce function accepts all pairs for a given
word, sorts the corresponding document IDs and emits a
⟨word, list(document ID )⟩ pair. The set of all output
pairs forms a simple inverted index. It is easy to augment
this computation to keep track of word positions.
Distributed Sort: The map function extracts the key
from each record, and emits a ⟨key, record⟩ pair. The
reduce function emits all pairs unchanged. This compu-
tation depends on the partitioning facilities described in
Section 4.1 and the ordering properties described in Sec-
tion 4.2.
3 Implementation
Many different implementations of the MapReduce in-
terface are possible. The right choice depends on the
environment. For example, one implementation may be
suitable for a small shared-memory machine, another for
a large NUMA multi-processor, and yet another for an
even larger collection of networked machines.
This section describes an implementation targeted
to the computing environment in wide use at Google:
large clusters of commodity PCs connected together with
switched Ethernet [4]. In our environment:
(1) Machines are typically dual-processor x86 processors
running Linux, with 2-4 GB of memory per machine.
(2) Commodity networking hardware is used – typically
either 100 megabits/second or 1 gigabit/second at the
machine level, but averaging considerably less in over-
all bisection bandwidth.
(3) A cluster consists of hundreds or thousands of ma-
chines, and therefore machine failures are common.
(4) Storage is provided by inexpensive IDE disks at-
tached directly to individual machines. A distributed file
system [8] developed in-house is used to manage the data
stored on these disks. The file system uses replication to
provide availability and reliability on top of unreliable
hardware.
(5) Users submit jobs to a scheduling system. Each job
consists of a set of tasks, and is mapped by the scheduler
to a set of available machines within a cluster.
3.1 Execution Overview
The Map invocations are distributed across multiple
machines by automatically partitioning the input data
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 139

into a set of M splits. The input splits can be pro-
cessed in parallel by different machines. Reduce invoca-
tions are distributed by partitioning the intermediate key
space into R pieces using a partitioning function (e.g.,
hash(key) mod R). The number of partitions ( R) and
the partitioning function are specified by the user.
Figure 1 shows the overall flow of a MapReduce op-
eration in our implementation. When the user program
calls the MapReduce function, the following sequence
of actions occurs (the numbered labels in Figure 1 corre-
spond to the numbers in the list below):
1. The MapReduce library in the user program first
splits the input files into M pieces of typically 16
megabytes to 64 megabytes (MB) per piece (con-
trollable by the user via an optional parameter). It
then starts up many copies of the program on a clus-
ter of machines.
2. One of the copies of the program is special – the
master. The rest are workers that are assigned work
by the master. There are M map tasks and R reduce
tasks to assign. The master picks idle workers and
assigns each one a map task or a reduce task.
3. A worker who is assigned a map task reads the
contents of the corresponding input split. It parses
key/value pairs out of the input data and passes each
pair to the user-defined Map function. The interme-
diate key/value pairs produced by the Map function
are buffered in memory.
4. Periodically, the buffered pairs are written to local
disk, partitioned into R regions by the partitioning
function. The locations of these buffered pairs on
the local disk are passed back to the master, who
is responsible for forwarding these locations to the
reduce workers.
5. When a reduce worker is notified by the master
about these locations, it uses remote procedure calls
to read the buffered data from the local disks of the
map workers. When a reduce worker has read all in-
termediate data, it sorts it by the intermediate keys
so that all occurrences of the same key are grouped
together. The sorting is needed because typically
many different keys map to the same reduce task. If
the amount of intermediate data is too large to fit in
memory, an external sort is used.
6. The reduce worker iterates over the sorted interme-
diate data and for each unique intermediate key en-
countered, it passes the key and the corresponding
set of intermediate values to the user’s Reduce func-
tion. The output of the Reduce function is appended
to a final output file for this reduce partition.
7. When all map tasks and reduce tasks have been
completed, the master wakes up the user program.
At this point, the MapReduce call in the user pro-
gram returns back to the user code.
After successful completion, the output of the mapre-
duce execution is available in the R output files (one per
reduce task, with file names as specified by the user).
Typically, users do not need to combine these R output
files into one file – they often pass these files as input to
another MapReduce call, or use them from another dis-
tributed application that is able to deal with input that is
partitioned into multiple files.
3.2 Master Data Structures
The master keeps several data structures. For each map
task and reduce task, it stores the state ( idle, in-progress,
or completed), and the identity of the worker machine
(for non-idle tasks).
The master is the conduit through which the location
of intermediate file regions is propagated from map tasks
to reduce tasks. Therefore, for each completed map task,
the master stores the locations and sizes of the R inter-
mediate file regions produced by the map task. Updates
to this location and size information are received as map
tasks are completed. The information is pushed incre-
mentally to workers that have in-progress reduce tasks.
3.3 Fault T olerance
Since the MapReduce library is designed to help process
very large amounts of data using hundreds or thousands
of machines, the library must tolerate machine failures
gracefully.
Worker Failure
The master pings every worker periodically. If no re-
sponse is received from a worker in a certain amount of
time, the master marks the worker as failed. Any map
tasks completed by the worker are reset back to their ini-
tial idle state, and therefore become eligible for schedul-
ing on other workers. Similarly, any map task or reduce
task in progress on a failed worker is also reset to idle
and becomes eligible for rescheduling.
Completed map tasks are re-executed on a failure be-
cause their output is stored on the local disk(s) of the
failed machine and is therefore inaccessible. Completed
reduce tasks do not need to be re-executed since their
output is stored in a global file system.
When a map task is executed first by worker A and
then later executed by worker B (because A failed), all
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association140

workers executing reduce tasks are notified of the re-
execution. Any reduce task that has not already read the
data from worker A will read the data from worker B.
MapReduce is resilient to large-scale worker failures.
For example, during one MapReduce operation, network
maintenance on a running cluster was causing groups of
80 machines at a time to become unreachable for sev-
eral minutes. The MapReduce master simply re-executed
the work done by the unreachable worker machines, and
continued to make forward progress, eventually complet-
ing the MapReduce operation.
Master Failure
It is easy to make the master write periodic checkpoints
of the master data structures described above. If the mas-
ter task dies, a new copy can be started from the last
checkpointed state. However, given that there is only a
single master, its failure is unlikely; therefore our cur-
rent implementation aborts the MapReduce computation
if the master fails. Clients can check for this condition
and retry the MapReduce operation if they desire.
Semantics in the Presence of Failures
When the user-supplied map and reduce operators are de-
terministic functions of their input values, our distributed
implementation produces the same output as would have
been produced by a non-faulting sequential execution of
the entire program.
We rely on atomic commits of map and reduce task
outputs to achieve this property. Each in-progress task
writes its output to private temporary files. A reduce task
produces one such file, and a map task produces R such
files (one per reduce task). When a map task completes,
the worker sends a message to the master and includes
the names of the R temporary files in the message. If
the master receives a completion message for an already
completed map task, it ignores the message. Otherwise,
it records the names of R files in a master data structure.
When a reduce task completes, the reduce worker
atomically renames its temporary output file to the final
output file. If the same reduce task is executed on multi-
ple machines, multiple rename calls will be executed for
the same final output file. We rely on the atomic rename
operation provided by the underlying file system to guar-
antee that the final file system state contains just the data
produced by one execution of the reduce task.
The vast majority of our map and reduce operators are
deterministic, and the fact that our semantics are equiv-
alent to a sequential execution in this case makes it very
easy for programmers to reason about their program’s be-
havior. When the map and/or reduce operators are non-
deterministic, we provide weaker but still reasonable se-
mantics. In the presence of non-deterministic operators,
the output of a particular reduce task R
1 is equivalent to
the output for R1 produced by a sequential execution of
the non-deterministic program. However, the output for
a different reduce task R2 may correspond to the output
for R2 produced by a different sequential execution of
the non-deterministic program.
Consider map task M and reduce tasks R1 and R2.
Let e(Ri) be the execution of Ri that committed (there
is exactly one such execution). The weaker semantics
arise because e(R1) may have read the output produced
by one execution of M and e(R2) may have read the
output produced by a different execution of M .
3.4 Locality
Network bandwidth is a relatively scarce resource in our
computing environment. We conserve network band-
width by taking advantage of the fact that the input data
(managed by GFS [8]) is stored on the local disks of the
machines that make up our cluster. GFS divides each
file into 64 MB blocks, and stores several copies of each
block (typically 3 copies) on different machines. The
MapReduce master takes the location information of the
input files into account and attempts to schedule a map
task on a machine that contains a replica of the corre-
sponding input data. Failing that, it attempts to schedule
a map task near a replica of that task’s input data (e.g., on
a worker machine that is on the same network switch as
the machine containing the data). When running large
MapReduce operations on a significant fraction of the
workers in a cluster, most input data is read locally and
consumes no network bandwidth.
3.5 T ask Granularity
We subdivide the map phase into M pieces and the re-
duce phase into R pieces, as described above. Ideally, M
and R should be much larger than the number of worker
machines. Having each worker perform many different
tasks improves dynamic load balancing, and also speeds
up recovery when a worker fails: the many map tasks
it has completed can be spread out across all the other
worker machines.
There are practical bounds on how large M and R can
be in our implementation, since the master must make
O(M + R) scheduling decisions and keeps O(M ∗ R)
state in memory as described above. (The constant fac-
tors for memory usage are small however: the O(M ∗ R)
piece of the state consists of approximately one byte of
data per map task/reduce task pair.)
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 141

Furthermore, R is often constrained by users because
the output of each reduce task ends up in a separate out-
put file. In practice, we tend to choose M so that each
individual task is roughly 16 MB to 64 MB of input data
(so that the locality optimization described above is most
effective), and we make R a small multiple of the num-
ber of worker machines we expect to use. We often per-
form MapReduce computations with M = 200 , 000 and
R =5 , 000, using 2,000 worker machines.
3.6 Backup T asks
One of the common causes that lengthens the total time
taken for a MapReduce operation is a “straggler”: a ma-
chine that takes an unusually long time to complete one
of the last few map or reduce tasks in the computation.
Stragglers can arise for a whole host of reasons. For ex-
ample, a machine with a bad disk may experience fre-
quent correctable errors that slow its read performance
from 30 MB/s to 1 MB/s. The cluster scheduling sys-
tem may have scheduled other tasks on the machine,
causing it to execute the MapReduce code more slowly
due to competition for CPU, memory, local disk, or net-
work bandwidth. A recent problem we experienced was
a bug in machine initialization code that caused proces-
sor caches to be disabled: computations on affected ma-
chines slowed down by over a factor of one hundred.
We have a general mechanism to alleviate the prob-
lem of stragglers. When a MapReduce operation is close
to completion, the master schedules backup executions
of the remaining in-progress tasks. The task is marked
as completed whenever either the primary or the backup
execution completes. We have tuned this mechanism so
that it typically increases the computational resources
used by the operation by no more than a few percent.
We have found that this significantly reduces the time
to complete large MapReduce operations. As an exam-
ple, the sort program described in Section 5.3 takes 44%
longer to complete when the backup task mechanism is
disabled.
4 Refinements
Although the basic functionality provided by simply
writing Map and Reduce functions is sufficient for most
needs, we have found a few extensions useful. These are
described in this section.
4.1 Partitioning Function
The users of MapReduce specify the number of reduce
tasks/output files that they desire ( R). Data gets parti-
tioned across these tasks using a partitioning function on
the intermediate key. A default partitioning function is
provided that uses hashing (e.g. “ hash(key) mod R”).
This tends to result in fairly well-balanced partitions. In
some cases, however, it is useful to partition data by
some other function of the key. For example, sometimes
the output keys are URLs, and we want all entries for a
single host to end up in the same output file. To support
situations like this, the user of the MapReduce library
can provide a special partitioning function. For example,
using “hash(Hostname(urlkey )) mod R” as the par-
titioning function causes all URLs from the same host to
end up in the same output file.
4.2 Ordering Guarantees
We guarantee that within a given partition, the interme-
diate key/value pairs are processed in increasing key or-
der. This ordering guarantee makes it easy to generate
a sorted output file per partition, which is useful when
the output file format needs to support efficient random
access lookups by key, or users of the output find it con-
venient to have the data sorted.
4.3 Combiner Function
In some cases, there is significant repetition in the inter-
mediate keys produced by each map task, and the user-
specified Reduce function is commutative and associa-
tive. A good example of this is the word counting exam-
ple in Section 2.1. Since word frequencies tend to follow
a Zipf distribution, each map task will produce hundreds
or thousands of records of the form <the, 1> . All of
these counts will be sent over the network to a single re-
duce task and then added together by the Reduce function
to produce one number. We allow the user to specify an
optional Combiner function that does partial merging of
this data before it is sent over the network.
The Combiner function is executed on each machine
that performs a map task. Typically the same code is used
to implement both the combiner and the reduce func-
tions. The only difference between a reduce function and
a combiner function is how the MapReduce library han-
dles the output of the function. The output of a reduce
function is written to the final output file. The output of
a combiner function is written to an intermediate file that
will be sent to a reduce task.
Partial combining significantly speeds up certain
classes of MapReduce operations. Appendix A contains
an example that uses a combiner.
4.4 Input and Output Types
The MapReduce library provides support for reading in-
put data in several different formats. For example, “text”
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association142

mode input treats each line as a key/value pair: the key
is the offset in the file and the value is the contents of
the line. Another common supported format stores a
sequence of key/value pairs sorted by key. Each input
type implementation knows how to split itself into mean-
ingful ranges for processing as separate map tasks (e.g.
text mode’s range splitting ensures that range splits oc-
cur only at line boundaries). Users can add support for a
new input type by providing an implementation of a sim-
ple reader interface, though most users just use one of a
small number of predefined input types.
A reader does not necessarily need to provide data
read from a file. For example, it is easy to define a reader
that reads records from a database, or from data struc-
tures mapped in memory.
In a similar fashion, we support a set of output types
for producing data in different formats and it is easy for
user code to add support for new output types.
4.5 Side-effects
In some cases, users of MapReduce have found it con-
venient to produce auxiliary files as additional outputs
from their map and/or reduce operators. We rely on the
application writer to make such side-effects atomic and
idempotent. Typically the application writes to a tempo-
rary file and atomically renames this file once it has been
fully generated.
We do not provide support for atomic two-phase com-
mits of multiple output files produced by a single task.
Therefore, tasks that produce multiple output files with
cross-file consistency requirements should be determin-
istic. This restriction has never been an issue in practice.
4.6 Skipping Bad Records
Sometimes there are bugs in user code that cause theMap
or Reduce functions to crash deterministically on certain
records. Such bugs prevent a MapReduce operation from
completing. The usual course of action is to fix the bug,
but sometimes this is not feasible; perhaps the bug is in
a third-party library for which source code is unavail-
able. Also, sometimes it is acceptable to ignore a few
records, for example when doing statistical analysis on
a large data set. We provide an optional mode of execu-
tion where the MapReduce library detects which records
cause deterministic crashes and skips these records in or-
der to make forward progress.
Each worker process installs a signal handler that
catches segmentation violations and bus errors. Before
invoking a user Map or Reduce operation, the MapRe-
duce library stores the sequence number of the argument
in a global variable. If the user code generates a signal,
the signal handler sends a “last gasp” UDP packet that
contains the sequence number to the MapReduce mas-
ter. When the master has seen more than one failure on
a particular record, it indicates that the record should be
skipped when it issues the next re-execution of the corre-
sponding Map or Reduce task.
4.7 Local Execution
Debugging problems in Map or Reduce functions can be
tricky, since the actual computation happens in a dis-
tributed system, often on several thousand machines,
with work assignment decisions made dynamically by
the master. To help facilitate debugging, profiling, and
small-scale testing, we have developed an alternative im-
plementation of the MapReduce library that sequentially
executes all of the work for a MapReduce operation on
the local machine. Controls are provided to the user so
that the computation can be limited to particular map
tasks. Users invoke their program with a special flag and
can then easily use any debugging or testing tools they
find useful (e.g. gdb).
4.8 Status Information
The master runs an internal HTTP server and exports
a set of status pages for human consumption. The sta-
tus pages show the progress of the computation, such as
how many tasks have been completed, how many are in
progress, bytes of input, bytes of intermediate data, bytes
of output, processing rates, etc. The pages also contain
links to the standard error and standard output files gen-
erated by each task. The user can use this data to pre-
dict how long the computation will take, and whether or
not more resources should be added to the computation.
These pages can also be used to figure out when the com-
putation is much slower than expected.
In addition, the top-level status page shows which
workers have failed, and which map and reduce tasks
they were processing when they failed. This informa-
tion is useful when attempting to diagnose bugs in the
user code.
4.9 Counters
The MapReduce library provides a counter facility to
count occurrences of various events. For example, user
code may want to count total number of words processed
or the number of German documents indexed, etc.
To use this facility, user code creates a named counter
object and then increments the counter appropriately in
the Map and/or Reduce function. For example:
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 143

Counter* uppercase;
uppercase = GetCounter("uppercase");
map(String name, String contents):
for each word w in contents:
if (IsCapitalized(w)):
uppercase->Increment();
EmitIntermediate(w, "1");
The counter values from individual worker machines
are periodically propagated to the master (piggybacked
on the ping response). The master aggregates the counter
values from successful map and reduce tasks and returns
them to the user code when the MapReduce operation
is completed. The current counter values are also dis-
played on the master status page so that a human can
watch the progress of the live computation. When aggre-
gating counter values, the master eliminates the effects of
duplicate executions of the same map or reduce task to
avoid double counting. (Duplicate executions can arise
from our use of backup tasks and from re-execution of
tasks due to failures.)
Some counter values are automatically maintained
by the MapReduce library, such as the number of in-
put key/value pairs processed and the number of output
key/value pairs produced.
Users have found the counter facility useful for san-
ity checking the behavior of MapReduce operations. For
example, in some MapReduce operations, the user code
may want to ensure that the number of output pairs
produced exactly equals the number of input pairs pro-
cessed, or that the fraction of German documents pro-
cessed is within some tolerable fraction of the total num-
ber of documents processed.
5 Performance
In this section we measure the performance of MapRe-
duce on two computations running on a large cluster of
machines. One computation searches through approxi-
mately one terabyte of data looking for a particular pat-
tern. The other computation sorts approximately one ter-
abyte of data.
These two programs are representative of a large sub-
set of the real programs written by users of MapReduce –
one class of programs shuffles data from one representa-
tion to another, and another class extracts a small amount
of interesting data from a large data set.
5.1 Cluster Configuration
All of the programs were executed on a cluster that
consisted of approximately 1800 machines. Each ma-
chine had two 2GHz Intel Xeon processors with Hyper-
Threading enabled, 4GB of memory, two 160GB IDE
20 40 60 80 100
Seconds
0
10000
20000
30000Input (MB/s)
Figure 2: Data transfer rate over time
disks, and a gigabit Ethernet link. The machines were
arranged in a two-level tree-shaped switched network
with approximately 100-200 Gbps of aggregate band-
width available at the root. All of the machines were
in the same hosting facility and therefore the round-trip
time between any pair of machines was less than a mil-
lisecond.
Out of the 4GB of memory, approximately 1-1.5GB
was reserved by other tasks running on the cluster. The
programs were executed on a weekend afternoon, when
the CPUs, disks, and network were mostly idle.
5.2 Grep
The grep program scans through 1010 100-byte records,
searching for a relatively rare three-character pattern (the
pattern occurs in 92,337 records). The input is split into
approximately 64MB pieces ( M = 15000 ), and the en-
tire output is placed in one file ( R =1 ).
Figure 2 shows the progress of the computation over
time. The Y -axis shows the rate at which the input data is
scanned. The rate gradually picks up as more machines
are assigned to this MapReduce computation, and peaks
at over 30 GB/s when 1764 workers have been assigned.
As the map tasks finish, the rate starts dropping and hits
zero about 80 seconds into the computation. The entire
computation takes approximately 150 seconds from start
to finish. This includes about a minute of startup over-
head. The overhead is due to the propagation of the pro-
gram to all worker machines, and delays interacting with
GFS to open the set of 1000 input files and to get the
information needed for the locality optimization.
5.3 Sort
The sort program sorts 1010 100-byte records (approxi-
mately 1 terabyte of data). This program is modeled after
the TeraSort benchmark [10].
The sorting program consists of less than 50 lines of
user code. A three-line Map function extracts a 10-byte
sorting key from a text line and emits the key and the
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association144

500 1000
0
5000
10000
15000
20000Input (MB/s)
500 1000
0
5000
10000
15000
20000Shuffle (MB/s)
500 1000
Seconds
0
5000
10000
15000
20000Output (MB/s)
Done
(a) Normal execution
500 1000
0
5000
10000
15000
20000Input (MB/s)
500 1000
0
5000
10000
15000
20000Shuffle (MB/s)
500 1000
Seconds
0
5000
10000
15000
20000Output (MB/s)
Done
(b) No backup tasks
500 1000
0
5000
10000
15000
20000Input (MB/s)
500 1000
0
5000
10000
15000
20000Shuffle (MB/s)
500 1000
Seconds
0
5000
10000
15000
20000Output (MB/s)
Done
(c) 200 tasks killed
Figure 3: Data transfer rates over time for different executions of the sort program
original text line as the intermediate key/value pair. We
used a built-in Identity function as the Reduce operator.
This functions passes the intermediate key/value pair un-
changed as the output key/value pair. The final sorted
output is written to a set of 2-way replicated GFS files
(i.e., 2 terabytes are written as the output of the program).
As before, the input data is split into 64MB pieces
(M = 15000 ). We partition the sorted output into 4000
files (R = 4000 ). The partitioning function uses the ini-
tial bytes of the key to segregate it into one of R pieces.
Our partitioning function for this benchmark has built-
in knowledge of the distribution of keys. In a general
sorting program, we would add a pre-pass MapReduce
operation that would collect a sample of the keys and
use the distribution of the sampled keys to compute split-
points for the final sorting pass.
Figure 3 (a) shows the progress of a normal execution
of the sort program. The top-left graph shows the rate
at which input is read. The rate peaks at about 13 GB/s
and dies off fairly quickly since all map tasks finish be-
fore 200 seconds have elapsed. Note that the input rate
is less than for grep. This is because the sort map tasks
spend about half their time and I/O bandwidth writing in-
termediate output to their local disks. The corresponding
intermediate output for grep had negligible size.
The middle-left graph shows the rate at which data
is sent over the network from the map tasks to the re-
duce tasks. This shuffling starts as soon as the first
map task completes. The first hump in the graph is for
the first batch of approximately 1700 reduce tasks (the
entire MapReduce was assigned about 1700 machines,
and each machine executes at most one reduce task at a
time). Roughly 300 seconds into the computation, some
of these first batch of reduce tasks finish and we start
shuffling data for the remaining reduce tasks. All of the
shuffling is done about 600 seconds into the computation.
The bottom-left graph shows the rate at which sorted
data is written to the final output files by the reduce tasks.
There is a delay between the end of the first shuffling pe-
riod and the start of the writing period because the ma-
chines are busy sorting the intermediate data. The writes
continue at a rate of about 2-4 GB/s for a while. All of
the writes finish about 850 seconds into the computation.
Including startup overhead, the entire computation takes
891 seconds. This is similar to the current best reported
result of 1057 seconds for the TeraSort benchmark [18].
A few things to note: the input rate is higher than the
shuffle rate and the output rate because of our locality
optimization – most data is read from a local disk and
bypasses our relatively bandwidth constrained network.
The shuffle rate is higher than the output rate because
the output phase writes two copies of the sorted data (we
make two replicas of the output for reliability and avail-
ability reasons). We write two replicas because that is
the mechanism for reliability and availability provided
by our underlying file system. Network bandwidth re-
quirements for writing data would be reduced if the un-
derlying file system used erasure coding [14] rather than
replication.
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 145

5.4 Effect of Backup T asks
In Figure 3 (b), we show an execution of the sort pro-
gram with backup tasks disabled. The execution flow is
similar to that shown in Figure 3 (a), except that there is
a very long tail where hardly any write activity occurs.
After 960 seconds, all except 5 of the reduce tasks are
completed. However these last few stragglers don’t fin-
ish until 300 seconds later. The entire computation takes
1283 seconds, an increase of 44% in elapsed time.
5.5 Machine Failures
In Figure 3 (c), we show an execution of the sort program
where we intentionally killed 200 out of 1746 worker
processes several minutes into the computation. The
underlying cluster scheduler immediately restarted new
worker processes on these machines (since only the pro-
cesses were killed, the machines were still functioning
properly).
The worker deaths show up as a negative input rate
since some previously completed map work disappears
(since the corresponding map workers were killed) and
needs to be redone. The re-execution of this map work
happens relatively quickly. The entire computation fin-
ishes in 933 seconds including startup overhead (just an
increase of 5% over the normal execution time).
6 Experience
We wrote the first version of the MapReduce library in
February of 2003, and made significant enhancements to
it in August of 2003, including the locality optimization,
dynamic load balancing of task execution across worker
machines, etc. Since that time, we have been pleasantly
surprised at how broadly applicable the MapReduce li-
brary has been for the kinds of problems we work on.
It has been used across a wide range of domains within
Google, including:
• large-scale machine learning problems,
• clustering problems for the Google News and
Froogle products,
• extraction of data used to produce reports of popular
queries (e.g. Google Zeitgeist),
• extraction of properties of web pages for new exper-
iments and products (e.g. extraction of geographi-
cal locations from a large corpus of web pages for
localized search), and
• large-scale graph computations.
2003/03
2003/06
2003/09
2003/12
2004/03
2004/06
2004/090
200
400
600
800
1000
Number of instances in source tree
Figure 4: MapReduce instances over time
Number of jobs 29,423
Average job completion time 634 secs
Machine days used 79,186 days
Input data read 3,288 TB
Intermediate data produced 758 TB
Output data written 193 TB
Average worker machines per job 157
Average worker deaths per job 1.2
Average map tasks per job 3,351
Average reduce tasks per job 55
Unique map implementations 395
Unique reduce implementations 269
Unique map/reduce combinations 426
Table 1: MapReduce jobs run in August 2004
Figure 4 shows the significant growth in the number of
separate MapReduce programs checked into our primary
source code management system over time, from 0 in
early 2003 to almost 900 separate instances as of late
September 2004. MapReduce has been so successful be-
cause it makes it possible to write a simple program and
run it efficiently on a thousand machines in the course
of half an hour, greatly speeding up the development and
prototyping cycle. Furthermore, it allows programmers
who have no experience with distributed and/or parallel
systems to exploit large amounts of resources easily.
At the end of each job, the MapReduce library logs
statistics about the computational resources used by the
job. In Table 1, we show some statistics for a subset of
MapReduce jobs run at Google in August 2004.
6.1 Large-Scale Indexing
One of our most significant uses of MapReduce to date
has been a complete rewrite of the production index-
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association146

ing system that produces the data structures used for the
Google web search service. The indexing system takes
as input a large set of documents that have been retrieved
by our crawling system, stored as a set of GFS files. The
raw contents for these documents are more than 20 ter-
abytes of data. The indexing process runs as a sequence
of five to ten MapReduce operations. Using MapReduce
(instead of the ad-hoc distributed passes in the prior ver-
sion of the indexing system) has provided several bene-
fits:
• The indexing code is simpler, smaller, and easier to
understand, because the code that deals with fault
tolerance, distribution and parallelization is hidden
within the MapReduce library. For example, the
size of one phase of the computation dropped from
approximately 3800 lines of C++ code to approx-
imately 700 lines when expressed using MapRe-
duce.
• The performance of the MapReduce library is good
enough that we can keep conceptually unrelated
computations separate, instead of mixing them to-
gether to avoid extra passes over the data. This
makes it easy to change the indexing process. For
example, one change that took a few months to
make in our old indexing system took only a few
days to implement in the new system.
• The indexing process has become much easier to
operate, because most of the problems caused by
machine failures, slow machines, and networking
hiccups are dealt with automatically by the MapRe-
duce library without operator intervention. Further-
more, it is easy to improve the performance of the
indexing process by adding new machines to the in-
dexing cluster.
7 Related Work
Many systems have provided restricted programming
models and used the restrictions to parallelize the com-
putation automatically. For example, an associative func-
tion can be computed over all prefixes of an N element
array in log N time on N processors using parallel prefix
computations [6, 9, 13]. MapReduce can be considered
a simplification and distillation of some of these models
based on our experience with large real-world compu-
tations. More significantly, we provide a fault-tolerant
implementation that scales to thousands of processors.
In contrast, most of the parallel processing systems have
only been implemented on smaller scales and leave the
details of handling machine failures to the programmer.
Bulk Synchronous Programming [17] and some MPI
primitives [11] provide higher-level abstractions that
make it easier for programmers to write parallel pro-
grams. A key difference between these systems and
MapReduce is that MapReduce exploits a restricted pro-
gramming model to parallelize the user program auto-
matically and to provide transparent fault-tolerance.
Our locality optimization draws its inspiration from
techniques such as active disks [12, 15], where compu-
tation is pushed into processing elements that are close
to local disks, to reduce the amount of data sent across
I/O subsystems or the network. We run on commodity
processors to which a small number of disks are directly
connected instead of running directly on disk controller
processors, but the general approach is similar.
Our backup task mechanism is similar to the eager
scheduling mechanism employed in the Charlotte Sys-
tem [3]. One of the shortcomings of simple eager
scheduling is that if a given task causes repeated failures,
the entire computation fails to complete. We fix some in-
stances of this problem with our mechanism for skipping
bad records.
The MapReduce implementation relies on an in-house
cluster management system that is responsible for dis-
tributing and running user tasks on a large collection of
shared machines. Though not the focus of this paper, the
cluster management system is similar in spirit to other
systems such as Condor [16].
The sorting facility that is a part of the MapReduce
library is similar in operation to NOW-Sort [1]. Source
machines (map workers) partition the data to be sorted
and send it to one of R reduce workers. Each reduce
worker sorts its data locally (in memory if possible). Of
course NOW-Sort does not have the user-definable Map
and Reduce functions that make our library widely appli-
cable.
River [2] provides a programming model where pro-
cesses communicate with each other by sending data
over distributed queues. Like MapReduce, the River
system tries to provide good average case performance
even in the presence of non-uniformities introduced by
heterogeneous hardware or system perturbations. River
achieves this by careful scheduling of disk and network
transfers to achieve balanced completion times. MapRe-
duce has a different approach. By restricting the pro-
gramming model, the MapReduce framework is able
to partition the problem into a large number of fine-
grained tasks. These tasks are dynamically scheduled
on available workers so that faster workers process more
tasks. The restricted programming model also allows
us to schedule redundant executions of tasks near the
end of the job which greatly reduces completion time in
the presence of non-uniformities (such as slow or stuck
workers).
BAD-FS [5] has a very different programming model
from MapReduce, and unlike MapReduce, is targeted to
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 147

the execution of jobs across a wide-area network. How-
ever, there are two fundamental similarities. (1) Both
systems use redundant execution to recover from data
loss caused by failures. (2) Both use locality-aware
scheduling to reduce the amount of data sent across con-
gested network links.
TACC [7] is a system designed to simplify con-
struction of highly-available networked services. Like
MapReduce, it relies on re-execution as a mechanism for
implementing fault-tolerance.
8 Conclusions
The MapReduce programming model has been success-
fully used at Google for many different purposes. We
attribute this success to several reasons. First, the model
is easy to use, even for programmers without experience
with parallel and distributed systems, since it hides the
details of parallelization, fault-tolerance, locality opti-
mization, and load balancing. Second, a large variety
of problems are easily expressible as MapReduce com-
putations. For example, MapReduce is used for the gen-
eration of data for Google’s production web search ser-
vice, for sorting, for data mining, for machine learning,
and many other systems. Third, we have developed an
implementation of MapReduce that scales to large clus-
ters of machines comprising thousands of machines. The
implementation makes efficient use of these machine re-
sources and therefore is suitable for use on many of the
large computational problems encountered at Google.
We have learned several things from this work. First,
restricting the programming model makes it easy to par-
allelize and distribute computations and to make such
computations fault-tolerant. Second, network bandwidth
is a scarce resource. A number of optimizations in our
system are therefore targeted at reducing the amount of
data sent across the network: the locality optimization al-
lows us to read data from local disks, and writing a single
copy of the intermediate data to local disk saves network
bandwidth. Third, redundant execution can be used to
reduce the impact of slow machines, and to handle ma-
chine failures and data loss.
Acknowledgements
Josh Levenberg has been instrumental in revising and
extending the user-level MapReduce API with a num-
ber of new features based on his experience with using
MapReduce and other people’s suggestions for enhance-
ments. MapReduce reads its input from and writes its
output to the Google File System [8]. We would like to
thank Mohit Aron, Howard Gobioff, Markus Gutschke,
David Kramer, Shun-Tak Leung, and Josh Redstone for
their work in developing GFS. We would also like to
thank Percy Liang and Olcan Sercinoglu for their work
in developing the cluster management system used by
MapReduce. Mike Burrows, Wilson Hsieh, Josh Leven-
berg, Sharon Perl, Rob Pike, and Debby Wallach pro-
vided helpful comments on earlier drafts of this pa-
per. The anonymous OSDI reviewers, and our shepherd,
Eric Brewer, provided many useful suggestions of areas
where the paper could be improved. Finally, we thank all
the users of MapReduce within Google’s engineering or-
ganization for providing helpful feedback, suggestions,
and bug reports.
References
[1] Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau,
David E. Culler, Joseph M. Hellerstein, and David A. Pat-
terson. High-performance sorting on networks of work-
stations. In Proceedings of the 1997 ACM SIGMOD In-
ternational Conference on Management of Data , Tucson,
Arizona, May 1997.
[2] Remzi H. Arpaci-Dusseau, Eric Anderson, Noah
Treuhaft, David E. Culler, Joseph M. Hellerstein, David
Patterson, and Kathy Yelick. Cluster I/O with River:
Making the fast case common. In Proceedings of the Sixth
W orkshop on Input/Output in Parallel and Distributed
Systems (IOPADS ’99) , pages 10–22, Atlanta, Georgia,
May 1999.
[3] Arash Baratloo, Mehmet Karaul, Zvi Kedem, and Peter
Wyckoff. Charlotte: Metacomputing on the web. In Pro-
ceedings of the 9th International Conference on Parallel
and Distributed Computing Systems , 1996.
[4] Luiz A. Barroso, Jeffrey Dean, and Urs H¨ olzle. Web
search for a planet: The Google cluster architecture. IEEE
Micro, 23(2):22–28, April 2003.
[5] John Bent, Douglas Thain, Andrea C.Arpaci-Dusseau,
Remzi H. Arpaci-Dusseau, and Miron Livny. Explicit
control in a batch-aware distributed file system. In Pro-
ceedings of the 1st USENIX Symposium on Networked
Systems Design and Implementation NSDI , March 2004.
[6] Guy E. Blelloch. Scans as primitive parallel operations.
IEEE Transactions on Computers , C-38(11), November
1989.
[7] Armando Fox, Steven D. Gribble, Yatin Chawathe,
Eric A. Brewer, and Paul Gauthier. Cluster-based scal-
able network services. In Proceedings of the 16th ACM
Symposium on Operating System Principles , pages 78–
91, Saint-Malo, France, 1997.
[8] Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Le-
ung. The Google file system. In 19th Symposium on Op-
erating Systems Principles , pages 29–43, Lake George,
New Y ork, 2003.
OSDI ’04: 6th Symposium on Operating Systems Design and Implementation USENIX Association148

[9] S. Gorlatch. Systematic efficient parallelization of scan
and other list homomorphisms. In L. Bouge, P . Fraigni-
aud, A. Mignotte, and Y . Robert, editors, Euro-Par’96.
Parallel Processing, Lecture Notes in Computer Science
1124, pages 401–408. Springer-V erlag, 1996.
[10] Jim Gray. Sort benchmark home page.
http://research.microsoft.com/barc/SortBenchmark/.
[11] William Gropp, Ewing Lusk, and Anthony Skjellum.
Using MPI: Portable Parallel Programming with the
Message-Passing Interface. MIT Press, Cambridge, MA,
1999.
[12] L. Huston, R. Sukthankar, R. Wickremesinghe, M. Satya-
narayanan, G. R. Ganger, E. Riedel, and A. Ailamaki. Di-
amond: A storage architecture for early discard in inter-
active search. In Proceedings of the 2004 USENIX File
and Storage Technologies F AST Conference, April 2004.
[13] Richard E. Ladner and Michael J. Fischer. Parallel prefix
computation. Journal of the ACM, 27(4):831–838, 1980.
[14] Michael O. Rabin. Efficient dispersal of information for
security, load balancing and fault tolerance. Journal of
the ACM, 36(2):335–348, 1989.
[15] Erik Riedel, Christos Faloutsos, Garth A. Gibson, and
David Nagle. Active disks for large-scale data process-
ing. IEEE Computer, pages 68–74, June 2001.
[16] Douglas Thain, Todd Tannenbaum, and Miron Livny.
Distributed computing in practice: The Condor experi-
ence. Concurrency and Computation: Practice and Ex-
perience, 2004.
[17] L. G. V aliant. A bridging model for parallel computation.
Communications of the ACM , 33(8):103–111, 1997.
[18] Jim Wyllie. Spsort: How to sort a terabyte quickly.
http://alme1.almaden.ibm.com/cs/spsort.pdf.
A Word Frequency
This section contains a program that counts the number
of occurrences of each unique word in a set of input files
specified on the command line.
#include "mapreduce/mapreduce.h"
// User’s map function
class WordCounter : public Mapper {
public:
virtual void Map(const MapInput& input) {
const string& text = input.value();
const int n = text.size();
for (int i = 0; i < n; ) {
// Skip past leading whitespace
while ((i < n) && isspace(text[i]))
i++;
// Find word end
int start = i;
while ((i < n) && !isspace(text[i]))
i++;
if (start < i)
Emit(text.substr(start,i-start),"1");
}
}
};
REGISTER_MAPPER(WordCounter);
// User’s reduce function
class Adder : public Reducer {
virtual void Reduce(ReduceInput* input) {
// Iterate over all entries with the
// same key and add the values
int64 value = 0;
while (!input->done()) {
value += StringToInt(input->value());
input->NextValue();
}
// Emit sum for input->key()
Emit(IntToString(value));
}
};
REGISTER_REDUCER(Adder);
int main(int argc, char** argv) {
ParseCommandLineFlags(argc, argv);
MapReduceSpecification spec;
// Store list of input files into "spec"
for (int i = 1; i < argc; i++) {
MapReduceInput* input = spec.add_input();
input->set_format("text");
input->set_filepattern(argv[i]);
input->set_mapper_class("WordCounter");
}
// Specify the output files:
// /gfs/test/freq-00000-of-00100
// /gfs/test/freq-00001-of-00100
// ...
MapReduceOutput* out = spec.output();
out->set_filebase("/gfs/test/freq");
out->set_num_tasks(100);
out->set_format("text");
out->set_reducer_class("Adder");
// Optional: do partial sums within map
// tasks to save network bandwidth
out->set_combiner_class("Adder");
// Tuning parameters: use at most 2000
// machines and 100 MB of memory per task
spec.set_machines(2000);
spec.set_map_megabytes(100);
spec.set_reduce_megabytes(100);
// Now run it
MapReduceResult result;
if (!MapReduce(spec, &result)) abort();
// Done: ’result’ structure contains info
// about counters, time taken, number of
// machines used, etc.
return 0;
}
OSDI ’04: 6th Symposium on Operating Systems Design and ImplementationUSENIX Association 149

6.824 Scaling Memcached at Facebook FAQ

Q: Does the paper's design eliminate the possibility of stale data?

A: No, the design allows clients to read stale data from memcached in
some fairly common situations. For example, if a client writes some
data in the database, there will be a delay before mcsqueal sends out
invalidates (delete()s) to all the memcached servers that may be
caching data derived from that write, in all the clusters in the
region. A client that reads during that delay may read old cached
data, not the newly written data.

Q: Why is it OK for memcache to yield stale data?

A: The cached data is typically displayed to users on web pages, for
example news feed items, friend status, and messages. If the data is
out of date by a fraction of a second, users will usually not notice.

The big danger they are avoiding is long-term caching of stale
data. It's OK to serve data that's out of date by a few seconds. It's
not OK to serve data that's out of date by hours. Without the paper's
machinery, unbounded memcached staleness could arise due to lost
deletes or out of order updates.

Q: What if a client reads stale data from memcached, computes
something based on it, and writes the result to the database?

A: Facebook's application programmers don't write code like that.
Instead, the client sends a transaction to the database; the
transaction includes both the reads and the writes. So updates get
strong consistency and don't involve stale data.

Q: Why do they use memcached at all? Why not just read directly from
the MySQL database servers in the "storage cluster"?

A: The MySQL servers are not nearly fast enough to serve the volume of
reads generated by Facebook's web servers. memcached is orders of
magnitude faster than MySQL.

Q: What would a no-compromises design look like?

A: Ideally a design would handle billions of requests per second, be
easy for application programmers to use, work well for users spread
all over the world, provide strong consistency, and not cost too much.
That's a hard set of goals, and I don't know of a satisfying answer.

One source of problems is that MySQL, while powerful and easy to use,
has relatively low performance. So one could imagine using a faster
database, such as FaRM, which might eliminate the need for a cache,
and thus eliminate problems with cache consistency.

Another source of problems is the lack of integration between
memcached and MySQL. Perhaps one could have the cache and database
cooperate more closely; it might help if the cache (rather than the
application) controlled the handling of cache misses, and if the
database was in sole charge of updating or invalidating cached data.

For geographic distribution, have a look at Yahoo's PNUTS, which was
designed from the start to have useful consistency properties (though
not linearizability) while supporting multiple regions.

Q: What's the difference between the paper's "memcached" and "memcache"?

A: "memcached" refers to the software, which you can find here:

  https://github.com/memcached/memcached

memcached is a simple and fast key/value server. It stores data in
RAM, with no fault tolerance, so people only use it for caching (not
for persistent storage).

The paper uses "memcache" to refer to Facebook's set of servers
running memcached.

Q: What is the "stale set" problem in 3.2.1, and how do leases solve it?

A: Here's an example of the "stale set" problem that could occur if
there were no leases:

1. Client C1 asks memcache for k; memcache says k doesn't exist.
2. C1 asks MySQL for k, MySQL replies with value 1.
   C1 is slow at this point for some reason...
3. Someone updates k's value in MySQL to 2.
4. MySQL/mcsqueal/mcrouter send an invalidate for k to memcache,
   though memcache is not caching k, so there's nothing to invalidate.
5. C2 asks memcache for k; memcache says k doesn't exist.
6. C2 asks MySQL for k, mySQL replies with value 2.
7. C2 installs k=2 in memcache.
8. C1 installs k=1 in memcache.

Now memcache has a stale version of k, and it may never be updated.

The paper's leases fix the example:

1. Client C1 asks memcache for k; memcache says k doesn't exist,
   returns lease L1 to C1, and remembers the lease.
2. C1 asks MySQL for k, MySQL replies with value 1.
   C1 is slow at this point for some reason...
3. Someone updates k's value in MySQL to 2.
4. MySQL/mcsqueal/mcrouter send an invalidate for k to memcache,
   though memcache is not caching k, so there's nothing to invalidate.
   But memcache does invalidate C1's lease L1 (deletes L1 from its set
   of valid leases).
5. C2 asks memcache for k; memcache says k doesn't exist,
   and returns lease L2 to C2 (since there was no current lease for k).
6. C2 asks MySQL for k, mySQL replies with value 2.
7. C2 installs k=2 in memcache, supplying valid lease L2.
8. C1 installs k=1 in memcache, supplying invalid lease L1,
   so memcache ignores C1.

Now memcache is left caching the correct k=2.

Q: What is the "thundering herd" problem in 3.2.1, and how do leases
solve it?

A: The thundering herd problem:

* key k is popular -- lots of clients read it.
* ordinarily clients read k from memcache, which is fast.
* but suppose someone writes k, causing it to be invalidated in memcache.
* for a while, every client that tries to read k will miss in memcache.
* they will all ask MySQL for k.
* MySQL may be overloaded with too many simultaneous requests.

The paper's leases solve this problem by allowing only the first
client that misses to ask MySQL for the latest data. The other clients
wait for a bit to give the first client a chance to fetch the data
from MySQL and install it in memcache, then the other clients re-try
memcache.

Q: Why do writing clients delete() from memcache, rather than updating
the values in memcache?

A: Suppose two clients, C1 and C2, want to update the same item at the
same time; C1 wants to set the item to value "x", and C2 to "y". They
both send their updates to the MySQL database, which executes the
writes in one order or the other. Let's suppose the database executes
C1's write first, then C2's write, so that the final value in the
database is "y". Then C1 sends put(k, "x") to memcached, and C2 sends
put(k, "y") to memcached, at about the same time. Memcached may
execute the requests in either order, so it may execute C2's put("y")
first, and C1's put("x") second, so that memcached ends up caching
"x". Now memcached is caching a value that differs from the one in the
database, which is a bad situation.

This problem doesn't arise if C1 and C2 delete() instead of put().

Q: What is McRouter?

A: The point of mcrouter is to aggregate memcached RPCs from many
clients and send them in big batches to memcached servers. It's more
efficient to have a smallish number of mcrouter servers talk to
memcached than a large number of individual clients. One reason is
that there's overhead to each network (TCP) connection; better that
each memcached have a TCP connection per mcrouter than per client.
Another reason is that there's overhead (packet header space and
interrupt) for each packet, so it's helpful that a mcrouter can pack
many client requests into each TCP packet.

Q: Isn't it wasteful that the gutter servers are idle when they aren't
taking over for a failed server? Why not use the gutter servers for
ordinary memcached service as well as gutter?

A: I think non-gutter memcached servers are often close to fully
loaded, and have little spare capacity. If one fails, the replacement
server needs to have been more or less idle, in order to handle the
failed server's load.

Q: How do Section 4.2's regional pools reduce the number of replicas?

A: Each region has multiple clusters. Each cluster has a complete cache.
Thus a given data item may be cached in each of the clusters. If there
are N clusters in a region, there may be N distinct cached copies of a
data item, one per cluster.

Items that are cached in the regional pool are only cached once per
region, not N times.

The tradeoff is that the potential serving capacity is N times higher if
there are N copies.

Q: What storage system work has gone on at Facebook since this paper?

A: Here's a sample:

https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf
https://www.cs.princeton.edu/~wlloyd/papers/existential-sosp15.pdf
http://www.cs.cmu.edu/~beckmann/publications/papers/2020.osdi.cachelib.pdf
https://www.usenix.org/system/files/fast21-pan.pdf

Q: Why not just put a cache into MySQL, where it can be better
integrated to provide good consistency?

A: It would be fantastic if someone could add a transparent cache to
MySQL that made it as fast as a cluster of memcached servers. But
no-one has done that; it may not be possible. MySQL does cache, and
it's still much slower than memcached. Presumably a lot of the reason
is that MySQL presents a much more powerful and complex interface than
memcached (MySQL supports SQL queries, an interface which is about
1000x as complex as memcached's put()/get()/delete).

Q: Figure 11 shows that Memcache can serve data that is even a day
old. Although this happens with low probability, couldn't it
still cause significant, perhaps catastrophic, problems in
applications using Memcache?

A: Yes, indeed.  It is something FB has struggled with because it makes
writing applications more challenging. It is the topic of two
follow-on papers (see the references above).

The bottom-line of these papers is still roughly the same: the
probability of inconsistency is so low that they are willing to accept
it: even though a few users in principle might be able to notice the
inconsistency, they probably won't realize it or care.  (Their
target applications are not banking applications.)

Q: How does the MySQL replication system work?

A: See https://dev.mysql.com/doc/refman/8.0/en/replication.html. FB uses
the log-based replication scheme as a component of the
publish/subscribe system, as described in "Wormhole: Reliable Pub-Sub
to support Geo-replicated Internet Services", Sharma et al, 2015. The
core of the replication scheme is to read updates from MySQL's
transaction log and send those to the backup, which applies them to
its data.

Q: What does "look-aside" caching refer to?

A: The cache sits on the side as opposed in between the application
and the storage layer. If the application misses in the cache, the
application retrieves the database records and updates the cache,
instead of the cache doing it. This arrangement is relatively simple,
since the cache and database don't have to know about each other, and
the application is free to use different key schemes for the cache
versus the database.

Q: What is incast congestion?

A: A situation in which a computer receives many packets from
different sources at the same time, too many to process immediately,
and (in the worst case) more than it can buffer, leading to discarded
packets. The congestion and discarding can also occur inside the
network switch feeding this computer.

In the paper, this comes up when a web server needs many distinct
items of information from memcache (perhaps 100s or 1000s), which may
be stored on 100s of different memcached servers. For speed, the
client asks for many items in parallel. But that means it will receive
many replies in parallel, perhaps leading to incast congestion and
loss of replies. Section 3.1's window mechanism is intended to limit
the number of parallel requests, and thus the number of simultaneous
reply packets.

Q: What is the cold cluster warmup inconsistency which Section 4.3
describes?

A: Here's a scenario:

  0. key k starts out with value v1.
  1. client C1 updates k to v2 in the DB
  2. C1 and the DB send delete(k) to the memcache cold cluster
     but the DB is slow at sending delete(k) to the warm cluster
  3. client C2 sends get(k) to the cold cluster, which sends back a "miss"
  4. C2 sends get(k) to the warm cluster, receives v1
  5. C2 set(k, v1) into cold cluster
  6. the DB's delete(k) finally reaches the warm cluster

Now the cold memcache cluster holds the stale v1 value, but the delete()
has already happened. So the value will stay stale indefinitely, until
the key is next written.

The two-second hold-off scheme solves this. After C1 calls delete(k),
the cold cluster memcached ignores any set(k) for two seconds. By then,
the DB's delete(k) should have reached the warm cluster.

Q: How is the privacy of user data stored in memcached protected?

A: The paper does not touch on this question, so we can only guess.

The first line of defense is that only Facebook's own computers can
talk to their memcached and MySQL servers. Probably there are
firewalls between Facebook's datacenters and the Internet so that no
outsider can directly contact any of Facebook's internal servers.

At a higher level, Facebook's web servers have code that decides what
data to reveal to who. If you entrust sensitive information to
Facebook, you have to trust that their code has similar notions to
your own about who should see your information.

One also has to think about the possibility of bugs in Facebook's
permissions code; or bugs that allow outside hackers to break into
Facebook's computers; or corrupt or malicious or careless Facebook
employees. Dealing with such threats requires internal controls that
limit how data can be used even within Facebook.

USENIX Association  1 0th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13)  3 85
Scaling Memcache at Facebook
Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li,
Ryan McElroy , Mike Paleczny , Daniel Peek, Paul Saab, David Stafford, T ony T ung,
Venkateshwaran Venkataramani
{rajeshn,hans}@fb.com, {sgrimm, marc}@facebook.com, {herman, hcli, rm, mpal, dpeek, ps, dstaff, ttung, veeve }@fb.com
Facebook Inc.
Abstract: Memcached is a well known, simple, in-
memory caching solution. This paper describes how
Facebook leverages memcached as a building block to
construct and scale a distributed key-value store that
supports the world’s largest social network. Our system
handles billions of requests per second and holds tril-
lions of items to deliver a rich experience for over a bil-
lion users around the world.
1 Introduction
Popular and engaging social networking sites present
significant infrastructure challenges. Hundreds of mil-
lions of people use these networks every day and im-
pose computational, network, and I/O demands that tra-
ditional web architectures struggle to satisfy. A social
network’s infrastructure needs to (1) allow near real-
time communication, (2) aggregate content on-the-fly
from multiple sources, (3) be able to access and update
very popular shared content, and (4) scale to process
millions of user requests per second.
We describe how we improved the open source ver-
sion of memcached [14] and used it as a building block to
construct a distributed key-value store for the largest so-
cial network in the world. We discuss our journey scal-
ing from a single cluster of servers to multiple geograph-
ically distributed clusters. To the best of our knowledge,
this system is the largest memcached installation in the
world, processing over a billion requests per second and
storing trillions of items.
This paper is the latest in a series of works that have
recognized the flexibility and utility of distributed key-
value stores [1, 2, 5, 6, 12, 14, 34, 36]. This paper fo-
cuses on memcached—an open-source implementation
of an in-memory hash table—as it provides low latency
access to a shared storage pool at low cost. These quali-
ties enable us to build data-intensive features that would
otherwise be impractical. For example, a feature that
issues hundreds of database queries per page request
would likely never leave the prototype stage because it
would be too slow and expensive. In our application,
however, web pages routinely fetch thousands of key-
value pairs from memcached servers.
One of our goals is to present the important themes
that emerge at different scales of our deployment. While
qualities like performance, efficiency, fault-tolerance,
and consistency are important at all scales, our experi-
ence indicates that at specific sizes some qualities re-
quire more effort to achieve than others. For exam-
ple, maintaining data consistency can be easier at small
scales if replication is minimal compared to larger ones
where replication is often necessary. Additionally, the
importance of finding an optimal communication sched-
ule increases as the number of servers increase and net-
working becomes the bottleneck.
This paper includes four main contributions: (1)
We describe the evolution of Facebook’s memcached-
based architecture. (2) We identify enhancements to
memcached that improve performance and increase
memory efficiency. (3) We highlight mechanisms that
improve our ability to operate our system at scale. (4)
We characterize the production workloads imposed on
our system.
2 Overview
The following properties greatly influence our design.
First, users consume an order of magnitude more con-
tent than they create. This behavior results in a workload
dominated by fetching data and suggests that caching
can have significant advantages. Second, our read op-
erations fetch data from a variety of sources such as
MySQL databases, HDFS installations, and backend
services. This heterogeneity requires a flexible caching
strategy able to store data from disparate sources.
Memcached provides a simple set of operations (set,
get, and delete) that makes it attractive as an elemen-
tal component in a large-scale distributed system. The
open-source version we started with provides a single-
machine in-memory hash table. In this paper, we discuss
how we took this basic building block, made it more ef-
ficient, and used it to build a distributed key-value store
that can process billions of requests per second. Hence-

386 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
database
web
server
memcache
1. get k 2. SELECT ...
3. set (k,v)
database
web
server
memcache
2. delete k
1. UPDATE ...
Figure 1: Memcache as a demand-filled look-aside
cache. The left half illustrates the read path for a web
server on a cache miss. The right half illustrates the
write path.
forth, we use ‘memcached’ to refer to the source code
or a running binary and ‘memcache’ to describe the dis-
tributed system.
Query cache: We rely on memcache to lighten the read
load on our databases. In particular, we use memcache
as a demand-filled look-aside cache as shown in Fig-
ure 1. When a web server needs data, it first requests
the value from memcache by providing a string key. If
the item addressed by that key is not cached, the web
server retrieves the data from the database or other back-
end service and populates the cache with the key-value
pair. For write requests, the web server issues SQL state-
ments to the database and then sends a delete request to
memcache that invalidates any stale data. We choose to
delete cached data instead of updating it because deletes
are idempotent. Memcache is not the authoritative source
of the data and is therefore allowed to evict cached data.
While there are several ways to address excessive
read traffic on MySQL databases, we chose to use
memcache. It was the best choice given limited engi-
neering resources and time. Additionally, separating our
caching layer from our persistence layer allows us to ad-
just each layer independently as our workload changes.
Generic cache:We also leverage memcache as a more
general key-value store. For example, engineers use
memcache to store pre-computed results from sophisti-
cated machine learning algorithms which can then be
used by a variety of other applications. It takes little ef-
fort for new services to leverage the existing marcher
infrastructure without the burden of tuning, optimizing,
provisioning, and maintaining a large server fleet.
As is, memcached provides no server-to-server co-
ordination; it is an in-memory hash table running on
a single server. In the remainder of this paper we de-
scribe how we built a distributed key-value store based
on memcached capable of operating under Facebook’s
workload. Our system provides a suite of configu-
ration, aggregation, and routing services to organize
memcached instances into a distributed system.
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g7/g19/g18/g17/g21/g4/g6/g17/g12/g1
/g5/g16/g22/g20/g21/g13/g19/g20/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g4/g8/g6/g1/g3/g8/g11/g12/g8/g11/g1
/g2/g8/g10/g7/g5/g7/g9/g8/g1
/g10/g21/g18/g19/g11/g14/g13/g1/g5/g16/g22/g20/g21/g13/g19/g1/g2/g8/g11/g20/g21/g13/g19/g3/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g7/g19/g18/g17/g21/g4/g6/g17/g12/g1
/g5/g16/g22/g20/g21/g13/g19/g20/g1
/g1
/g1
/g1
/g1
/g1
/g1
/g4/g8/g6/g1/g3/g8/g11/g12/g8/g11/g1
/g2/g8/g10/g7/g5/g7/g9/g8/g1
/g10/g21/g18/g19/g11/g14/g13/g1/g5/g16/g22/g20/g21/g13/g19/g1/g2/g10/g16/g11/g23/g13/g3/g1
/g9/g13/g14/g15/g18/g17/g1/g2/g8/g11/g20/g21/g13/g19/g3/g1/g9/g13/g14/g15/g18/g17/g1/g2/g10/g16/g11/g23/g13/g3/g1
Figure 2: Overall architecture
We structure our paper to emphasize the themes that
emerge at three different deployment scales. Our read-
heavy workload and wide fan-out is the primary con-
cern when we have one cluster of servers. As it becomes
necessary to scale to multiple frontend clusters, we ad-
dress data replication between these clusters. Finally, we
describe mechanisms to provide a consistent user ex-
perience as we spread clusters around the world. Op-
erational complexity and fault tolerance is important at
all scales. We present salient data that supports our de-
sign decisions and refer the reader to work by Atikoglu
et al.[8] for a more detailed analysis of our workload. At
a high-level, Figure 2 illustrates this final architecture in
which we organize co-located clusters into a region and
designate a master region that provides a data stream to
keep non-master regions up-to-date.
While evolving our system we prioritize two ma-
jor design goals. (1) Any change must impact a user-
facing or operational issue. Optimizations that have lim-
ited scope are rarely considered. (2) We treat the prob-
ability of reading transient stale data as a parameter to
be tuned, similar to responsiveness. We are willing to
expose slightly stale data in exchange for insulating a
backend storage service from excessive load.
3 In a Cluster: Latency and Load
We now consider the challenges of scaling to thousands
of servers within a cluster. At this scale, most of our
efforts focus on reducing either the latency of fetching
cached data or the load imposed due to a cache miss.
3.1 Reducing Latency
Whether a request for data results in a cache hit or miss,
the latency of memcache’s response is a critical factor
in the response time of a user’s request. A single user
web request can often result in hundreds of individual

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 387
memcache get requests. For example, loading one of our
popular pages results in an average of 521 distinct items
fetched from memcache. 1
We provision hundreds of memcached servers in a
cluster to reduce load on databases and other services.
Items are distributed across the memcached servers
through consistent hashing [22]. Thus web servers have
to routinely communicate with many memcached servers
to satisfy a user request. As a result, all web servers
communicate with every memcached server in a short
period of time. This all-to-allcommunication pattern
can cause incast congestion [30] or allow a single server
to become the bottleneck for many web servers. Data
replication often alleviates the single-server bottleneck
but leads to significant memory inefficiencies in the
common case.
We reduce latency mainly by focusing on the
memcache client, which runs on each web server. This
client serves a range of functions, including serializa-
tion, compression, request routing, error handling, and
request batching. Clients maintain a map of all available
servers, which is updated through an auxiliary configu-
ration system.
Parallel requests and batching:We structure our web-
application code to minimize the number of network
round trips necessary to respond to page requests. We
construct a directed acyclic graph (DAG) representing
the dependencies between data. A web server uses this
DAG to maximize the number of items that can be
fetched concurrently. On average these batches consist
of 24 keys per request
2 .
Client-server communication:Memcached servers do
not communicate with each other. When appropriate,
we embed the complexity of the system into a stateless
client rather than in the memcached servers. This greatly
simplifies memcached and allows us to focus on making
it highly performant for a more limited use case. Keep-
ing the clients stateless enables rapid iteration in the
software and simplifies our deployment process. Client
logic is provided as two components: a library that can
be embedded into applications or as a standalone proxy
named mcrouter. This proxy presents a memcached
server interface and routes the requests/replies to/from
other servers.
Clients use UDP and TCP to communicate with
memcached servers. We rely on UDP for get requests to
reduce latency and overhead. Since UDP is connection-
less, each thread in the web server is allowed to directly
communicate with memcached servers directly, bypass-
ing mcrouter, without establishing and maintaining a
1 The 95 th percentile of fetches for that page is 1,740 items.
2 The 95 th percentile is 95 keys per request.
Average of Medians Average of 95th Percentiles
microseconds
0 200 600 1000 1400
UDP direct
by mcrouter (TCP)
Figure 3: Get latency for UDP , TCP via mcrouter
connection thereby reducing the overhead. The UDP
implementation detects packets that are dropped or re-
ceived out of order (using sequence numbers) and treats
them as errors on the client side. It does not provide
any mechanism to try to recover from them. In our in-
frastructure, we find this decision to be practical. Un-
der peak load, memcache clients observe that 0.25% of
get requests are discarded. About 80% of these drops
are due to late or dropped packets, while the remainder
are due to out of order delivery. Clients treat get er-
rors as cache misses, but web servers will skip insert-
ing entries into memcached after querying for data to
avoid putting additional load on a possibly overloaded
network or server.
For reliability, clients perform set and delete opera-
tions over TCP through an instance of mcrouter run-
ning on the same machine as the web server. For opera-
tions where we need to confirm a state change (updates
and deletes) TCP alleviates the need to add a retry mech-
anism to our UDP implementation.
Web servers rely on a high degree of parallelism and
over-subscription to achieve high throughput. The high
memory demands of open TCP connections makes it
prohibitively expensive to have an open connection be-
tween every web thread and memcached server without
some form of connection coalescing via mcrouter. Co-
alescing these connections improves the efficiency of
the server by reducing the network, CPU and memory
resources needed by high throughput TCP connections.
Figure 3 shows the average, median, and 95
th percentile
latencies of web servers in production getting keys over
UDP and through mcrouter via TCP . In all cases, the
standard deviation from these averages was less than
1%. As the data show, relying on UDP can lead to a
20% reduction in latency to serve requests.
Incast congestion:Memcache clients implement flow-
control mechanisms to limit incast congestion. When a

388 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
100 200 300 400 500
0 10 20 30 40
Window Size
milliseconds
95th Percentile
Median
Figure 4: Average time web requests spend waiting to
be scheduled
client requests a large number of keys, the responses
can overwhelm components such as rack and cluster
switches if those responses arrive all at once. Clients
therefore use a sliding window mechanism [11] to con-
trol the number of outstanding requests. When the client
receives a response, the next request can be sent. Similar
to TCP’s congestion control, the size of this sliding win-
dow grows slowly upon a successful request and shrinks
when a request goes unanswered. The window applies
to all memcache requests independently of destination;
whereas TCP windows apply only to a single stream.
Figure 4 shows the impact of the window size on the
amount of time user requests are in the runnable state
but are waiting to be scheduled inside the web server.
The data was gathered from multiple racks in one fron-
tend cluster. User requests exhibit a Poisson arrival pro-
cess at each web server. According to Little’s Law [26],
L = λW , the number of requests queued in the server
(L ) is directly proportional to the average time a request
takes to process ( W ), assuming that the input request
rate is constant (which it was for our experiment). The
time web requests are waiting to be scheduled is a di-
rect indication of the number of web requests in the
system. With lower window sizes, the application will
have to dispatch more groups of memcache requests se-
rially, increasing the duration of the web request. As the
window size gets too large, the number of simultaneous
memcache requests causes incast congestion. The result
will be memcache errors and the application falling back
to the persistent storage for the data, which will result
in slower processing of web requests. There is a balance
between these extremes where unnecessary latency can
be avoided and incast congestion can be minimized.
3.2 Reducing Load
We use memcache to reduce the frequency of fetch-
ing data along more expensive paths such as database
queries. Web servers fall back to these paths when the
desired data is not cached. The following subsections
describe three techniques for decreasing load.
3.2.1 Leases
We introduce a new mechanism we call leasesto address
two problems: stale sets and thundering herds. A stale
set occurs when a web server sets a value in memcache
that does not reflect the latest value that should be
cached. This can occur when concurrent updates to
memcache get reordered. A thundering herd happens
when a specific key undergoes heavy read and write ac-
tivity. As the write activity repeatedly invalidates the re-
cently set values, many reads default to the more costly
path. Our lease mechanism solves both problems.
Intuitively, a memcached instance gives a leaseto a
client to set data back into the cache when that client ex-
periences a cache miss. The lease is a 64-bit token bound
to the specific key the client originally requested. The
client provides the lease token when setting the value
in the cache. With the lease token, memcached can ver-
ify and determine whether the data should be stored and
thus arbitrate concurrent writes. V erification can fail if
memcached has invalidated the lease token due to re-
ceiving a delete request for that item. Leases prevent
stale sets in a manner similar to how load-link/store-
conditional operates [20].
A slight modification to leasesalso mitigates thunder-
ing herds. Each memcached server regulates the rate at
which it returns tokens. By default, we configure these
servers to return a token only once every 10 seconds per
key. Requests for a key’s value within 10 seconds of a
token being issued results in a special notification telling
the client to wait a short amount of time. Typically, the
client with the lease will have successfully set the data
within a few milliseconds. Thus, when waiting clients
retry the request, the data is often present in cache.
To illustrate this point we collect data for all cache
misses of a set of keys particularly susceptible to thun-
dering herds for one week. Without leases, all of the
cache misses resulted in a peak database query rate of
17K/s. With leases, the peak database query rate was
1.3K/s. Since we provision our databases based on peak
load, our lease mechanism translates to a significant ef-
ficiency gain.
Stale values: With leases, we can minimize the appli-
cation’s wait time in certain use cases. We can further
reduce this time by identifying situations in which re-
turning slightly out-of-date data is acceptable. When a
key is deleted, its value is transferred to a data struc-

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 389
Minimum, mean, and maximum
T erabytes 20
40
60
80
Daily Weekly
Low−churn
Daily Weekly
High−churn
Figure 5: Daily and weekly working set of a high-churn
family and a low-churn key family
ture that holds recently deleted items, where it lives for
a short time before being flushed. A get request can re-
turn a lease token or data that is marked as stale. Appli-
cations that can continue to make forward progress with
stale data do not need to wait for the latest value to be
fetched from the databases. Our experience has shown
that since the cached value tends to be a monotonically
increasing snapshot of the database, most applications
can use a stale value without any changes.
3.2.2 Memcache Pools
Using memcache as a general-purpose caching layer re-
quires workloads to share infrastructure despite differ-
ent access patterns, memory footprints, and quality-of-
service requirements. Different applications’ workloads
can produce negative interference resulting in decreased
hit rates.
To accommodate these differences, we partition a
cluster’s memcached servers into separate pools. We
designate one pool (named wildcard) as the default and
provision separate pools for keys whose residence in
wildcard is problematic. For example, we may provi-
sion a small pool for keys that are accessed frequently
but for which a cache miss is inexpensive. We may also
provision a large pool for infrequently accessed keys for
which cache misses are prohibitively expensive.
Figure 5 shows the working set of two different sets
of items, one that is low-churn and another that is high-
churn. The working set is approximated by sampling all
operations on one out of every one million items. For
each of these items, we collect the minimum, average,
and maximum item size. These sizes are summed and
multiplied by one million to approximate the working
set. The difference between the daily and weekly work-
ing sets indicates the amount of churn. Items with differ-
ent churn characteristics interact in an unfortunate way:
low-churn keys that are still valuable are evicted before
high-churn keys that are no longer being accessed. Plac-
ing these keys in different pools prevents this kind of
negative interference, and allows us to size high-churn
pools appropriate to their cache miss cost. Section 7 pro-
vides further analysis.
3.2.3 Replication Within Pools
Within some pools, we use replication to improve the la-
tency and efficiency of memcached servers. We choose
to replicate a category of keys within a pool when (1)
the application routinely fetches many keys simultane-
ously, (2) the entire data set fits in one or two memcached
servers and (3) the request rate is much higher than what
a single server can manage.
We favor replication in this instance over further di-
viding the key space. Consider a memcached server
holding 100 items and capable of responding to 500k
requests per second. Each request asks for 100 keys.
The difference in memcached overhead for retrieving
100 keys per request instead of 1 key is small. To scale
the system to process 1M requests/sec, suppose that we
add a second server and split the key space equally be-
tween the two. Clients now need to split each request for
100 keys into two parallel requests for ∼50 keys. Con-
sequently, both servers still have to process 1M requests
per second. However, if we replicate all 100 keys to mul-
tiple servers, a client’s request for 100 keys can be sent
to any replica. This reduces the load per server to 500k
requests per second. Each client chooses replicas based
on its own IP address. This approach requires delivering
invalidations to all replicas to maintain consistency.
3.3 Handling Failures
The inability to fetch data from memcache results in ex-
cessive load to backend services that could cause fur-
ther cascading failures. There are two scales at which
we must address failures: (1) a small number of hosts
are inaccessible due to a network or server failure or (2)
a widespread outage that affects a significant percent-
age of the servers within the cluster. If an entire clus-
ter has to be taken offline, we divert user web requests
to other clusters which effectively removes all the load
from memcache within that cluster.
For small outages we rely on an automated remedi-
ation system [3]. These actions are not instant and can
take up to a few minutes. This duration is long enough to
cause the aforementioned cascading failures and thus we
introduce a mechanism to further insulate backend ser-
vices from failures. We dedicate a small set of machines,
named Gutter, to take over the responsibilities of a few
failed servers. Gutter accounts for approximately 1% of
the memcached servers in a cluster.
When a memcached client receives no response to its
get request, the client assumes the server has failed and
issues the request again to a special Gutter pool. If this
second request misses, the client will insert the appropri-
ate key-value pair into the Gutter machine after querying
the database. Entries in Gutter expire quickly to obviate

390 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
Gutter invalidations. Gutter limits the load on backend
services at the cost of slightly stale data.
Note that this design differs from an approach in
which a client rehashes keys among the remaining
memcached servers. Such an approach risks cascading
failures due to non-uniform key access frequency. For
example, a single key can account for 20% of a server’s
requests. The server that becomes responsible for this
hot key might also become overloaded. By shunting load
to idle servers we limit that risk.
Ordinarily, each failed request results in a hit on the
backing store, potentially overloading it. By using Gut-
ter to store these results, a substantial fraction of these
failures are converted into hits in the gutter pool thereby
reducing load on the backing store. In practice, this sys-
tem reduces the rate of client-visible failures by 99%
and converts 10%–25% of failures into hits each day. If
a memcached server fails entirely, hit rates in the gutter
pool generally exceed 35% in under 4 minutes and often
approach 50%. Thus when a few memcached servers are
unavailable due to failure or minor network incidents,
Gutter protects the backing store from a surge of traffic.
4 In a Region: Replication
It is tempting to buy more web and memcached servers
to scale a cluster as demand increases. However, na ¨ıvely
scaling the system does not eliminate all problems.
Highly requested items will only become more popular
as more web servers are added to cope with increased
user traffic. Incast congestion also worsens as the num-
ber of memcached servers increases. We therefore split
our web and memcached servers into multiple frontend
clusters. These clusters, along with a storage cluster that
contain the databases, define a region. This region ar-
chitecture also allows for smaller failure domains and
a tractable network configuration. We trade replication
of data for more independent failure domains, tractable
network configuration, and a reduction of incast conges-
tion.
This section analyzes the impact of multiple frontend
clusters that share the same storage cluster. Specifically
we address the consequences of allowing data replica-
tion across these clusters and the potential memory effi-
ciencies of disallowing this replication.
4.1 Regional Invalidations
While the storage cluster in a region holds the authori-
tative copy of data, user demand may replicate that data
into frontend clusters. The storage cluster is responsi-
ble for invalidating cached data to keep frontend clus-
ters consistent with the authoritative versions. As an op-
timization, a web server that modifies data also sends
invalidations to its own cluster to provide read-after-
Memcache
Mcrouter
Update
Operations Storage
MySQL McSqueal
Commit Log
Storage Server
Figure 6: Invalidation pipeline showing keys that need
to be deleted via the daemon (mcsqueal).
write semantics for a single user request and reduce the
amount of time stale data is present in its local cache.
SQL statements that modify authoritative state are
amended to include memcache keys that need to be
invalidated once the transaction commits [7]. We de-
ploy invalidation daemons (named mcsqueal) on every
database. Each daemon inspects the SQL statements that
its database commits, extracts any deletes, and broad-
casts these deletes to the memcache deployment in every
frontend cluster in that region. Figure 6 illustrates this
approach. We recognize that most invalidations do not
delete data; indeed, only 4% of all deletes issued result
in the actual invalidation of cached data.
Reducing packet rates: While mcsqueal could con-
tact memcached servers directly, the resulting rate of
packets sent from a backend cluster to frontend clus-
ters would be unacceptably high. This packet rate prob-
lem is a consequence of having many databases and
many memcached servers communicating across a clus-
ter boundary. Invalidation daemons batch deletes into
fewer packets and send them to a set of dedicated servers
running mcrouter instances in each frontend cluster.
These mcrouters then unpack individual deletes from
each batch and route those invalidations to the right
memcached server co-located within the frontend clus-
ter. The batching results in an 18× improvement in the
median number of deletes per packet.
Invalidation via web servers: It is simpler for web
servers to broadcast invalidations to all frontend clus-
ters. This approach unfortunately suffers from two prob-
lems. First, it incurs more packet overhead as web
servers are less effective at batching invalidations than
mcsqueal pipeline. Second, it provides little recourse
when a systemic invalidation problem arises such as
misrouting of deletes due to a configuration error. In the
past, this would often require a rolling restart of the en-
tire memcache infrastructure, a slow and disruptive pro-

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 391
A (Cluster) B (Region)
Median number of users 30 1
Gets per second 3.26 M 458 K
Median value size 10.7 kB 4.34 kB
Table 1: Deciding factors for cluster or regional replica-
tion of two item families
cess we want to avoid. In contrast, embedding invalida-
tions in SQL statements, which databases commit and
store in reliable logs, allows mcsqueal to simply replay
invalidations that may have been lost or misrouted.
4.2 Regional Pools
Each cluster independently caches data depending on
the mix of the user requests that are sent to it. If
users’ requests are randomly routed to all available fron-
tend clusters then the cached data will be roughly the
same across all the frontend clusters. This allows us to
take a cluster offline for maintenance without suffer-
ing from reduced hit rates. Over-replicating the data can
be memory inefficient, especially for large, rarely ac-
cessed items. We can reduce the number of replicas by
having multiple frontend clusters share the same set of
memcached servers. We call this a regional pool.
Crossing cluster boundaries incurs more latency. In
addition, our networks have 40% less average available
bandwidth over cluster boundaries than within a single
cluster. Replication trades more memcached servers for
less inter-cluster bandwidth, lower latency, and better
fault tolerance. For some data, it is more cost efficient
to forgo the advantages of replicating data and have a
single copy per region. One of the main challenges of
scaling memcache within a region is deciding whether
a key needs to be replicated across all frontend clusters
or have a single replica per region. Gutter is also used
when servers in regional pools fail.
Table 1 summarizes two kinds of items in our appli-
cation that have large values. We have moved one kind
(B) to a regional pool while leaving the other (A) un-
touched. Notice that clients access items falling into cat-
egory B an order of magnitude less than those in cate-
gory A. Category B’s low access rate makes it a prime
candidate for a regional pool since it does not adversely
impact inter-cluster bandwidth. Category B would also
occupy 25% of each cluster’s wildcard pool so region-
alization provides significant storage efficiencies. Items
in category A, however, are twice as large and accessed
much more frequently, disqualifying themselves from
regional consideration. The decision to migrate data into
regional pools is currently based on a set of manual
heuristics based on access rates, data set size, and num-
ber of unique users accessing particular items.
4.3 Cold Cluster Warmup
When we bring a new cluster online, an existing one
fails, or perform scheduled maintenance the caches will
have very poor hit rates diminishing the ability to in-
sulate backend services. A system called Cold Clus-
ter Warmup mitigates this by allowing clients in the
“cold cluster” (i.e. the frontend cluster that has an empty
cache) to retrieve data from the “warm cluster” (i.e. a
cluster that has caches with normal hit rates) rather than
the persistent storage. This takes advantage of the afore-
mentioned data replication that happens across frontend
clusters. With this system cold clusters can be brought
back to full capacity in a few hours instead of a few days.
Care must be taken to avoid inconsistencies due to
race conditions. For example, if a client in the cold clus-
ter does a database update, and a subsequent request
from another client retrieves the stale value from the
warm cluster before the warm cluster has received the
invalidation, that item will be indefinitely inconsistent
in the cold cluster. Memcached deletes support nonzero
hold-off times that reject add operations for the spec-
ified hold-off time. By default, all deletes to the cold
cluster are issued with a two second hold-off. When a
miss is detected in the cold cluster, the client re-requests
the key from the warm cluster and adds it into the cold
cluster. The failure of the add indicates that newer data
is available on the database and thus the client will re-
fetch the value from the databases. While there is still a
theoretical possibility that deletes get delayed more than
two seconds, this is not true for the vast majority of the
cases. The operational benefits of cold cluster warmup
far outweigh the cost of rare cache consistency issues.
We turn it off once the cold cluster’s hit rate stabilizes
and the benefits diminish.
5 Across Regions: Consistency
There are several advantages to a broader geographic
placement of data centers. First, putting web servers
closer to end users can significantly reduce latency.
Second, geographic diversity can mitigate the effects
of events such as natural disasters or massive power
failures. And third, new locations can provide cheaper
power and other economic incentives. We obtain these
advantages by deploying to multiple regions. Each re-
gion consists of a storage cluster and several frontend
clusters. We designate one region to hold the master
databases and the other regions to contain read-only
replicas; we rely on MySQL’s replication mechanism
to keep replica databases up-to-date with their mas-
ters. In this design, web servers experience low latency
when accessing either the local memcached servers or
the local database replicas. When scaling across mul-

392 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
tiple regions, maintaining consistency between data in
memcache and the persistent storage becomes the pri-
mary technical challenge. These challenges stem from
a single problem: replica databases may lag behind the
master database.
Our system represents just one point in the wide
spectrum of consistency and performance trade-offs.
The consistency model, like the rest of the system, has
evolved over the years to suit the scale of the site. It
mixes what can be practically built without sacrificing
our high performance requirements. The large volume
of data that the system manages implies that any minor
changes that increase network or storage requirements
have non-trivial costs associated with them. Most ideas
that provide stricter semantics rarely leave the design
phase because they become prohibitively expensive. Un-
like many systems that are tailored to an existing use
case, memcache and Facebook were developed together.
This allowed the applications and systems engineers to
work together to find a model that is sufficiently easy
for the application engineers to understand yet perfor-
mant and simple enough for it to work reliably at scale.
We provide best-effort eventual consistency but place an
emphasis on performance and availability. Thus the sys-
tem works very well for us in practice and we think we
have found an acceptable trade-off.
Writes from a master region: Our earlier decision re-
quiring the storage cluster to invalidate data via daemons
has important consequences in a multi-region architec-
ture. In particular, it avoids a race condition in which
an invalidation arrives before the data has been repli-
cated from the master region. Consider a web server in
the master region that has finished modifying a database
and seeks to invalidate now stale data. Sending invalida-
tions within the master region is safe. However, having
the web server invalidate data in a replica region may be
premature as the changes may not have been propagated
to the replica databases yet. Subsequent queries for the
data from the replica region will race with the replica-
tion stream thereby increasing the probability of setting
stale data into memcache. Historically, we implemented
mcsqueal after scaling to multiple regions.
Writes from a non-master region: Now consider a
user who updates his data from a non-master region
when replication lag is excessively large. The user’s next
request could result in confusion if his recent change is
missing. A cache refill from a replica’s database should
only be allowed after the replication stream has caught
up. Without this, subsequent requests could result in the
replica’s stale data being fetched and cached.
We employ a remote marker mechanism to minimize
the probability of reading stale data. The presence of the
marker indicates that data in the local replica database
are potentially stale and the query should be redirected
to the master region. When a web server wishes to up-
date data that affects a key k, that server (1) sets a re-
mote marker r
k in the region, (2) performs the write to
the master embedding k and rk to be invalidated in the
SQL statement, and (3) deletes k in the local cluster. On
a subsequent request for k, a web server will be unable
to find the cached data, check whether rk exists, and di-
rect its query to the master or local region depending on
the presence of r
k . In this situation, we explicitly trade
additional latency when there is a cache miss, for a de-
creased probability of reading stale data.
We implement remote markers by using a regional
pool. Note that this mechanism may reveal stale in-
formation during concurrent modifications to the same
key as one operation may delete a remote marker that
should remain present for another in-flight operation. It
is worth highlighting that our usage of memcache for re-
mote markers departs in a subtle way from caching re-
sults. As a cache, deleting or evicting keys is always a
safe action; it may induce more load on databases, but
does not impair consistency. In contrast, the presence of
a remote marker helps distinguish whether a non-master
database holds stale data or not. In practice, we find both
the eviction of remote markers and situations of concur-
rent modification to be rare.
Operational considerations:Inter-region communica-
tion is expensive since data has to traverse large geo-
graphical distances (e.g. across the continental United
States). By sharing the same channel of communication
for the delete stream as the database replication we gain
network efficiency on lower bandwidth connections.
The aforementioned system for managing deletes in
Section 4.1 is also deployed with the replica databases to
broadcast the deletes to memcached servers in the replica
regions. Databases and mcrouters buffer deletes when
downstream components become unresponsive. A fail-
ure or delay in any of the components results in an in-
creased probability of reading stale data. The buffered
deletes are replayed once these downstream components
are available again. The alternatives involve taking a
cluster offline or over-invalidating data in frontend clus-
ters when a problem is detected. These approaches result
in more disruptions than benefits given our workload.
6 Single Server Improvements
The all-to-allcommunication pattern implies that a sin-
gle server can become a bottleneck for a cluster. This
section describes performance optimizations and mem-
ory efficiency gains in memcached which allow better

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 393
scaling within clusters. Improving single server cache
performance is an active research area [9, 10, 28, 25].
6.1 Performance Optimizations
We began with a single-threaded memcached which used
a fixed-size hash table. The first major optimizations
were to: (1) allow automatic expansion of the hash ta-
ble to avoid look-up times drifting to O(n), (2) make the
server multi-threaded using a global lock to protect mul-
tiple data structures, and (3) giving each thread its own
UDP port to reduce contention when sending replies and
later spreading interrupt processing overhead. The first
two optimizations were contributed back to the open
source community. The remainder of this section ex-
plores further optimizations that are not yet available in
the open source version.
Our experimental hosts have an Intel Xeon
CPU (X5650) running at 2.67GHz (12 cores and
12 hyperthreads), an Intel 82574L gigabit ethernet
controller and 12GB of memory. Production servers
have additional memory. Further details have been
previously published [4]. The performance test setup
consists of fifteen clients generating memcache traffic
to a single memcached server with 24 threads. The
clients and server are co-located on the same rack and
connected through gigabit ethernet. These tests measure
the latency of memcached responses over two minutes
of sustained load.
Get Performance: We first investigate the effect of re-
placing our original multi-threaded single-lock imple-
mentation with fine-grained locking. We measured hits
by pre-populating the cache with 32-byte values before
issuing memcached requests of 10 keys each. Figure 7
shows the maximum request rates that can be sustained
with sub-millisecond average response times for differ-
ent versions of memcached. The first set of bars is our
memcached before fine-grained locking, the second set
is our current memcached, and the final set is the open
source version 1.4.10 which independently implements
a coarser version of our locking strategy.
Employing fine-grained locking triples the peak get
rate for hits from 600k to 1.8M items per second. Per-
formance for misses also increased from 2.7M to 4.5M
items per second. Hits are more expensive because the
return value has to be constructed and transmitted, while
misses require a single static response (END) for the en-
tire multiget indicating that all keys missed.
We also investigated the performance effects of us-
ing UDP instead of TCP . Figure 8 shows the peak re-
quest rate we can sustain with average latencies of less
than one millisecond for single gets and multigets of 10
keys. We found that our UDP implementation outper-
Facebook Facebook−μlocks 1.4.10
Max sustained
items / second
hits
misses
0 2M 4M 6M
Figure 7: Multiget hit and miss performance comparison
by memcached version
Get 10−key multiget
Max sustained
items / second
TCP
UDP
0 1M 2M
Figure 8: Get hit performance comparison for single
gets and 10-key multigets over TCP and UDP
forms our TCP implementation by 13% for single gets
and 8% for 10-key multigets.
Because multigets pack more data into each request
than single gets, they use fewer packets to do the same
work. Figure 8 shows an approximately four-fold im-
provement for 10-key multigets over single gets.
6.2 Adaptive Slab Allocator
Memcached employs a slab allocator to manage memory.
The allocator organizes memory into slab classes, each
of which contains pre-allocated, uniformly sized chunks
of memory. Memcached stores items in the smallest pos-
sible slab class that can fit the item’s metadata, key, and
value. Slab classes start at 64 bytes and exponentially in-
crease in size by a factor of 1.07 up to 1 MB, aligned on
4-byte boundaries
3 . Each slab class maintains a free-list
of available chunks and requests more memory in 1MB
slabs when its free-list is empty. Once a memcached
server can no longer allocate free memory, storage for
new items is done by evicting the least recently used
(LRU) item within that slab class. When workloads
change, the original memory allocated to each slab class
may no longer be enough resulting in poor hit rates.
3 This scaling factor ensures that we have both 64 and 128 byte
items which are more amenable to hardware cache lines.

394 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
We implemented an adaptive allocator that period-
ically re-balances slab assignments to match the cur-
rent workload. It identifies slab classes as needing more
memory if they are currently evicting items and if the
next item to be evicted was used at least 20% more re-
cently than the average of the least recently used items in
other slab classes. If such a class is found, then the slab
holding the least recently used item is freed and trans-
ferred to the needy class. Note that the open-source com-
munity has independently implemented a similar allo-
cator that balances the eviction rates across slab classes
while our algorithm focuses on balancing the age of the
oldest items among classes. Balancing age provides a
better approximation to a single global Least Recently
Used (LRU) eviction policy for the entire server rather
than adjusting eviction rates which can be heavily influ-
enced by access patterns.
6.3 The Transient Item Cache
While memcached supports expiration times, entries
may live in memory well after they have expired.
Memcached lazily evicts such entries by checking ex-
piration times when serving a get request for that item
or when they reach the end of the LRU. Although effi-
cient for the common case, this scheme allows short-
lived keys that see a single burst of activity to waste
memory until they reach the end of the LRU.
We therefore introduce a hybrid scheme that relies on
lazy eviction for most keys and proactively evicts short-
lived keys when they expire. We place short-lived items
into a circular buffer of linked lists (indexed by sec-
onds until expiration) – called the Transient Item Cache
– based on the expiration time of the item. Every sec-
ond, all of the items in the bucket at the head of the
buffer are evicted and the head advances by one. When
we added a short expiration time to a heavily used set of
keys whose items have short useful lifespans; the pro-
portion of memcache pool used by this key family was
reduced from 6% to 0.3% without affecting the hit rate.
6.4 Software Upgrades
Frequent software changes may be needed for upgrades,
bug fixes, temporary diagnostics, or performance test-
ing. A memcached server can reach 90% of its peak hit
rate within a few hours. Consequently, it can take us over
12 hours to upgrade a set of memcached servers as the re-
sulting database load needs to be managed carefully. We
modified memcached to store its cached values and main
data structures in System V shared memory regions so
that the data can remain live across a software upgrade
and thereby minimize disruption.
distinct memcached servers
percentile of requests
20 100 200 300 400 500 600
0 20 40 60 80 100
All requests
A popular data intensive page
Figure 9: Cumulative distribution of the number of dis-
tinct memcached servers accessed
7 Memcache Workload
We now characterize the memcache workload using data
from servers that are running in production.
7.1 Measurements at the Web Server
We record all memcache operations for a small percent-
age of user requests and discuss the fan-out, response
size, and latency characteristics of our workload.
Fanout: Figure 9 shows the distribution of distinct
memcached servers a web server may need to contact
when responding to a page request. As shown, 56%
of all page requests contact fewer than 20 memcached
servers. By volume, user requests tend to ask for small
amounts of cached data. There is, however, a long tail to
this distribution. The figure also depicts the distribution
for one of our more popular pages that better exhibits
the all-to-all communication pattern. Most requests of
this type will access over 100 distinct servers; accessing
several hundred memcached servers is not rare.
Response size:Figure 10 shows the response sizes from
memcache requests. The difference between the median
(135 bytes) and the mean (954 bytes) implies that there
is a very large variation in the sizes of the cached items.
In addition there appear to be three distinct peaks at ap-
proximately 200 bytes and 600 bytes. Larger items tend
to store lists of data while smaller items tend to store
single pieces of content.
Latency: We measure the round-trip latency to request
data from memcache, which includes the cost of rout-
ing the request and receiving the reply, network transfer
time, and the cost of deserialization and decompression.
Over 7 days the median request latency is 333 microsec-
onds while the 75
th and 95 th percentiles (p75 and p95)
are 475μs and 1.135ms respectively. Our median end-
to-end latency from an idle web server is 178μs while
the p75 and p95 are 219μs and 374μs, respectively. The

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 395
Bytes
percentile of requests
0 200 400 600
0 20 40 60 80 100
Figure 10: Cumulative distribution of value sizes
fetched
wide variance between the p95 latencies arises from
handling large responses and waiting for the runnable
thread to be scheduled as discussed in Section 3.1.
7.2 Pool Statistics
We now discuss key metrics of four memcache pools.
The pools are wildcard (the default pool), app (a pool
devoted for a specific application), a replicated pool for
frequently accessed data, and a regional pool for rarely
accessed information. In each pool, we collect average
statistics every 4 minutes and report in Table 2 the high-
est average for one month collection period. This data
approximates the peak load seen by those pools. The ta-
ble shows the widely different get, set, and delete rates
for different pools. Table 3 shows the distribution of re-
sponse sizes for each pool. Again, the different char-
acteristics motivate our desire to segregate these work-
loads from one another.
As discussed in Section 3.2.3, we replicate data
within a pool and take advantage of batching to handle
the high request rates. Observe that the replicated pool
has the highest get rate (about 2.7 × that of the next high-
est one) and the highest ratio of bytes to packets despite
having the smallest item sizes. This data is consistent
with our design in which we leverage replication and
batching to achieve better performance. In the app pool,
a higher churn of data results in a naturally higher miss
rate. This pool tends to have content that is accessed for
a few hours and then fades away in popularity in favor
of newer content. Data in the regional pool tends to be
large and infrequently accessed as shown by the request
rates and the value size distribution.
7.3 Invalidation Latency
We recognize that the timeliness of invalidations is a
critical factor in determining the probability of expos-
ing stale data. To monitor this health, we sample one out
master region replica region
seconds of delay
fraction of deletes that failed
1s 10s 1m 10m 1h 1d 1s 10s 1m 10m 1h 1d
1e−06 1e−05 1e−04 1e−03
Figure 11: Latency of the Delete Pipeline
of a million deletes and record the time the delete was is-
sued. We subsequently query the contents of memcache
across all frontend clusters at regular intervals for the
sampled keys and log an error if an item remains cached
despite a delete that should have invalidated it.
In Figure 11, we use this monitoring mechanism to re-
port our invalidation latencies across a 30 day span. We
break this data into two different components: (1) the
delete originated from a web server in the master region
and was destined to a memcached server in the master re-
gion and (2) the delete originated from a replica region
and was destined to another replica region. As the data
show, when the source and destination of the delete are
co-located with the master our success rates are much
higher and achieve four 9s of reliability within 1 second
and five 9s after one hour. However when the deletes
originate and head to locations outside of the master re-
gion our reliability drops to three 9s within a second and
four 9s within 10 minutes. In our experience, we find
that if an invalidation is missing after only a few sec-
onds the most common reason is that the first attempt
failed and subsequent retrials will resolve the problem.
8 Related Work
Several other large websites have recognized the util-
ity of key-value stores. DeCandia et al. [12] present
a highly available key-value store that is used by a
variety of application services at Amazon.com. While
their system is optimized for a write heavy workload,
ours targets a workload dominated by reads. Similarly,
LinkedIn uses V oldemort [5], a system inspired by Dy-
namo. Other major deployments of key-value caching
solutions include Redis [6] at Github, Digg, and Bliz-
zard, and memcached at Twitter [33] and Zynga. Lak-
shman et al. [1] developed Cassandra, a schema-based
distributed key-value store. We preferred to deploy and
scale memcached due to its simpler design.
Our work in scaling memcache builds on extensive
work in distributed data structures. Gribble et al. [19]

396 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
pool miss rate get
s
set
s
delete
s
packets
s outbound
bandwidth (MB/s)
wildcard 1.76% 262k 8.26k 21.2k 236k 57.4
app 7.85% 96.5k 11.9k 6.28k 83.0k 31.0
replicated 0.053% 710k 1.75k 3.22k 44.5k 30.1
regional 6.35% 9.1k 0.79k 35.9k 47.2k 10.8
Table 2: Traffic per server on selected memcache pools averaged over 7 days
pool mean std dev p5 p25 p50 p75 p95 p99
wildcard 1.11 K 8.28 K 77 102 169 363 3.65 K 18.3 K
app 881 7.70 K 103 247 269 337 1.68K 10.4 K
replicated 66 2 62 68 68 68 68 68
regional 31.8 K 75.4 K 231 824 5.31 K 24.0 K 158 K 381 K
Table 3: Distribution of item sizes for various pools in bytes
present an early version of a key-value storage system
useful for Internet scale services. Ousterhout et al.[29]
also present the case for a large scale in-memory key-
value storage system. Unlike both of these solutions,
memcache does not guarantee persistence. We rely on
other systems to handle persistent data storage.
Ports et al. [31] provide a library to manage the
cached results of queries to a transactional database.
Our needs require a more flexible caching strategy. Our
use of leases [18] and stale reads [23] leverages prior
research on cache consistency and read operations in
high-performance systems. Work by Ghandeharizadeh
and Y ap [15] also presents an algorithm that addresses
the stale set problem based on time-stamps rather than
explicit version numbers.
While software routers are easier to customize and
program, they are often less performant than their hard-
ware counterparts. Dobrescu et al. [13] address these
issues by taking advantage of multiple cores, multiple
memory controllers, multi-queue networking interfaces,
and batch processing on general purpose servers. Ap-
plying these techniques to mcrouter’s implementation
remains future work. Twitter has also independently de-
veloped a memcache proxy similar to mcrouter [32].
In Coda [35], Satyanarayanan et al.demonstrate how
datasets that diverge due to disconnected operation can
be brought back into sync. Glendenning et al.[17] lever-
age Paxos [24] and quorums [16] to build Scatter, a dis-
tributed hash table with linearizable semantics [21] re-
silient to churn. Lloyd et al.[27] examine causal consis-
tency in COPS, a wide-area storage system.
TAO [37] is another Facebook system that relies heav-
ily on caching to serve large numbers of low-latency
queries. TAO differs from memcache in two fundamental
ways. (1) TAO implements a graph data model in which
nodes are identified by fixed-length persistent identifiers
(64-bit integers). (2) TAO encodes a specific mapping of
its graph model to persistent storage and takes respon-
sibility for persistence. Many components, such as our
client libraries and mcrouter, are used by both systems.
9 Conclusion
In this paper, we show how to scale a memcached-based
architecture to meet the growing demand of Facebook.
Many of the trade-offs discussed are not fundamental,
but are rooted in the realities of balancing engineering
resources while evolving a live system under continu-
ous product development. While building, maintaining,
and evolving our system we have learned the following
lessons. (1) Separating cache and persistent storage sys-
tems allows us to independently scale them. (2) Features
that improve monitoring, debugging and operational ef-
ficiency are as important as performance. (3) Managing
stateful components is operationally more complex than
stateless ones. As a result keeping logic in a stateless
client helps iterate on features and minimize disruption.
(4) The system must support gradual rollout and roll-
back of new features even if it leads to temporary het-
erogeneity of feature sets. (5) Simplicity is vital.
Acknowledgements
We would like to thank Philippe Ajoux, Nathan Bron-
son, Mark Drayton, David Fetterman, Alex Gartrell, An-
drii Grynenko, Robert Johnson, Sanjeev Kumar, Anton
Likhtarov, Mark Marchukov, Scott Marlette, Ben Mau-
rer, David Meisner, Konrad Michels, Andrew Pope, Jeff
Rothschild, Jason Sobel, and Y ee Jiun Song for their
contributions. We would also like to thank the anony-
mous reviewers, our shepherd Michael Piatek, Tor M.
Aamodt, Remzi H. Arpaci-Dusseau, and Tayler Hether-
ington for their valuable feedback on earlier drafts of
the paper. Finally we would like to thank our fellow en-
gineers at Facebook for their suggestions, bug-reports,
and support which makes memcache what it is today.

USENIX Association  10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) 397
References
[1] Apache Cassandra. http://cassandra.apache.org/.
[2] Couchbase. http://www.couchbase.com/.
[3] Making Facebook Self-Healing. https://www.facebook.
com/note.php?note_id=10150275248698920.
[4] Open Compute Project. http://www.opencompute.org.
[5] Project V oldemort. http://project-voldemort.com/.
[6] Redis. http://redis.io/.
[7] Scaling Out. https://www.facebook.com/note.php?note_
id=23844338919.
[8] A TIKOGLU , B., X U ,Y . ,F RACHTENBERG , E., J IANG , S., AND
P ALECZNY , M. Workload analysis of a large-scale key-value
store. ACM SIGMETRICS Performance Evaluation Review 40 ,
1 (June 2012), 53–64.
[9] B EREZECKI , M., F RACHTENBERG , E., P ALECZNY , M., AND
S TEELE , K. Power and performance evaluation of memcached
on the tilepro64 architecture. Sustainable Computing: Informat-
ics and Systems 2, 2 (June 2012), 81 – 90.
[10] B OYD -W ICKIZER , S., C LEMENTS , A. T., M AO , Y. ,
P ESTEREV , A., K AASHOEK ,M .F . ,M ORRIS , R., AND
Z ELDOVICH , N. An analysis of linux scalability to many cores.
In Proceedings of the 9th USENIX Symposium on Operating
Systems Design & Implementation (2010), pp. 1–8.
[11] C ERF , V . G., AND K AHN , R. E. A protocol for packet network
intercommunication. ACM SIGCOMM Compututer Communi-
cation Review 35, 2 (Apr. 2005), 71–82.
[12] D E C ANDIA , G., H ASTORUN , D., J AMPANI , M., K AKULAP -
AT I , G., L AKSHMAN , A., P ILCHIN , A., S IV ASUBRAMANIAN ,
S., V OSSHALL , P. , AND V OGELS , W. Dynamo: amazon’s
highly available key-value store. ACM SIGOPS Operating Sys-
tems Review 41, 6 (Dec. 2007), 205–220.
[13] F ALL , K., I ANNACCONE , G., M ANESH , M., R A TNASAMY , S.,
A RGYRAKI , K., D OBRESCU , M., AND E GI , N. Routebricks:
enabling general purpose network infrastructure. ACM SIGOPS
Operating Systems Review 45, 1 (Feb. 2011), 112–125.
[14] F ITZPA TRICK , B. Distributed caching with memcached. Linux
Journal 2004, 124 (Aug. 2004), 5.
[15] G HANDEHARIZADEH , S., AND Y AP , J. Gumball: a race con-
dition prevention technique for cache augmented sql database
management systems. In Proceedings of the 2nd ACM SIGMOD
Workshop on Databases and Social Networks (2012), pp. 1–6.
[16] G IFFORD , D. K. Weighted voting for replicated data. In Pro-
ceedings of the 7th ACM Symposium on Operating Systems Prin-
ciples (1979), pp. 150–162.
[17] G
LENDENNING , L., B ESCHASTNIKH , I., K RISHNAMURTHY ,
A., AND A NDERSON , T. Scalable consistency in Scatter. In
Proceedings of the 23rd ACM Symposium on Operating Systems
Principles (2011), pp. 15–28.
[18] G
R AY , C., AND C HERITON , D. Leases: An efficient fault-
tolerant mechanism for distributed file cache consistency. ACM
SIGOPS Operating Systems Review 23, 5 (Nov. 1989), 202–210.
[19] G RIBBLE , S. D., B REWER , E. A., H ELLERSTEIN , J. M., AND
C ULLER , D. Scalable, distributed data structures for internet
service construction. In Proceedings of the 4th USENIX Sym-
posium on Operating Systems Design & Implementation (2000),
pp. 319–332.
[20] H EINRICH , J. MIPS R4000 Microprocessor User’s Manual.
MIPS technologies, 1994.
[21] H ERLIHY , M. P ., AND W ING , J. M. Linearizability: a correct-
ness condition for concurrent objects. ACM Transactions on
Programming Languages and Systems 12, 3 (July 1990), 463–
492.
[22] K ARGER , D., L EHMAN , E., L EIGHTON ,T . ,P ANIGRAHY , R.,
L EVINE , M., AND L EWIN , D. Consistent Hashing and Random
trees: Distributed Caching Protocols for Relieving Hot Spots on
the World Wide Web. In Proceedings of the 29th annual ACM
Symposium on Theory of Computing (1997), pp. 654–663.
[23] K EETON , K., M ORREY , III, C. B., S OULES , C. A., AND
V EITCH , A. Lazybase: freshness vs. performance in informa-
tion management. ACM SIGOPS Operating Systems Review 44,
1 (Dec. 2010), 15–19.
[24] L AMPORT , L. The part-time parliament. ACM Transactions on
Computer Systems 16, 2 (May 1998), 133–169.
[25] L IM , H., F AN , B., A NDERSEN , D. G., AND K AMINSKY , M.
Silt: a memory-efficient, high-performance key-value store. In
Proceedings of the 23rd ACM Symposium on Operating Systems
Principles (2011), pp. 1–13.
[26] L
ITTLE , J., AND G RA VES , S. Little’s law. Building Intuition
(2008), 81–100.
[27] L LOYD ,W . ,F REEDMAN , M., K AMINSKY , M., AND A NDER -
SEN , D. Don’t settle for eventual: scalable causal consistency for
wide-area storage with COPS. In Proceedings of the 23rd ACM
Symposium on Operating Systems Principles (2011), pp. 401–
416.
[28] M ETREVELI , Z., Z ELDOVICH , N., AND K AASHOEK , M.
Cphash: A cache-partitioned hash table. In Proceedings of the
17th ACM SIGPLAN symposium on Principles and Practice of
Parallel Programming (2012), pp. 319–320.
[29] O USTERHOUT , J., A GRAW AL ,P . ,E RICKSON , D.,
K OZYRAKIS , C., L EVERICH , J., M AZI `ERES , D., M I -
TRA , S., N ARA Y ANAN , A., O NGARO , D., P ARULKAR , G.,
R OSENBLUM , M., R UMBLE , S. M., S TRA TMANN , E., AND
S TUTSMAN , R. The case for ramcloud. Communications of the
ACM 54, 7 (July 2011), 121–130.
[30] P HANISHA YEE , A., K REV A T , E., V ASUDEV AN ,V . ,A NDER -
SEN , D. G., G ANGER , G. R., G IBSON , G. A., AND S E -
SHAN , S. Measurement and analysis of tcp throughput col-
lapse in cluster-based storage systems. In Proceedings of the 6th
USENIX Conference on File and Storage Technologies (2008),
pp. 12:1–12:14.
[31] P ORTS , D. R. K., C LEMENTS ,A .T . ,Z HANG , I., M ADDEN ,
S., AND L ISKOV , B. Transactional consistency and automatic
management in an application data cache. In Proceedings of
the 9th USENIX Symposium on Operating Systems Design &
Implementation (2010), pp. 1–15.
[32] R
AJASHEKHAR , M. Twemproxy: A fast, light-weight proxy for
memcached. https://dev.twitter.com/blog/twemproxy.
[33] R AJASHEKHAR , M., AND Y UE , Y. Caching with twem-
cache. http://engineering.twitter.com/2012/07/
caching-with-twemcache.html.
[34] R A TNASAMY , S., F RANCIS ,P . ,H ANDLEY , M., K ARP , R.,
AND S HENKER , S. A scalable content-addressable network.
ACM SIGCOMM Computer Communication Review 31, 4 (Oct.
2001), 161–172.
[35] S A TY ANARA Y ANAN , M., K ISTLER , J., K UMAR ,P . ,O KASAKI ,
M., S IEGEL , E., AND S TEERE , D. Coda: A highly available file
system for a distributed workstation environment. IEEE Trans-
actions on Computers 39, 4 (Apr. 1990), 447–459.

398 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’13) USENIX Association
[36] S TOICA , I., M ORRIS , R., K ARGER , D., K AASHOEK , M., AND
B ALAKRISHNAN , H. Chord: A scalable peer-to-peer lookup
service for internet applications. ACM SIGCOMM Computer
Communication Review 31, 4 (Oct. 2001), 149–160.
[37] V ENKA TARAMANI ,V . ,A MSDEN , Z., B RONSON , N., C ABR -
ERA III, G., C HAKKA ,P . ,D IMOV ,P . ,D ING , H., F ERRIS , J.,
G IARDULLO , A., H OON , J., K ULKARNI , S., L AWRENCE , N.,
M ARCHUKOV , M., P ETROV , D., AND P UZAR , L. Tao: how
facebook serves the social graph. In Proceedings of the ACM
SIGMOD International Conference on Management of Data
(2012), pp. 791–792.

Linearizability: A Correctness Condition for
Concurrent Objects
MAURICE P. HERLIHY and JEANNETTE M. WING
Carnegie Mellon University
A concurrent object is a data object shared by concurrent processes. Linearizability is a correctness
condition for concurrent objects that exploits the semantics of abstract data types. It permits a high
degree of concurrency, yet it permits programmers to specify and reason about concurrent objects
using known techniques from the sequential domain. Linearizability provides the illusion that each
operation applied by concurrent processes takes effect instantaneously at some point between its
invocation and its response, implying that the meaning of a concurrent object’s operations can be
given by pre- and post-conditions. This paper defines linearizability, compares it to other correctness
conditions, presents and demonstrates a method for proving the correctness of implementations, and
shows how to reason about concurrent objects, given they are linearizable.
Categories and Subject Descriptors: D.1.3 [Programming Techniques]: Concurrent Programming;
D.2.1 [Software Engineering]: Requirements/Specifications; D.3.3 [Programming Lan-
guages]: Language Constructs--abstract data types, concurrent programming structures, data types
and structures; F.1.2 [Computation by Abstract Devices]: Modes of Computation-parallelism;
F.3.1 [Logics and Meanings of Programs]: Specifying and Verifying and Reasoning about
Programs-pre- and post-conditions, specification techniques
General Terms: Theory, Verification
Additional Key Words and Phrases: Concurrrency, correctness, Larch, linearizability, multi-
processing, serializability, shared memory, specification
1. INTRODUCTION
1 .l Overview
Informally, a concurrent system consists of a collection of sequential processes
that communicate through shared typed objects. This model encompasses both
message-passing architectures in which the shared objects are message queues,
A preliminary version of this paper appeared in the Proceedings of the 14th ACM Symposium on
Principles of Programming Languages, January 1987 [21].
This research was sponsored by IBM and the Defense Advanced Research Projects Agents (DOD),
ARPA order 4976 (Amendment 20), under contract F33615-87-C-1499, monitored by the Avionics
Laboratory, Air Force Wright Aeronautical Laboratories, Wright-Patterson AFB. Additional spport
for J. M. Wing was provided in part by the National Science Foundation under grant CCR-8620027.
The views and conclusions contained in this document are those of the authors and should not be
interpreted as representing the official policies, either expressed or implied, of the Defense Advanced
Research Projects Agency or the US Government.
Authors’ address: Department of Computer Science, Carnegie Mellon University, Pittsburgh, PA
15213-3890.
Permission to copy without fee all or part of this material is granted provided that the copies are not
made or distributed for direct commercial advantage, the ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is by permission of the Association
for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific
permission.
0 1990 ACM 0164-0925/90/0700-0463 $01.50
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990, Pages 463-492.

464 l M. Herlihy and J. Wing
and shared-memory architectures in which the shared objects are data structures
in memory. Each object has a type, which defines a set of possible values and a
set of primitive operations that provide the only means to create and manipulate
that object. In a sequential system, where an object’s operations are invoked one
at a time by a single process, the meaning of the operations can be given by pre-
and postconditions. In a concurrent system, however, an object’s operations can
be invoked by concurrent processes, and it is necessary to give a meaning to
possible interleavings of operation invocations.
A concurrent computation is linearizable if it is “equivalent,” in a sense formally
defined in Section 2, to a legal sequential computation. We interpret a data type’s
(sequential) axiomatic specification as permitting only linearizable interleavings.
Instead of leaving data uninterpreted, linearizability exploits the semantics of
abstract data types; it permits a high degree of concurrency, yet it permits
programmers to specify and reason about concurrent objects using standard
verification techniques. Unlike alternative correctness conditions such as sequen-
tial consistency [31] or serializability [40], linearizability is a local property: a
system is linearizable if each individual object is linearizable. Locality enhances
modularity and concurrency, since objects can be implemented and verified
independently, and run-time scheduling can be completely decentralized. Linear-
izability is also a nonblocking property: processes invoking totally-defined oper-
ations are never forced to wait. Nonblocking enhances concurrency and implies
that linearizability is an appropriate condition for systems for which real-time
response is critical. Linearizability is a simple and intuitively appealing correct-
ness condition that generalizes and unifies a number of correctness conditions
both implicit and explicit in theliterature.
Using axiomatic specifications and our notion of linearizability, we can reason
about two kinds of problems:
(1) We reason about the correctness of linearizable object implementations using
new techniques that generalize the notions of representation invariant and
abstraction function [18, 251 to the concurrent domain.
(2) We reason about computations that use linearizable objects by transforming
assertions about concurrent computations into simpler assertions about their
sequential counterparts.
Section 2 presents our model of a concurrent system and the formal definition
of linearizability. Section 3 discusses linearizability’s locality and nonblocking
properties and compares it to other correctness conditions. Section 4 presents
our proof technique for reasoning about implementations of linearizable objects,
and illustrates this technique on two novel implementations of a highly concur-
rent queue. Section 5 presents examples of reasoning about concurrent registers
and queues, given that they are linearizable. Section 6 surveys some related work
and discusses the significance of linearizability as a correctness condition.
1.2 Motivation
When defining a correctness condition for concurrent objects, two requirements
seem to make intuitive sense: First, each operation should appear to “take effect”
instantaneously, and second, the order of nonconcurrent operations should be
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 465
E(x) A D(Y) A E(z) A
t 4 I 4 I- . . . .
E(Y) B D(x) B
i 4 I I
(a) H1 (acceptable).
E(x) A D(Y) A
t I I I
(b) Hz (not acceptable).
E(x) A
I- . . .
D(x) B
I t
E(x) A
E(Y) B
t- I
(c) I-I3 (acceptable).
D(Y) A
I I
D(Y) C
I 4
(d) H, (not acceptable).
Fig. 1. FIFO queue histories.
preserved. These requirements allow us to describe acceptable concurrent behav-
ior directly in terms of acceptable sequential behavior, an approach that simplifies
both formal and informal reasoning about concurrent programs. We capture
these notions formally in the next section; here we informally review some
examples to illustrate what we do and do not consider intuitively acceptable
concurrent behavior. Our examples employ a first in, first out (FIFO) queue, a
simple data type that provides two operations: Enq inserts an item in the queue,
and Deq returns and removes the oldest item from the queue. Figure 1 shows
four different ways in which a FIFO queue might behave when manipulated by
concurrent processes. Here, a time axis runs from left to right, and each operation
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

466 l M. Herlihy and J. Wing
W(o) A R(l) A W(O) c
t I --
t
W(l) B
(a) H6 (acceptable).
R(O) B
I I t
W(O) A R(l) A W(O) c
t t --
W(l) B R(1) B
t I t t
(b) Ha (not acceptable).
Fig. 2. Register histories.
is associated with an interval. Overlapping intervals indicate concurrent opera-
tions. We use “E(x) A” (“D(X) A”) to stand for the enqueue (dequeue) operation
of item x by process A.
The behavior shown in H1 (Figure la) corresponds to our intuitive notion of
how a concurrent FIFO queue should behave. In this scenario, processes A and
B concurrently enqueue x and y. Later, B dequeues x, and then A dequeues y and
begins enqueuing z. Since the dequeue for x precedes the dequeue for y, the FIFO
property implies that their enqueues must have taken effect in the same order.
In fact, their enqueues were concurrent, thus they could indeed have taken effect
in that order. The uncompleted enqueue of z by A illustrates that we are interested
in behaviors in which processes are continually executing operations, perhaps
forever.
The behavior shown in HP, however, is not intuitively acceptable. Here, it is
clear to an external observer that x was enqueued before y, yet y is dequeued
without x having been dequeued. To be consistent with our informal require-
ments, A should have dequeued x. We consider the behavior shown in H3 to be
acceptable, even though x is dequeued before its enqueuing operation has re-
turned. Intuitively, the enqueue of x took effect before it completed. Finally, Hq
is clearly unacceptable because y is dequeued twice.
To decide whether a concurrent history is acceptable, it is necessary to take
into account the object’s intended semantics. For example, acceptable concurrent
behaviors for FIFO queues would not be acceptable for stacks, sets, directories,
etc. When restricted to register objects providing read and write operations, our
intuitive notion of acceptability corresponds exactly to the notion used in Misra’s
careful axiomatization of concurrent registers [35]. Our approach can be thought
of as generalizing Misra’s approach to objects with richer sets of operations. For
example, H5 in Figure 2a is acceptable, but H6 is not (examples are taken from
[35]). These two behaviors differ at one point: In Hg, B reads a 0, and in Hg,
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 467
B reads a 1. The latter is intuitively unacceptable because A did a previous read
of a 1, implying that B’s write of 1 must have occurred before A’s read. C’s
subsequent write of 0, though concurrent with B’s write of 1, strictly follows A’s
read of 1.
In the next section, we formalize the intuition presented here by defining
the notion of linearizability to encompass those histories we have argued are
intuitively acceptable.
2. SYSTEM MODEL AND DEFINITION OF LINEARIZABILITY
2.1 Histories
Informally, a concurrent system consists of a collection of sequential threads of
control called processes that communicate through shared data structures called
objects. Each object has a unique name and a type. The type defines a set of
possible values, and a set of primitive operations that provide the only means to
manipulate that object. Processes are sequential: each process applies a sequence
of operations to objects, alternately issuing an invocation and then receiving the
associated response. (Dynamic process creation can be modeled simply by treating
each child process as an additional process that executes no operations before
the fork or after the join.)
Formally, an execution of a concurrent system is modeled by a history, which
is a finite sequence of operation invocation and response events. A subhistory of
a history H is a subsequence of the events of H. An operation invocation is
written as (x op(args*) A), where x is an object name, op is an operation name,
args* denotes a sequence of argument values, and A is a process name. The
response to an operation invocation is written as (x term(res*) A), where term
is a termination condition, and res* is a sequence of results. We use “Ok” for
normal termination. A response matches an invocation if their object names
agree and their process names agree. An invocation is pending in a history if
no matching response follows the invocation. If H is a history, complete(H) is
the maximal subsequence of H consisting only of invocations and matching
responses.
A history H is sequential if:
(1) The first event of H is an invocation.
(2) Each invocation, except possibly the last, is immediately followed by a
matching response. Each response is immediately followed by a matching
invocation.
A history that is not sequential is concurrent.
A process subhistory, H 1 P (H at P), of a history H is the subsequence of all
events in H whose process names are P. An object subhistory H 1 x is similarly
defined for an object x. Two histories H and H’ are equivalent if for every process
P, H 1 P = H’ ) P. A history H is well-formed if each process subhistory H ) P of
H is sequential. All histories considered in this paper are assumed to be well-
formed. Notice that whereas process subhistories of a well-formed history are
necessarily sequential, object subhistories are not.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

468 l M. Herlihy and J. Wing
An operation, e, in a history is a pair consisting of an invocation, inv(e), and
the next matching response, res(e). We denote an operation by [q inv/res A],
where q is an object and A a process. An operation e. lies within another operation
e, in H if inu (e,) precedes inu (eo) and res (eo) precedes res (ei) in H. Angle brackets
for events and square brackets for operations are omitted where they would
otherwise be unnecessarily confusing; object and process names are omitted
where they are clear from context.
For example, H, of Figure 1 is the following well-formed history for a FIFO
queue q.
Q Endx) A
4 EwW B
q Ok( 1 B
q W 1 A
q De4 1 B
q Ok(x) B
q Ded 1 A
q Ok(y) A
q Endz) A
The first event in H, is an invocation of Enq with argument x by process
A, and the fourth event is the matching response with termination condition
Ok and no results. The [q Enq(y)/Ok( ) B] operation lies within the
[q Enq(x)/Ok( ) A] operation. The subhistory, complete (H,), is H1 with the last
(pending) invocation of Enq removed. Reordering the first two events yields one
of many histories equivalent to H,.
A set S of histories is prefix-closed if, whenever H is in S, every prefix of H is
also in S. A single-object history is one in which all events are associated with
the same object. A sequential specification for an object is a prefix-closed set of
single-object sequential histories for that object. A sequential history H is legal
if each object subhistory H ] x belongs to the sequential specification for x. Many
conventional techniques exist for defining sequential specifications. In this paper,
we use the axiomatic style of Larch [19], in which an object’s sequential history
is summarized by a value, which (informally speaking) reflects the object’s state
at the end of the history. These values are used in axioms giving the pre- and
postconditions on the objects operations. For example, axioms for the Enq and
Deq operations for FIFO queues are shown in Figure 3. The post-condition for
Enq states that on termination, the new queue value is the old queue value with
e inserted. The specification for Deq states that applying that operation to a
non-empty queue removes the first item from the queue. An operation is total if,
like Enq, it is defined for every object value, otherwise it is partial, like Deq
which is left undefined for the empty queue.
2.2 Definition of Linearizability
A history H induces an irreflexive partial order <u on operations:
e. <u e, if res(e,) precedes inv(el) in H.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects l 469
Axiom E:
WeI
EnqW / Ok0
(4’ = Wq, e))
Fig. 3. Axioms for queue operations.
Axiom D:
(9 f Ill
DeqO /Ok(e) (q’ = rest(q) A e = first(q))
(Where appropriate, subscripts on partial orders are omitted). Informally, <n
captures the “real-time” precedence ordering of operations in H. Operations
unrelated by <n are said to be concurrent. If H is sequential, <n is a total order.
A history H is linearizable if it can be extended (by appending zero or more
response events) to some history H’ such that:
Ll: complete(H’) is equivalent to some legal sequential history S, and
L2: <H 2 CS.
Informally, extending H to H’ captures the notion that some pending invoca-
tions may have taken effect even though their responses have not yet been
returned to the caller (as in the pending Enq in history H, in Figure 1). Restricting
attention to complete(H’) captures the notion that the remaining pending
invocations have not yet had an effect. Ll states that processes act as if they
were interleaved at the granularity of complete operations. L2 states that this
apparent sequential interleaving respects the real-time precedence ordering of
operations.
We call S a linearization of H. Nondeterminism is inherent in the notion of
linearizability: (1) For each H, there may be more than one extension H’
satisfying the two conditions, Ll and L2, and (2) for each extension H’, there
may be more than one linearization S. A linearizable object is one whose concur-
rent histories are linearizable with respect to some sequential specification.
2.3 Queue Examples Revisited
Let “ . ” denote concatenation of events. The history H1 shown in Figure 1 is
linearizable, because H, . (q Ok( ) A) is equivalent to the following sequential
history:
q h(x) A (History Hi)
q ON ) A
q End B
q Ok( 1 B
q Ded ) B
q Ok(x) B
q De4 1 A
q Ok(y) A
q End4 A
q W 1 A
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

470 l M. Herlihy and J. Wing
Hz is not linear&able:
q h(x) A
q W 1 A
q Endy) B
q De4 1 A
q W 1 B
q Ok(y) A
(History HJ
because the complete Enq operation of x precedes the Enq of y, but y is dequeued
before x.
Linearizability does not rule out histories such as H3, in which an operation
“takes effect” before its return event occurs:
q EnqW A
q De4 1 B
q 0%) B
(History HJ
H, can be extended to Hi = H3 . (q Ok( ) A), which is equivalent to the sequential
history in which the enqueue operation occurs before the dequeue.
Finally, H4,
q J%(x) A
q Endy) B
q ON 1 A
q ON 1 B
q Ded ) A
q Ded 1 C
q Ok(y) A
q Ok(y) C
(History HJ
is not linearizable because y is enqueued once but dequeued twice, and hence H,
is not equivalent to any sequential FIFO queue history.
3. PROPERTIES OF LINEARIZABILITY
This section proves that linearizability is a local and nonblocking property, and
discusses the differences between it and other correctness conditions.
3.1 Locality
A property P of a concurrent system is said to be local if the system as a whole
satisfies P whenever each individual object satisfies P. Linearizability is a local
property:
THEOREM 1. H is linearizable if and only if, for each object x, H 1 x is linearizable.
PROOF. The “only if” part is obvious.
For each X, pick a linearization of H 1~. Let R, be the set of responses appended
to H 1 x to construct that linearization, and let cX be the corresponding lineari-
zation order. Let H’ be the history constructed by appending to H each response
in R,. We will construct a partial order < on the operations of complete(H’)
such that: (1) For each X, <X G <, and (2) <n C <. Let S be the sequential history
constructed by ordering the operations of complete(H’) in any total order that
extends <. Condition (1) implies that S is legal, hence that Ll is satisfied, and
Condition (2) implies that L2 is satisfied.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 471
Let < be the transitive closure of the union of all <X with <n. It is immediate
from the construction that < satisfies Conditions (1) and (2), but it remains to
be shown that < is a partial order. We argue by contradiction. If not, then there
exists a set of operations el, . . . , e,, such that e, < e2 < . . . < e,, e, < el, and
each pair is directly related by some <X or by <H. Choose a cycle whose length is
minimal.
Suppose all operations are associated with the same object X. Since <x is a
total order, there must exist two operations ei-1 and ei such that ei-1 < n ei and
ei <x ei-1, contradicting the linearizability of x.
The cycle must therefore include operations of at least two objects. By rein-
dexing if necessary, let el and e2 be operations of distinct objects. Let z be the
object associated with el. We claim that none of e2, . . . , e, can be an operation
of X. The claim holds for e2 by construction. Let ei be the first operation in
e3, . . . . e, associated with x. Since ei-1 and ei are unrelated by <%, they must be
related by <n; hence the response of ei-1 precedes the invocation of ei. The
invocation of e2 precedes the response of ei-1, since otherwise ei-1 <H e2, yielding
the shorter cycle e2, . . . , ei-1. Finally, the response of el precedes the invocation
of e2, since e, <n e2 by construction. It follows that the response to el precedes
the invocation of ei, hence el <n ei, yielding the shorter cycle el, ei, . . . , e,.
Since e, is not an operation of x, but e, < el, it follows that e, <n e,. But
el <u e2 by construction, and because < H is transitive, e, <n e2, yielding the
shorter cycle e2, . . . , e,, the final contradiction. II
Henceforth, we need consider only single-object histories.
Locality is important because it allows concurrent systems to be designed and
constructed in a modular fashion; linearizable objects can be implemented,
verified, and executed independently. A concurrent system based on a nonlocal
correctness property must either rely on a centralized scheduler for all objects,
or else satisfy additional constraints placed on objects to ensure that they follow
compatible scheduling protocols. Locality should not be taken for granted; as
discussed below, the literature includes proposals for alternative correctness
properties that are not local.
3.2 Blocking versus Nonblocking
Linearizability is a nonblocking property: a pending invocation of a totally-
defined operation is never required to wait for another pending invocation to
complete.
THEOREM 2. Let inv be an invocation of a total operation. If (x inv P) is
a pending invocation in a linearizable history H, then there exists a response
(x res P) such that H . (x res P) is linearizable.
PROOF. Let S be any linearization of H. If S includes a response (x res P) to
(x inv P), we are done, since S is also a linearization of H e (x res P). Otherwise,
(x inv P) does not appear in S either, since linearizations, by definition, include
no pending invocations. Because the operation is total, there exists a response
(x res P) such that
S’ = S - (-2 inv P) . (x res P)
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

472 l M. Herlihy and J. Wing
is legal. S’, however, is a linearization of H . (X res P), and hence is also a
linearization of H. 0
This theorem implies that linearizability per se never forces a process with a
pending invocation of a total operation to block. Of course, blocking (or even
deadlock) may occur as artifacts of particular implementations of linearizability,
but is is not inherent to the correctness property itself. (Techniques for con-
structing nonblocking implementations of linearizable objects are discussed
elsewhere [23].) This theorem suggests that linearizability is an appropriate
correctness condition for systems where concurrency and real-time response
are important. We shall see that alternative correctness conditions, such as
serializability, do not share this nonblocking property.
The nonblocking property does not rule out blocking in situations where it is
explicitly intended. For example, it may be sensible for a process attempting to
dequeue from an empty queue to block, waiting until another process enqueues
an item. Our queue specification captures this intention by making Deq’s speci-
fication partial, leaving it undefined for the empty queue. The most natural
concurrent interpretation of a partial sequential specification is simply to wait
until the object reaches a state in which the operation is defined.
3.3 Comparison to Other Correctness Conditions
Lamport’s notion of sequential consistency [31] requires that a history be equiv-
alent to a legal sequential history. Sequential consistency is weaker than linear-
izability, because it does not require the original history’s precedence ordering
to be preserved. For example, history H7 is sequentially consistent, but not
linearizable:
q End4 A
q Ok( 1 A
q Endy) B
q ON 1 B
q De4 1 B
q Ok(y) B
(History H7)
Sequential consistency is not a local property. Consider the following history
H8, in which processes A and B operate on queue objects p and q.
P End4 A (History H8)
P ON 1 A
q Endy) B
q W ) B
q Enqbd A
q ON 1 A
P Enqb) B
POkOB
P De4 1 A
P Ok(y) A
q D-d ) B
q Ok(x) B
It is easily checked that H, ] p and H8 ] q are sequentially consistent, but Hs itself
it not.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 473
Much work on databases and distributed systems uses serializability [40] as
the basic correctness condition for concurrent computations.l In this model, a
transaction is a thread of control that applies a finite sequence of primitive
operations to a set of objects shared with other transactions.’ A history is
serializable if it is equivalent to one in which transactions appear to execute
sequentially, i.e., without interleaving. A (partial) precedence order can be defined
on non-overlapping pairs of transactions in the obvious way. A history is strictly
serializable if the transactions’ order in the sequential history is compatible with
their precedence order. Strict serializability is ensured by some synchronization
mechanisms, such as two-phase locking [12], but not by others, such as multi-
version timestamp schemes [41], or schemes that provide high levels of availa-
bility in the presence of network partitions [22].
Linearizability can be viewed as a special case of strict serializability where
transactions are restricted to consist of a single operation applied to a single
object. Nevertheless, this single-operation restriction has far-reaching practical
and formal consequences, giving linearizable computations a different flavor from
their serializable counterparts. An immediate practical consequence is that con-
currency control mechanisms appropriate for serializability are typically inap-
propriate for linearizability because they introduce unnecessary overhead and
place unnecessary restrictions on concurrency. For example, the queue imple-
mentation given below in Section 4 is much more efficient and much more
concurrent than an analogous implementation using conventional serializability-
oriented techniques such as two-phase locking or multi-version timestamping.
One important formal difference between linearizability and serializability is
that neither serializability nor strict serializability is a local property. For
example, in history Hs shown above, if we interpret A and B as transactions
instead of processes, then it is easily seen that both Hs ] p and Hs ] q are strictly
serializable but He is not. (Because A and B overlap at each object, they are
unrelated by transaction precedence in either subhistory.) Moreover, since A and
B each dequeues an item enqueued by the other, H8 is not even serializable. A
practical consequence of this observation is that implementors of objects in
serializable systems must rely on global conventions to ensure that all objects’
concurrency control mechanisms are compatible with one another. For example,
it is well known that two-phase locking is incompatible with multiversion
timestamping [46].
Another important formal difference is that serializability places more rigorous
restrictions on concurrency. Serializability is inherently a blocking property:
under certain circumstances, a transaction may be unable to complete a pending
operation without violating serializability, even if the operation is total. Such a
transaction must be rolled back and restarted, implying that additional mecha-
nisms must be provided for that purpose. For example, consider the following
i In practice, serializability is almost always provided in conjunction with failure atomicity, ensuring
that a transaction unable to execute to completion will be automatically rolled back. There is no
counterpart to failure atomicity for linearizability.
* Some models permit transactions to be nested, or to encompass concurrent threads of control. Our
remarks about locality and nonblocking hold for these more elaborate models as well.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

474 l M. Herlihy and J. Wing
history involving two register objects: x and y, and two transactions: A and B.
x Read( ) A (History H,)
y Read( ) B
x Ok(O) A
Y Ok(O) B
x Write(l) B
y Write(l) A
Here, A and B respectively read x and y and then attempt to write new values to
y and x. It is easy to see that both pending invocations cannot be completed
without violating serializability. Although different concurrency control mecha-
nisms would resolve this conflict in different ways, such deadlocks are not an
artifact of any particular mechanism; they are inherent to the notion of serializ-
ability itself. By contrast, we have seen that linearizability never forces processes
executing total operations to wait for one another.
Perhaps the major practical distinction between serializability and lineariza-
bility is that the two notions are appropriate for different problem domains.
Serializability is appropriate for systems such as databases in which it must be
easy for application programmers to preserve complex application-specific invar-
iants spanning multiple objects. A general-purpose serialization protocol, such as
two-phase locking, enables programmers to reason about transactions as if they
were sequential programs (setting aside questions of deadlock or performance).
Linearizability, by contrast, is intended for applications such as multiprocessor
operating systems in which concurrency is of primary interest, and where pro-
grammers are willing to apply special-purpose synchronization protocols, and to
reason explicitly about the effects of concurrency.
4. VERIFYING THAT IMPLEMENTATIONS ARE LINEARIZABLE
In this section, we motivate and describe our method for verifying implementa-
tions of linearizable objects. We begin with our definition of when an implemen-
tation is correct. In order to prove correctness, we reexamine the notions of
representation invariant and abstraction function (Section 4.2), and use their
new interpretation in our proof method (Section 4.3).
4.1 Definition of Correctness
An implementation is a set of histories in which events of two objects, a
representation (or rep) object REP of type REP and an abstract object ABS of type
ABS, are interleaved in a constrained way: for each history H in the implemen-
tation, (1) the subhistories H ( REP and H ( ABS satisfy the usual well-formedness
conditions; and (2) for each process P, each rep operation in H ] P lies within an
abstract operation in H ] P. Informally, an abstract operation is implemented by
the sequence of rep operations that occur within it.
An implementation is correct with respect to the specification of ABS if for
every history H in the implementation, H ( ABS is linearizable.
4.2 Representation Invariant and Abstraction Function
We first review how to verify the correctness of sequential objects [18, 251. In
the sequential domain, an implementation consists of an abstract type ABS, the
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990

Linearizability: A Correctness Condition for Concurrent Objects 475
type being implemented, and a representation type REP, the type used to
implement ABS. The subset of REP values that are legal representations
is characterized by a predicate called the rep invariant, I: REP + BOOL.
The meaning of a legal representation is given by an abstraction function,
A: REP + ABS, defined for representation values that satisfy the invariant.
An abstract operation (Y is implemented by a sequence, p, of rep operations
that carries the rep from one legal value to another, perhaps passing through
intermediate values where the abstraction function is undefined. The rep invar-
iant is thus part of both the precondition and postcondition for each operation’s
implementation; it must be satisfied between abstract operations, although it
may be temporarily violated while an operation is in progress. An implementation,
p, of an abstract operation, a, is correct if there exists a rep invariant, I, and
abstraction function, A, such that whenever p carries one legal rep value r to
another r’, CY carries the abstract value from A(r) to A(r’).
This verification technique must be substantially modified before it can be
applied to concurrent objects: we change both the meaning of the rep invariant
and the signature of the abstraction function. To help motivate these changes
and to make our discussion as concrete as possible, consider the following highly
concurrent implementation of a linearizable FIFO queue. The queue’s represen-
tation is a record with two components: items is an array having a low bound of
1 and a (conceptually) infinite high bound, and buck is the (integer) index of the
next unused position in items.
rep = record [back: int, items: array [item]]
Each element of items is initialized to a special null value, and back is initialized
to 1. Enq and Deq are implemented as follows:
Enq = proc (q: queue, x: item)
i: int := INC(q.back) %Allocate a new slot.
STORE (q.items[i], x) % Fill it.
end Enq
Deq = proc (q: queue) returns (item)
while true do
range: int := READ(q.back) - 1
for i: int in 1 . . range do
x: item := SWAP(q.items[i], null)
if x -= null then return(x) end
end
end
end Deq
An Enq execution occurs in two distinct steps, which may be interleaved with
steps of other concurrent operations: an array slot is reserved by atomically
incrementing back, and the new item is stored in items.3 Deq traverses the array
in ascending order, starting at index 1. For each element, it atomically swaps
null with the current contents. If the value returned is not equal to null,
3 Like the FETCH-AND-ADD operation [30], INC returns the value of its argument from before the
invocation, not the newly incremented value.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

476 l M. Herlihy and J. Wing
Deq returns that value, otherwise it tries the next slot. If the index reaches
q.back - 1 without encountering a nonnull element, the operation is restarted.
(Note that there is a small chance that a dequeuing process may starve if it is
continually overtaken by other dequeuing processes. Any queue item, however,
will eventually be dequeued as long as there are active dequeuers.) All atomic
steps can be interleaved with steps of other operations. An interesting aspect of
this implementation is that there is no mutual exclusion: no process can delay
other processes by halting in a critical section. As an aside, we note that this
implementation could be rendered more efficient by reclaiming slots from which
items have been dequeued, reducing both the overall size of the rep of the queue
and the cost of dequeuing an item. Such optimizations, however, would add
nothing to our discussion of verification, so we ignore them in this paper.
The first difficulty arises when trying to define a rep invariant for this
implementation. For sequential objects, the rep invariant must be satisfied at the
start and tinish of each abstract operation, but it may be violated temporarily
while an operation is in progress. For concurrent objects, however, it no longer
makes sense to view the object’s representation as assuming meaningful values
only between abstract operations. For example, our queue implementation per-
mits operations to be in progress at every instant, thus the object may never be
“between operations.” When implementing a queue operation, one must be
prepared to encounter a rep value that reflects the incomplete effects of concur-
rent operations, a problem that has no analog in the sequential domain. To
assign a meaning to such transient values, the abstraction function must be
defined continually, not just between abstract operations. As a consequence, the
rep invariant must be preserved by each rep operation in the sequence imple-
menting each abstract operation.
Another, more subtle difficulty arises when attempting to define an abstraction
function. One natural approach is the following, proposed by Lamport [32]. A
(continually defined) abstraction function A is chosen so that each abstract
operation “takes effect” instantaneously at some step in its execution. In our
queue example, when a process enqueues an item X, exactly one of the opera-
tions implementing the Enq would carry the rep from r to r’, where A(r’) =
ins(A(r), x). Surprisingly, perhaps, this technique fails to work for our queue
implementation. To see why, we assume that such a function A exists, and we
derive a contradiction. Consider the following scenario. Processes A and B invoke
concurrent Enq operations, respectively enqueuing x and y. By incrementing the
back counter, A reserves array position 1 and B reserves array position 2. B stores
y in the array and returns. This computation is represented by the following
history, where rep operations are indented and shown in upper-case.
End4 A
EnqW B
INC(q.back) A
OK(l) A
INC(q.back) B
OK(2) B
STORE(q.items[2], y) B
OK( 1 B
ON 1 B
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 477
Let r be the rep value after this history. Because B’s Enq operation has
returned, A(r) must reflect B’s Enq. Because A’s Enq operation is still in progress,
A(r) may or may not reflect A’s Enq, depending on how A is defined. Thus, since
no other operations have occurred, A(r) must be one of [y], [y, n], or [x, y], where
the leftmost item is at the head of the queue.
We now derive a contradiction by showing that each of these values is
contradicted by some future computation. First, assume A(r) is [x, y]. If we now
suspend A and allow a third process C to execute a Deq, C’s Deq will return y,
contradicting our assumption.
De4 1 C
READ(q.back) C
OK(2) C
SWAP(q.items[l], y) C
OK(nul1) C
SWAP(q.items[B],y) C
OK(Y) C
Ok(y) C
Second, assume A(r) is [y] or [y, x]. Allow A to complete its Enq, leaving a rep
value r’. Now x must be in the queue, since its Enq is complete, and moreover it
must follow y in the queue since, by hypothesis, A’s enqueue appears to take
effect after B’s. It follows that A(r’) must be [y, x]. If C then executes a Deq,
however, it will return x, a contradiction.
STORE(q.items[l], x) A
OK( 1 A
Ok( ) A
Ded 1 C
READ(q.back) C
OK(2) C
SWAP(q.items[l], y) C
OK(x) C
Ok(x) C
The problem here is that the linearization order depends on a race condition:
A’s Enq will appear to occur before B’s if A stores into location 1 before C reads
from it, otherwise the order is reversed. Such nondeterminism is perfectly
acceptable, however, because all resulting histories are linearizable. We circum-
vent this difficulty by redefining the abstraction function to map a rep value to
a set of abstract values. This set represents the possible set of linearizations
permitted by the current value of the rep. For objects that permit low levels of
concurrrency, the value of the abstraction function might be a singleton set.
In conclusion, the rep invariant I must be continually satisfied and the
abstraction function continually defined, not only between abstract operations,
but also between rep operations implementing abstract operations. The abstrac-
tion function maps each rep value to a nonempty set of abstract values:
A: REP + 2ABS
The nondeterminism inherent in a concurrent computation thus gives our notions
of abstraction function and rep invariant a different flavor from their sequential
counterparts.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

478 l M. Herlihy and J. Wing
4.3 Verification Method
In the next three sections we show how we use our new interpretation of
representation invariant and abstraction function for proofs of correctness. We
illustrate these ideas on the queue example presented in the previous section, as
well as for an alternative implementation that uses critical sections.
4.3.1 Linearized Values. So far, linearizability is discussed in terms of histo-
ries. This characterization is useful for motivating the property, and for demon-
strating properties such as locality, but it is awkward for verification. For
linearizable histories, however, assertions about interleaved histories can be
transformed into assertions about sets of sequential histories, and thus, sets of
values. The transformed assertions can be stated and proved with the help of
familiar axiomatic methods developed for sequential programs.
For a given history H, we call the value of an object at the end of a linearization
of H a linearized value. Since a given history may have more than one lineariza-
tion, an object may have more than one linearized value at the end of a history.
We let Lin(H) denote the set of all linearized values of H. Informally, a history’s
linearized values represent the object’s possible values from the point of view of
an external observer. Figure 4 shows a queue history with its set of linearized
values after each event. Initially, only the empty queue is associated with the
empty history. After the invocation of Enq(x), there are two linearized values,
since the enqueue may or may not have taken effect. After the invocation of
Enq(y), there are five linearized values: either Enq may or may not have occurred,
and if both have occurred, either ordering is possible. After the response to
Enq(y), y is known to have been enqueued, and after the response to Enq(x),
both x and y must have been enqueued, although their order remains ambiguous
until x is dequeued.
4.3.2 Proof Method. To show correctness, the verification technique for se-
quential implementations is generalized as follows. Assume that the implemen-
tation of r is correct, hence H 1 REP is linearizable for all H in the implementation.
Our verification technique focuses on showing the following property:
For all r in Lin(H 1 REP), I(r) holds and A(r) G Lin(H 1 ABS)
This condition implies that Lin(H 1 ABS) is nonempty, hence that H 1 ABS is
linearizable. Note that the set inclusion is necessary in one direction only; there
may be linearized abstract values that have no corresponding representation
values. Such a situation arises when the representation “chooses” to linearize
concurrent operations in one of several permissible ways.
4.3.3 The Queue Example. Returning to our queue example, our verification
method is applied as follows. Let H I REP be a complete history for a queue
representation, REP. If r is a linearized value for H I REP, define items(r) to be
the set of non-null items in the array r.items. Let cr be the partial order such
that x cr y if the STORE operation for x precedes the INC operation for y in
H 1 REP. We can encode the partial order cr as auxiliary data. For a queue q, let
c4 denote the total order on its items, and items(q), the set of its items.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 479
History
Et-W) A
W(y) 13
Ok0 6
Ok0 A
DeqO C
Ok(x) C
Fig. 4. Linearized values.
The implementation has the following rep invariant:
I(r) = (r.back L 1)
A (Vi. i 2 r.back + r.items[i] = null)
A (lbound(r.items) = 1)
where Ibound is the lowest array index, and the following abstraction function:
A(r) = 14 ] items(r) = items(q) A cr C <J
In other words, a queue representation value corresponds to the set of queues
whose items are the items in the array, sorted in some order consistent with the
precedence order of their Enq operations. Thus, our implementation allows for
an item with a higher index to be removed from the array before an item with a
lower index, but only if the items were enqueued concurrently.
Figure 5 shows a sequence of abstract operations of Figure 4 along with their
implementing sequence of rep operations. Column two is the set of abstracted
linearized rep values. Column three is the set of linearized abstract values. Our
correctness criterion requires showing that each set in column two is a subset of
the corresponding set in column three.
Appendix II outlines a complete formal proof of correctness (see also [45]). It
relies on two key facts: (1) Enq enqueues an item x that is maximal with respect
to <,., and (2) Deq removes and returns an item x that is minimal with
respect to Cr.
4.3.4 Critical Sections. So far our method for proving the correctness of an
implementation assumes there exists a continually defined abstraction function.
If the object’s implementation includes critical sections, however, it may not
always be possible to define such a function. Within the critical section, the rep
invariant may be temporarily violated, leaving the abstraction function unde-
fined. We show here how to overcome this difficulty relying on the standard trick
of using (auxiliary) hidden data [37], thereby permitting us to reintroduce a
continually defined abstraction function with the extended representation as its
domain.
Both the problem and the solution are best illustrated by a simple example.
Let us replace the atomic SWAP operation with a sequence of rotations executed
within a critical section. Items are represented by 32-bit quantities, and the
queue representation is expanded to associate a lock with each item:
rep = recordtback: int, items: array[item],
locks: array[mutex])
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

480 l M. Herlihy and J. Wing
ROT& y) atomically rotates the 64-bit quantity by one bit. The Deq operation
is implemented as follows:
Deq = proc(q: queue) returns (item)
while true do
range: int := READ(q.back)-1
x: item := null
for i: int in l..range do
LOCK(q.locks[i] ) % start critical section
for k: int in 1..32 do
ROT(q.items[i], x)
end
UNLOCK(q.items[i]) % end critical section
if x -= null then return(x) end
end
end Deq
Although it is clear that this implementation is linearizable, its correctness
cannot be proved directly using the method outlined so far. While the rotation is
in progress, the abstraction function is undefined because necessary state infor-
mation is encoded in the process’s program counter and local variables, not in
the representation itself. Thus, we introduce an auxiliary array of items to hold
the value being shifted out of the queue, shown here as an additional field in the
representation. Auxiliary data and statements are shown in italics. Statements
enclosed in angle brackets are executed atomically.
rep = record(back: int
items: array[item] ,
aux: array[item],
locks: array[mutex]
Enq = proc(q: queue, x: item)
i: int := INC(q.back)
(STORE(q.items[i], x)
STORE(q.aux[i], x)) % Make a redundant copy.
end Enq
Deq = proc(q: queue) returns (item)
while true do
range: int := READ(q.back)-1
x: item := null
for i: int in l..range do
LOCK(q.locks[i]) % start critical section
for k: int in 1..32 do
ROT(q.items[i], x)
end
STURE(q.aux[i], null) % Update auxiliary array.
UNLOCK(q.items[i]) % end critical section
if x -= null then return(x) end
end
end Deq
By embedding the representation object in an extended representation, we can
give a continually defined abstraction function, one that agrees with the original
abstraction function when the object is quiescent. We can use our proof method
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 481
History A(Lin(H 1 REP)) Lin(H I AM)
EnW A INC(q.back) A
OK(l) A
STORE(q.items[l], x) A
OK0 A
EnW B
INC(q.back) B
OK(Z) B
STORE(q.items[P], y) B
OK0 B
Ok0 B
Ok0 A
DeqO C
READ(q.back) C
OK(2) C
SWAP(q.items[l], null) C
OK(x) C
Ok(x) C
Fig. 5. A queue history.
to show the correctness of the extended representation, which then implies the
correctness of the original.
The implementation has the following rep invariant:
I(r) = (r.back > 1)
A (Vi. i 2 r.back + (r.items[i] = null A r.aux[i] = null))
A (Vi. (i < r.back A r.locks[i] = FREE) + r.items[i] = r.aux[i])
A (lbound(r.items) = 1 A lbound(r.aux) = 1)
The third conjunct is the most interesting since it states that the auxiliary array
and the “real” array agree on all unlocked items.
Below, let A’ be the extended abstraction function defined on the object r
of the original rep type, and z, the auxiliary data. As before, we define cr to be
the partial order on items in the r.items array, and similarly define <= to be the
partial order on items in the r.aux array. The abstraction function is:
A’(r, z) = (q ] (3i. (i < r.back A r.locks[i] # FREE))
+ (items(q) = items(z) A <* C c4)
A (Vi. (i < r.back A r.locks[i] = FREE))
4 (items(q) = items(r) A cr C <,)I
If a rotation is in progress the extended abstraction function simply uses the
auxiliary value. When the object is quiescent, each lock is free, and A’ agrees
with the original A.
5. REASONING ABOUT LINEARIZABLE OBJECTS
In the previous section we showed how to reason about the correctness of an
implementation, given that linearizability is our correctness condition. In this
section we show how we reason about properties of concurrent objects given just
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

482 l M. Herlihy and J. Wing
their (sequential) specifications and the assumption that they are implemented
correctly, i.e., that they are linearizable.
5.1 Concurrent Registers
Here are axioms for Read and Write operations for all concurrent register
objects, r:
(true)
Read( )/Ok(u)
(r.val = r’.val = u)
(true]
Write(u)/Ok( )
(r’.val = u)
These sequential axioms can be combined with our linearizability condition to
prove assertions about the interleavings permitted by concurrent registers. Below,
in a linearization H of a register history, let ui denote the value of the register
after the ith (complete) operation of H.
Every value read was written, but not overwritten.
THEOREM 3. If r is a Read( )/Ok(u) operation in H, then there exists a
Write(u)/Ok( ) operation w such that r does not precede w, and there is no other
Write operation w ’ such that w precedes w ’ and w ’ precedes r.
PROOF. Let r be the kth operation in a linearization of H, and let i < k be
the greatest index such that ai = u. By construction, the ith operation in H is the
Write(u) operation. If w ’ exists, then there exists j such that i < j < k and
uj # u, a contradiction. 0
Register values are persistent in the absence of Write operations.
THEOREM 4. An interval in a history is a sequence of contiguous euents. If I is
an interval that does not overlap any Write operations, then all Read operations
that lie within I return the same ualue.
PROOF. Pick two Read operations ei and ej, i < j, that lie within the interval
I. If ui # uj, then a Write operation must be linearized after e; and before ej,
contradicting the assumption that no Writes overlap 1. Cl
5.2 Concurrent Queues
The proofs of the following properties of concurrent queues use the following
fact, which follows from Axioms E and D in Figure 3. For simplicity, we assume
all values of items in a queue are unique.
LEMMA 5. In any sequential queue history where x is enqueued before y, x is
not dequeued after y.
THEOREM 6. If the Enq of x, Enq of y, Deq of x, and Deq of y are complete
operations of H such that x’s Enq precedes y’s Enq, then y’s Deq does not precede
x’s Deq (i.e., either x’s Deq precedes y’s, or they are concurrent).
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 483
PROOF. Suppose not, i.e., y’s Deq precedes x’s Deq. Pick a linearization, and
let qi and qj be queue values following the Deq operations of x and y respectively.
From the assumption that j < i, qj-1 = [y, . . . , X, . . .I, which implies that y is
enqueued before x, a contradiction. 0
Gottlieb, Lubachevsky, and Rudolph [15] adopt the property proved in Theo-
rem 6 as the (informal) correctness property for a linearizable queue implemen-
tation. The difficulty of reasoning informally about concurrent histories is
illustrated by observing that Theorem 6 by itself is incomplete as a concurrent
queue specification, since it does not prohibit implementations in which enqueued
items spontaneously disappear from the queue, or new items spontaneously
appear. Such behavior is easily ruled out by the following two theorems:
Items do not spontaneously vanish from the queue.
THEOREM 7. If the Enq of x precedes the Enq of y, and if y has been dequeued,
then either x has been dequeued or there is a pending Deq concurrent with the
Deq of Y.
PROOF. Pick a linearization. Suppose x has not been dequeued. Let qj be the
value of the queue following the Deq of y, If y has been dequeued, but x has not,
qj-1 = [Yt * * - 9 X9 * * -19 contradicting the assumption that the Enq of x precedes
the Enq of y. 0
Items do not spontaneously appear in the queue.
THEOREM 8. If x has been dequeued, then it was enqueued, and the Deq
operation does not precede the Enq.
PROOF. Suppose not. Pick a linearization, and let qi and qj be the queue values
after the Enq and Deq operations respectively. From our assumption, j < i. Then
qj-1 = [X, e a .] and qi = [. . . , x], implying by the uniqueness of the values of the
items, that i I j - 1 < j, a contradiction. 0
6. DISCUSSION
6.1 Related Work
The axiomatic approach to specifying sequential programs has its origins in
Hoare’s early work on verification [ 241. Owicki and Gries extended Hoare’s work
to handle concurrent programs [37] by including axioms for general concurrent
programming language constructs such as the parallel operator. Apt et al. [3] use
an axiomatic approach for CSP [27]. Many researchers have also developed proof
techniques for concurrent programs using conditional critical regions and moni-
tors [7, 14,28,44]. We appeal to this past work when we perform syntax-directed
reasoning about our implementations. In particular, we rely on standard tech-
niques to deal with noninterference, using auxiliary data to encode both the
program counters of other processes (e.g., the auxiliary array of Section 4.3.4)
and history information (e.g., the cr partial order on items). All of this work,
however, differs from ours by focusing on control structures. Data are either left
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

484 - M. Herlihy and J. Wing
completely uninterpreted or assumed to be of simple primitive types like booleans
and integers. In contrast, our work on specifying and verifying concurrent objects
focuses on data entirely, exploiting the semantics of the data type to increase the
degree of concurrency. Our work builds upon, not replaces, older verification
technology.
Related axiomatic work in abstract data types deals with proofs of correctness
of their implementations [25], where, typically, first-order predicate logic pre-
and post conditions are used for the specification of each operation of the type.
Standish [43] and Nakajima [36] use a similar approach. The algebraic approach,
which defines data types to be heterogeneous algebras [5], uses axioms to specify
properties of programs and abstract data types, but the axioms are restricted to
equations. Much work has been done on algebraic specifications for abstract data
types [2,8, 10,171. Any one of these approaches would be adequate for specifying
the sequential behavior of a data type as required by our definition of when a
sequential history is legal. In practice, we use Larch [19, 201. Our contribution to
the area of specifying abstract data types is that we can work with data in a
concurrent, not just sequential, domain.
In short, whereas verification of concurrent programs focused on control, we
focus on data; whereas past verification of abstract data types is applicable for
sequential programs, ours is applicable for concurrent ones.
One notable exception is Lamport’s work [32] in which he proposed a model
and assertion language for specifying safety and liveness properties of concurrent
objects. His approach is more general than ours, as it addresses liveness as well
as safety properties, and nonlinearizable as well as linearizable behavior. Our
approach, however, focuses exclusively on a subset of concurrent computations
that we believe to be the most interesting and useful. In place of a specification
language powerful enough to specify all conceivable concurrent behaviors, we
re-interpret assertions about “well-behaved” concurrent computations as asser-
tions about their equivalent sequential computations.
Moreover, Lamport’s technique is based on a continually defined abstraction
function (called a state function) that maps the representation to a single
abstract value. This abstraction function defines the instant at which each
operation appears to take effect: each primitive step of each operation either
leaves the function’s value unchanged, or it instantaneously causes the operation
to take effect. This technique is not powerful enough to verify highly concurrent
objects such as the queue implementation given in Section 4. Indeed, our
linearizable queue example has since inspired Abadi and Lamport to extend
Lamport’s original technique to include not only history variables, but prophecy
variables [ 11. Prophecy variables are related to hidden variables called possibilities
which we use in our proofs in the Appendices.
Our notion of linearizability generalizes and unifies similar notions found
in specific examples in the literature. The use of concurrency control
mechanisms such as monitors [26] or Ada tasks [9] is usually illustrated by
simple implementations of linearizable objects such as bounded FIFO queues.
These implementations permit very little concurrency, since operations exe-
cute one at a time. A more interesting example is due to Lamport 1321, who
verifies linearizability and liveness for a queue implementation that permits one
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects l 485
enqueuing process to execute concurrently with one dequeuing process. There
exists extensive literature on concurrent B-trees [4, 33, 421 and related search
structures [6, 11, 13, 16, 291. Although the correctness properties for these data
structures are often stated in ad hoc terms, it is clear that they are meant to be
linearizable. The algorithms cited above provide excellent additional examples of
nontrivial techniques for implementing linearizable objects.
Misra [35] has proposed an axiomatic treatment of concurrent hardware
registers in which the register’s value is expressed as a function of time. Restricted
to registers, our axiomatic treatment is equivalent to his in the sense that both
characterize the full set of linearizable register histories. Theorems 3 and 4
capture two properties of Misra’s registers. Misra’s explicit use of time in axioms
is appropriate for hardware, where reasoning in terms of the register’s hypothet-
ical value is useful as a guide to hardware designers. Our approach, however, is
also appropriate for objects implemented in software, as we have found that
reasoning directly in terms of partial orders generalizes more effectively to data
types having a richer set of operations.
Gottlieb et al. [15] have investigated architectural support for implementing
concurrent objects without critical sections, an approach illustrated by our
linearizable implementation of a FIFO queue. They present a linearizable imple-
mentation of a concurrent queue (different from ours). The correctness condition
asserted for their queue, however, is the property stated in Theorem 6, which by
itself is incomplete as a concurrent queue specification since it does not prohibit
implementations in which enqueued items spontaneously disappear from the
queue, or new items spontaneously appear. As shown by Theorems 7 and 8, such
anomalous behavior is easily ruled out by our queue axioms and the assumption
of linearizability.
6.2 Final Remarks
Without linearizability, the meaning of an operation may depend on how it is
interleaved with concurrent operations. Specifying such behavior would require
a more complex specification language, as well as producing more complex
specifications. Linearizability provides the illusion that each operation takes
effect instantaneously at some point between its invocation and its response,
implying that the meaning of a concurrent object’s operations can still be given
by pre- and post conditions.
The role of linearizability for concurrent objects is analogous to the role of
linearizability for database theory: it facilitates certain kinds of formal (and
informal) reasoning by transforming assertions about complex concurrent behav-
ior into assertions about simpler sequential behavior. Like serializability, linear-
izability is a safety property; it states that certain interleavings cannot occur, but
makes no guarantees about what must occur. Other techniques, such as temporal
logic [32,34,39], must be used to reason about liveness properties such as fairness
or priority.
An implementation of a concurrent object need not realize all interleavings
permitted by linearizability, but all interleavings it does realize must be linear-
izable. The actual set of interleavings permitted by a particular implementation
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

486 l M. Herlihy and J. Wing
may be quite difficult to specify at the abstract level, being the result of
engineering trade-offs at lower levels. As long as the object’s client relies only on
linearizability to reason about safety properties, the object’s implementor is free
to support any level of concurrency that appears to be cost-effective.
In conclusion, linearizability provides benefits for specifying, implementing,
and verifying concurrent objects in multiprocessor systems. Rather than intro-
ducing complex new formalisms to reason directly about concurrent computa-
tions, we feel it is more effective to transform problems in the concurrent domain
into simpler problems in the sequential domain.
I. GENERAL PROOFS OF CORRECTNESS
The proofs of the lemmas in this section are given elsewhere [45].
1.1 Possibilities and Linearized Values
For each linearized value, it is sometimes useful to keep track of which invocations
were completed in the linearization that yielded that value, and what their
responses were. A possibility for a history H is a triple (v, P, R), where v is
a linearized value of H, P is the subset of pending invocations in H not com-
pleted when forming the linearization that yielded u, and R is the set
of responses appended to H to form u. We let Pass(H) denote the set of
possibilities of a history H. The relationship between the set of possibilities
and set of linearized values for a given history H is the following: for each
(u, P, R) u E Pass(H), u E Lin(H). For the example in Figure 4, the possibilities
([ I, 1Ensb4 AJ,Q9 and ([xl, 0, VW ) 4) are in Poss((Enq(x) A)). In the first
case, the linearization is the empty history: the queue is empty, the pending Enq
invocation was not completed, and no responses were appended. In the second
case, the linearization is a single Enq operation: the queue holds x, no pending
invocations were left incomplete, and A’s Enq was completed normally. Similarly,
(ix, ~1, 0, VW 1 A, W 1 Bl) and ([Y, 4, 0, 04 1 A, W 1 W) are two of the
possibilities (among many others) in Poss( (Enq(x) A) . (Enq(y) B)).
1.2 Four Generic Axioms
In order to carry out a formal proof of correctness for our queue example, it helps
to appeal to the following four type-independent axioms. These axioms are used
to derive a history’s set of possibilities, and hence its set of linearized values.
Let x be the object whose operations appear in H. The following closure axiom
states that if u is in Lin(H) and (inv A) is a pending invocation in H that is not
completed to form u, but could be completed with a response (res A) to yield a
legal value u ’ for X, then u ’ is also in Lin(H):
Axiom C:
(u, P, R) E Pass(H) A (inv A) E P A (x = u) inv/res (X = u’l
- (v’, P - (inv A}, R U {res A}) E Pass(H)
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 487
We write “(x = u) inv/res (x = ~‘1” to indicate that the condition must be
derivable from the sequential axioms for X.
The following invocation axiom states that any linearization of H is also a
linearization of H . (inv A):
Axiom I:
(u, P, R) E Pass(H)
4 (u, P U (inv A], R) E Poss(H . (inv A))
The following response axiom states that any linearization of H in which the
pending (inv A) is completed with (res A) is also a linearization of H . (res A):
Axiom R:
(u, P, R) E Pass(H) and (res A) E R
+ (u, P, R - (res A]) E Poss(H a (res A))
The following initialization axiom states that the possibility for the initial value
ug of an object corresponds to the empty history.
Axiom S:
((uo, 0, @>I = Pass(A)
For each operation of a typed object, Axioms C, I, R, and S are instantiated to
yield type-specific axioms.
For a given history H with m events, we use Possi(H) to denote the set of
possibilities for the ith prefix of H, for 0 I i 5 m. A derivation that shows that
(u, P, R) E Pass,(H) is a sequence of implications of the form:
(uo, PO, Ro) E Posse(H)
. . .
f (Uj, Pit Rj) E POSS,(H)
*...
where u, = u, P, = P, R, = R, and each implication is justified by Axiom C, I,
or R.
Intuitively, a derivation is like a history. Each implication in a derivation is
like a step in a proof, and each such step is justified by an axiom.
The axioms C, I, R, and S are sound:
THEOREM 9. If there exists a deriuation showing that (u, P, R) is a possibility
for H, then u is a linearized value for H.
Axioms C, I, R, and S are complete.
THEOREM 10. If u E Lin(H), then there exists a deriuation that (u, P, R) E
Pass(H).
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

488 l M. Herlihy and J. Wing
II. PROOF OF CORRECTNESS FOR THE QUEUE
11.1 Two Lemmas About Concurrent Queues
In a derivation, an Enq inference for x is an instantiation of Axiom C of the
form:
(qj, Pi, Rj) E PoSSk
3 (ins(qj, x), Pj - {Ens(x) A), Rj U (Ok( ) A)) E POSSE
A Deq inference is defined analogously.
Two inferences commute in a derivation if their order can be reversed without
invalidating the derivation. A derivation showing (q, P, R) E Poss, is in canonical
form if each Enq inference for an item in q occurs “as late as possible,” i.e., it
does not commute with the next inference in the derivation.
Lemma 11 implies that if x is in q, the event following the Enq inference for x
is either the return event for x, or the return event for an item that follows x
in q.
LEMMA 11. If 6 is a canonical derivation showing that (q, P, R) E Poss,,,, and
x is an item in q, then the inference following the Enq inference for x is either
the Enq inference for the item following x in q, or an application of Axiom R for
the matching response to Enq(x).
Lemma 12 states that we can consider equivalence classes of queues rather
than individual queues.
LEMMA 12. If (q, P, R) E Pass,, and q* is a queue value constructed by
rearranging the items of q in an order consistent with the partial precedence order
of their Enq operations, then (q *, P, R) E Poss,,, .
11.2 Main Proof
Figure 6 shows the Enq and Deq implementation annotated with assertions that
are true before and after each abstract invocation and response and each rep
operation. To avoid distraction, we assume queue values are unique. It is conven-
ient to keep as implicit auxiliary data the partial order, cr, on items in the array,
defined in Section 4.3.3. The set of possibilities, Poss, referred to in the annota-
tions can also be encoded as auxiliary data in terms of the sets, P (pending
invocations) and R (possible responses), which are components of a possibility.
If I is a set of items partially ordered by <, define:
and
(I, <) = (q 1 I = items(q) and < 5 c,J
((1, <), P, RI = ((4, P, R) I 4 E (I, 41.
The partially ordered set of queue items (I, <), captures the nonquiescent
abstract state of the queue, i.e., the possible values of the queue while there are
concurrent Enq and Deq operations or pending invocations. Notice that we can
rewrite the abstraction function as A(r) = (items(r), <,). The set [(I, <), P, R]
identifies each of the possible sets of queue values with a set of pending
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 489
lS(q, P, R) E Possl
Enq = proc (q: queue, x: item)
(3(q’, P’, R’) E Poss’ . q’ = q A P’ U (Enq(x) A) A R’ = R)
{3(q, P, R) E Poss . (En&) A) E PI
: int := INC(q.back)
;Posd = Poss)
(3(q, P, R) E Poss. (Enq(x) A) E Pj
STORE(q.items[i], x)
{3(q’, P’, R’) E Poss’ .P’ = P - (Enq(r) A} A R’ = R U (Ok() A}
A index(q.items’, x) = i A z E max(items(q’)) A q.back 5 q.back’)
(3(q, P, R) E Poss . (Ok() A) E RJ
end Enq
(3(q’, P’, R’) E Poss’.q’ = q A P’ = P A R’ = R - {Ok( ) A))
13(q, P, R) E Possl
Deq = Proc (q: queue) returns (item)
(3(q’, P’, R’) E Poss’ . q’ = q A P’ = P U {Deq( ) A) A R’ = R)
(3(q, P, R) E Poss . (Deq() A) E PJ
while true do
range : int := READ(q.back) - 1
(Poss’ = POSSJ
for i: int in 1 . . range do
(3(q, P, R) E Poss . (Deq() A) E P)
x: item := SWAP(q.items[i], null)
[3(q’, P’, R’) E Poss’ .P’ = P - (Deq( ) A) A R’ = R U {Ok(r) A) A
(z = null V z E min(items(q’)))l
if x -= null then return(x) end
end
end
end
(3(q, P, R) E Pass. (Ok(r) A) E R]
end Deq
(3(q’, P’, R’) E Poss’ . q’ = q A P’ = P A R’ = R - (Ok(x) A)]
Fig. 6. Annotated queue implementation.
invocations and a set of possible responses, thereby forming a set of (queue)
possibilities. The following two lemmas make use of Lemma 12, stated in the
previous section.
LEMMA 13. If x is a maximal element with respect to <, x 4 I, (Enq(x) A) B P,
(Ok() A) E R, and [(I, <), P U iEnq(x) A], R - (Ok() Al] C_ Pass, then
[(I U (xl, c), P, R] G Poss.
LEMMA 14. If (Deq( ) A) 4 P, (Ok(x) A) E R, and [(I, <), P U {Deq( ) Al,
R - (Ok(x) A]] C Poss, then for all x such that x is a minimal element of I,
[(I - (x), 4, P, R] C Pass.
Lemma 13 will allow us to show that the set of linearized queue values does not
change over a STORE operation and similarly, Lemma 14, for a SWAP operation,
by using cr for < and by recalling that for each (u, P, R) E Poss, u is a linearized
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

490 l M. Herlihy and J. Wing
value. We use the next two lemmas to satisfy the conditions of the previous two
lemmas.
LEMMA 15. Enq enqueues an item x that is maximal with respect to c,..
LEMMA 16. Deq removes and returns an item x that is minimal with respect
to -cr.
Here is a proof of correctness.
THEOREM 17. The queue implementation is correct.
PROOF. Assuming every rep history is linearizable, we need to show that every
queue history, H 1 q, is linearizable. It suffices to show that the “subset” property,
UrELin(H ( r) A(r) C Lin(H 1 q), remains invariant over abstract invocation and
responses and over complete rep operations. Thus, it can be conjoined to the pre-
and post conditions of Figure 6 as justified by the Owicki-Gries proof method
[38]. Axioms I and R give us the result for abstract invocation and response
events. INC and READ leave the abstraction function the same. Thus, we are
left with two cases, STORE and SWAP. By Lemma 15 we know that STORE
adds a maximal item and thus, we can apply Lemma 13 to show that the subset
property is preserved. Similarly, by Lemma 16 we know that SWAP removes a
minimal item and thus, we can apply Lemma 14 to show that the subset property
is preserved. 0
ACKNOWLEDGMENTS
The authors thank Jim Horning, Leslie Lamport, Larry Rudolph, and William
Weihl for lively verbal and electronic discussions about our notions of lineariza-
bility and correctness. We also thank James Aspnes, Stewart Clamen, David
Detlefs, Richard Lerner, and Mark Maimone for their comments on earlier
versions of this paper. Finally, we would like to thank Jim Gray and the
anonymous referees for their comments and suggestions.
REFERENCES
1. ABADI, M., AND LAMPORT, L. The existence of refinement mappings. Tech. Rep. 29, DEC
Systems Research Center, Aug. 1988.
2. GOGUEN, J. A., THATCHER, J. W., WAGNER, E. G., AND WRIGHT, J. B. Abstract data types as
initial algebras and correctness of data representations. In Proceedings of the Conference on
Computer Graphics, Pattern Recognition and Data Structures (May 1975). ACM, New York, 1975,
89-93.
3. APT, K. R., FRANCEZ, N., AND DEROEVER, W. P. A proof system for communicating sequential
processes. ACM Trans. Program. Lang. Syst. 2, 3 (July 1980), 359-385.
4. BAYER, R., AND SCHKOLNICK, M. Concurrency of operations on B-trees. Acta Znf. I, 1 (1977),
1-21.
5. BIRKHOFF, G., AND LIPSON, J. D. Heterogeneous algebras. J. Comb. Z’heor. 8 (1970), 115-133.
6. BISWAS, J., AND BROWNE, J. C. Simultaneous update of priority structures. In Proceedings of
the 1987 International Conference on Parallel Processing (St. Charles, Ill., 1987). 124-131.
7. BROOKES, S. D. An axiomatic treatment of a parallel language. In Proceedings of Conference on
Logics of Programs. Lecture Notes in Computer Science. Vol. 193. Springer-Verlag, Berlin, 1985.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Linearizability: A Correctness Condition for Concurrent Objects 491
8. BURSTALL, R. M., AND GOGUEN, J. A. Putting theories together to make specifications. In
Fifth International Joint Conference on Artificial Intelligence (Cambridge, Mass., Aug. 1977).
1045-1058. Invited paper.
9. DEPARTMENT OF DEFENSE. Reference Manual for the ADA Programming Language. ANSI/
MIL-STD-1815A-1983,1983.
10. EHRIG, H., AND MAHR, B. Fundamentals of Algebraic Specification 1. Springer-Verlag, Berlin,
1985.
11. ELLIS, C. S. Concurrent search and insertion in 2-3 trees. Acta Znf. 14 (1980), 63-86.
12. ESWARAN, K. P., GRAY, J. N., LORIE, R. A., AND TRAIGER, I. L. The notion of consistency and
predicate locks in a database system. Commun. ACM 19, 11 (Nov. 1976), 624-633.
13. FORD, R., AND CALHOUN, J. Concurrency control mechanisms and the serializability of concur-
rent tree algorithms. In 3rd ACM Symposium on Principles of Database Systems (1984). ACM,
New York, 1984,51-60.
14. GERTH, R., AND DEROEVER, W. P. Proving monitors revisited: A first step towards verifying
object oriented systems. Fundamental Znf. 9 (1986), 371-400.
15. GOTTLIEB, A., LUBACHEVSKY, B. D., AND RUDOLPH, L. Basic techniques for the efficient
coordination of very large numbers of cooperating sequential processors. ACM Trans. Program.
Lang. Syst. 5, 2 (April 1983), 164-189.
16. GUIBAS, L., AND SEDGEWICK, R. A dichromatic framework for balanced trees. In 19th ACM
Symposium on Foundations of Computer Science (Providence, R.I., 1978). ACM, New York, 1978,
8-21.
17. GUTTAG, J. V. The specification and application to programming of abstract data types. Ph.D.
thesis, Univ. of Toronto, Toronto, Sept. 1975.
18. GUTTAG, 3. V., HOROWITZ, E., AND MUSSER, D. R. Abstract data types and software validation.
Commun. ACM 21,12 (Dec. 1978), 1048-1064.
19. GUTTAG, J. V., HORNING, J. J., AND WING, J. M. Larch in five easy pieces. Tech. Rep. 5, DEC
Systems Research Center, July 1985.
20. GU~AG, J. V., HORNING, J. J., AND WING, J. M. The Larch family of specification languages.
IEEE Softw. 2,5 (Sept. 1985), 24-36.
21. HERLIHY, M., AND WING, J. Axioms for concurrent objects. In 14th ACM Symposium on
Principles of Programming Languages (Jan. 1987). ACM, New York, 1987, 13-26.
22. HERLIHY, M. P. Dynamic quorum adjustment for partitioned data. ACM Trans. Database Syst.
12,2 (June 1987), 170-194.
23. HERLIHY, M. P. Impossibility and universality results for wait-free synchronization. In Seuenth
ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing (PODC) (Toronto,
Ont., Aug. 1988). ACM, New York, 1988, 276-290.
24. HOARE, C. A. R. An axiomatic basis for computer programming. Commun. ACM 12, 10 (Oct.
1969), 576-583.
25. HOARE, C. A. R. Proof of correctness of data representations. Acta Inf. 1, 1 (1972), 271-281.
26. HOARE, C. A. R. Monitors: An operating system structuring concept. Commun. ACM 17, 10
(Oct. 1974), 549-557.
27. HOARE, C. A. R. Communicating sequential processes. Commun. ACM 21, 8 (Aug. 1978),
666-677.
28. HOWARD, J. H. Proving monitors. Commun. ACM 19,5 (May 1976), 273-279.
29. JONES, C. B. Software Development: A Rigorous Approach. Prentice-Hall, Englewood Cliffs,
N.J., 1980.
30. KRUSKAL, C. P., RUDOLPH, L., AND SNIR, M. Efficient synchronization on multiprocessors
with shared memory. In Fifth ACM SZGACT-SZGOPS Symposium on Principles of Distributed
Computing (Aug. 1986). ACM, New York, 1986.
31. LAMPORT, L. How to make a multiprocessor computer that correctly executes multiprocess
programs. IEEE Trans. Comput. C-28,9 (Sept. 1979), 690-691.
32. LAMPORT, L. Specifying concurrent program modules. ACM Trans. Program. Lang. Syst. 5, 2
(April 1983), 190-222.
33. LEHMAN, P. L., AND YAO, S. B. Efficient locking for concurrent operations on B-trees. ACM
Trans. Database Syst. 6, 4 (Dec. 1981), 650-670.
34. MANNA, Z., AND PNUELI, A. Verification of concurrent programs, Part I: The temporal frame-
work. Tech. Rep. STAN-CS-81-836, Dept. of Computer Science, Stanford Univ., June 1981.
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

492 l M. Herlihy and J. Wing
35. MISRA, J. Axioms for memory access in asynchronous hardware systems. ACM Trans. Program.
Lang. Syst. 8, 1 (Jan. 1986), 142-153.
36. NAKAJIMA, R., HONDA, M., AND NAKAHARA, H. Hierarchical program specification and verifi-
cation-A many-sorted logical approach. Acta Znf. 14 (1980), 135-155.
37. OWICKI, S., AND GRIES, D. Verifying properties of parallel programs: An axiomatic approach.
Commun. ACM 19,5 (May 1976), 279-285.
38. OWICKI, S., AND GRIES, D. An axiomatic proof technique for parallel programs. Acta Znf. 6, 4
(1976), 319-340.
39. OWICKI, S., AND LAMPORT, L. Proving liveness properties of concurrent programs. ACM Trans.
Program. Lang. Syst. 4, 3 (July 1982), 455-495.
40. PAPADIMITRIOU, C. H. The serializability of concurrent database updates. J. ACM 26, 4 (Oct.
1979), 631-653.
41. REED, D. P. Implementing atomic actions on decentralized data. ACM Trans. Comput. Syst. 1,
1 (Feb. 1983), 3-23.
42. SAGIV, Y. Concurrent operations on B-trees with overtaking. In Symposium on Principles of
Database Sy&ems (Waterloo, Ont., Jan. 1985). ACM, New York, 1985, 28-37.
43. STANDISH, T. A. Data structures: An axiomatic approach. Rep. 2639, Bolt, Beranek, and
Newman, Cambridge, Mass., Aug. 1973.
44. STIRLING, C. A generalization of Owicki-Gries-Hoare logic for a concurrent while language.
Tech. Rep., Edinburgh Univ., March 1987.
45. HERLIHY, M. P., AND WING, J. M. Axioms for concurrent objects. Tech. Rep. CMU-CS-86-154,
Computer Science Dept., Carnegie Mellon Univ., 1986.
46. WEIHL, W. E. Local atomicity properties: Modular concurrent control for abstract data types.
ACM Trans. Program. Lang. Syst. 11, 2 (April 1989), 249-283.
Received January 1988, revised November 1988 and July 1989, accepted October 1989
ACM Transactions on Programming Languages and Systems, Vol. 12, No. 3, July 1990.

Paxos FAQ

Q: How does Paxos prevent split-brain?

A: For Paxos, split brain means that different servers "agree" on
different values. The places in Paxos where it must wait for a
majority to respond are enough to prevent this. If there are two
active proposers, the Paxos algorithm requires them both to get
prepare replies from a majority of acceptors, and accept replies from
a majority of acceptors. The two proposers' majorities must overlap in
at least one server; the reply from that server will tell the losing
proposer (the one with the smaller n) that it has lost, or (if the
lower number proposer has already reached agreement) will tell the
higher-numbered server what the agreed value was.

If a network failure has partitioned the servers, then at most one of
the partitions can possibly contain a majority of the servers. So
Paxos will only be able to reach agreement in that partition. Servers
in the other partitions won't be able to agree on anything. Thus split
brain is avoided under network partition.

Q: What are some ways to ensure unique proposal numbers?

A: A proposer can put an ID that's different for each proposer in the
low bits of the proposal number. For example, a proposer could put its
IP address in the low bits. Then different proposers will never use
the same proposal number. Each proposer can avoid re-using the same
proposal number by keeping a variable holding the highest proposal
number it has used so far, and, when it starts a new proposal, make
sure the new proposal number is higher.

If you are feeling lucky, you could use numbers that have the current
time in the high bits and a random number in the low bits. You'd want
lots of bits for the random number to keep the probability of
collision super low.

Q: In 2.4, what is the distinguished proposer about?

A: The idea is that the implementation should try to make it likely
that there's at most one active proposal at a time. This is not a
requirement for correctness; it is advice about how to reach agreement
faster. One way to do this is to have each proposer wait a random
amount of time before sending prepare messages. Then there the
proposer who chose the smallest random number will go first and will
likely complete agreement before the other proposers finish their
sleeping. There can still be more than one proposer, but this
randomization scheme breaks the symmetry among the proposers and
causes it to be likely that one of them wins.

Q: How does the algorithm terminate?

A: A Paxos agreement doesn't have a well-defined termination point.
However, if there are no failures, a proposer can observe that
agreement has been reached (because it gets accept replies from
everyone), and it can safely stop proposing at that point.

Q: What are some use cases for Paxos?

A: Paxos helps you build replicated services, which can increase fault
tolerance. For example, one can build a Paxos-replicated database with
three servers, such that if one of the servers fails, the other two
can continue to execute database requests.

You can see various uses if you search the web for papers about Chubby,
Spanner, Megastore, Spinnaker, and Zookeeper.

Q: How fast is Paxos?

A: There are two main limits to performance: message exchanges, and
writing updated n and v information to disk (so they can be recovered
if the server crashes and restarts). Paxos requires at least two
message exchanges in order to agree; if all hosts are within the same
datacenter this might take a few hundred microseconds. Paxos also
requires two writes to the disk (one for prepare, one for accept);
depending on disk technology this could take anywhere from a few
hundred microseconds to a few tens of milliseconds.

So without clever optimizations Paxos will be relatively slow. This
paper describes some of what's needed for good performance:

  http://research.google.com/archive/paxos_made_live.html

Q: What does the paper mean by a leader on page 9? Does Paxos have a
leader?

A: Section 3 of the paper is not about Paxos itself, but about one way to
implement and use Paxos. Paxos itself doesn't require a leader. I do not
know whether any real-world Paxos implementations use the ideas in
Section 3. It may be that when the paper speaks of electing a leader, it
means using Paxos to agree on a leader. Or one could use any of a number
of schemes that usually (but not always) result in agreement on a leader
(since Paxos will be correct regardless), such as letting the
participant with the lowest IP address propose with no delay, but
requiring other participants to sleep for a random amount of time before
proposing. Much of the point of a leader is to avoid simultaneous
different proposals, which require multiple Paxos rounds to sort out.

The best paper I know about real-life implementation and use of Paxos is
Paxos Made Live:

  http://research.google.com/archive/paxos_made_live.html

Section 5.2 on master leases may be the most relevant.

Paxos Made Simple
Leslie Lamport
01 Nov 2001

Abstract
The Paxos algorithm, when presented in plain English, is very simple.

Contents
1 Introduction 1
2 The Consensus Algorithm 1
2.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2.2 Choosing a Value . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.3 Learning a Chosen Value . . . . . . . . . . . . . . . . . . . . . 6
2.4 Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 The Implementation . . . . . . . . . . . . . . . . . . . . . . . 7
3 Implementing a State Machine 8
References 11

1 Introduction
The Paxos algorithm for implementing a fault-tolerant distributed system
has been regarded as difficult to understand, perhaps because the original
presentation was Greek to many readers [5]. In fact, it is among the sim-
plest and most obvious of distributed algorithms. At its heart is a consensus
algorithm—the “synod” algorithm of [5]. The next section shows that this
consensus algorithm follows almost unavoidably from the properties we want
it to satisfy. The last section explains the complete Paxos algorithm, which
is obtained by the straightforward application of consensus to the state ma-
chine approach for building a distributed system—an approach that should
be well-known, since it is the subject of what is probably the most often-cited
article on the theory of distributed systems [4].
2 The Consensus Algorithm
2.1 The Problem
Assume a collection of processes that can propose values. A consensus al-
gorithm ensures that a single one among the proposed values is chosen. If
no value is proposed, then no value should be chosen. If a value has been
chosen, then processes should be able to learn the chosen value. The safety
requirements for consensus are:
• Only a value that has been proposed may be chosen,
• Only a single value is chosen, and
• A process never learns that a value has been chosen unless it actually
has been.
We won’t try to specify precise liveness requirements. However, the goal is
to ensure that some proposed value is eventually chosen and, if a value has
been chosen, then a process can eventually learn the value.
We let the three roles in the consensus algorithm be performed by three
classes of agents: proposers, acceptors, and learners. In an implementation,
a single process may act as more than one agent, but the mapping from
agents to processes does not concern us here.
Assume that agents can communicate with one another by sending mes-
sages. We use the customary asynchronous, non-Byzantine model, in which:
1

• Agents operate at arbitrary speed, may fail by stopping, and may
restart. Since all agents may fail after a value is chosen and then
restart, a solution is impossible unless some information can be re-
membered by an agent that has failed and restarted.
• Messages can take arbitrarily long to be delivered, can be duplicated,
and can be lost, but they are not corrupted.
2.2 Choosing a Value
The easiest way to choose a value is to have a single acceptor agent. A pro-
poser sends a proposal to the acceptor, who chooses the first proposed value
that it receives. Although simple, this solution is unsatisfactory because the
failure of the acceptor makes any further progress impossible.
So, let’s try another way of choosing a value. Instead of a single acceptor,
let’s use multiple acceptor agents. A proposer sends a proposed value to a
set of acceptors. An acceptor may accept the proposed value. The value is
chosen when a large enough set of acceptors have accepted it. How large is
large enough? To ensure that only a single value is chosen, we can let a large
enough set consist of any majority of the agents. Because any two majorities
have at least one acceptor in common, this works if an acceptor can accept
at most one value. (There is an obvious generalization of a majority that
has been observed in numerous papers, apparently starting with [3].)
In the absence of failure or message loss, we want a value to be chosen
even if only one value is proposed by a single proposer. This suggests the
requirement:
P1. An acceptor must accept the first proposal that it receives.
But this requirement raises a problem. Several values could be proposed by
different proposers at about the same time, leading to a situation in which
every acceptor has accepted a value, but no single value is accepted by a
majority of them. Even with just two proposed values, if each is accepted by
about half the acceptors, failure of a single acceptor could make it impossible
to learn which of the values was chosen.
P1 and the requirement that a value is chosen only when it is accepted
by a majority of acceptors imply that an acceptor must be allowed to accept
more than one proposal. We keep track of the different proposals that an
acceptor may accept by assigning a (natural) number to each proposal, so a
proposal consists of a proposal number and a value. To prevent confusion,
we require that different proposals have different numbers. How this is
2

achieved depends on the implementation, so for now we just assume it. A
value is chosen when a single proposal with that value has been accepted by
a majority of the acceptors. In that case, we say that the proposal (as well
as its value) has been chosen.
We can allow multiple proposals to be chosen, but we must guarantee
that all chosen proposals have the same value. By induction on the proposal
number, it suffices to guarantee:
P2. If a proposal with value v is chosen, then every higher-numbered pro-
posal that is chosen has value v .
Since numbers are totally ordered, condition P2 guarantees the crucial safety
property that only a single value is chosen.
To be chosen, a proposal must be accepted by at least one acceptor. So,
we can satisfy P2 by satisfying:
P2a . If a proposal with value v is chosen, then every higher-numbered pro-
posal accepted by any acceptor has value v .
We still maintain P1 to ensure that some proposal is chosen. Because com-
munication is asynchronous, a proposal could be chosen with some particu-
lar acceptor c never having received any proposal. Suppose a new proposer
“wakes up” and issues a higher-numbered proposal with a different value.
P1 requires c to accept this proposal, violating P2 a . Maintaining both P1
and P2 a requires strengthening P2 a to:
P2b. If a proposal with value v is chosen, then every higher-numbered pro-
posal issued by any proposer has value v .
Since a proposal must be issued by a proposer before it can be accepted by
an acceptor, P2 b implies P2 a , which in turn implies P 2.
To discover how to satisfy P2 b, let’s consider how we would prove that
it holds. We would assume that some proposal with number m and value
v is chosen and show that any proposal issued with number n > m also
has value v . We would make the proof easier by using induction on n,
so we can prove that proposal number n has value v under the additional
assumption that every proposal issued with a number in m . . (n − 1) has
value v , where i . . j denotes the set of numbers from i through j . For the
proposal numbered m to be chosen, there must be some set C consisting of a
majority of acceptors such that every acceptor in C accepted it. Combining
this with the induction assumption, the hypothesis that m is chosen implies:
3

Every acceptor in C has accepted a proposal with number in
m . . (n − 1), and every proposal with number in m . . (n − 1)
accepted by any acceptor has value v .
Since any set S consisting of a majority of acceptors contains at least one
member of C , we can conclude that a proposal numbered n has value v by
ensuring that the following invariant is maintained:
P2c. For any v and n, if a proposal with value v and number n is issued,
then there is a set S consisting of a majority of acceptors such that
either (a) no acceptor in S has accepted any proposal numbered less
than n, or (b) v is the value of the highest-numbered proposal among
all proposals numbered less than n accepted by the acceptors in S .
We can therefore satisfy P2 b by maintaining the invariance of P2 c.
To maintain the invariance of P2 c, a proposer that wants to issue a pro-
posal numbered n must learn the highest-numbered proposal with number
less than n, if any, that has been or will be accepted by each acceptor in
some majority of acceptors. Learning about proposals already accepted is
easy enough; predicting future acceptances is hard. Instead of trying to pre-
dict the future, the proposer controls it by extracting a promise that there
won’t be any such acceptances. In other words, the proposer requests that
the acceptors not accept any more proposals numbered less than n. This
leads to the following algorithm for issuing proposals.
1. A proposer chooses a new proposal number n and sends a request to
each member of some set of acceptors, asking it to respond with:
(a) A promise never again to accept a proposal numbered less than
n, and
(b) The proposal with the highest number less than n that it has
accepted, if any.
I will call such a request a prepare request with number n.
2. If the proposer receives the requested responses from a majority of
the acceptors, then it can issue a proposal with number n and value
v , where v is the value of the highest-numbered proposal among the
responses, or is any value selected by the proposer if the responders
reported no proposals.
4

A proposer issues a proposal by sending, to some set of acceptors, a request
that the proposal be accepted. (This need not be the same set of acceptors
that responded to the initial requests.) Let’s call this an accept request.
This describes a proposer’s algorithm. What about an acceptor? It can
receive two kinds of requests from proposers: prepare requests and accept
requests. An acceptor can ignore any request without compromising safety.
So, we need to say only when it is allowed to respond to a request. It can
always respond to a prepare request. It can respond to an accept request,
accepting the proposal, iff it has not promised not to. In other words:
P1a . An acceptor can accept a proposal numbered n iff it has not responded
to a prepare request having a number greater than n.
Observe that P1 a subsumes P1.
We now have a complete algorithm for choosing a value that satisfies the
required safety properties—assuming unique proposal numbers. The final
algorithm is obtained by making one small optimization.
Suppose an acceptor receives a prepare request numbered n, but it has
already responded to a prepare request numbered greater than n, thereby
promising not to accept any new proposal numbered n. There is then no
reason for the acceptor to respond to the new prepare request, since it will
not accept the proposal numbered n that the proposer wants to issue. So
we have the acceptor ignore such a prepare request. We also have it ignore
a prepare request for a proposal it has already accepted.
With this optimization, an acceptor needs to remember only the highest-
numbered proposal that it has ever accepted and the number of the highest-
numbered prepare request to which it has responded. Because P2 c must
be kept invariant regardless of failures, an acceptor must remember this
information even if it fails and then restarts. Note that the proposer can
always abandon a proposal and forget all about it—as long as it never tries
to issue another proposal with the same number.
Putting the actions of the proposer and acceptor together, we see that
the algorithm operates in the following two phases.
Phase 1. (a) A proposer selects a proposal number n and sends a prepare
request with number n to a majority of acceptors.
(b) If an acceptor receives a prepare request with number n greater
than that of any prepare request to which it has already responded,
then it responds to the request with a promise not to accept any more
proposals numbered less than n and with the highest-numbered pro-
posal (if any) that it has accepted.
5

Phase 2. (a) If the proposer receives a response to its prepare requests
(numbered n) from a majority of acceptors, then it sends an accept
request to each of those acceptors for a proposal numbered n with a
value v , where v is the value of the highest-numbered proposal among
the responses, or is any value if the responses reported no proposals.
(b) If an acceptor receives an accept request for a proposal numbered
n, it accepts the proposal unless it has already responded to a prepare
request having a number greater than n.
A proposer can make multiple proposals, so long as it follows the algorithm
for each one. It can abandon a proposal in the middle of the protocol at any
time. (Correctness is maintained, even though requests and/or responses
for the proposal may arrive at their destinations long after the proposal
was abandoned.) It is probably a good idea to abandon a proposal if some
proposer has begun trying to issue a higher-numbered one. Therefore, if an
acceptor ignores a prepare or accept request because it has already received
a prepare request with a higher number, then it should probably inform
the proposer, who should then abandon its proposal. This is a performance
optimization that does not affect correctness.
2.3 Learning a Chosen Value
To learn that a value has been chosen, a learner must find out that a pro-
posal has been accepted by a majority of acceptors. The obvious algorithm
is to have each acceptor, whenever it accepts a proposal, respond to all
learners, sending them the proposal. This allows learners to find out about
a chosen value as soon as possible, but it requires each acceptor to respond
to each learner—a number of responses equal to the product of the number
of acceptors and the number of learners.
The assumption of non-Byzantine failures makes it easy for one learner
to find out from another learner that a value has been accepted. We can
have the acceptors respond with their acceptances to a distinguished learner,
which in turn informs the other learners when a value has been chosen. This
approach requires an extra round for all the learners to discover the chosen
value. It is also less reliable, since the distinguished learner could fail. But
it requires a number of responses equal only to the sum of the number of
acceptors and the number of learners.
More generally, the acceptors could respond with their acceptances to
some set of distinguished learners, each of which can then inform all the
learners when a value has been chosen. Using a larger set of distinguished
6

learners provides greater reliability at the cost of greater communication
complexity.
Because of message loss, a value could be chosen with no learner ever
finding out. The learner could ask the acceptors what proposals they have
accepted, but failure of an acceptor could make it impossible to know whether
or not a majority had accepted a particular proposal. In that case, learners
will find out what value is chosen only when a new proposal is chosen. If
a learner needs to know whether a value has been chosen, it can have a
proposer issue a proposal, using the algorithm described above.
2.4 Progress
It’s easy to construct a scenario in which two proposers each keep issuing
a sequence of proposals with increasing numbers, none of which are ever
chosen. Proposer p completes phase 1 for a proposal number n 1. Another
proposer q then completes phase 1 for a proposal number n 2 > n 1. Proposer
p’s phase 2 accept requests for a proposal numbered n 1 are ignored because
the acceptors have all promised not to accept any new proposal numbered
less than n 2. So, proposer p then begins and completes phase 1 for a new
proposal number n 3 > n 2, causing the second phase 2 accept requests of
proposer q to be ignored. And so on.
To guarantee progress, a distinguished proposer must be selected as the
only one to try issuing proposals. If the distinguished proposer can com-
municate successfully with a majority of acceptors, and if it uses a proposal
with number greater than any already used, then it will succeed in issuing a
proposal that is accepted. By abandoning a proposal and trying again if it
learns about some request with a higher proposal number, the distinguished
proposer will eventually choose a high enough proposal number.
If enough of the system (proposer, acceptors, and communication net-
work) is working properly, liveness can therefore be achieved by electing a
single distinguished proposer. The famous result of Fischer, Lynch, and Pat-
terson [1] implies that a reliable algorithm for electing a proposer must use
either randomness or real time—for example, by using timeouts. However,
safety is ensured regardless of the success or failure of the election.
2.5 The Implementation
The Paxos algorithm [5] assumes a network of processes. In its consensus
algorithm, each process plays the role of proposer, acceptor, and learner.
The algorithm chooses a leader, which plays the roles of the distinguished
7

proposer and the distinguished learner. The Paxos consensus algorithm is
precisely the one described above, where requests and responses are sent as
ordinary messages. (Response messages are tagged with the corresponding
proposal number to prevent confusion.) Stable storage, preserved during
failures, is used to maintain the information that the acceptor must re-
member. An acceptor records its intended response in stable storage before
actually sending the response.
All that remains is to describe the mechanism for guaranteeing that no
two proposals are ever issued with the same number. Different proposers
choose their numbers from disjoint sets of numbers, so two different pro-
posers never issue a proposal with the same number. Each proposer remem-
bers (in stable storage) the highest-numbered proposal it has tried to issue,
and begins phase 1 with a higher proposal number than any it has already
used.
3 Implementing a State Machine
A simple way to implement a distributed system is as a collection of clients
that issue commands to a central server. The server can be described as
a deterministic state machine that performs client commands in some se-
quence. The state machine has a current state; it performs a step by taking
as input a command and producing an output and a new state. For ex-
ample, the clients of a distributed banking system might be tellers, and
the state-machine state might consist of the account balances of all users.
A withdrawal would be performed by executing a state machine command
that decreases an account’s balance if and only if the balance is greater than
the amount withdrawn, producing as output the old and new balances.
An implementation that uses a single central server fails if that server
fails. We therefore instead use a collection of servers, each one independently
implementing the state machine. Because the state machine is deterministic,
all the servers will produce the same sequences of states and outputs if they
all execute the same sequence of commands. A client issuing a command
can then use the output generated for it by any server.
To guarantee that all servers execute the same sequence of state machine
commands, we implement a sequence of separate instances of the Paxos
consensus algorithm, the value chosen by the i th instance being the i th state
machine command in the sequence. Each server plays all the roles (proposer,
acceptor, and learner) in each instance of the algorithm. For now, I assume
that the set of servers is fixed, so all instances of the consensus algorithm
8

use the same sets of agents.
In normal operation, a single server is elected to be the leader, which
acts as the distinguished proposer (the only one that tries to issue proposals)
in all instances of the consensus algorithm. Clients send commands to the
leader, who decides where in the sequence each command should appear.
If the leader decides that a certain client command should be the 135 th
command, it tries to have that command chosen as the value of the 135 th
instance of the consensus algorithm. It will usually succeed. It might fail
because of failures, or because another server also believes itself to be the
leader and has a different idea of what the 135 th command should be. But
the consensus algorithm ensures that at most one command can be chosen
as the 135 th one.
Key to the efficiency of this approach is that, in the Paxos consensus
algorithm, the value to be proposed is not chosen until phase 2. Recall that,
after completing phase 1 of the proposer’s algorithm, either the value to be
proposed is determined or else the proposer is free to propose any value.
I will now describe how the Paxos state machine implementation works
during normal operation. Later, I will discuss what can go wrong. I consider
what happens when the previous leader has just failed and a new leader has
been selected. (System startup is a special case in which no commands have
yet been proposed.)
The new leader, being a learner in all instances of the consensus algo-
rithm, should know most of the commands that have already been chosen.
Suppose it knows commands 1–134, 138, and 139—that is, the values cho-
sen in instances 1–134, 138, and 139 of the consensus algorithm. (We will
see later how such a gap in the command sequence could arise.) It then
executes phase 1 of instances 135–137 and of all instances greater than 139.
(I describe below how this is done.) Suppose that the outcome of these ex-
ecutions determine the value to be proposed in instances 135 and 140, but
leaves the proposed value unconstrained in all other instances. The leader
then executes phase 2 for instances 135 and 140, thereby choosing commands
135 and 140.
The leader, as well as any other server that learns all the commands
the leader knows, can now execute commands 1–135. However, it can’t
execute commands 138–140, which it also knows, because commands 136
and 137 have yet to be chosen. The leader could take the next two commands
requested by clients to be commands 136 and 137. Instead, we let it fill the
gap immediately by proposing, as commands 136 and 137, a special “no-
op” command that leaves the state unchanged. (It does this by executing
phase 2 of instances 136 and 137 of the consensus algorithm.) Once these
9

no-op commands have been chosen, commands 138–140 can be executed.
Commands 1–140 have now been chosen. The leader has also completed
phase 1 for all instances greater than 140 of the consensus algorithm, and it
is free to propose any value in phase 2 of those instances. It assigns command
number 141 to the next command requested by a client, proposing it as the
value in phase 2 of instance 141 of the consensus algorithm. It proposes the
next client command it receives as command 142, and so on.
The leader can propose command 142 before it learns that its proposed
command 141 has been chosen. It’s possible for all the messages it sent
in proposing command 141 to be lost, and for command 142 to be chosen
before any other server has learned what the leader proposed as command
141. When the leader fails to receive the expected response to its phase 2
messages in instance 141, it will retransmit those messages. If all goes well,
its proposed command will be chosen. However, it could fail first, leaving a
gap in the sequence of chosen commands. In general, suppose a leader can
get α commands ahead—that is, it can propose commands i + 1 through
i +α after commands 1 through i are chosen. A gap of up to α−1 commands
could then arise.
A newly chosen leader executes phase 1 for infinitely many instances
of the consensus algorithm—in the scenario above, for instances 135–137
and all instances greater than 139. Using the same proposal number for
all instances, it can do this by sending a single reasonably short message
to the other servers. In phase 1, an acceptor responds with more than a
simple OK only if it has already received a phase 2 message from some
proposer. (In the scenario, this was the case only for instances 135 and
140.) Thus, a server (acting as acceptor) can respond for all instances with
a single reasonably short message. Executing these infinitely many instances
of phase 1 therefore poses no problem.
Since failure of the leader and election of a new one should be rare
events, the effective cost of executing a state machine command—that is, of
achieving consensus on the command/value—is the cost of executing only
phase 2 of the consensus algorithm. It can be shown that phase 2 of the
Paxos consensus algorithm has the minimum possible cost of any algorithm
for reaching agreement in the presence of faults [2]. Hence, the Paxos algo-
rithm is essentially optimal.
This discussion of the normal operation of the system assumes that there
is always a single leader, except for a brief period between the failure of the
current leader and the election of a new one. In abnormal circumstances,
the leader election might fail. If no server is acting as leader, then no new
commands will be proposed. If multiple servers think they are leaders, then
10

they can all propose values in the same instance of the consensus algo-
rithm, which could prevent any value from being chosen. However, safety is
preserved—two different servers will never disagree on the value chosen as
the i th state machine command. Election of a single leader is needed only
to ensure progress.
If the set of servers can change, then there must be some way of deter-
mining what servers implement what instances of the consensus algorithm.
The easiest way to do this is through the state machine itself. The current
set of servers can be made part of the state and can be changed with ordi-
nary state-machine commands. We can allow a leader to get α commands
ahead by letting the set of servers that execute instance i + α of the con-
sensus algorithm be specified by the state after execution of the i th state
machine command. This permits a simple implementation of an arbitrarily
sophisticated reconfiguration algorithm.
References
[1] Michael J. Fischer, Nancy Lynch, and Michael S. Paterson. Impossibility
of distributed consensus with one faulty process. Journal of the ACM ,
32(2):374–382, April 1985.
[2] Idit Keidar and Sergio Rajsbaum. On the cost of fault-tolerant consensus
when there are no faults—a tutorial. TechnicalReport MIT-LCS-TR-821,
Laboratory for Computer Science, Massachusetts Institute Technology,
Cambridge, MA, 02139, May 2001. also published in SIGACT News
32(2) (June 2001).
[3] Leslie Lamport. The implementation of reliable distributed multiprocess
systems. Computer Networks , 2:95–114, 1978.
[4] Leslie Lamport. Time, clocks, and the ordering of events in a distributed
system. Communications of the ACM , 21(7):558–565, July 1978.
[5] Leslie Lamport. The part-time parliament. ACM Transactions on Com-
puter Systems , 16(2):133–169, May 1998.
11

In Search of an Understandable Consensus Algorithm
(Extended Version)
Diego Ongaro and John Ousterhout
Stanford University
Abstract
Raft is a consensus algorithm for managing a replicated
log. It produces a result equivalent to (multi-)Paxos, and
it is as efficient as Paxos, but its structure is different
from Paxos; this makes Raft more understandable than
Paxos and also provides a better foundation for build-
ing practical systems. In order to enhance understandabil-
ity, Raft separates the key elements of consensus, such as
leader election, log replication, and safety, and it enforc es
a stronger degree of coherency to reduce the number of
states that must be considered. Results from a user study
demonstrate that Raft is easier for students to learn than
Paxos. Raft also includes a new mechanism for changing
the cluster membership, which uses overlapping majori-
ties to guarantee safety.
1 Introduction
Consensus algorithms allow a collection of machines
to work as a coherent group that can survive the fail-
ures of some of its members. Because of this, they play a
key role in building reliable large-scale software systems .
Paxos [15, 16] has dominated the discussion of consen-
sus algorithms over the last decade: most implementations
of consensus are based on Paxos or influenced by it, and
Paxos has become the primary vehicle used to teach stu-
dents about consensus.
Unfortunately, Paxos is quite difficult to understand, in
spite of numerous attempts to make it more approachable.
Furthermore, its architecture requires complex changes
to support practical systems. As a result, both system
builders and students struggle with Paxos.
After struggling with Paxos ourselves, we set out to
find a new consensus algorithm that could provide a bet-
ter foundation for system building and education. Our ap-
proach was unusual in that our primary goal was under-
standability: could we define a consensus algorithm for
practical systems and describe it in a way that is signifi-
cantly easier to learn than Paxos? Furthermore, we wanted
the algorithm to facilitate the development of intuitions
that are essential for system builders. It was important not
just for the algorithm to work, but for it to be obvious why
it works.
The result of this work is a consensus algorithm called
Raft. In designing Raft we applied specific techniques to
improve understandability, including decomposition (Raft
separates leader election, log replication, and safety) an d
This tech report is an extended version of [32]; additional m aterial is
noted with a gray bar in the margin. Published May 20, 2014.
state space reduction (relative to Paxos, Raft reduces the
degree of nondeterminism and the ways servers can be in-
consistent with each other). A user study with 43 students
at two universities shows that Raft is significantly easier
to understand than Paxos: after learning both algorithms,
33 of these students were able to answer questions about
Raft better than questions about Paxos.
Raft is similar in many ways to existing consensus al-
gorithms (most notably, Oki and Liskov’s Viewstamped
Replication [29, 22]), but it has several novel features:
• Strong leader: Raft uses a stronger form of leader-
ship than other consensus algorithms. For example,
log entries only flow from the leader to other servers.
This simplifies the management of the replicated log
and makes Raft easier to understand.
• Leader election: Raft uses randomized timers to
elect leaders. This adds only a small amount of
mechanism to the heartbeats already required for any
consensus algorithm, while resolving conflicts sim-
ply and rapidly.
• Membership changes: Raft’s mechanism for
changing the set of servers in the cluster uses a new
joint consensus approach where the majorities of
two different configurations overlap during transi-
tions. This allows the cluster to continue operating
normally during configuration changes.
We believe that Raft is superior to Paxos and other con-
sensus algorithms, both for educational purposes and as a
foundation for implementation. It is simpler and more un-
derstandable than other algorithms; it is described com-
pletely enough to meet the needs of a practical system;
it has several open-source implementations and is used
by several companies; its safety properties have been for-
mally specified and proven; and its efficiency is compara-
ble to other algorithms.
The remainder of the paper introduces the replicated
state machine problem (Section 2), discusses the strengths
and weaknesses of Paxos (Section 3), describes our gen-
eral approach to understandability (Section 4), presents
the Raft consensus algorithm (Sections 5–8), evaluates
Raft (Section 9), and discusses related work (Section 10).
2 Replicated state machines
Consensus algorithms typically arise in the context of
replicated state machines [37]. In this approach, state ma-
chines on a collection of servers compute identical copies
of the same state and can continue operating even if some
of the servers are down. Replicated state machines are
1

Figure 1: Replicated state machine architecture. The con-
sensus algorithm manages a replicated log containing state
machine commands from clients. The state machines process
identical sequences of commands from the logs, so they pro-
duce the same outputs.
used to solve a variety of fault tolerance problems in dis-
tributed systems. For example, large-scale systems that
have a single cluster leader, such as GFS [8], HDFS [38],
and RAMCloud [33], typically use a separate replicated
state machine to manage leader election and store config-
uration information that must survive leader crashes. Ex-
amples of replicated state machines include Chubby [2]
and ZooKeeper [11].
Replicated state machines are typically implemented
using a replicated log, as shown in Figure 1. Each server
stores a log containing a series of commands, which its
state machine executes in order. Each log contains the
same commands in the same order, so each state ma-
chine processes the same sequence of commands. Since
the state machines are deterministic, each computes the
same state and the same sequence of outputs.
Keeping the replicated log consistent is the job of the
consensus algorithm. The consensus module on a server
receives commands from clients and adds them to its log.
It communicates with the consensus modules on other
servers to ensure that every log eventually contains the
same requests in the same order, even if some servers fail.
Once commands are properly replicated, each server’s
state machine processes them in log order, and the out-
puts are returned to clients. As a result, the servers appear
to form a single, highly reliable state machine.
Consensus algorithms for practical systems typically
have the following properties:
• They ensure safety (never returning an incorrect re-
sult) under all non-Byzantine conditions, including
network delays, partitions, and packet loss, duplica-
tion, and reordering.
• They are fully functional ( available) as long as any
majority of the servers are operational and can com-
municate with each other and with clients. Thus, a
typical cluster of five servers can tolerate the failure
of any two servers. Servers are assumed to fail by
stopping; they may later recover from state on stable
storage and rejoin the cluster.
• They do not depend on timing to ensure the consis-
tency of the logs: faulty clocks and extreme message
delays can, at worst, cause availability problems.
• In the common case, a command can complete as
soon as a majority of the cluster has responded to a
single round of remote procedure calls; a minority of
slow servers need not impact overall system perfor-
mance.
3 What’s wrong with Paxos?
Over the last ten years, Leslie Lamport’s Paxos proto-
col [15] has become almost synonymous with consensus:
it is the protocol most commonly taught in courses, and
most implementations of consensus use it as a starting
point. Paxos first defines a protocol capable of reaching
agreement on a single decision, such as a single replicated
log entry. We refer to this subset as single-decree Paxos.
Paxos then combines multiple instances of this protocol to
facilitate a series of decisions such as a log ( multi-Paxos).
Paxos ensures both safety and liveness, and it supports
changes in cluster membership. Its correctness has been
proven, and it is efficient in the normal case.
Unfortunately, Paxos has two significant drawbacks.
The first drawback is that Paxos is exceptionally diffi-
cult to understand. The full explanation [15] is notori-
ously opaque; few people succeed in understanding it, and
only with great effort. As a result, there have been several
attempts to explain Paxos in simpler terms [16, 20, 21].
These explanations focus on the single-decree subset, yet
they are still challenging. In an informal survey of atten-
dees at NSDI 2012, we found few people who were com-
fortable with Paxos, even among seasoned researchers.
We struggled with Paxos ourselves; we were not able to
understand the complete protocol until after reading sev-
eral simplified explanations and designing our own alter-
native protocol, a process that took almost a year.
We hypothesize that Paxos’ opaqueness derives from
its choice of the single-decree subset as its foundation.
Single-decree Paxos is dense and subtle: it is divided into
two stages that do not have simple intuitive explanations
and cannot be understood independently. Because of this,
it is difficult to develop intuitions about why the single-
decree protocol works. The composition rules for multi-
Paxos add significant additional complexity and subtlety.
We believe that the overall problem of reaching consensus
on multiple decisions (i.e., a log instead of a single entry)
can be decomposed in other ways that are more direct and
obvious.
The second problem with Paxos is that it does not pro-
vide a good foundation for building practical implemen-
tations. One reason is that there is no widely agreed-
upon algorithm for multi-Paxos. Lamport’s descriptions
are mostly about single-decree Paxos; he sketched possi-
ble approaches to multi-Paxos, but many details are miss-
ing. There have been several attempts to flesh out and op-
timize Paxos, such as [26], [39], and [13], but these differ
2

from each other and from Lamport’s sketches. Systems
such as Chubby [4] have implemented Paxos-like algo-
rithms, but in most cases their details have not been pub-
lished.
Furthermore, the Paxos architecture is a poor one for
building practical systems; this is another consequence of
the single-decree decomposition. For example, there is lit-
tle benefit to choosing a collection of log entries indepen-
dently and then melding them into a sequential log; this
just adds complexity. It is simpler and more efficient to
design a system around a log, where new entries are ap-
pended sequentially in a constrained order. Another prob-
lem is that Paxos uses a symmetric peer-to-peer approach
at its core (though it eventually suggests a weak form of
leadership as a performance optimization). This makes
sense in a simplified world where only one decision will
be made, but few practical systems use this approach. If a
series of decisions must be made, it is simpler and faster
to first elect a leader, then have the leader coordinate the
decisions.
As a result, practical systems bear little resemblance
to Paxos. Each implementation begins with Paxos, dis-
covers the difficulties in implementing it, and then de-
velops a significantly different architecture. This is time -
consuming and error-prone, and the difficulties of under-
standing Paxos exacerbate the problem. Paxos’ formula-
tion may be a good one for proving theorems about its cor-
rectness, but real implementations are so different from
Paxos that the proofs have little value. The following com-
ment from the Chubby implementers is typical:
There are significant gaps between the description of
the Paxos algorithm and the needs of a real-world
system. . . . the final system will be based on an un-
proven protocol [4].
Because of these problems, we concluded that Paxos
does not provide a good foundation either for system
building or for education. Given the importance of con-
sensus in large-scale software systems, we decided to see
if we could design an alternative consensus algorithm
with better properties than Paxos. Raft is the result of that
experiment.
4 Designing for understandability
We had several goals in designing Raft: it must provide
a complete and practical foundation for system building,
so that it significantly reduces the amount of design work
required of developers; it must be safe under all conditions
and available under typical operating conditions; and it
must be efficient for common operations. But our most
important goal—and most difficult challenge—was un-
derstandability. It must be possible for a large audience to
understand the algorithm comfortably. In addition, it must
be possible to develop intuitions about the algorithm, so
that system builders can make the extensions that are in-
evitable in real-world implementations.
There were numerous points in the design of Raft
where we had to choose among alternative approaches.
In these situations we evaluated the alternatives based on
understandability: how hard is it to explain each alterna-
tive (for example, how complex is its state space, and does
it have subtle implications?), and how easy will it be for a
reader to completely understand the approach and its im-
plications?
We recognize that there is a high degree of subjectiv-
ity in such analysis; nonetheless, we used two techniques
that are generally applicable. The first technique is the
well-known approach of problem decomposition: wher-
ever possible, we divided problems into separate pieces
that could be solved, explained, and understood relatively
independently. For example, in Raft we separated leader
election, log replication, safety, and membership changes.
Our second approach was to simplify the state space
by reducing the number of states to consider, making the
system more coherent and eliminating nondeterminism
where possible. Specifically, logs are not allowed to have
holes, and Raft limits the ways in which logs can become
inconsistent with each other. Although in most cases we
tried to eliminate nondeterminism, there are some situ-
ations where nondeterminism actually improves under-
standability. In particular, randomized approaches intro -
duce nondeterminism, but they tend to reduce the state
space by handling all possible choices in a similar fashion
(“choose any; it doesn’t matter”). We used randomization
to simplify the Raft leader election algorithm.
5 The Raft consensus algorithm
Raft is an algorithm for managing a replicated log of
the form described in Section 2. Figure 2 summarizes the
algorithm in condensed form for reference, and Figure 3
lists key properties of the algorithm; the elements of these
figures are discussed piecewise over the rest of this sec-
tion.
Raft implements consensus by first electing a distin-
guished leader, then giving the leader complete responsi-
bility for managing the replicated log. The leader accepts
log entries from clients, replicates them on other servers,
and tells servers when it is safe to apply log entries to
their state machines. Having a leader simplifies the man-
agement of the replicated log. For example, the leader can
decide where to place new entries in the log without con-
sulting other servers, and data flows in a simple fashion
from the leader to other servers. A leader can fail or be-
come disconnected from the other servers, in which case
a new leader is elected.
Given the leader approach, Raft decomposes the con-
sensus problem into three relatively independent subprob-
lems, which are discussed in the subsections that follow:
• Leader election: a new leader must be chosen when
an existing leader fails (Section 5.2).
• Log replication: the leader must accept log entries
3

Invoked by candidates to gather votes (§5.2).
Arguments:
term candidate’s term
candidateId candidate requesting vote
lastLogIndex index of candidate’s last log entry (§5.4)
lastLogTerm term of candidate’s last log entry (§5.4)
Results:
term currentTerm, for candidate to update itself
voteGranted true means candidate received vote
Receiver implementation:
1. Reply false if term < currentTerm (§5.1)
2. If votedFor is null or candidateId, and candidate’s log is at
least as up-to-date as receiver’s log, grant vote (§5.2, §5.4)
RequestVote RPC
Invoked by leader to replicate log entries (§5.3); also used as
heartbeat (§5.2).
Arguments:
term leader’s term
leaderId so follower can redirect clients
prevLogIndex index of log entry immediately preceding
new ones
prevLogTerm term of prevLogIndex entry
entries[] log entries to store (empty for heartbeat;
may send more than one for efficiency)
leaderCommit leader’s commitIndex
Results:
term currentTerm, for leader to update itself
success true if follower contained entry matching
prevLogIndex and prevLogTerm
Receiver implementation:
1. Reply false if term < currentTerm (§5.1)
2. Reply false if log doesn’t contain an entry at prevLogIndex
whose term matches prevLogTerm (§5.3)
3. If an existing entry conflicts with a new one (same index
but different terms), delete the existing entry and all that
follow it (§5.3)
4. Append any new entries not already in the log
5. If leaderCommit > commitIndex, set commitIndex =
min(leaderCommit, index of last new entry)
AppendEntries RPC
Persistent state on all servers:
(Updated on stable storage before responding to RPCs)
currentTerm latest term server has seen (initialized to 0
on first boot, increases monotonically)
votedFor candidateId that received vote in current
term (or null if none)
log[] log entries; each entry contains command
for state machine, and term when entry
was received by leader (first index is 1)
Volatile state on all servers:
commitIndex index of highest log entry known to be
committed (initialized to 0, increases
monotonically)
lastApplied index of highest log entry applied to state
machine (initialized to 0, increases
monotonically)
Volatile state on leaders:
(Reinitialized after election)
nextIndex[] for each server, index of the next log entry
to send to that server (initialized to leader
last log index + 1)
matchIndex[] for each server, index of highest log entry
known to be replicated on server
(initialized to 0, increases monotonically)
State
All Servers:
• If commitIndex > lastApplied: increment lastApplied, apply
log[lastApplied] to state machine (§5.3)
• If RPC request or response contains term T > currentTerm:
set currentTerm = T, convert to follower (§5.1)
Followers (§5.2):
• Respond to RPCs from candidates and leaders
• If election timeout elapses without receiving AppendEntries
RPC from current leader or granting vote to candidate:
convert to candidate
Candidates (§5.2):
• On conversion to candidate, start election:
• Increment currentTerm
• V ote for self
• Reset election timer
• Send RequestV ote RPCs to all other servers
• If votes received from majority of servers: become leader
• If AppendEntries RPC received from new leader: convert to
follower
• If election timeout elapses: start new election
Leaders:
• Upon election: send initial empty AppendEntries RPCs
(heartbeat) to each server; repeat during idle periods to
prevent election timeouts (§5.2)
• If command received from client: append entry to local log,
respond after entry applied to state machine (§5.3)
• If last log index ≥ nextIndex for a follower: send
AppendEntries RPC with log entries starting at nextIndex
• If successful: update nextIndex and matchIndex for
follower (§5.3)
• If AppendEntries fails because of log inconsistency:
decrement nextIndex and retry (§5.3)
• If there exists an N such that N > commitIndex, a majority
of matchIndex[i] ≥ N, and log[N].term == currentTerm:
set commitIndex = N (§5.3, §5.4).
Rules for Servers
Figure 2: A condensed summary of the Raft consensus algorithm (excluding membership changes and log compaction). The server
behavior in the upper-left box is described as a set of rules t hat trigger independently and repeatedly. Section numbers such as §5.2
indicate where particular features are discussed. A formal specification [31] describes the algorithm more precisely.
4

Election Safety: at most one leader can be elected in a
given term. §5.2
Leader Append-Only: a leader never overwrites or deletes
entries in its log; it only appends new entries. §5.3
Log Matching: if two logs contain an entry with the same
index and term, then the logs are identical in all entries
up through the given index. §5.3
Leader Completeness: if a log entry is committed in a
given term, then that entry will be present in the logs
of the leaders for all higher-numbered terms. §5.4
State Machine Safety: if a server has applied a log entry
at a given index to its state machine, no other server
will ever apply a different log entry for the same index.
§5.4.3
Figure 3: Raft guarantees that each of these properties is true
at all times. The section numbers indicate where each prop-
erty is discussed.
from clients and replicate them across the cluster,
forcing the other logs to agree with its own (Sec-
tion 5.3).
• Safety: the key safety property for Raft is the State
Machine Safety Property in Figure 3: if any server
has applied a particular log entry to its state machine,
then no other server may apply a different command
for the same log index. Section 5.4 describes how
Raft ensures this property; the solution involves an
additional restriction on the election mechanism de-
scribed in Section 5.2.
After presenting the consensus algorithm, this section dis-
cusses the issue of availability and the role of timing in the
system.
5.1 Raft basics
A Raft cluster contains several servers; five is a typical
number, which allows the system to tolerate two failures.
At any given time each server is in one of three states:
leader, follower, or candidate. In normal operation there
is exactly one leader and all of the other servers are fol-
lowers. Followers are passive: they issue no requests on
their own but simply respond to requests from leaders
and candidates. The leader handles all client requests (if
a client contacts a follower, the follower redirects it to th e
leader). The third state, candidate, is used to elect a new
leader as described in Section 5.2. Figure 4 shows the
states and their transitions; the transitions are discusse d
below.
Raft divides time into terms of arbitrary length, as
shown in Figure 5. Terms are numbered with consecutive
integers. Each term begins with an election, in which one
or more candidates attempt to become leader as described
in Section 5.2. If a candidate wins the election, then it
serves as leader for the rest of the term. In some situations
an election will result in a split vote. In this case the term
will end with no leader; a new term (with a new election)
Figure 4: Server states. Followers only respond to requests
from other servers. If a follower receives no communication ,
it becomes a candidate and initiates an election. A candidat e
that receives votes from a majority of the full cluster becomes
the new leader. Leaders typically operate until they fail.
Figure 5: Time is divided into terms, and each term begins
with an election. After a successful election, a single lead er
manages the cluster until the end of the term. Some elections
fail, in which case the term ends without choosing a leader.
The transitions between terms may be observed at different
times on different servers.
will begin shortly. Raft ensures that there is at most one
leader in a given term.
Different servers may observe the transitions between
terms at different times, and in some situations a server
may not observe an election or even entire terms. Terms
act as a logical clock [14] in Raft, and they allow servers
to detect obsolete information such as stale leaders. Each
server stores a current term number, which increases
monotonically over time. Current terms are exchanged
whenever servers communicate; if one server’s current
term is smaller than the other’s, then it updates its current
term to the larger value. If a candidate or leader discovers
that its term is out of date, it immediately reverts to fol-
lower state. If a server receives a request with a stale term
number, it rejects the request.
Raft servers communicate using remote procedure calls
(RPCs), and the basic consensus algorithm requires only
two types of RPCs. RequestV ote RPCs are initiated by
candidates during elections (Section 5.2), and Append-
Entries RPCs are initiated by leaders to replicate log en-
tries and to provide a form of heartbeat (Section 5.3). Sec-
tion 7 adds a third RPC for transferring snapshots between
servers. Servers retry RPCs if they do not receive a re-
sponse in a timely manner, and they issue RPCs in parallel
for best performance.
5.2 Leader election
Raft uses a heartbeat mechanism to trigger leader elec-
tion. When servers start up, they begin as followers. A
server remains in follower state as long as it receives valid
5

RPCs from a leader or candidate. Leaders send periodic
heartbeats (AppendEntries RPCs that carry no log entries)
to all followers in order to maintain their authority. If a
follower receives no communication over a period of time
called the election timeout, then it assumes there is no vi-
able leader and begins an election to choose a new leader.
To begin an election, a follower increments its current
term and transitions to candidate state. It then votes for
itself and issues RequestV ote RPCs in parallel to each of
the other servers in the cluster. A candidate continues in
this state until one of three things happens: (a) it wins the
election, (b) another server establishes itself as leader, or
(c) a period of time goes by with no winner. These out-
comes are discussed separately in the paragraphs below.
A candidate wins an election if it receives votes from
a majority of the servers in the full cluster for the same
term. Each server will vote for at most one candidate in a
given term, on a first-come-first-served basis (note: Sec-
tion 5.4 adds an additional restriction on votes). The ma-
jority rule ensures that at most one candidate can win the
election for a particular term (the Election Safety Prop-
erty in Figure 3). Once a candidate wins an election, it
becomes leader. It then sends heartbeat messages to all of
the other servers to establish its authority and prevent new
elections.
While waiting for votes, a candidate may receive an
AppendEntries RPC from another server claiming to be
leader. If the leader’s term (included in its RPC) is at least
as large as the candidate’s current term, then the candidate
recognizes the leader as legitimate and returns to follower
state. If the term in the RPC is smaller than the candidate’s
current term, then the candidate rejects the RPC and con-
tinues in candidate state.
The third possible outcome is that a candidate neither
wins nor loses the election: if many followers become
candidates at the same time, votes could be split so that
no candidate obtains a majority. When this happens, each
candidate will time out and start a new election by incre-
menting its term and initiating another round of Request-
V ote RPCs. However, without extra measures split votes
could repeat indefinitely.
Raft uses randomized election timeouts to ensure that
split votes are rare and that they are resolved quickly. To
prevent split votes in the first place, election timeouts are
chosen randomly from a fixed interval (e.g., 150–300ms).
This spreads out the servers so that in most cases only a
single server will time out; it wins the election and sends
heartbeats before any other servers time out. The same
mechanism is used to handle split votes. Each candidate
restarts its randomized election timeout at the start of an
election, and it waits for that timeout to elapse before
starting the next election; this reduces the likelihood of
another split vote in the new election. Section 9.3 shows
that this approach elects a leader rapidly.
Figure 6: Logs are composed of entries, which are numbered
sequentially. Each entry contains the term in which it was
created (the number in each box) and a command for the state
machine. An entry is considered committed if it is safe for that
entry to be applied to state machines.
Elections are an example of how understandability
guided our choice between design alternatives. Initially
we planned to use a ranking system: each candidate was
assigned a unique rank, which was used to select between
competing candidates. If a candidate discovered another
candidate with higher rank, it would return to follower
state so that the higher ranking candidate could more eas-
ily win the next election. We found that this approach
created subtle issues around availability (a lower-ranked
server might need to time out and become a candidate
again if a higher-ranked server fails, but if it does so too
soon, it can reset progress towards electing a leader). We
made adjustments to the algorithm several times, but after
each adjustment new corner cases appeared. Eventually
we concluded that the randomized retry approach is more
obvious and understandable.
5.3 Log replication
Once a leader has been elected, it begins servicing
client requests. Each client request contains a command to
be executed by the replicated state machines. The leader
appends the command to its log as a new entry, then is-
sues AppendEntries RPCs in parallel to each of the other
servers to replicate the entry. When the entry has been
safely replicated (as described below), the leader applies
the entry to its state machine and returns the result of that
execution to the client. If followers crash or run slowly,
or if network packets are lost, the leader retries Append-
Entries RPCs indefinitely (even after it has responded to
the client) until all followers eventually store all log en-
tries.
Logs are organized as shown in Figure 6. Each log en-
try stores a state machine command along with the term
number when the entry was received by the leader. The
term numbers in log entries are used to detect inconsis-
tencies between logs and to ensure some of the properties
in Figure 3. Each log entry also has an integer index iden-
6

tifying its position in the log.
The leader decides when it is safe to apply a log en-
try to the state machines; such an entry is called commit-
ted. Raft guarantees that committed entries are durable
and will eventually be executed by all of the available
state machines. A log entry is committed once the leader
that created the entry has replicated it on a majority of
the servers (e.g., entry 7 in Figure 6). This also commits
all preceding entries in the leader’s log, including entrie s
created by previous leaders. Section 5.4 discusses some
subtleties when applying this rule after leader changes,
and it also shows that this definition of commitment is
safe. The leader keeps track of the highest index it knows
to be committed, and it includes that index in future
AppendEntries RPCs (including heartbeats) so that the
other servers eventually find out. Once a follower learns
that a log entry is committed, it applies the entry to its
local state machine (in log order).
We designed the Raft log mechanism to maintain a high
level of coherency between the logs on different servers.
Not only does this simplify the system’s behavior and
make it more predictable, but it is an important component
of ensuring safety. Raft maintains the following proper-
ties, which together constitute the Log Matching Property
in Figure 3:
• If two entries in different logs have the same index
and term, then they store the same command.
• If two entries in different logs have the same index
and term, then the logs are identical in all preceding
entries.
The first property follows from the fact that a leader
creates at most one entry with a given log index in a given
term, and log entries never change their position in the
log. The second property is guaranteed by a simple con-
sistency check performed by AppendEntries. When send-
ing an AppendEntries RPC, the leader includes the index
and term of the entry in its log that immediately precedes
the new entries. If the follower does not find an entry in
its log with the same index and term, then it refuses the
new entries. The consistency check acts as an induction
step: the initial empty state of the logs satisfies the Log
Matching Property, and the consistency check preserves
the Log Matching Property whenever logs are extended.
As a result, whenever AppendEntries returns successfully,
the leader knows that the follower’s log is identical to its
own log up through the new entries.
During normal operation, the logs of the leader and
followers stay consistent, so the AppendEntries consis-
tency check never fails. However, leader crashes can leave
the logs inconsistent (the old leader may not have fully
replicated all of the entries in its log). These inconsisten -
cies can compound over a series of leader and follower
crashes. Figure 7 illustrates the ways in which followers’
logs may differ from that of a new leader. A follower may
Figure 7: When the leader at the top comes to power, it is
possible that any of scenarios (a–f) could occur in follower
logs. Each box represents one log entry; the number in the
box is its term. A follower may be missing entries (a–b), may
have extra uncommitted entries (c–d), or both (e–f). For ex-
ample, scenario (f) could occur if that server was the leader
for term 2, added several entries to its log, then crashed before
committing any of them; it restarted quickly, became leader
for term 3, and added a few more entries to its log; before any
of the entries in either term 2 or term 3 were committed, the
server crashed again and remained down for several terms.
be missing entries that are present on the leader, it may
have extra entries that are not present on the leader, or
both. Missing and extraneous entries in a log may span
multiple terms.
In Raft, the leader handles inconsistencies by forcing
the followers’ logs to duplicate its own. This means that
conflicting entries in follower logs will be overwritten
with entries from the leader’s log. Section 5.4 will show
that this is safe when coupled with one more restriction.
To bring a follower’s log into consistency with its own,
the leader must find the latest log entry where the two
logs agree, delete any entries in the follower’s log after
that point, and send the follower all of the leader’s entries
after that point. All of these actions happen in response
to the consistency check performed by AppendEntries
RPCs. The leader maintains a nextIndex for each follower,
which is the index of the next log entry the leader will
send to that follower. When a leader first comes to power,
it initializes all nextIndex values to the index just after t he
last one in its log (11 in Figure 7). If a follower’s log is
inconsistent with the leader’s, the AppendEntries consis-
tency check will fail in the next AppendEntries RPC. Af-
ter a rejection, the leader decrements nextIndex and retries
the AppendEntries RPC. Eventually nextIndex will reach
a point where the leader and follower logs match. When
this happens, AppendEntries will succeed, which removes
any conflicting entries in the follower’s log and appends
entries from the leader’s log (if any). Once AppendEntries
succeeds, the follower’s log is consistent with the leader’ s,
and it will remain that way for the rest of the term.
If desired, the protocol can be optimized to reduce the
number of rejected AppendEntries RPCs. For example,
when rejecting an AppendEntries request, the follower
7

can include the term of the conflicting entry and the first
index it stores for that term. With this information, the
leader can decrement nextIndex to bypass all of the con-
flicting entries in that term; one AppendEntries RPC will
be required for each term with conflicting entries, rather
than one RPC per entry. In practice, we doubt this opti-
mization is necessary, since failures happen infrequently
and it is unlikely that there will be many inconsistent en-
tries.
With this mechanism, a leader does not need to take any
special actions to restore log consistency when it comes to
power. It just begins normal operation, and the logs auto-
matically converge in response to failures of the Append-
Entries consistency check. A leader never overwrites or
deletes entries in its own log (the Leader Append-Only
Property in Figure 3).
This log replication mechanism exhibits the desirable
consensus properties described in Section 2: Raft can ac-
cept, replicate, and apply new log entries as long as a ma-
jority of the servers are up; in the normal case a new entry
can be replicated with a single round of RPCs to a ma-
jority of the cluster; and a single slow follower will not
impact performance.
5.4 Safety
The previous sections described how Raft elects lead-
ers and replicates log entries. However, the mechanisms
described so far are not quite sufficient to ensure that each
state machine executes exactly the same commands in the
same order. For example, a follower might be unavailable
while the leader commits several log entries, then it could
be elected leader and overwrite these entries with new
ones; as a result, different state machines might execute
different command sequences.
This section completes the Raft algorithm by adding a
restriction on which servers may be elected leader. The
restriction ensures that the leader for any given term con-
tains all of the entries committed in previous terms (the
Leader Completeness Property from Figure 3). Given the
election restriction, we then make the rules for commit-
ment more precise. Finally, we present a proof sketch for
the Leader Completeness Property and show how it leads
to correct behavior of the replicated state machine.
5.4.1 Election restriction
In any leader-based consensus algorithm, the leader
must eventually store all of the committed log entries. In
some consensus algorithms, such as Viewstamped Repli-
cation [22], a leader can be elected even if it doesn’t
initially contain all of the committed entries. These al-
gorithms contain additional mechanisms to identify the
missing entries and transmit them to the new leader, ei-
ther during the election process or shortly afterwards. Un-
fortunately, this results in considerable additional mech a-
nism and complexity. Raft uses a simpler approach where
it guarantees that all the committed entries from previous
Figure 8: A time sequence showing why a leader cannot de-
termine commitment using log entries from older terms. In
(a) S1 is leader and partially replicates the log entry at ind ex
2. In (b) S1 crashes; S5 is elected leader for term 3 with votes
from S3, S4, and itself, and accepts a different entry at log
index 2. In (c) S5 crashes; S1 restarts, is elected leader, an d
continues replication. At this point, the log entry from ter m 2
has been replicated on a majority of the servers, but it is not
committed. If S1 crashes as in (d), S5 could be elected leader
(with votes from S2, S3, and S4) and overwrite the entry with
its own entry from term 3. However, if S1 replicates an en-
try from its current term on a majority of the servers before
crashing, as in (e), then this entry is committed (S5 cannot
win an election). At this point all preceding entries in the l og
are committed as well.
terms are present on each new leader from the moment of
its election, without the need to transfer those entries to
the leader. This means that log entries only flow in one di-
rection, from leaders to followers, and leaders never over-
write existing entries in their logs.
Raft uses the voting process to prevent a candidate from
winning an election unless its log contains all committed
entries. A candidate must contact a majority of the cluster
in order to be elected, which means that every committed
entry must be present in at least one of those servers. If the
candidate’s log is at least as up-to-date as any other log
in that majority (where “up-to-date” is defined precisely
below), then it will hold all the committed entries. The
RequestV ote RPC implements this restriction: the RPC
includes information about the candidate’s log, and the
voter denies its vote if its own log is more up-to-date than
that of the candidate.
Raft determines which of two logs is more up-to-date
by comparing the index and term of the last entries in the
logs. If the logs have last entries with different terms, the n
the log with the later term is more up-to-date. If the logs
end with the same term, then whichever log is longer is
more up-to-date.
5.4.2 Committing entries from previous terms
As described in Section 5.3, a leader knows that an en-
try from its current term is committed once that entry is
stored on a majority of the servers. If a leader crashes be-
fore committing an entry, future leaders will attempt to
finish replicating the entry. However, a leader cannot im-
mediately conclude that an entry from a previous term is
committed once it is stored on a majority of servers. Fig-
8

Figure 9: If S1 (leader for term T) commits a new log entry
from its term, and S5 is elected leader for a later term U, then
there must be at least one server (S3) that accepted the log
entry and also voted for S5.
ure 8 illustrates a situation where an old log entry is stored
on a majority of servers, yet can still be overwritten by a
future leader.
To eliminate problems like the one in Figure 8, Raft
never commits log entries from previous terms by count-
ing replicas. Only log entries from the leader’s current
term are committed by counting replicas; once an entry
from the current term has been committed in this way,
then all prior entries are committed indirectly because
of the Log Matching Property. There are some situations
where a leader could safely conclude that an older log en-
try is committed (for example, if that entry is stored on ev-
ery server), but Raft takes a more conservative approach
for simplicity.
Raft incurs this extra complexity in the commitment
rules because log entries retain their original term num-
bers when a leader replicates entries from previous
terms. In other consensus algorithms, if a new leader re-
replicates entries from prior “terms,” it must do so with
its new “term number.” Raft’s approach makes it easier
to reason about log entries, since they maintain the same
term number over time and across logs. In addition, new
leaders in Raft send fewer log entries from previous terms
than in other algorithms (other algorithms must send re-
dundant log entries to renumber them before they can be
committed).
5.4.3 Safety argument
Given the complete Raft algorithm, we can now ar-
gue more precisely that the Leader Completeness Prop-
erty holds (this argument is based on the safety proof; see
Section 9.2). We assume that the Leader Completeness
Property does not hold, then we prove a contradiction.
Suppose the leader for term T (leader T) commits a log
entry from its term, but that log entry is not stored by the
leader of some future term. Consider the smallest term U
> T whose leader (leader U) does not store the entry.
1. The committed entry must have been absent from
leaderU’s log at the time of its election (leaders never
delete or overwrite entries).
2. leader T replicated the entry on a majority of the clus-
ter, and leader U received votes from a majority of
the cluster. Thus, at least one server (“the voter”)
both accepted the entry from leader T and voted for
leaderU, as shown in Figure 9. The voter is key to
reaching a contradiction.
3. The voter must have accepted the committed entry
from leader T before voting for leader U; otherwise it
would have rejected the AppendEntries request from
leaderT (its current term would have been higher than
T).
4. The voter still stored the entry when it voted for
leaderU, since every intervening leader contained the
entry (by assumption), leaders never remove entries,
and followers only remove entries if they conflict
with the leader.
5. The voter granted its vote to leader U, so leader U’s
log must have been as up-to-date as the voter’s. This
leads to one of two contradictions.
6. First, if the voter and leader U shared the same last
log term, then leader U’s log must have been at least
as long as the voter’s, so its log contained every entry
in the voter’s log. This is a contradiction, since the
voter contained the committed entry and leaderU was
assumed not to.
7. Otherwise, leader U’s last log term must have been
larger than the voter’s. Moreover, it was larger than
T, since the voter’s last log term was at least T (it con-
tains the committed entry from term T). The earlier
leader that created leader U’s last log entry must have
contained the committed entry in its log (by assump-
tion). Then, by the Log Matching Property, leaderU’s
log must also contain the committed entry, which is
a contradiction.
8. This completes the contradiction. Thus, the leaders
of all terms greater than T must contain all entries
from term T that are committed in term T.
9. The Log Matching Property guarantees that future
leaders will also contain entries that are committed
indirectly, such as index 2 in Figure 8(d).
Given the Leader Completeness Property, we can prove
the State Machine Safety Property from Figure 3, which
states that if a server has applied a log entry at a given
index to its state machine, no other server will ever apply a
different log entry for the same index. At the time a server
applies a log entry to its state machine, its log must be
identical to the leader’s log up through that entry and the
entry must be committed. Now consider the lowest term
in which any server applies a given log index; the Log
Completeness Property guarantees that the leaders for all
higher terms will store that same log entry, so servers that
apply the index in later terms will apply the same value.
Thus, the State Machine Safety Property holds.
Finally, Raft requires servers to apply entries in log in-
dex order. Combined with the State Machine Safety Prop-
erty, this means that all servers will apply exactly the same
set of log entries to their state machines, in the same order.
9

5.5 Follower and candidate crashes
Until this point we have focused on leader failures. Fol-
lower and candidate crashes are much simpler to han-
dle than leader crashes, and they are both handled in the
same way. If a follower or candidate crashes, then fu-
ture RequestV ote and AppendEntries RPCs sent to it will
fail. Raft handles these failures by retrying indefinitely;
if the crashed server restarts, then the RPC will complete
successfully. If a server crashes after completing an RPC
but before responding, then it will receive the same RPC
again after it restarts. Raft RPCs are idempotent, so this
causes no harm. For example, if a follower receives an
AppendEntries request that includes log entries already
present in its log, it ignores those entries in the new re-
quest.
5.6 Timing and availability
One of our requirements for Raft is that safety must
not depend on timing: the system must not produce incor-
rect results just because some event happens more quickly
or slowly than expected. However, availability (the ability
of the system to respond to clients in a timely manner)
must inevitably depend on timing. For example, if mes-
sage exchanges take longer than the typical time between
server crashes, candidates will not stay up long enough to
win an election; without a steady leader, Raft cannot make
progress.
Leader election is the aspect of Raft where timing is
most critical. Raft will be able to elect and maintain a
steady leader as long as the system satisfies the follow-
ing timing requirement:
broadcastTime ≪ electionTimeout ≪ MTBF
In this inequality broadcastTime is the average time it
takes a server to send RPCs in parallel to every server
in the cluster and receive their responses; electionTime-
out is the election timeout described in Section 5.2; and
MTBF is the average time between failures for a single
server. The broadcast time should be an order of mag-
nitude less than the election timeout so that leaders can
reliably send the heartbeat messages required to keep fol-
lowers from starting elections; given the randomized ap-
proach used for election timeouts, this inequality also
makes split votes unlikely. The election timeout should be
a few orders of magnitude less than MTBF so that the sys-
tem makes steady progress. When the leader crashes, the
system will be unavailable for roughly the election time-
out; we would like this to represent only a small fraction
of overall time.
The broadcast time and MTBF are properties of the un-
derlying system, while the election timeout is something
we must choose. Raft’s RPCs typically require the recip-
ient to persist information to stable storage, so the broad-
cast time may range from 0.5ms to 20ms, depending on
storage technology. As a result, the election timeout is
likely to be somewhere between 10ms and 500ms. Typical
Figure 10: Switching directly from one configuration to an-
other is unsafe because different servers will switch at dif -
ferent times. In this example, the cluster grows from three
servers to five. Unfortunately, there is a point in time where
two different leaders can be elected for the same term, one
with a majority of the old configuration ( Cold) and another
with a majority of the new configuration ( Cnew).
server MTBFs are several months or more, which easily
satisfies the timing requirement.
6 Cluster membership changes
Up until now we have assumed that the cluster config-
uration (the set of servers participating in the consensus
algorithm) is fixed. In practice, it will occasionally be nec-
essary to change the configuration, for example to replace
servers when they fail or to change the degree of replica-
tion. Although this can be done by taking the entire cluster
off-line, updating configuration files, and then restarting
the cluster, this would leave the cluster unavailable dur-
ing the changeover. In addition, if there are any manual
steps, they risk operator error. In order to avoid these is-
sues, we decided to automate configuration changes and
incorporate them into the Raft consensus algorithm.
For the configuration change mechanism to be safe,
there must be no point during the transition where it
is possible for two leaders to be elected for the same
term. Unfortunately, any approach where servers switch
directly from the old configuration to the new configura-
tion is unsafe. It isn’t possible to atomically switch all of
the servers at once, so the cluster can potentially split int o
two independent majorities during the transition (see Fig-
ure 10).
In order to ensure safety, configuration changes must
use a two-phase approach. There are a variety of ways
to implement the two phases. For example, some systems
(e.g., [22]) use the first phase to disable the old configura-
tion so it cannot process client requests; then the second
phase enables the new configuration. In Raft the cluster
first switches to a transitional configuration we call joint
consensus; once the joint consensus has been committed,
the system then transitions to the new configuration. The
joint consensus combines both the old and new configu-
rations:
• Log entries are replicated to all servers in both con-
figurations.
10

Figure 11: Timeline for a configuration change. Dashed lines
show configuration entries that have been created but not
committed, and solid lines show the latest committed configu-
ration entry. The leader first creates the Cold,new configuration
entry in its log and commits it to Cold,new (a majority of Cold
and a majority of Cnew). Then it creates the Cnew entry and
commits it to a majority of Cnew. There is no point in time in
which Cold and Cnew can both make decisions independently.
• Any server from either configuration may serve as
leader.
• Agreement (for elections and entry commitment) re-
quires separate majorities from both the old and new
configurations.
The joint consensus allows individual servers to transition
between configurations at different times without com-
promising safety. Furthermore, joint consensus allows the
cluster to continue servicing client requests throughout
the configuration change.
Cluster configurations are stored and communicated
using special entries in the replicated log; Figure 11 illus -
trates the configuration change process. When the leader
receives a request to change the configuration from Cold
to Cnew, it stores the configuration for joint consensus
(Cold,new in the figure) as a log entry and replicates that
entry using the mechanisms described previously. Once a
given server adds the new configuration entry to its log,
it uses that configuration for all future decisions (a server
always uses the latest configuration in its log, regardless
of whether the entry is committed). This means that the
leader will use the rules of Cold,new to determine when the
log entry for Cold,new is committed. If the leader crashes,
a new leader may be chosen under either Cold or Cold,new,
depending on whether the winning candidate has received
Cold,new. In any case, Cnew cannot make unilateral deci-
sions during this period.
Once Cold,new has been committed, neither Cold nor Cnew
can make decisions without approval of the other, and the
Leader Completeness Property ensures that only servers
with the Cold,new log entry can be elected as leader. It is
now safe for the leader to create a log entry describing
Cnew and replicate it to the cluster. Again, this configura-
tion will take effect on each server as soon as it is seen.
When the new configuration has been committed under
the rules of Cnew, the old configuration is irrelevant and
servers not in the new configuration can be shut down. As
shown in Figure 11, there is no time when Cold and Cnew
can both make unilateral decisions; this guarantees safety.
There are three more issues to address for reconfigura-
tion. The first issue is that new servers may not initially
store any log entries. If they are added to the cluster in
this state, it could take quite a while for them to catch
up, during which time it might not be possible to com-
mit new log entries. In order to avoid availability gaps,
Raft introduces an additional phase before the configu-
ration change, in which the new servers join the cluster
as non-voting members (the leader replicates log entries
to them, but they are not considered for majorities). Once
the new servers have caught up with the rest of the cluster,
the reconfiguration can proceed as described above.
The second issue is that the cluster leader may not be
part of the new configuration. In this case, the leader steps
down (returns to follower state) once it has committed the
Cnew log entry. This means that there will be a period of
time (while it is committing Cnew) when the leader is man-
aging a cluster that does not include itself; it replicates l og
entries but does not count itself in majorities. The leader
transition occurs when Cnew is committed because this is
the first point when the new configuration can operate in-
dependently (it will always be possible to choose a leader
from Cnew). Before this point, it may be the case that only
a server from Cold can be elected leader.
The third issue is that removed servers (those not in
Cnew) can disrupt the cluster. These servers will not re-
ceive heartbeats, so they will time out and start new elec-
tions. They will then send RequestV ote RPCs with new
term numbers, and this will cause the current leader to
revert to follower state. A new leader will eventually be
elected, but the removed servers will time out again and
the process will repeat, resulting in poor availability.
To prevent this problem, servers disregard RequestV ote
RPCs when they believe a current leader exists. Specif-
ically, if a server receives a RequestV ote RPC within
the minimum election timeout of hearing from a cur-
rent leader, it does not update its term or grant its vote.
This does not affect normal elections, where each server
waits at least a minimum election timeout before starting
an election. However, it helps avoid disruptions from re-
moved servers: if a leader is able to get heartbeats to its
cluster, then it will not be deposed by larger term num-
bers.
7 Log compaction
Raft’s log grows during normal operation to incorpo-
rate more client requests, but in a practical system, it can-
not grow without bound. As the log grows longer, it oc-
cupies more space and takes more time to replay. This
will eventually cause availability problems without some
mechanism to discard obsolete information that has accu-
mulated in the log.
Snapshotting is the simplest approach to compaction.
In snapshotting, the entire current system state is written
to a snapshot on stable storage, then the entire log up to
11

Figure 12: A server replaces the committed entries in its log
(indexes 1 through 5) with a new snapshot, which stores just
the current state (variables x and y in this example). The snap-
shot’s last included index and term serve to position the sna p-
shot in the log preceding entry 6.
that point is discarded. Snapshotting is used in Chubby
and ZooKeeper, and the remainder of this section de-
scribes snapshotting in Raft.
Incremental approaches to compaction, such as log
cleaning [36] and log-structured merge trees [30, 5], are
also possible. These operate on a fraction of the data at
once, so they spread the load of compaction more evenly
over time. They first select a region of data that has ac-
cumulated many deleted and overwritten objects, then
they rewrite the live objects from that region more com-
pactly and free the region. This requires significant addi-
tional mechanism and complexity compared to snapshot-
ting, which simplifies the problem by always operating
on the entire data set. While log cleaning would require
modifications to Raft, state machines can implement LSM
trees using the same interface as snapshotting.
Figure 12 shows the basic idea of snapshotting in Raft.
Each server takes snapshots independently, covering just
the committed entries in its log. Most of the work con-
sists of the state machine writing its current state to the
snapshot. Raft also includes a small amount of metadata
in the snapshot: the last included index is the index of the
last entry in the log that the snapshot replaces (the last en-
try the state machine had applied), and the last included
term is the term of this entry. These are preserved to sup-
port the AppendEntries consistency check for the first log
entry following the snapshot, since that entry needs a pre-
vious log index and term. To enable cluster membership
changes (Section 6), the snapshot also includes the latest
configuration in the log as of last included index. Once a
server completes writing a snapshot, it may delete all log
entries up through the last included index, as well as any
prior snapshot.
Although servers normally take snapshots indepen-
dently, the leader must occasionally send snapshots to
followers that lag behind. This happens when the leader
has already discarded the next log entry that it needs to
send to a follower. Fortunately, this situation is unlikely
in normal operation: a follower that has kept up with the
Invoked by leader to send chunks of a snapshot to a follower.
Leaders always send chunks in order.
Arguments:
term leader’s term
leaderId so follower can redirect clients
lastIncludedIndex the snapshot replaces all entries up through
and including this index
lastIncludedTerm term of lastIncludedIndex
offset byte offset where chunk is positioned in the
snapshot file
data[] raw bytes of the snapshot chunk, starting at
offset
done true if this is the last chunk
Results:
term currentTerm, for leader to update itself
Receiver implementation:
1. Reply immediately if term < currentTerm
2. Create new snapshot file if first chunk (offset is 0)
3. Write data into snapshot file at given offset
4. Reply and wait for more data chunks if done is false
5. Save snapshot file, discard any existing or partial snapshot
with a smaller index
6. If existing log entry has same index and term as snapshot’s
last included entry, retain log entries following it and reply
7. Discard the entire log
8. Reset state machine using snapshot contents (and load
snapshot’s cluster configuration)
InstallSnapshot RPC
Figure 13: A summary of the InstallSnapshot RPC. Snap-
shots are split into chunks for transmission; this gives the fol-
lower a sign of life with each chunk, so it can reset its election
timer.
leader would already have this entry. However, an excep-
tionally slow follower or a new server joining the cluster
(Section 6) would not. The way to bring such a follower
up-to-date is for the leader to send it a snapshot over the
network.
The leader uses a new RPC called InstallSnapshot to
send snapshots to followers that are too far behind; see
Figure 13. When a follower receives a snapshot with this
RPC, it must decide what to do with its existing log en-
tries. Usually the snapshot will contain new information
not already in the recipient’s log. In this case, the followe r
discards its entire log; it is all superseded by the snapshot
and may possibly have uncommitted entries that conflict
with the snapshot. If instead the follower receives a snap-
shot that describes a prefix of its log (due to retransmis-
sion or by mistake), then log entries covered by the snap-
shot are deleted but entries following the snapshot are still
valid and must be retained.
This snapshotting approach departs from Raft’s strong
leader principle, since followers can take snapshots with-
out the knowledge of the leader. However, we think this
departure is justified. While having a leader helps avoid
conflicting decisions in reaching consensus, consensus
has already been reached when snapshotting, so no de-
cisions conflict. Data still only flows from leaders to fol-
12

lowers, just followers can now reorganize their data.
We considered an alternative leader-based approach in
which only the leader would create a snapshot, then it
would send this snapshot to each of its followers. How-
ever, this has two disadvantages. First, sending the snap-
shot to each follower would waste network bandwidth and
slow the snapshotting process. Each follower already has
the information needed to produce its own snapshots, and
it is typically much cheaper for a server to produce a snap-
shot from its local state than it is to send and receive one
over the network. Second, the leader’s implementation
would be more complex. For example, the leader would
need to send snapshots to followers in parallel with repli-
cating new log entries to them, so as not to block new
client requests.
There are two more issues that impact snapshotting per-
formance. First, servers must decide when to snapshot. If
a server snapshots too often, it wastes disk bandwidth and
energy; if it snapshots too infrequently, it risks exhaust-
ing its storage capacity, and it increases the time required
to replay the log during restarts. One simple strategy is
to take a snapshot when the log reaches a fixed size in
bytes. If this size is set to be significantly larger than the
expected size of a snapshot, then the disk bandwidth over-
head for snapshotting will be small.
The second performance issue is that writing a snap-
shot can take a significant amount of time, and we do
not want this to delay normal operations. The solution is
to use copy-on-write techniques so that new updates can
be accepted without impacting the snapshot being writ-
ten. For example, state machines built with functional data
structures naturally support this. Alternatively, the operat-
ing system’s copy-on-write support (e.g., fork on Linux)
can be used to create an in-memory snapshot of the entire
state machine (our implementation uses this approach).
8 Client interaction
This section describes how clients interact with Raft,
including how clients find the cluster leader and how Raft
supports linearizable semantics [10]. These issues apply
to all consensus-based systems, and Raft’s solutions are
similar to other systems.
Clients of Raft send all of their requests to the leader.
When a client first starts up, it connects to a randomly-
chosen server. If the client’s first choice is not the leader,
that server will reject the client’s request and supply in-
formation about the most recent leader it has heard from
(AppendEntries requests include the network address of
the leader). If the leader crashes, client requests will tim e
out; clients then try again with randomly-chosen servers.
Our goal for Raft is to implement linearizable seman-
tics (each operation appears to execute instantaneously,
exactly once, at some point between its invocation and
its response). However, as described so far Raft can exe-
cute a command multiple times: for example, if the leader
crashes after committing the log entry but before respond-
ing to the client, the client will retry the command with a
new leader, causing it to be executed a second time. The
solution is for clients to assign unique serial numbers to
every command. Then, the state machine tracks the latest
serial number processed for each client, along with the as-
sociated response. If it receives a command whose serial
number has already been executed, it responds immedi-
ately without re-executing the request.
Read-only operations can be handled without writing
anything into the log. However, with no additional mea-
sures, this would run the risk of returning stale data, since
the leader responding to the request might have been su-
perseded by a newer leader of which it is unaware. Lin-
earizable reads must not return stale data, and Raft needs
two extra precautions to guarantee this without using the
log. First, a leader must have the latest information on
which entries are committed. The Leader Completeness
Property guarantees that a leader has all committed en-
tries, but at the start of its term, it may not know which
those are. To find out, it needs to commit an entry from
its term. Raft handles this by having each leader com-
mit a blank no-op entry into the log at the start of its
term. Second, a leader must check whether it has been de-
posed before processing a read-only request (its informa-
tion may be stale if a more recent leader has been elected).
Raft handles this by having the leader exchange heart-
beat messages with a majority of the cluster before re-
sponding to read-only requests. Alternatively, the leader
could rely on the heartbeat mechanism to provide a form
of lease [9], but this would rely on timing for safety (it
assumes bounded clock skew).
9 Implementation and evaluation
We have implemented Raft as part of a replicated
state machine that stores configuration information for
RAMCloud [33] and assists in failover of the RAMCloud
coordinator. The Raft implementation contains roughly
2000 lines of C++ code, not including tests, comments, or
blank lines. The source code is freely available [23]. There
are also about 25 independent third-party open source im-
plementations [34] of Raft in various stages of develop-
ment, based on drafts of this paper. Also, various compa-
nies are deploying Raft-based systems [34].
The remainder of this section evaluates Raft using three
criteria: understandability, correctness, and performan ce.
9.1 Understandability
To measure Raft’s understandability relative to Paxos,
we conducted an experimental study using upper-level un-
dergraduate and graduate students in an Advanced Oper-
ating Systems course at Stanford University and a Dis-
tributed Computing course at U.C. Berkeley. We recorded
a video lecture of Raft and another of Paxos, and created
corresponding quizzes. The Raft lecture covered the con-
tent of this paper except for log compaction; the Paxos
13

 0
 10
 20
 30
 40
 50
 60
 0  10  20  30  40  50  60
Raft grade
Paxos grade
Raft then Paxos
Paxos then Raft
Figure 14: A scatter plot comparing 43 participants’ perfor-
mance on the Raft and Paxos quizzes. Points above the diag-
onal (33) represent participants who scored higher for Raft .
lecture covered enough material to create an equivalent
replicated state machine, including single-decree Paxos,
multi-decree Paxos, reconfiguration, and a few optimiza-
tions needed in practice (such as leader election). The
quizzes tested basic understanding of the algorithms and
also required students to reason about corner cases. Each
student watched one video, took the corresponding quiz,
watched the second video, and took the second quiz.
About half of the participants did the Paxos portion first
and the other half did the Raft portion first in order to
account for both individual differences in performance
and experience gained from the first portion of the study.
We compared participants’ scores on each quiz to deter-
mine whether participants showed a better understanding
of Raft.
We tried to make the comparison between Paxos and
Raft as fair as possible. The experiment favored Paxos in
two ways: 15 of the 43 participants reported having some
prior experience with Paxos, and the Paxos video is 14%
longer than the Raft video. As summarized in Table 1, we
have taken steps to mitigate potential sources of bias. All
of our materials are available for review [28, 31].
On average, participants scored 4.9 points higher on the
Raft quiz than on the Paxos quiz (out of a possible 60
points, the mean Raft score was 25.7 and the mean Paxos
score was 20.8); Figure 14 shows their individual scores.
A paired t-test states that, with 95% confidence, the true
distribution of Raft scores has a mean at least 2.5 points
larger than the true distribution of Paxos scores.
We also created a linear regression model that predicts
a new student’s quiz scores based on three factors: which
quiz they took, their degree of prior Paxos experience, and
 0
 5
 10
 15
 20
implement explain
number of participants
Paxos much easier
Paxos somewhat easier
Roughly equal
Raft somewhat easier
Raft much easier
Figure 15: Using a 5-point scale, participants were asked
(left) which algorithm they felt would be easier to implemen t
in a functioning, correct, and efficient system, and (right)
which would be easier to explain to a CS graduate student.
the order in which they learned the algorithms. The model
predicts that the choice of quiz produces a 12.5-point dif-
ference in favor of Raft. This is significantly higher than
the observed difference of 4.9 points, because many of the
actual students had prior Paxos experience, which helped
Paxos considerably, whereas it helped Raft slightly less.
Curiously, the model also predicts scores 6.3 points lower
on Raft for people that have already taken the Paxos quiz;
although we don’t know why, this does appear to be sta-
tistically significant.
We also surveyed participants after their quizzes to see
which algorithm they felt would be easier to implement
or explain; these results are shown in Figure 15. An over-
whelming majority of participants reported Raft would be
easier to implement and explain (33 of 41 for each ques-
tion). However, these self-reported feelings may be less
reliable than participants’ quiz scores, and participants
may have been biased by knowledge of our hypothesis
that Raft is easier to understand.
A detailed discussion of the Raft user study is available
at [31].
9.2 Correctness
We have developed a formal specification and a proof
of safety for the consensus mechanism described in Sec-
tion 5. The formal specification [31] makes the informa-
tion summarized in Figure 2 completely precise using the
TLA+ specification language [17]. It is about 400 lines
long and serves as the subject of the proof. It is also use-
ful on its own for anyone implementing Raft. We have
mechanically proven the Log Completeness Property us-
ing the TLA proof system [7]. However, this proof relies
on invariants that have not been mechanically checked
(for example, we have not proven the type safety of the
specification). Furthermore, we have written an informal
proof [31] of the State Machine Safety property which
is complete (it relies on the specification alone) and rela-
Concern Steps taken to mitigate bias Materials for review [28, 31]
Equal lecture quality Same lecturer for both. Paxos lecture based on and improved from exist-
ing materials used in several universities. Paxos lecture i s 14% longer.
videos
Equal quiz difficulty Questions grouped in difficulty and pai red across exams. quizzes
Fair grading Used rubric. Graded in random order, alternati ng between quizzes. rubric
Table 1: Concerns of possible bias against Paxos in the study, steps t aken to counter each, and additional materials available.
14

0%
20%
40%
60%
80%
100%
 100  1000  10000  100000
cumulative percent
150-150ms
150-151ms
150-155ms
150-175ms
150-200ms
150-300ms
0%
20%
40%
60%
80%
100%
 0  100  200  300  400  500  600
cumulative percent
time without leader (ms)
12-24ms
25-50ms
50-100ms
100-200ms
150-300ms
Figure 16: The time to detect and replace a crashed leader.
The top graph varies the amount of randomness in election
timeouts, and the bottom graph scales the minimum election
timeout. Each line represents 1000 trials (except for 100 tr i-
als for “150–150ms”) and corresponds to a particular choice
of election timeouts; for example, “150–155ms” means that
election timeouts were chosen randomly and uniformly be-
tween 150ms and 155ms. The measurements were taken on a
cluster of five servers with a broadcast time of roughly 15ms.
Results for a cluster of nine servers are similar.
tively precise (it is about 3500 words long).
9.3 Performance
Raft’s performance is similar to other consensus algo-
rithms such as Paxos. The most important case for per-
formance is when an established leader is replicating new
log entries. Raft achieves this using the minimal number
of messages (a single round-trip from the leader to half the
cluster). It is also possible to further improve Raft’s per-
formance. For example, it easily supports batching and
pipelining requests for higher throughput and lower la-
tency. V arious optimizations have been proposed in the
literature for other algorithms; many of these could be ap-
plied to Raft, but we leave this to future work.
We used our Raft implementation to measure the per-
formance of Raft’s leader election algorithm and answer
two questions. First, does the election process converge
quickly? Second, what is the minimum downtime that can
be achieved after leader crashes?
To measure leader election, we repeatedly crashed the
leader of a cluster of five servers and timed how long it
took to detect the crash and elect a new leader (see Fig-
ure 16). To generate a worst-case scenario, the servers in
each trial had different log lengths, so some candidates
were not eligible to become leader. Furthermore, to en-
courage split votes, our test script triggered a synchro-
nized broadcast of heartbeat RPCs from the leader before
terminating its process (this approximates the behavior
of the leader replicating a new log entry prior to crash-
ing). The leader was crashed uniformly randomly within
its heartbeat interval, which was half of the minimum
election timeout for all tests. Thus, the smallest possible
downtime was about half of the minimum election time-
out.
The top graph in Figure 16 shows that a small amount
of randomization in the election timeout is enough to
avoid split votes in elections. In the absence of random-
ness, leader election consistently took longer than 10 sec-
onds in our tests due to many split votes. Adding just 5ms
of randomness helps significantly, resulting in a median
downtime of 287ms. Using more randomness improves
worst-case behavior: with 50ms of randomness the worst-
case completion time (over 1000 trials) was 513ms.
The bottom graph in Figure 16 shows that downtime
can be reduced by reducing the election timeout. With
an election timeout of 12–24ms, it takes only 35ms on
average to elect a leader (the longest trial took 152ms).
However, lowering the timeouts beyond this point violates
Raft’s timing requirement: leaders have difficulty broad-
casting heartbeats before other servers start new elections.
This can cause unnecessary leader changes and lower
overall system availability. We recommend using a con-
servative election timeout such as 150–300ms; such time-
outs are unlikely to cause unnecessary leader changes and
will still provide good availability.
10 Related work
There have been numerous publications related to con-
sensus algorithms, many of which fall into one of the fol-
lowing categories:
• Lamport’s original description of Paxos [15], and at-
tempts to explain it more clearly [16, 20, 21].
• Elaborations of Paxos, which fill in missing details
and modify the algorithm to provide a better founda-
tion for implementation [26, 39, 13].
• Systems that implement consensus algorithms, such
as Chubby [2, 4], ZooKeeper [11, 12], and Span-
ner [6]. The algorithms for Chubby and Spanner
have not been published in detail, though both claim
to be based on Paxos. ZooKeeper’s algorithm has
been published in more detail, but it is quite different
from Paxos.
• Performance optimizations that can be applied to
Paxos [18, 19, 3, 25, 1, 27].
• Oki and Liskov’s Viewstamped Replication (VR), an
alternative approach to consensus developed around
the same time as Paxos. The original description [29]
was intertwined with a protocol for distributed trans-
actions, but the core consensus protocol has been
separated in a recent update [22]. VR uses a leader-
based approach with many similarities to Raft.
The greatest difference between Raft and Paxos is
Raft’s strong leadership: Raft uses leader election as an
essential part of the consensus protocol, and it concen-
15

trates as much functionality as possible in the leader. This
approach results in a simpler algorithm that is easier to
understand. For example, in Paxos, leader election is or-
thogonal to the basic consensus protocol: it serves only as
a performance optimization and is not required for achiev-
ing consensus. However, this results in additional mecha-
nism: Paxos includes both a two-phase protocol for basic
consensus and a separate mechanism for leader election.
In contrast, Raft incorporates leader election directly in to
the consensus algorithm and uses it as the first of the two
phases of consensus. This results in less mechanism than
in Paxos.
Like Raft, VR and ZooKeeper are leader-based and
therefore share many of Raft’s advantages over Paxos.
However, Raft has less mechanism that VR or ZooKeeper
because it minimizes the functionality in non-leaders. For
example, log entries in Raft flow in only one direction:
outward from the leader in AppendEntries RPCs. In VR
log entries flow in both directions (leaders can receive
log entries during the election process); this results in
additional mechanism and complexity. The published de-
scription of ZooKeeper also transfers log entries both to
and from the leader, but the implementation is apparently
more like Raft [35].
Raft has fewer message types than any other algo-
rithm for consensus-based log replication that we are
aware of. For example, we counted the message types VR
and ZooKeeper use for basic consensus and membership
changes (excluding log compaction and client interaction,
as these are nearly independent of the algorithms). VR
and ZooKeeper each define 10 different message types,
while Raft has only 4 message types (two RPC requests
and their responses). Raft’s messages are a bit more dense
than the other algorithms’, but they are simpler collec-
tively. In addition, VR and ZooKeeper are described in
terms of transmitting entire logs during leader changes;
additional message types will be required to optimize
these mechanisms so that they are practical.
Raft’s strong leadership approach simplifies the algo-
rithm, but it precludes some performance optimizations.
For example, Egalitarian Paxos (EPaxos) can achieve
higher performance under some conditions with a lead-
erless approach [27]. EPaxos exploits commutativity in
state machine commands. Any server can commit a com-
mand with just one round of communication as long as
other commands that are proposed concurrently commute
with it. However, if commands that are proposed con-
currently do not commute with each other, EPaxos re-
quires an additional round of communication. Because
any server may commit commands, EPaxos balances load
well between servers and is able to achieve lower latency
than Raft in W AN settings. However, it adds significant
complexity to Paxos.
Several different approaches for cluster member-
ship changes have been proposed or implemented in
other work, including Lamport’s original proposal [15],
VR [22], and SMART [24]. We chose the joint consensus
approach for Raft because it leverages the rest of the con-
sensus protocol, so that very little additional mechanism
is required for membership changes. Lamport’s
α-based
approach was not an option for Raft because it assumes
consensus can be reached without a leader. In comparison
to VR and SMART, Raft’s reconfiguration algorithm has
the advantage that membership changes can occur with-
out limiting the processing of normal requests; in con-
trast, VR stops all normal processing during configura-
tion changes, and SMART imposes an
α-like limit on the
number of outstanding requests. Raft’s approach also adds
less mechanism than either VR or SMART.
11 Conclusion
Algorithms are often designed with correctness, effi-
ciency, and/or conciseness as the primary goals. Although
these are all worthy goals, we believe that understandabil-
ity is just as important. None of the other goals can be
achieved until developers render the algorithm into a prac-
tical implementation, which will inevitably deviate from
and expand upon the published form. Unless developers
have a deep understanding of the algorithm and can cre-
ate intuitions about it, it will be difficult for them to retai n
its desirable properties in their implementation.
In this paper we addressed the issue of distributed con-
sensus, where a widely accepted but impenetrable algo-
rithm, Paxos, has challenged students and developers for
many years. We developed a new algorithm, Raft, which
we have shown to be more understandable than Paxos.
We also believe that Raft provides a better foundation
for system building. Using understandability as the pri-
mary design goal changed the way we approached the de-
sign of Raft; as the design progressed we found ourselves
reusing a few techniques repeatedly, such as decomposing
the problem and simplifying the state space. These tech-
niques not only improved the understandability of Raft
but also made it easier to convince ourselves of its cor-
rectness.
12 Acknowledgments
The user study would not have been possible with-
out the support of Ali Ghodsi, David Mazi` eres, and the
students of CS 294-91 at Berkeley and CS 240 at Stan-
ford. Scott Klemmer helped us design the user study,
and Nelson Ray advised us on statistical analysis. The
Paxos slides for the user study borrowed heavily from
a slide deck originally created by Lorenzo Alvisi. Spe-
cial thanks go to David Mazi` eres and Ezra Hoch for
finding subtle bugs in Raft. Many people provided help-
ful feedback on the paper and user study materials,
including Ed Bugnion, Michael Chan, Hugues Evrard,
16

Daniel Giffin, Arjun Gopalan, Jon Howell, Vimalkumar
Jeyakumar, Ankita Kejriwal, Aleksandar Kracun, Amit
Levy, Joel Martin, Satoshi Matsushita, Oleg Pesok, David
Ramos, Robbert van Renesse, Mendel Rosenblum, Nico-
las Schiper, Deian Stefan, Andrew Stone, Ryan Stutsman,
David Terei, Stephen Y ang, Matei Zaharia, 24 anony-
mous conference reviewers (with duplicates), and espe-
cially our shepherd Eddie Kohler. Werner V ogels tweeted
a link to an earlier draft, which gave Raft significant ex-
posure. This work was supported by the Gigascale Sys-
tems Research Center and the Multiscale Systems Cen-
ter, two of six research centers funded under the Fo-
cus Center Research Program, a Semiconductor Research
Corporation program, by STARnet, a Semiconductor Re-
search Corporation program sponsored by MARCO and
DARP A, by the National Science Foundation under Grant
No. 0963859, and by grants from Facebook, Google, Mel-
lanox, NEC, NetApp, SAP , and Samsung. Diego Ongaro
is supported by The Junglee Corporation Stanford Gradu-
ate Fellowship.
References
[1] B OLOSKY , W. J., B RADSHAW , D., H AAGENS , R. B.,
KUSTERS , N. P., AND LI, P. Paxos replicated state
machines as the basis of a high-performance data store.
In Proc. NSDI’11, USENIX Conference on Networked
Systems Design and Implementation (2011), USENIX,
pp. 141–154.
[2] B URROWS , M. The Chubby lock service for loosely-
coupled distributed systems. In Proc. OSDI’06, Sympo-
sium on Operating Systems Design and Implementation
(2006), USENIX, pp. 335–350.
[3] C AMARGOS , L. J., S CHMIDT , R. M., AND PEDONE , F.
Multicoordinated Paxos. In Proc. PODC’07, ACM Sym-
posium on Principles of Distributed Computing (2007),
ACM, pp. 316–317.
[4] C HANDRA , T. D., G RIESEMER , R., AND REDSTONE , J.
Paxos made live: an engineering perspective. In Proc.
PODC’07, ACM Symposium on Principles of Distributed
Computing (2007), ACM, pp. 398–407.
[5] C HANG , F., D EAN , J., G HEMAWAT, S., H SIEH , W. C.,
WALLACH , D. A., B URROWS , M., C HANDRA , T.,
FIKES , A., AND GRUBER , R. E. Bigtable: a distributed
storage system for structured data. In Proc. OSDI’06,
USENIX Symposium on Operating Systems Design and
Implementation (2006), USENIX, pp. 205–218.
[6] C ORBETT , J. C., D EAN , J., E PSTEIN , M., F IKES , A.,
FROST , C., F URMAN , J. J., G HEMAWAT, S., G UBAREV ,
A., H EISER , C., H OCHSCHILD , P., H SIEH , W., K AN -
THAK , S., K OGAN , E., L I, H., L LOYD , A., M ELNIK ,
S., M WAURA , D., N AGLE , D., Q UINLAN , S., R AO, R.,
ROLIG , L., S AITO , Y., S ZYMANIAK , M., T AYLOR , C.,
WANG , R., AND WOODFORD , D. Spanner: Google’s
globally-distributed database. In Proc. OSDI’12, USENIX
Conference on Operating Systems Design and Implemen-
tation (2012), USENIX, pp. 251–264.
[7] C OUSINEAU , D., D OLIGEZ , D., L AMPORT , L., M ERZ ,
S., R ICKETTS , D., AND VANZETTO , H. TLA + proofs.
In Proc. FM’12, Symposium on F ormal Methods (2012),
D. Giannakopoulou and D. M´ ery, Eds., vol. 7436 of Lec-
ture Notes in Computer Science , Springer, pp. 147–154.
[8] G HEMAWAT, S., G OBIOFF , H., AND LEUNG , S.-T. The
Google file system. In Proc. SOSP’03, ACM Symposium
on Operating Systems Principles (2003), ACM, pp. 29–43.
[9] G RAY, C., AND CHERITON , D. Leases: An efficient fault-
tolerant mechanism for distributed file cache consistency.
In Proceedings of the 12th ACM Ssymposium on Operating
Systems Principles (1989), pp. 202–210.
[10] H ERLIHY , M. P., AND WING , J. M. Linearizability: a
correctness condition for concurrent objects. ACM Trans-
actions on Programming Languages and Systems 12 (July
1990), 463–492.
[11] H UNT, P., K ONAR , M., J UNQUEIRA , F. P., AND REED ,
B. ZooKeeper: wait-free coordination for internet-scale
systems. In Proc ATC’10, USENIX Annual Technical Con-
ference (2010), USENIX, pp. 145–158.
[12] J UNQUEIRA , F. P., R EED , B. C., AND SERAFINI , M.
Zab: High-performance broadcast for primary-backup sys-
tems. In Proc. DSN’11, IEEE/IFIP Int’l Conf. on Depend-
able Systems & Networks (2011), IEEE Computer Society,
pp. 245–256.
[13] K IRSCH , J., AND AMIR , Y. Paxos for system builders.
Tech. Rep. CNDS-2008-2, Johns Hopkins University,
2008.
[14] L AMPORT , L. Time, clocks, and the ordering of events in
a distributed system. Commununications of the ACM 21, 7
(July 1978), 558–565.
[15] L AMPORT , L. The part-time parliament. ACM Transac-
tions on Computer Systems 16 , 2 (May 1998), 133–169.
[16] L AMPORT , L. Paxos made simple. ACM SIGACT News
32, 4 (Dec. 2001), 18–25.
[17] L AMPORT , L. Specifying Systems, The TLA+ Language
and Tools for Hardware and Software Engineers. Addison-
Wesley, 2002.
[18] L AMPORT , L. Generalized consensus and Paxos. Tech.
Rep. MSR-TR-2005-33, Microsoft Research, 2005.
[19] L AMPORT , L. Fast paxos. Distributed Computing 19 , 2
(2006), 79–103.
[20] L AMPSON , B. W. How to build a highly available system
using consensus. In Distributed Algorithms, O. Baboaglu
and K. Marzullo, Eds. Springer-V erlag, 1996, pp. 1–17.
[21] L AMPSON , B. W. The ABCD’s of Paxos. In Proc.
PODC’01, ACM Symposium on Principles of Distributed
Computing (2001), ACM, pp. 13–13.
[22] L ISKOV, B., AND COWLING , J. Viewstamped replica-
tion revisited. Tech. Rep. MIT-CSAIL-TR-2012-021, MIT,
July 2012.
[23] LogCabin source code. http://github.com/
logcabin/logcabin.
17

[24] L ORCH , J. R., A DYA, A., B OLOSKY , W. J., C HAIKEN ,
R., D OUCEUR , J. R., AND HOWELL , J. The SMART
way to migrate replicated stateful services. In Proc. Eu-
roSys’06, ACM SIGOPS/EuroSys European Conference on
Computer Systems (2006), ACM, pp. 103–115.
[25] M AO, Y., J UNQUEIRA , F. P., AND MARZULLO , K.
Mencius: building efficient replicated state machines for
W ANs. In Proc. OSDI’08, USENIX Conference on
Operating Systems Design and Implementation (2008),
USENIX, pp. 369–384.
[26] M AZI `ERES , D. Paxos made practical. http:
//www.scs.stanford.edu/˜dm/home/
papers/paxos.pdf, Jan. 2007.
[27] M ORARU , I., A NDERSEN , D. G., AND KAMINSKY , M.
There is more consensus in egalitarian parliaments. In
Proc. SOSP’13, ACM Symposium on Operating System
Principles (2013), ACM.
[28] Raft user study. http://ramcloud.stanford.
edu/˜ongaro/userstudy/.
[29] O KI , B. M., AND LISKOV, B. H. Viewstamped
replication: A new primary copy method to support
highly-available distributed systems. In Proc. PODC’88,
ACM Symposium on Principles of Distributed Computing
(1988), ACM, pp. 8–17.
[30] O’N EIL , P., C HENG , E., G AWLICK , D., AND ON EIL , E.
The log-structured merge-tree (LSM-tree). Acta Informat-
ica 33, 4 (1996), 351–385.
[31] O NGARO , D. Consensus: Bridging Theory and Practice .
PhD thesis, Stanford University, 2014 (work in progress).
http://ramcloud.stanford.edu/˜ongaro/
thesis.pdf.
[32] O NGARO , D., AND OUSTERHOUT , J. In search of an
understandable consensus algorithm. In Proc ATC’14,
USENIX Annual Technical Conference (2014), USENIX.
[33] O USTERHOUT , J., A GRAWAL , P., E RICKSON , D.,
KOZYRAKIS , C., L EVERICH , J., M AZI `ERES , D., M I-
TRA , S., N ARAYANAN , A., O NGARO , D., P ARULKAR ,
G., R OSENBLUM , M., R UMBLE , S. M., S TRATMANN ,
E., AND STUTSMAN , R. The case for RAMCloud. Com-
munications of the ACM 54 (July 2011), 121–130.
[34] Raft consensus algorithm website.
http://raftconsensus.github.io.
[35] R EED , B. Personal communications, May 17, 2013.
[36] R OSENBLUM , M., AND OUSTERHOUT , J. K. The design
and implementation of a log-structured file system. ACM
Trans. Comput. Syst. 10 (February 1992), 26–52.
[37] S CHNEIDER , F. B. Implementing fault-tolerant services
using the state machine approach: a tutorial. ACM Com-
puting Surveys 22, 4 (Dec. 1990), 299–319.
[38] S HVACHKO , K., K UANG , H., R ADIA , S., AND
CHANSLER , R. The Hadoop distributed file system.
In Proc. MSST’10, Symposium on Mass Storage Sys-
tems and Technologies (2010), IEEE Computer Society,
pp. 1–10.
[39] VAN RENESSE , R. Paxos made moderately complex.
Tech. rep., Cornell University, 2012.
18

Raft FAQ

Q: What do people use Raft for?

A: The most frequent use for Raft (and Paxos) is to build
fault-tolerant "configuration services" whose job is to keep track of
how responsibilities are currently assigned to servers in a large
deployment. This job is particularly sensitive for deployments with
replication; Raft-based configuration services are often used to
select primaries in a way that avoids split brain. The VMware FT
test-and-set server is a simple example of a configuration service.
Chubby, ZooKeeper, and etcd are more powerful fault-tolerant
configuration services based on Raft or Paxos; they are widely used.

Some databases, such as Spanner, CockroachDB, and Lab 3, use Raft or
Paxos to replicate the data. (In contrast, GFS, VMware FT, and Chain
Replication use simpler primary-backup for the data.) Some databases
use Raft or Paxos in two different ways: for the configuration service
that assigns responsibilities to servers (for every shard, who is
currently primary and who are backups), and separately to handle the
data within each shard.

Q: Does Raft sacrifice anything for simplicity?

A: Raft gives up some performance in return for clarity; for example:

* Every operation must be written to disk for persistence; performance
  probably requires batching many operations into each disk write.

* There can only usefully be a single AppendEntries in flight from the
  leader to each follower: followers reject out-of-order
  AppendEntries, and the sender's nextIndex[] mechanism requires
  one-at-a-time. A provision for pipelining many AppendEntries would
  be better.

* The snapshotting design is only practical for relatively small
  states, since it writes the entire state to disk. If the state is
  big (e.g. if it's a big database), you'd want a way to write just
  parts of the state that have changed recently.

* Similarly, bringing recovering replicas up to date by sending them a
  complete snapshot will be slow, needlessly so if the replica already
  has a snapshot that's only somewhat out of date.

* Servers may not be able to take much advantage of multi-core because
  operations must be executed one at a time (in log order).

These could be fixed by modifying Raft, but the result might have less
value as a tutorial.

Q: Is Raft used in real-world software, or do companies generally roll
their own flavor of Paxos (or use a different consensus protocol)?

A: There are several real-world users of Raft: Docker
(https://docs.docker.com/engine/swarm/raft/), etcd (https://etcd.io),
and MongoDB. Other systems said to be using Raft include CockroachDB,
RethinkDB, and TiKV. Maybe you can find more starting at
http://raft.github.io/

On the other hand, many real-world state-machine replication systems
(Google's Chubby, ZooKeeper's ZAB) are derived from the older
Multi-Paxos and Viewstamped Replication protocols.

Q: What is Paxos? In what sense is Raft simpler?

A: There is a protocol called Paxos that allows a set of servers to
agree on a single value. While Paxos requires some thought to
understand, it is far simpler than Raft. Here's an easy-to-read paper
about Paxos:

  http://css.csail.mit.edu/6.824/2014/papers/paxos-simple.pdf

However, Paxos solves a smaller problem than Raft. To build a
real-world replicated service, the replicas need to agree on an
indefinite sequence of values (the client commands), and they need
ways to efficiently recover when servers crash and restart or miss
messages. People have built such systems with Paxos as the starting
point; look up Google's Chubby and Paxos Made Live papers, and
ZooKeeper/ZAB. There is also a protocol called Viewstamped
Replication; it's a good design, and similar to Raft, but the paper
about it is hard to understand.

These real-world protocols are complex, and (before Raft) there was
not a good introductory paper describing how they work. The Raft
paper, in contrast, is relatively easy to read and fairly detailed.
That's a big contribution.

Whether the Raft protocol is inherently easier to understand than
something else is not clear. The issue is clouded by a lack of good
descriptions of other real-world protocols. In addition, Raft
sacrifices performance for clarity in a number of ways; that's fine
for a tutorial but not always desirable in a real-world protocol.

Q: How long had Paxos existed before the authors created Raft?

A: Paxos was invented in the late 1980s. Raft was developed around
2012.

Raft closely resembles a protocol called Viewstamped Replication,
originally published in 1988. There were replicated fault-tolerant file
servers built on top of Viewstamped Replication in the early 1990s,
though not in production use.

A bunch of real-world systems are derived from Paxos: Chubby, Spanner,
Megastore, and Zookeeper/ZAB. Starting in the early 2000s big web
sites and cloud providers needed fault-tolerant services, and Paxos
was dusted off at that time and put into production.

Q: How does Raft's performance compare to Paxos in real-world applications?

A: The fastest Paxos-derived protocols are probably faster than
Raft as described in the paper; have a look at ZAB/ZooKeeper and Paxos
Made Live. On the other hand, etcd3 (using Raft) claims to have
achieved better performance than zookeeper and many Paxos-based
implementations (https://www.youtube.com/watch?v=hQigKX0MxPw).

There are situations where Raft's leader is not so great. If the
datacenters containing replicas and clients are distant from each
other, people sometimes use agreement protocols derived from original
Paxos. The reason is that Paxos has no leader; any replica can start
an agreement; so clients can talk to the replica in their local
datacenter rather than having to talk to a leader in a distant
datacenter. ePaxos is an example.

Q: Why are we learning/implementing Raft instead of Paxos?

A: We're using Raft in 6.824 because there is a paper that clearly
describes how to build a complete replicated service using Raft. I
know of no satisfactory paper that describes how to build a complete
replicated server system based on Paxos.

Q: Are there systems like Raft that can survive and continue to
operate when only a minority of the cluster is active?

A: Not with Raft's properties. But you can do it with different
assumptions, or different client-visible semantics. The basic problem
is split-brain -- the possibility of multiple diverging copies of the
state, caused by multiple subsets of the replicas mutating the state
without being aware of each other. There are two solution approaches
that I know of.

If somehow clients and servers can learn exactly which servers are
live and which are dead (as opposed to live but unreachable due to
network failure), then one can build a system that can function as
long as one is alive, picking (say) the lowest-numbered server known
to be alive. However, it's usually impractical for one computer to
decide if another computer is dead, as opposed to the network losing
the messages between them. One way to do it is to have a human decide:
the human can inspect the servers and decide which are alive and dead.

The other approach is to allow split-brain operation, and to have a
way for servers to reconcile the resulting diverging state after
partitions are healed. This can be made to work for some kinds of
services, but has complex client-visible semantics (usually called
"eventual consistency"). Have a look at the COPS, FuzzyLog, and
Bitcoin papers which are assigned later in the course.

Q: In Raft, the service which is being replicated is not available to
the clients during an election process. In practice how much of a
problem does this cause?

A: The client-visible pause seems likely to be on the order of a tenth of a
second. The authors expect failures (and thus elections) to be rare,
since they only happen if machines or the network fails. Many servers
and networks stay up continuously for months or even years at a time, so
this doesn't seem like a huge problem for many applications.

Q: Are there other consensus systems that don't have leader-election
pauses?

A: There are versions of Paxos-based replication that do not have a leader
or elections, and thus don't suffer from pauses during elections.
Instead, any server can effectively act as leader at any time. The cost
of not having a leader is that more messages are required for each
agreement.

Q: How are Raft and VMware FT related?

A: Raft has no single point of failure, while VMware FT does have a
single point of failure in the form of the test-and-set server. In
that sense Raft is fundamentally more fault-tolerant than VMware FT.
One could fix this by implementing FT's test-and-set server as a
replicated service using Raft or Paxos.

VMware-FT can replicate any virtual machine guest, and thus any
server-style software, even software that has no idea that it is being
replicated. Raft is used as a library integrated into the application
software, which is usually designed specifically to work well with
replication.

Q: Why can't a malicious person take over a Raft server, or forge
incorrect Raft messages?

A: Raft doesn't include defenses against attacks like this. It assumes
that all participants are following the protocol, and that only the
correct set of servers is participating.

A real deployment would have to keep out malicious attackers. The most
straightforward option is to place the servers behind a firewall to
filter out packets from random people on the Internet, and to ensure
that all computers and people inside the firewall are trustworthy.

There may be situations where Raft has to operate on the same network as
potential attackers. In that case a good plan would be to authenticate
the Raft packets with some cryptographic scheme. For example, give each
legitimate Raft server a public/private key pair, have it sign all the
packets it sends, give each server a list of the public keys of
legitimate Raft servers, and have the servers ignore packets that aren't
signed by a key on that list.

Q: The paper mentions that Raft works under all non-Byzantine
conditions. What are Byzantine conditions and why could they make Raft
fail?

A: "Non-Byzantine conditions" means that the servers are fail-stop:
they either follow the Raft protocol correctly, or they halt. For
example, most power failures are non-Byzantine because they cause
computers to simply stop executing instructions; if a power failure
occurs, Raft may stop operating, but it won't send incorrect results
to clients.

Byzantine failure refers to situations in which some computers execute
incorrectly, because of bugs or because someone malicious is
controlling the computers. If a failure like this occurs, Raft may
send incorrect results to clients.

Most of 6.824 is about tolerating non-Byzantine faults. Correct
operation despite Byzantine faults is more difficult; we'll touch on
this topic at the end of the term.

Q: Is the assumption that a Raft cluster is provisioned in the same
physical location or can you deploy peers in
geographically-distributed data centers?

A: The typical deployment is a single data center.  We will see later
some systems that run Paxos across data centers (e.g., Spanner), which
is better done with a leaderless design so that a client can talk to a
local peer (instead of the potentially faraway leader).

Google's Chubby paper reports that their Chubby deployments are
typically a single data center, except for the root Chubby (which
spans geographically-separated data centers).  (Chubby isn't based on
Raft but uses Google's replicated-state machine library, which is
based on Paxos.)

Q: Are there variations of the Raft concensus algorithm that don't
require strict ordering of the operations?  (that is, they don't have
to follow the Leader Completeness Property.)

A: Yes, if you know whether operations commute.  Google "generalized
paxos" or "exploiting commutativity for practical fast replication".

Q: In Figure 1, what does the interface between client and
server look like?

A: Typically an RPC interface to the server. For a key/value storage
server such as you'll build in Lab 3, it's Put(key,value) and
Get(value) RPCs. The RPCs are handled by a key/value module in the
server, which calls Raft.Start() to ask Raft to put a client RPC in
the log, and reads the applyCh to learn of newly committed log
entries.

Q: What if a client sends a request to a leader, but the leader
crashes before sending the client request to all followers, and the
new leader doesn't have the request in its log? Won't that cause the
client request to be lost?

A: Yes, the request may be lost. If a log entry isn't committed, Raft
may not preserve it across a leader change.

That's OK because the client could not have received a reply to its
request if Raft didn't commit the request. The client will know (by
seeing a timeout or leader change) that its request may have been
lost, and will re-send it.

The fact that clients can re-send requests means that the system has
to be on its guard against duplicate requests; you'll deal with this
in Lab 3.

Q: If there's a network partition, can Raft end up with two leaders
and split brain?

A: No. There can be at most one active leader.

A new leader can only be elected if it can contact a majority of servers
(including itself) with RequestVote RPCs. So if there's a partition, and
one of the partitions contains a majority of the servers, that one
partition can elect a new leader. Other partitions must have only a
minority, so they cannot elect a leader. If there is no majority
partition, there will be no leader (until someone repairs the network).

Q: Suppose a new leader is elected while the network is partitioned,
but the old leader is in a different partition. How will the old
leader know to stop committing new entries?

A: The old leader will either not be able to get a majority of
successful responses to its AppendEntries RPCs (if it's in a minority
partition), or if it can talk to a majority, that majority must
overlap with the new leader's majority, and the servers in the overlap
will tell the old leader that there's a higher term. That will cause
the old leader to switch to follower.

Q: When some servers have failed, does "majority" refer to a majority
of the live servers, or a majority of all servers (even the dead
ones)?

A: Always a majority of all servers. So if there are 5 Raft peers in
total, but two have failed, a candidate must still get 3 votes
(including itself) in order to be elected leader.

There are many reasons for this. It could be that the two "failed"
servers are actually up and running in a different partition. From
their point of view, there are three failed servers. If they were
allowed to elect a leader using just two votes (from just the two
live-looking servers), we would get split brain. Another reason is
that we need the majorities of any two leader to overlap at at least
one server, to guarantee that a new leader sees the previous term
number and any log entries committed in previous terms; this requires
a majority out of all servers, dead and alive.

Q: What if the election timeout is too short? Will that cause Raft to
malfunction?

A: A bad choice of election timeout does not affect safety, it only
affects liveness.

If the election timeout is too small, then followers may repeatedly
time out before the leader has a chance to send out any AppendEntries.
In that case Raft may spend all its time electing new leaders, and no
time processing client requests. If the election timeout is too large,
then there will be a needlessly large pause after a leader failure
before a new leader is elected.

Q: Why randomize election timeouts?

A: To reduce the chance that multiple peers simultaneously become
candidates and divide the votes among themselves so that no-one gets a
majority.

Q: Can a candidate declare itself the leader as soon as it receives
votes from a majority, and not bother waiting for further RequestVote
replies?

A: Yes -- a majority is sufficient. It would be a mistake to wait
longer, because some peers might have failed and thus not ever reply.

Q: What network does Raft assume?

A: The network is unreliable: it may lose requests and replies and
delay them.  Raft's RPC library doesn't provide reliability; it is
best effort (e.g,. it sends a request but the network may drop
it). The lab's RPC library provides similar semantics: it may lose
requests, lose replies, delay messages, and entirely disconnect
particular hosts.

Q: What is the purpose of the votedFor check in the requestVote RPC?

A: Two candidates may start an election at the same time for the same
new term. A follower should vote only for one of them.

Q: Can a leader ever stop being a leader except by crashing?

A: Yes. If a leader's CPU is slow, or its network connection breaks,
or loses too many packets, or delivers packets too slowly, the other
servers won't see its AppendEntries RPCs, and will start an election.

Q: When are followers' log entries sent to their state machines?

A: Only after the leader says that an entry is committed, using the
leaderCommit field of the AppendEntries RPC. At that point the
follower can execute (or apply) the log entry, which for us means send
it on the applyCh.

Q: Should the leader wait for replies to AppendEntries RPCs?

A: The leader should send the AppendEntries RPCs concurrently, without
waiting. As replies come back, the leader should count them, and mark
the log entry as committed only when it has replies from a majority of
servers (including itself).

One way to do this in Go is for the leader to send each AppendEntries
RPC in a separate goroutine, so that the leader sends the RPCs
concurrently. Something like this:

  for each server {
    go func() {
      send the AppendEntries RPC and wait for the reply
      if reply.success == true {
        increment count
        if count == nservers/2 + 1 {
          this entry is committed
        }
      }
    } ()
  }

Q: What happens if a half (or more) of the servers die?

A: The service can't make any progress; it will keep trying to elect a
leader over and over. If/when enough servers come back to life with
persistent Raft state intact, they will be able to elect a leader and
continue.

Q: Why is the Raft log 1-indexed?

A: You should view it as zero-indexed, but starting out with an entry
(at index=0) that has term 0. That allows the very first AppendEntries
RPC to contain 0 as PrevLogIndex, and be a valid index into the log.

Q: When the network partitions, won't client requests in minority
partitions be lost?

A: Only the partition with a majority of servers can commit and
execute client operations. The servers in the minority partition(s)
won't be able to commit client operations, so they won't reply to
client requests. Clients will keep re-sending the requests until they
can contact a majority Raft partition, so these clients' requests
won't be lost forever.

Q: Is the argument in 5.4.3 a complete proof?

A: 5.4.3 is not a complete proof. Here are some places to look:

http://ramcloud.stanford.edu/~ongaro/thesis.pdf
http://verdi.uwplse.org/raft-proof.pdf

Q: Are there any limitations to what applications can be built on top of Raft?

A: I think that in order to fit cleanly into a replicated state
machine framework like Raft, the replicated service has to be
self-contained -- it can have private state, and accept commands from
clients that update the state, but it can't contact outside entities
without special precautions. If the replicated application interacts
with the outside world, the outside world has to be able to deal
correctly with repeated requests (due to replication and replay of
logs after reboot), and it has to never contradict itself (i.e. it has
to be careful to send exactly the same answer to all replicas, and to
all re-executions of log entries). That in turn seems to require that
any outside entity that a Raft-based application contacts must itself
be fault-tolerant, i.e. probably has to use Raft or something like it.
That's fairly limiting.

As an example, imagine a replicated online ordering system sending
credit card charging requests to some external credit card processor
service. That external processor will see repeated requests (one or more
from each replica). Will it respond exactly the same way to each
request? Will it charge the credit card more than once? If it does do
the right thing, will it still do the right thing if it crashes and
reboots at an awkward time?

Q: What is the copy-on-write optimization in section 7?

A: The basic idea is for the server to fork() when the service wants
to make a snapshot, giving the child a complete copy of the in-memory
state. If fork() really copied all the memory, and the state was
large, this would be slow. But most operating systems don't copy all
the memory in fork(); instead they mark the memory pages as
"copy-on-write", and make them read-only in both parent and
child. Then the operating system will see a page fault the server
tries to write a page, and the operating system will only copy the
page at that point.  The net effect is usually that the child sees a
copy of its parent process' memory at the time of the fork(), but with
relatively little copying.  (This optimization is not necessary for
your labs.)

Q: Why is it called Raft?

A: https://groups.google.com/g/raft-dev/c/95rZqptGpmU

Q: Is it important for Raft/Paxos to work correctly?

A: Raft/Paxos are often the foundation of a distributed system. For
example, many of Google's services (GFS, Spanner, BigTable, etc.) rely
on Chubby, a configuration service based on Paxos.  As another
example, many compananies use Kubernetes to manage their containers;
Kubernetes in turn stores the configuration information about the
containers in Etcd, a key/value service based on Raft.  Occasionally
issues with systems based on Paxos/Raft materialize and lead to major
outages; see, for example,
https://decentralizedthoughts.github.io/2020-12-12-raft-liveness-full-omission/

Raft (2) FAQ

Q: What are some uses of Raft?

A: You could (and will) build a fault-tolerant key/value database using Raft.

You could make the MapReduce master fault-tolerant with Raft.

You could build a fault-tolerant locking service.

Q: In Section 8, why does a new leader need to commit a no-op entry at
the start of its term?

A: The problem situation is shown in Figure 8, where if S1 becomes
leader after (b), it cannot know if its last log entry (2) is
committed or not. The situation in which the last log entry will turn
out not to be committed is if S1 immediately fails, and S5 is the next
leader; in that case S5 will force all peers (including S1) to have
logs identical to S5's log, which does not include entry 2.

But suppose S1 manages to commit a new entry during its term (term 4).
If S5 sees the new entry, S5 will erase 3 from its log and accept 2 in
its place. If S5 does not see the new entry, S5 cannot be the next
leader if S1 fails, because it will fail the Election Restriction.
Either way, once S1 has committed a new entry for its term, it can
correctly conclude that every preceding entry in its log is committed.

The no-op text at the end of Section 8 is talking about an optimization
in which the leader executes and answers read-only commands (e.g.
get("k1")) without committing those commands in the log. For example,
for get("k1"), the leader just looks up "k1" in its key/value table and
sends the result back to the client. If the leader has just started, it
may have at the end of its log a put("k1", "v99"). Should the leader
send "v99" back to the client, or the value in the leader's key/value
table? At first, the leader doesn't know whether that v99 log entry is
committed (and must be returned to the client) or not committed (and
must not be sent back). So (if you are using this optimization) a new
Raft leader first tries to commit a no-op to the log; if the commit
succeeds (i.e. the leader doesn't crash), then the leader knows
everything before that point is committed.

Q: How does using the heartbeat mechanism to provide leases (for
read-only) operations work, and why does this require timing for
safety (e.g. bounded clock skew)?

A: I don't know exactly what the authors had in mind. Perhaps every
AppendEntries RPC the leader sends out says or implies that the no other
leader is allowed to be elected for the next 100 milliseconds. If the
leader gets positive responses from a majority, then the leader can
serve read-only requests for the next 100 milliseconds without further
communication with the followers.

This requires the servers to have the same definition of what 100
milliseconds means, i.e. they must have clocks that tick at close to
the same rate.

Q: What exactly do the C_old and C_new variables in Section 6 (and
Figure 11) represent? Are they the leader in each configuration?

A: They are the set of servers in the old/new configuration. The paper
doesn't provide details. I believe it's the identities (network names
or addresses) of the servers.

Q: When transitioning from cluster C_old to cluster C_new, how can we
create a hybrid cluster C_{old,new}? I don't really understand what
that means. Isn't it following either the network configuration of
C_old or of C_new? What if the two networks disagreed on a connection?

A: During the period of joint consensus (while Cold,new is active), the
leader is required to get a majority from both the servers in Cold and
the servers in Cnew.

There can't really be disagreement, because after Cold,new is committed
into the logs of both Cold and Cnew (i.e. after the period of joint
consensus has started), any new leader in either Cold or Cnew is
guaranteed to see the log entry for Cold,Cnew.

Q: I'm confused about Figure 11 in the paper. I'm unsure about how
exactly the transition from 'C_old' to 'C_old,new' to 'C_new' goes.
Why is there the issue of the cluster leader not being a part of the
new configuration, where the leader steps down once it has committed
the 'C_new' log entry? (The second issue mentioned in Section 6)

A: Suppose C_old={S1,S2,S3} and C_new={S4,S5,S6}, and that S1 is the leader
at the start of the configuration change. At the end of the
configuration change, after S1 has committed C_new, S1 should not be
participating any more, since S1 isn't in C_new. One of S4, S5, or S6
should take over as leader.

Q: About cluster configuration: During the configuration change time, if
we have to stop receiving requests from the clients, then what's the
point of having this automated configuration step? Doesn't it suffice
to just 1) stop receiving requests 2) change the configurations 3)
restart the system and continue?

A: The challenge here is ensuring that the system is correct even if there
are failures during this process, and even if not all servers get the
"stop receiving requests" and "change the configuration" commands at the
same time. Any scheme has to cope with the possibility of a mix of
servers that have and have not seen or completed the configuration
change -- this is true even of a non-automated system. The paper's
protocol is one way to solve this problem.

Q: The last two paragraphs of section 6 discuss removed servers
interfering with the cluster by trying to get elected even though
they've been removed from the configuration. Wouldn't a simpler
solution be to require servers to be shut down when they leave the
configuration? It seems that leaving the cluster implies that a server
can't send or receive RPCs to the rest of the cluster anymore, but the
paper doesn't assume that. Why not? Why can't you assume that the
servers will shut down right away?

A: I think the immediate problem is that the Section 6 protocol doesn't
commit Cnew to the old servers, it only commits Cnew to the servers in
Cnew. So the servers that are not in Cnew never learn when Cnew takes
over from Cold,new.

The paper does say this:

  When the new configuration has been committed under the rules of Cnew,
  the old configuration is irrelevant and servers not in the new
  configuration can be shut down.

So perhaps the problem only exists during the period of time between the
configuration change and when an administrator shuts down the old
servers. I don't know why they don't have a more automated scheme.

Q: How common is it to get a majority from both the old and new
configurations when doing things like elections and entry commitment,
if it's uncommon, how badly would this affect performance?

A: I imagine that in most cases there is no failure, and the leader gets
both majorities right away. Configuration change probably only takes a
few round trip times, i.e. a few dozen milliseconds, so the requirement
to get both majorities will slow the system down for only a small amount
of time. Configuration change is likely to be uncommon (perhaps every
few months); a few milliseconds of delay every few months doesn't seem
like a high price.

Q: And how important is the decision to have both majorities?

A: The requirement for both majorities is required for correctness, to
cover the possibility that the leader fails during the configuration
change.

Q: Just to be clear, the process of having new members join as non-voting
entities isn't to speed up the process of replicating the log, but
rather to influence the election process? How does this increase
availability? These servers that need to catch up are not going to be
available regardless, right?

A: The purpose of non-voting servers is to allow those servers to get a
complete copy of the leader's log without holding up new commits. The
point is to allow a subsequent configuration change to be quick. If the
new servers didn't already have nearly-complete logs, then the leader
wouldn't be able to commit Cold,new until they caught up; and no new
client commands can be executed between the time Cold,new is first sent
out and the time at which it is committed.

Q: If the cluster leader does not have the new configuration, why doesn't
it just remove itself from majority while committing C_new, and then
when done return to being leader? Is there a need for a new election
process?

A: Is this about the "second issue" in Section 6? The situation they
describe is one in which the leader isn't in the new configuration at
all. So after Cnew is committed, the leader shouldn't be participating
in Raft at all.

Q: How does the non-voting membership status work in the configuration change
portion of Raft. Does that server state only last during the changeover
(i.e. while c_new not committed) or do servers only get full voting privileges
after being fully "caught up"? If so, at what point are they considered
"caught up"?

A: The paper doesn't have much detail here. I imagine that the leader won't
start the configuration change until the servers to be added (the
non-voting members) are close to completely caught up. When the leader
sends out the Cold,new log entry in AppendEntries RPCs to those new
servers, the leader will bring them fully up to date (using the Figure 2
machinery). The leader won't be able to commit the Cold,new message
until a majority of those new servers are fully caught up. Once the
Cold,new message is committed, those new servers can vote.

Q: I don't disagree that having servers deny RequestVotes that are less than
the minimum election timeout from the last heartbeat is a good idea (it
helps prevent unnecessary elections in general), but why did they choose
that method specifically to prevent servers not in a configuration running
for election? It seems like it would make more sense to check if a given
server is in the current configuration. E.g., in the lab code we are using,
each server has the RPC addresses of all the servers (in the current
configuration?), and so should be able to check if a requestVote RPC came
from a valid (in-configuration) server, no?

Q: I agree that the paper's design seems a little awkward, and I don't know
why they designed it that way. Your idea seems like a reasonable
starting point. One complication is that there may be situations in
which a server in Cnew is leader during the joint consensus phase, but
at that time some servers in Cold may not know about the joint consensus
phase (i.e. they only know about Cold, not Cold,new); we would not want
the latter servers to ignore the legitimate leader.

Q: When exactly does joint consensus begin, and when does it end? Does joint
consensus begin at commit time of "C_{o,n}"?

A: Joint consensus is in progress when the current leader is aware of
Cold,new. If the leader doesn't manage to commit Cold,new, and crashes,
and the new leader doesn't have Cold,new in its log, then joint
consensus ends early. If a leader manages to commit Cold,new, then joint
consensus has not just started but will eventually complete, when a
leader commits Cnew.

Q: Can the configuration log entry be overwritten by a subsequent leader
(assuming that the log entry has not been committed)?

A: Yes, that is possible, if the original leader trying to send out Cold,new
crashes before it commits the Cold,new.

Q: How can the "C_{o,n}" log entry ever be committed? It seems like it must be
replicated to a majority of "old" servers (as well as the "new" servers), but
the append of "C_{o,n}" immediately transitions the old server to new, right?

A: The commit does not change the set of servers in Cold or Cnew. For example,
perhaps the original configuration contains servers S1, S2, S3; then Cold
is {S1,S2,S3}. Perhaps the desired configuration is S4, S5, S6; then
Cnew is {S4,S5,S6}. Once Cnew is committed to the log, the configuration
is Cnew={S4,S5,S6}; S1,S2, and S3 are no longer part of the configuration.

Q: When snapshots are created, is the data and state used the one for the
client application? If it's the client's data then is this something
that the client itself would need to support in addition to the
modifications mentioned in the raft paper?

A: Example: if you are building a key/value server that uses Raft for
replication, then there will be a key/value module in the server that
stores a table of keys and values. It is that table of keys and values
that is saved in the snapshot.

Q: The paper says that "if the follower receives a snapshot that
describes a prefix of its log, then log entries covered by the
snapshot are deleted but entries following the snapshot are retained".
This means that we could potentially be deleting operations on the
state machine.

A: I don't think information will be lost. If the snapshot covers a prefix
of the log, that means the snapshot includes the effects of all the
operations in that prefix. So it's OK to discard that prefix.

Q: It seems that snapshots are useful when they are a lot smaller than
applying the sequence of updates (e.g., frequent updates to a few
keys). What happens when a snapshot is as big as the sum of its
updates (e.g., each update inserts a new unique key)? Are there any
cost savings from doing snapshots at all in this case?

A: If the snapshot is about as big as the log, then there may not be a lot
of value in having snapshots. On the other hand, perhaps the snapshot
organizes the data in a way that's easier to access than a log, e.g. in
a sorted table. Then it might be faster to re-start the service after a
crash+reboot from a snapshotted table than from the log (which you would
have to sort).

It's much more typical, however, for the log to be much bigger than the
state.

Q: Also wouldn't a InstallSnapshot incur heavy bandwidth costs?

A: Yes, if the state is large (as it would be for e.g. a database).
However, this is not an easy problem to solve. You'd probably want the
leader to keep enough of its log to cover all common cases of followers
lagging or being temporarily offline. You might also want a way to
transfer just the differences in server state, e.g. just the parts of
the database that have changed recently.


Q: Is there a concern that writing the snapshot can take longer than the
election timeout because of the amount of data that needs to be
appended to the log?

A: You're right that it's a potential problem for a large server. For
example if you're replicating a database with a gigabyte of data, and
your disk can only write at 100 megabytes per second, writing the
snapshot will take ten seconds. One possibility is to write the snapshot
in the background (i.e. arrange to not wait for the write, perhaps by
doing the write from a child process), and to make sure that snapshots
are created less often than once per ten seconds.


Q: Under what circumstances would a follower receive a snapshot that is a
prefix of its own log?

A: The network can deliver messages out of order, and the RPC handling
system can execute them out of order. So for example if the leader sends
a snapshot for log index 100, and then one for log index 110, but the
network delivers the second one first.

Q: Additionally, if the follower receives a snapshot that is a
prefix of its log, and then replaces the entries in its log up to that
point, the entries after that point are ones that the leader is not
aware of, right?

A: The follower might have log entries that are not in a received snapshot
if the network delays delivery of the snapshot, or if the leader has
sent out log entires but not yet committed them.

Q: Will those entries ever get committed?

A: They could get committed.


Q: How does the processing of InstallSnapshot RPC handle reordering, when the
check at step 6 references log entries that have been compacted? Specifically,
shouldn't Figure 13 include: 1.5: If lastIncludedIndex < commitIndex, return
immediately.  or alternatively 1.5: If there is already a snapshot and
lastIncludedIndex < currentSnapshot.lastIncludedIndex, return immediately.

A: I agree -- for Lab 3C the InstallSnapshot RPC handler must reject stale
snapshots. I don't know why Figure 13 doesn't include this test; perhaps
the authors' RPC system is better about order than ours. Or perhaps the
authors intend that we generalize step 6 in Figure 13 to cover this case.


Q: What happens when the leader sends me an InstallSnapshot command that
is for a prefix of my log, but I've already undergone log compaction
and my snapshot is ahead? Is it safe to assume that my snapshot that
is further forward subsumes the smaller snapshot?


A: Yes, it is correct for the recipient to ignore an InstallSnapshot if the
recipient is already ahead of that snapshot. This case can arise in Lab
3, for example if the RPC system delivers RPCs out of order.


Q: How do leaders decide which servers are lagging and need to be sent a
snapshot to install?

A: If a follower rejects an AppendEntries RPC for log index i1 due to rule
#2 or #3 (under AppendEntries RPC in Figure 2), and the leader has
discarded its log before i1, then the leader will send an
InstallSnapshot rather than backing up nextIndex[].


Q: In actual practical use of raft, how often are snapshots sent?

A: I have not seen an analysis of real-life Raft use. I imagine people
using Raft would tune it so that snapshots were rarely needed (e.g. by
having leaders keep lots of log entries with which to update lagging
followers).  The etcd pages give some guidance:
https://etcd.io/docs/v3.4/op-guide/maintenance/


Q: Is InstallSnapshot atomic? If a server crashes after partially
installing a snapshot, and the leader re-sends the InstallSnapshot
RPC, is this idempotent like RequestVote and AppendEntries RPCs?

A: The implementation of InstallSnapshot must be atomic.

It's harmless for the leader to re-send a snapshot.

Q: Why is an offset needed to index into the data[] of an InstallSNapshot
RPC, is there data not related to the snapshot? Or does it overlap
previous/future chunks of the same snapshot? Thanks!

A: The complete snapshot may be sent in multiple RPCs, each containing a
different part ("chunk") of the complete snapshot. The offset field
indicates where this RPC's data should go in the complete snapshot.

Q: How does a leader know when to send a snapshot to a follower?

A: When the matchIndex for the follower is smaller than the index for
the beginning of the leader's log.

Q: How does copy-on-write help with the performance issue of creating snapshots?

A: The basic idea is for the server to fork() when the service wants
to make a checkpoint, giving the child a complete copy of the
in-memory state. If fork() really copied all the memory, and the state
was large, this would be slow. But most operating systems don't copy
all the memory in fork(); instead they mark the pages as
"copy-on-write", and make them read-only in both parent and
child. Then the operating system will see a page fault if either tries
to write a page, and the operating system will only copy the page at
that point.  The net effect is usually that the child sees a copy of
its parent process' memory at the time of the fork(), but with
relatively little copying.


Q: What data compression schemes, such as VIZ, ZIP, Huffman encoding,
etc. are most efficient for Raft snapshotting?

A: It depends on what data the service stores. If it stores images, for
example, then maybe you'd want to compress them with JPEG.

If you are thinking that each snapshot probably shares a lot of content
with previous snapshots, then perhaps you'd want to use some kind of
tree structure which can share nodes across versions.


Q: Does adding an entry to the log count as an executed operation?

A: No. A server should only execute an operation in a log entry after the
leader has indicated that the log entry is committed. "Execute" means
handing the operation to the service that's using Raft. In Lab 3,
"execute" means that Raft gives the committed log entry to your
key/value software, which applies the Put(key,value) or Get(key) to its
table of key/value pairs.


Q: According to the paper, a server disregards RequestVoteRPCs when they
think a current leader exists, but then the moment they think a
current leader doesn't exist, I thought they try to start their own
election. So in what case would they actually cast a vote for another
server?

For the second question, I'm still confused: what does the paper mean
when it says a server should disregard a RequestVoteRPC when it thinks
a current leader exists at the end of Section 6? In what case would a
server think a current leader doesn't exist but hasn't started its own
election? Is it if the server thinks it hasn't yet gotten a heartbeat
from the server but before its election timeout?

A: Each server waits for a randomly chosen election timeout; if it hasn't heard
from the leader for that whole period, and no other server has started an
election, then the server starts an election. Whichever server's election timer
expires first is likely to get votes from most or all of the servers before any
other server's timer expires, and thus is likely to win the election.

Suppose the heartbeat interval is 10 milliseconds (ms). The leader sends
out heartbeats at times 10, 20, and 30.

Suppose server S1 doesn't hear the heartbeat at time 30. S1's election timer
goes off at time 35, and S1 sends out RequestVote RPCs.

Suppose server S2 does hear the heartbeat at time 30, so it knows the
server was alive at that time. S2 will set its election timer to go off
no sooner than time 40, since only a missing heartbeat indicates a
possibly dead server, and the next heartbeat won't come until time 40.
When S2 hears S1'a RequestVote at time 35, S2 can ignore the
RequestVote, because S2 knows that it heard a heartbeat less than one
heartbeat interval ago.


Q: I'm a little confused by the "how to roll back quickly" part of the
Lecture 6 notes (and the corresponding part of the paper):

  paper outlines a scheme towards end of Section 5.3:
  if follower rejects, includes this in reply:
    the term of the conflicting entry
    the index of the first entry for conflicting term
  if leader knows about the conflicting term:
    move nextIndex[i] back to its last entry for the conflicting term
  else:
    move nextIndex[i] back to follower's first index

I think according to the paper, the leader should move nextIndex[i] to
the index of the first entry for conflicting term. What does the
situation "if leader knows about the conflicting term" mean?

A: The paper's description of the algorithm is not complete, so we have to
invent the details for ourselves. The notes have the version I invented;
I don't know if it's what the authors had in mind.

The specific problem with the paper's "index of the first entry for the
conflicting term" is that the leader might no have entries at all for
the conflicting term. Thus my notes cover two cases -- if the server
knows about the conflicting term, and if it doesn't.


Q: What are the tradeoffs in network/ performance in decreasing nextIndex
by a factor of 2 each time at each mismatch? i.e. first by 1,2,4, 8
and so on

A: The leader will overshoot by up to a factor of two, and thus have to
send more entries than needed. Of course the Figure 2 approach is also
wasteful if one has to back up a lot. Best might be to implement
something more precise, for example the optimization outlined towards
the end of section 5.3.

Q: Why are read-only operations avoiding the log?

A: It is more efficient to avoid running read-only operations through
Raft and store them in the log.  Read-only operations are typically
more common than write operations, so this could save both time and
space.

Q: The paper says that before responding to read-only requests, the
leader exchanges heart-beats with a majority of the servers. How does
this fit into the existing Raft framework?

A: The leader sends AppendEntry RPCs (most likely empty) to all the
peers and replies after a majority of them respond.  This is more
efficient than running Raft because the operations are not stored in
the log, and the leader doesn't have to wait until the operation
appears on the applyCh, which serializes all operations.


Q: Unrelatedly - How does your experience teaching Raft and Paxos
correspond to section 9.1 of the paper? Do your experiences support
their findings?


A: I was pretty happy with the 6.824 Paxos labs from a few years ago.
I'm pretty happy with the current Raft labs too. The Raft labs are
more ambitious: unlike the Paxos labs, the Raft labs have a leader,
persistence, and snapshots. I don't think we have any light to shed on
the findings in Section 9.1; we didn't perform a side-by-side
experiment on the students, and our Raft labs are noticeably more
ambitious.


Q: What has been the impact of Raft, from the perspective of academic
researchers in the field? Is it considered significant, inspiring,
non-incremental work? Or is it more of "okay, this seems like a
natural progression, and is a bit easier to teach, so let's teach
this?"

A: The Raft paper does a better job than any paper I know of in explaining
modern replicated state machine techniques. I think it has inspired lots
of people to build their own replication implementations.


Q: The paper states that there are a fair amount of implementations of Raft out
in the wild. Have there been any improvement suggestions that would make sense
to include in a revised version of the algorithm?

A: Here are two examples:

https://www.cl.cam.ac.uk/~ms705/pub/papers/2015-osr-raft.pdf
https://www.usenix.org/system/files/nsdi19-park.pdf

FAQ for Ray

Q: Is Ray used in practice?

A: Ray is an open-source project (https://github.com/ray-project/ray)
with many users, including OpenAI.  Anyscale offers a commericial
offering of Ray as a service.

Q: What application benefit from Ray?

A: Ray is in the line of systems such as MapReduce, Spark, etc., but
can handle a wider class of applications by supporting both functional
and stateful computations and providing low-latency for invoking
computations.

Q: What is the scheduling policy that Ray uses?

A: The default policy is try to run a task locally. It the local
scheduler doesn't have enough capacity for the request, asks a random
other scheduler if it can run the task (based on the requested
resources for the task).  The details are here:
https://docs.ray.io/en/latest/ray-core/scheduling/index.html

Q: What's the difference between actors and tasks?

A: Actors is a kind of task: a stateful one that can be invoked
multiple times and keeps its state between invocations.

Q: What is the actor model?

A: A model of computation where a computation can maintain local state
between invocations. For more specifics, see
https://en.wikipedia.org/wiki/Actor_model

Q: Why even bother supporting actors?

A: Developers may want to have long-running tasks that can maintain
state between invocations. For example, because the Router in 3(a) is
an actor, it can perform batching. Each model actor in 3(a) keeps
weights between invocation to keep them warm in the local GPU memory.

Q: What is lineage reconstruction?

A: Lineage reconstruction recomputes results by tracing back the call
graph to a task that hasn't crashed and then re-executing its tasks,
reproducing the tasks that crashed.  The computations must be
idempotent.


Q: What are the tradeoffs between recovering value through lineage
tree re-execution compared to recovering values through persistence
log?

A: Lineage recovery requires that the computations are
idempotent. Lineage recovery may take a long time and expensive since
the system may have to re-run many computations, which may produce
large amounts of data.  This is the reason that Ray tries to speed up
this recovery up by re-using secondary copies. If the second copy,
exists it doesn't have to re-run the computation to generate it.

Q: How would the system handle non-idempotent tasks, or what would be
the strategy to deal with side effects?

A: Ray by itself doesn't handle non-idempotent tasks. You would need,
for example, a transaction-like implementation plan with write-ahead
logs.  This out of scope of the Ray system itself, but actors could
implement a plan like that on their own.

Q: What is fate sharing?

A: Fate sharing refers the general idea of forcing a tasks to share
fate with the tasks that failed; that is, fail it too.  In Ray this is
important because task blocked on a dangling future may block forever
because the future will never complete because the task that might
have completed it has crashed.

Q: Why is the system called Ray?

A: There is no real reason behind the name. According to the authors,
it was one of several choices and it sounded nice.

Q: Is the idea of "ownership" here inspired by ownership in other
contexts, e.g. programming langugae design (such as Rust)?

A: The idea of ownership is a common idea in systems.  I don't think
there is much overlap between Ray and Rust in terms of their specific
use of ownership.  For example, a borrower can modify an object in
Ray, which is disallowed in Rust.

Q: If Ray were implemented in Go could one use a distributed garbage
collector instead of reference counting?

A: One could implement a distributed garbage collector that leverages
Go's local garbage collector but that is particular challenging in
distributed systems, because of failures and because of potentially
long pause times (because the GC has to contact a remote node).  Here
is a survey article on distributed garbage collectors:
http://portal.acm.org/citation.cfm?doid=292469.292471

Q: What is the security model in Ray?

A: Each Ray deployment is for a single application, isolated from
other deployments using standard approaches: renting private machines,
virtual machines, or containers.  That is, Ray itself is not
multi-tenant.

Ownership: A Distributed Futures System for Fine-Grained Tasks
Stephanie Wang*, Eric Liang*, Edward Oakes*, Ben Hindman, Frank Luan, Audrey Cheng, Ion Stoica
UC Berkeley, *and Anyscale
Abstract
The distributed futures interface is an increasingly popular
choice for building distributed applications that manipulate
large amounts of data. Distributed futures are an extension
of RPC that combines futures and distributed memory: a
distributed future is a reference whose eventual value may
be stored on a remote node. An application can then express
distributed computation without having to specify when or
where execution should occur and data should be moved.
Recent distributed futures applications require the ability
to execute fine-grained computations, i.e., tasks that run on
the order of milliseconds. Compared to coarse-grained tasks,
fine-grained tasks are difficult to execute with acceptable sys-
tem overheads. In this paper, we present a distributed futures
system for fine-grained tasks that provides fault tolerance
without sacrificing performance. Our solution is based on a
novel concept called ownership, which assigns each object a
leader for system operations. We show that this decentralized
architecture can achieve horizontal scaling, 1ms latency per
task, and fast failure handling.
1 Introduction
RPC is a standard for building distributed applications be-
cause of its generality and because its simple semantics yield
high-performance implementations. The original proposal
uses synchronous calls that copy return values back to the
caller (Figure 2a). Several recent systems [4, 34, 37, 45] have
extended RPC so that, in addition to distributed communica-
tion, the system may also manage data movement and paral-
lelism on behalf of the application.
Data movement. Pass-by-value semantics require all RPC
arguments to be sent to the executor by copying them directly
into the request body. Thus, performance degrades with large
data. Data copying is both expensive and unnecessary in cases
like Figure 2a, where a process executes an RPC over data
that it previously returned to the same caller.
To reduce data copies, some RPC systems use distributed
memory [16, 27, 37, 40, 41]. This allows large arguments to be
passed by reference (Figure 2b), while small arguments can
still be passed by value. In the best case, arguments passed
by reference to an RPC do not need to be copied if they are
already on the same node as the executor (Figure 2b). Note
that, like traditional RPC, we make all values immutable to
simplify the consistency model and implementation.
Parallelism. RPCs are traditionally blocking, so control is
only returned to the caller once the reply is received (Fig-
a_future = compute()
b_future = compute()
c_future = add(a_future, b_future)
c = system.get(c_future)
Figure 1: A distributed futures program. compute and add are state-
less. a_future, b_future, and c_future are distributed futures.
a=f()a=f()
Driver Worker 1 Worker 2
b=f()
a=f()
c=add(
  a,b)
a
b
c
ba
Pass by value
Pass by reference
(Distributed memory)
Blocking RPCsNonblocking RPCs (Futures)
(a)
(c)
(b)
(d)
Driver Worker 1 Worker 2
b=f()
c=add(
  a,b)
a
b
ba
c
a
Driver Worker 1 Worker 2
b=f()
b
Driver Worker 1 Worker 2
b=f()
a=f()
c=add(
  a,b)
a
b
c
a
c
c=add(
  a,b)
cc
a
Figure 2: Example executions of the program from Figure 1.(a) With
RPC. (b) With RPC and distributed memory, allowing the system to
reduce data copies. (c) With RPC and futures, allowing the system
to manage parallel execution. (d) With distributed futures.
ure 2a). Futures are a popular method for extending RPC
with asynchrony [8, 29], allowing the system to execute func-
tions in parallel with each other and the caller. With composi-
tion [29, 37], i.e., passing a future as an argument to another
RPC, the application can also express the parallelism and
dependencies of future RPCs. For example, in Figure 2c, add
is invoked at the beginning of the program but only executed
by the system once a and b are computed.
Distributed futures are an extension of RPC that combines
futures with distributed memory: a distributed future is a
reference whose eventual value may be stored on a remote
node (Figure 2d). An application can then express distributed
computation without having to specify when or where exe-
cution should occur and data should be moved. This is an
increasingly popular interface for developing distributed ap-
plications that manipulate large amounts of data [4,34,37,45].
As with traditional RPC, a key goal is generality. To achieve
this, the system must minimize the overhead of each function
call [13]. For example, the widely used gRPC provides hori-
zontal scalability and sub-millisecond RPC latency, making

it practical to execute millions of fine-grained functions, i.e.
millisecond-level “tasks”, per second [2].
Similarly, there are emerging examples of large-scale, fine-
grained applications of distributed futures, including rein-
forcement learning [34], video processing [22,43], and model
serving [49]. These applications must optimize parallelism
and data movement for performance [39, 43, 49], making dis-
tributed futures apt. Unfortunately, existing systems for dis-
tributed futures are limited to coarse-grained tasks [37].
In this paper, we present a distributed futures system for
fine-grained tasks. While others [34,37,45] have implemented
distributed futures before, our contribution is in identifying
and addressing the challenges of providing fault tolerance for
fine-grained tasks without sacrificing performance.
The primary challenge is that distributed futures introduce
shared state between processes. In particular, an object and
its metadata are shared by its reference holder(s), the RPC ex-
ecutor that creates the object, and its physical location(s). To
ensure that each reference holder can dereference the value,
the processes must coordinate, a difficult problem in the pres-
ence of failures. In contrast, traditional RPC has no shared
state, since data is passed by value, and naturally avoids coor-
dination, which is critical to scalability and low latency.
For example, in Figure 2a, once worker 1 copies a to the
driver, it does not need to be involved in the execution of
the downstream add task. In contrast, worker 1 stores a in
Figure 2d, so the two workers must coordinate to ensure thata
is available long enough for worker 2 to read. Also, worker 1
must garbage-collect a once worker 2 executes add and there
are no other references. Finally, the processes must coordinate
to detect and recover from the failure of another process.
The common solution in previous systems is to use a cen-
tralized master to store system state and coordinate these
operations [34, 37]. A simple way to ensure fault tolerance is
to record and replicate metadata at the master synchronously
with the associated operation. For example, in Figure 2d, the
master would record that add is scheduled to worker 2 before
dispatching the task. Then, it can correctly detect c’s failure
if worker 2 fails. However, this adds significant overhead for
applications with a high volume of fine-grained tasks [32, 51].
Thus, decentralizing the system state is necessary for scal-
ability. The question is how to do so without complicating co-
ordination. The key insight in our work is to exploit the appli-
cation structure: a distributed future may be shared by passing
by reference, but most distributed futures are shared within
the scope of the caller. For example, in Figure 1, a_future
is created then passed to add in the same scope.
We thus propose ownership, a method of decentralizing
system state across the RPC executors. In particular, the caller
of a task is the owner of the returned future and all related
metadata. In Figure 2d, the driver owns a, b, and c.
This solution has three advantages. First, for horizontal
scalability, the application can use nested tasks to “shard”
system state across the workers. Second, since a future’s
owner is the task’s caller, task latency is low because the
required metadata writes, though synchronous, are local. This
is in contrast to an application-agnostic method of sharding,
such as consistent hashing. Third, each worker becomes in
effect a centralized master for the distributed futures that it
owns, simplifying failure handling.
The system guarantees that if the owner of a future is alive,
any task that holds a reference to that future can eventually
dereference the value. This is because the owner will co-
ordinate system operations such as reference counting, for
memory safety, and lineage reconstruction, for recovery. Of
course, this is not sufficient if the owner fails.
Here, we rely on lineage reconstruction and a second key
insight into the application structure: in many cases, the ref-
erences to a distributed future are held by tasks that are a de-
scendant of the failed owner. The failed task can be recreated
through lineage reconstruction by its owner, and the descen-
dant tasks will also be recreated in the process. Therefore, it
is safe to fate-share any tasks that have a reference to a dis-
tributed future with the future’s owner. As we expect failures
to be relatively rare, we argue that this reduction in system
overheads and complexity outweighs the cost of additional
re-execution upon a failure.
In summary, our contributions are:
• A decentralized system for distributed futures with trans-
parent recovery and automatic memory management.
• A lightweight technique for transparent recovery based
on lineage reconstruction and fate sharing.
• An implementation in the Ray system [34] that provides
high throughput, low latency, and fast recovery.
2 Distributed Futures
2.1 API
The key benefit of distributed futures is that the system can
transparently manage parallelism and data movement on be-
half of the application. Here, we describe the API (Table 1).
To spawn a task, the caller invokes a remote function that
immediately returns a DFut (Table 1). The spawned task com-
prises the function and its arguments, resource requirements,
etc. The returned DFut refers to the object whose value will
be returned by the function. The caller can dereference the
DFut through get, a blocking call that returns a copy of the
object. The caller can delete the DFut, removing it from scope
and allowing the system to reclaim the value. Like other sys-
tems [34, 37, 45], all objects areimmutable.
After the creation of a DFut through task invocation, the
caller can create other references in two ways. First, the caller
can pass the DFut as an argument to another task. DFut task
arguments are implicitly dereferenced by the system. Thus,
the task will only begin once all upstream tasks have finished,
and the executor sees only the DFut values.

Operation Semantics
f(DFut x) →
DFut
Invoke the remote proceduref, and pass x by reference.
The system implicitly dereferences x to its Value be-
fore execution. Creates and returns a distributed future,
whose value is returned by f.
get(DFut x) →
Value
Dereference a distributed future. Blocks until the value
is computed and local.
del(DFut x) Delete a reference to a distributed future from the
caller’s scope. Must be called by the program.
Actor.f(DFut x)
→ DFut
Invoke a stateful remote procedure. f must execute on
the actor referred to by Actor.
shared(DFut x)
→ SharedDFut
Returns a SharedDFut that can be used to pass x to
another worker, without dereferencing the value.
f(SharedDFut x)
→ DFut
Passes x as a first-classDFut: The system dereferences
x to the corresponding DFut instead of the Value.
Table 1: Distributed futures API. The full API also includes an actor
creation call. A task may also return a DFut to its caller (nested
DFuts are automatically flattened).
Model
Preprocess
Router
Request
Invocation
Legend
Task (RPC)
Data dependency
State dependency
(a) Model serving
frame3
transform’transform
frame2
frame1
frame0
Decode Flow CumSum Smooth Sink
... ... ... ... ... (b) Video processing
Figure 3: Distributed futures applications.
Second, the DFut can be passed or returned as a first-class
value [21], i.e. passed to another task without dereferencing.
Table 1 shows how to cast a DFut to a SharedDFut, so the
system can differentiate when to dereference arguments. We
call the process that receives the DFut a borrower, to differ-
entiate it from the original caller. Like the original caller, a
borrower may create other references by passing the DFut or
casting again to a SharedDFut (creating further borrowers).
Like recent systems [4, 34, 45], we support stateful com-
putation with actors. The caller creates an actor by invoking
a remote constructor function. This immediately returns a
reference to the actor (an ARef) and asynchronously executes
the constructor on a remote process. The ARef can be used
to spawn tasks bound to the same process. Similar to DFuts,
ARefs are first-class, i.e. the caller may return or pass the
ARef to another task, and the system automatically collects
the actor process once all ARefs have gone out of scope.
2.2 Applications
Typical applications of distributed futures are those for whom
performance requires the flexibility of RPC, as well as op-
timization of data movement and parallelism. We describe
some examples here and evaluate them in Section 5.2.
Distributed futures have previously been explored for data-
intensive applications that cannot be expressed or executed
efficiently as data-parallel programs [34, 37]. Ciel identified
the key ability to dynamically specify tasks during execu-
tion, e.g., based on previous results, rather than specify the
entire graph upfront [37]. This enabled new workloads such
as dynamic programming, which is recursive by nature [54].
Our goal is to expand the application scope to include those
with fine-grained tasks that run in the milliseconds. We also
explore the use of actors and first-class distributed futures.
Model serving. The goal is to reduce request latency
while maximizing throughput, often by using model repli-
cas. Depending on the model, a latency target might be 10-
100ms [20]. Typically, an application-level scheduling policy
is required, e.g., for staged rollout of new models [46].
Figure 3a shows an example of a GPU-based image clas-
sification pipeline. Each client passes its input image to a
Preprocess task, e.g., for resizing, then shares the returned
DFut with a Router actor. Router implements the schedul-
ing policy and passes the DFut by reference to the chosen
Model actor. Router then returns the results to the clients.
Actors improve performance in two ways: (1) each Model
keeps weights warm in its local GPU memory, and (2)Router
buffers the preprocessed DFuts until it has a batch of requests
to pass to a Model, to leverage GPU parallelism for through-
put. With dynamic tasks, the Router can also choose to flush
its buffer on a timeout, to reduce latency from batching.
First-class distributed futures are important to reduce rout-
ing overhead. They allows the Router to pass the references
of the preprocessed images to the Model actors, instead of
copying these images. This avoids creating a bottleneck at
the Router, which we evaluate in Figure 15a. While the ap-
plication could use an intermediate storage system for pre-
processed images, it would then have to manage additional
concerns such as garbage collection and failures.
Online video processing. Video processing algorithms often
have complex data dependencies that are not well supported
by data-parallel systems such as Apache Spark [22, 43]. For
example, video stabilization (Figure 3b) works by tracking
objects between frames (Flow), taking a cumulative sum of
these trajectories (CumSum), then applying a moving average
(Smooth). Frame-to-frame dependencies are common, such
as the video decoding state stored in an actor in Figure 3b.
Each stage runs in 1-10s of milliseconds per frame.
Safe and timely garbage collection in this setting can be
challenging because a single object (e.g., a video frame) may
be referenced by multiple tasks. Live video processing is also
latency-sensitive: output must be produced at the same frame
rate as the input. Low latency relies on pipelined parallelism
between frames, as the application cannot afford to wait for
multiple input frames to appear before beginning execution.
With distributed futures, the application can specify the
logical task graph dynamically, as input frames appear.
Meanwhile, the system manages the physical execution, i.e.
pipelined parallelism and garbage collection, according to
the specified graph. Concurrent video streams can easily be

Distributed memory Distributed futures
a
Driver Worker 1 Worker 2
b=f()a=f()
c=add(
  a,b)
b
Driver Worker 1 Worker 2
b=f()
a=f()
c=add(
  a,b)
a
b
a?
?
(a) (b)
Figure 4: Failure detection. (a) a’s location is known by the time
worker 2 receives the reference. (b) a’s location may not be known
when worker 2 receives add, so worker 2 cannot detect the failure.
Futures
a
Driver Worker 1 Worker 2
a=f()b=f()
c=add(
  a,b)
b
Distributed futures
Driver Worker 1 Worker 2
b=f()a=f()
c=add(
  a,b)
a b
ba
c=add(
  a,b)
ba
c
(a) (b)
Figure 5: Failure recovery. (a) Data is passed by value, so the driver
recovers by resubmitting add. (b) b is also lost. f’s description must
be recorded during runtime so that b can be recomputed.
supported using nested tasks, one “driver” per stream. The
system can then manage inter-video parallelism.
3 Overview
3.1 Requirements
The system guarantees that each DFut can be dereferenced to
its value. This involves three problems: automatic memory
management, failure detection, and failure recovery.
Automatic memory management is a system for dynamic
memory allocation and reclamation of objects. The system
must decide at run time whether an object is currently refer-
enced by a live process, e.g., through reference counting [42].
Failure detection is the minimum functionality needed to
ensure progress in the presence of failures. The system detects
when a DFut cannot be dereferenced due to worker failure.
With distributed memory but no futures, this is straightfor-
ward because the location of the value is known by the time
the reference is created. In Figure 4a, for example, the driver
learns that a is stored on worker 1 and could then attach the
location when passing a to worker 2. Then, when worker 2
receives add, it can detect a’s failure.
The addition of futures complicates failure detection be-
cause references can be created before the value. Even the
future location of the value may not be known at reference
creation time. Of course, the system could wait until a task
has been scheduled before returning the reference to the caller.
However, this would defeat the purpose of futures as an asyn-
chronous construct. It is also impractical because a realistic
scheduler must be able to update its decision at run time, e.g.,
according to changes in the environment such as resource
availability and worker failures.
Thus, it is possible that there are no locations for a when
worker 2 receives the add RPC in Figure 4b. Then, worker 2
must decide whether f is still executing, or if it has failed. If
it is the former, then worker 2 should wait. But if there is a
failure, then the system must recovera. To solve this problem,
the system must record the locations of all tasks, i.e. pending
objects, in addition to created objects.
Failure recovery. The system must also provide a method
of recovering from a failed DFut. The minimum requirement
is to throw an error to the application if it tries to dereference
a failed DFut. We further provide an option for transparent
recovery, i.e. the system will recover a failed DFut’s value.
With futures but no distributed memory, if a process fails,
then we will lose the reply of any pending task on that process.
Assuming idempotence, this can be recovered through retries,
a common approach for pass-by-value RPC. For example,
in Figure 5a, the driver recovers by resubmitting add(a,b).
Failure recovery is simple because all data is passed by value.
With distributed memory, however, tasks can also contain
arguments passed by reference. Therefore, a node failure can
cause the loss of an object value that is still referenced, as
b is in Figure 4b. A common approach to this problem is to
record each object’s lineage, or the subgraph that produced
the object, during runtime [17,30,56]. The system then walks
a lost object’s lineage and recursively reconstructs the object
and its dependencies through task re-execution. This approach
reduces the runtime overhead of logging, since the data itself
is not recorded, and the work that must be redone after a partial
failure, since objects cached in distributed memory do not
need to be recomputed. Still, achieving low run-time overhead
is difficult because the lineage itself must be recorded and
collected at run time and it must survive failures.
Note that we focus specifically onobject recovery and, like
previous systems [34, 37, 56], assume idempotence for cor-
rectness. Thus, our techniques are directly applicable to idem-
potent functions and actors with read-only, checkpointable, or
transient state, as we evaluate in Figure 15c. Although it is not
our focus, these techniques may also be used in conjunction
with known recovery techniques for actor state [17, 34] such
as recovery for nondeterministic execution [52].
Metadata requirements. In summary, during normal opera-
tion, the system must at minimum record (1) the location(s)
of each object’s value, so that reference holders can retrieve it,
and (2) whether the object is still referenced, for safe garbage
collection. For failure detection and recovery, the system must
further record, respectively, (3) the location of each pending
object, i.e. the task location, and (4) the object lineage.
The key question is where and when to record this system
metadata such that it is consistent1 and fault-tolerant. By
consistent, we mean that the system metadata matches the
1Unrelated to the more standard definition of replica consistency [50].

W1 W2 W3
A B C
Master
Obj Task Loc
X B() W2
Y C(X) W3
(b) Centralized
master
CB
(a) Application
Task submission
Data dependency
A
B CX Y
def A():
  y = C(B())
...
Obj Task Loc
X B() W2
Y C(X) W3
Lease Manager
W1 W2 W3
(c) Distributed
leases
A B C
Obj Task Loc
X B() W2
Y C(X) W3
Worker1
W2 W3
(d) Ownership
A
B C
Figure 6: Distributed futures systems. (a) An application. (b) Master
manages metadata and object failures. (c) Workers write metadata
asynchronously, coordinate failure handling with leases.(d) Workers
manage metadata. Worker 1 handles failures for workers 2 and 3.
Worker 1 failure is handled by A’s owner elsewhere in the cluster.
current physical state of the cluster. By fault-tolerant, we mean
that the metadata should survive individual node failures.
In some cases, it is safe for metadata to be asynchronously
updated, i.e. there is a transient mismatch between the system
metadata and the system state. For example, the system may
transiently believe that an object x is still on node A even
though it has been removed. This is safe because a reference
holder can resolve the inconsistency by asking A if it has x.
On the other hand, metadata needed for failure handling
should ideally be synchronously updated. For example, the
metadata should never say that a task T is on node A when
it is really on node B. In particular, if node A then fails, the
system would incorrectly conclude that T has failed. As we
will see next, synchrony simplifies fault tolerance but can add
significant runtime overhead if done naively.
3.2 Existing solutions
Centralized master. Failure handling is simple with a syn-
chronously updated centralized master, but this design can
also add significant runtime overhead. For example, failure
detection requires that the master record a task’s scheduled
location before dispatch (Figure 6b). Similarly, the master
must record every new reference before it can be used. This
makes the master a bottleneck for scalability and latency.
The master can be sharded for scalability, but this can com-
plicate operations that coordinate multiple objects, such as
garbage collection and lineage reconstruction. Also, the la-
tency overhead is fundamental. Each task invocation must
first contact the master, adding at minimum one round-trip
to the critical path of execution, even without replicating the
metadata for fault tolerance. This overhead can be detrimental
when the task itself is milliseconds long, and especially so if
the return value is small enough to be passed by value. Small
values may be stored in the master directly as an optimization,
but still require 1 RTT for retrieval [38].
Distributed leases. Decentralization can remove such bot-
tlenecks, but often leads to complex coordination schemes.
One approach is to use distributed leases [19]. This is similar
to a centralized master that is updated asynchronously.
As an example, consider asynchronous task location up-
dates (Figure 6c). To account for a possibly stale master, the
worker nodes must coordinate to detect task failures, in this
case using leases. Each worker node acquires a lease for each
locally queued task and repeatedly renews the lease until the
task has finished. For example, in Figure 6c, worker 3 can
detect a failure of B by waiting for worker 2’s lease to expire.
This design is horizontally scalable through sharding and
reduces task latency, since metadata is written asynchronously.
However, the reliance on timing to reconcile system state
can slow recovery (Figure 14). Furthermore, this method
of decentralization introduces a new problem: the workers
must also coordinate on who should recover an object, i.e.
re-execute the creating task. This is trivial in the centralized
scheme, since the master coordinates all recovery operations.
3.3 Our solution: Ownership
The key insight in our work is to “shard” the centralized mas-
ter, for scalability, but to do so based on the application struc-
ture, for low run-time overhead and simple failure handling.
In ownership, the worker that calls a task stores the meta-
data related to the returned DFut. Like a centralized master, it
coordinates operations such as task scheduling, to ensure it
knows the task location, and garbage collection. For example,
in Figure 6d, worker 1 owns X and Y.
The reason for choosing the task’s caller as the owner is that
in general, it is the worker that accesses the metadata most
frequently. The caller is involved in the initial creation of
the DFut, via task invocation, as well as the creation of other
references, by passing the DFut to other RPCs. Thus, task
invocation latency is minimal because the scheduled location
is written locally. Similarly, if the DFut stays in the owner’s
scope, the overhead of garbage collection is low because the
DFut’s reference count can be updated locally when the owner
passes the DFut to another RPC. These overheads can be
further reduced for small objects, which can be passed by
value as if without distributed memory (see Section 4.2).
Of course, if all tasks are submitted by a single driver,
as in BSP programs, ownership will not scale beyond the
driver’s throughput. Nor indeed will any system for dynamic
tasks. However, with ownership, the application can scale
horizontally by distributing its control logic across multiple
nested tasks, as opposed to an application-agnostic method
such as consistent hashing (Figure 12e). Furthermore, the
worker processes hold much of the system metadata. This
is in contrast to previous solutions that push all metadata
into the system’s centralized or per-node processes, limiting
the vertical scalability of a single node with many worker
processes (Figure 12).
However, there are problems that are simpler to solve with
a fully centralized design, assuming sufficient performance:
First-class futures. First-class futures (Section 2) allow non-
owning processes to reference a DFut. While many applica-
tions can be written without first-class futures (Figure 3b),

they are sometimes essential for performance. For example,
the model serving application in Figure 3a uses first-class
futures to delegate task invocation to a nested task, without
having to dereference and copy the arguments.
A first-classDFut may leave the owner’s scope, so we must
account for this during garbage collection. We avoid centraliz-
ing the reference count at the owner, as this would defeat the
purpose of delegation. Instead, we use a distributed hierarchi-
cal reference counting protocol (Section 4.2). Each borrower
stores a local reference count for the DFut on behalf of the
owner (Table 2) and notifies the owner when the local refer-
ence count reaches zero. The owner decides when the object
is safe to reclaim. We use a reference counting approach as
opposed to tracing [42] to avoid global pauses.
Owner recovery. If a worker fails, then we will also lose its
owned metadata. For transparent recovery, the system must
recover the worker’s state on a new process and reassociate
state related to the previously owned DFuts, including any
copies of the value, reference holders, and pending tasks.
We choose a minimal approach that guarantees progress, at
the potential cost of additional re-execution on a failure: we
fate share the object and any reference holders with the owner,
then use lineage reconstruction to recover the object and any
of the owner’s fate-shared children tasks (Section 4.3). This
method adds minimal run-time overhead and is correct, i.e.
the application will recover to a previous state and the system
guarantees against resource leakage. A future extension is to
persist the owner’s state to minimize recovery time at the cost
of additional recovery complexity and run-time overhead.
4 Ownership Design
Each node in the cluster hosts one to many workers (usually
one per core), one scheduler, and one object store (Figure 7).
These processes implement future resolution, resource man-
agement, and distributed memory, respectively. Each node
and worker process is assigned a unique ID.
Workers are responsible for the resolution, reference count-
ing, and failure handling of distributed futures. Each worker
executes one task at a time and can invoke other tasks. The
root task is executed by the “driver”.
Each task has a unique TaskID that is a hash of the parent
task’s ID and the number of tasks invoked by the parent task so
far. The root TaskID is assigned randomly. Each task may re-
turn multiple objects, each of which is assigned an ObjectID
that concatenates the TaskID and the object’s index. A DFut
is a tuple of the ObjectID and the owner’s address (Owner).
The worker stores one record per future that it has in scope
in its local ownership table (Table 2). A DFut borrower
records a subset of these fields (* in Table 2). When aDFut is
passed as an argument to a task, the system implicitly resolves
the future’s value, and the executing worker stores only the
ID, Owner, and Value for the task duration. The worker also
caches the owner’s stored Locations.
Field Value
*ID The ObjectID. Also used as a distributed memory key.
*Owner Address of the owner (IP address, port, WorkerID).
*Value (1) Empty if not yet computed, (2) Pointer if in distributed
memory, or (3) Inlined value, for small objects (Section 4.2).
*References A list of reference holders: Number of dependent tasks and a
list of borrower addresses (Section 4.2 and appendix A).
Task Specification for the creating task. Includes theObjectIDs
and Owners of any DFuts passed as arguments.
Locations If Value is empty, the location of the task. If Value is a
pointer to distributed memory, then the locations of the object.
Table 2: Ownership table. The owner stores all fields. A bor-
rower (Section 3.2) only stores fields indicated by the *.
Node
Worker
Object Store
Obj Owner ...

A
Scheduler
Worker
     …
Object Store
Scheduler
Worker
     …
Distributed
memory
Resource
management
Future
resolution
(Ownership)
b
c
a
f
e
d
Figure 7: Architecture and protocol overview. (a) Task execution.
(b) Local task scheduling. (c) Remote task scheduling. (d) Object
transfer. (e) Task output storage and input retrieval. Ownership layer
manages distributed memory garbage collection and recovery. (f)
Scheduler fetches objects in distributed memory to fulfill task de-
pendencies.
An actor is a stateful task that can be invoked multiple times.
Like objects, an actor is created through task invocation and
owned by the caller. The ownership table is also used to locate
and manage actors: the Location is the actor’s address. Like
a DFut, an ARef (an actor reference) is a tuple of the ID and
Owner and can be passed as a first-class value to other tasks.
A worker requests resources from the scheduling layer to
determine task placement (Section 4.1). We assume a de-
centralized scheduler for scalability: each scheduler manages
local resources, can serve requests from remote workers, and
can redirect a worker to a remote scheduler.
The distributed memory layer (Section 4.2) consists of
an immutable distributed object store (Figure 7d) with
Locations stored at the owner. The Locations are updated
asynchronously. The object store uses shared memory to re-
duce copies between reference holders on the same node.
Workers store, retrieve, reclaim, and recover large objects
in distributed memory (Figure 7f). The scheduling layer sends
requests to distributed memory to fetch objects between nodes
according to worker requests (Figure 7g).
4.1 Task scheduling
We describe how the owner coordinates task scheduling. At
a high level, the owner dispatches each task to a location
chosen by the distributed scheduler. This ensures that the task
location in the ownership table is updated synchronously with
dispatch. We assume an abstract scheduling policy that takes

Node 2
Worker
Scheduler
Node 1
Worker
Scheduler
A1
2
Master
Obj Task Loc
X B() N2 3
B
5
4
(a)
Node 2
Worker
Scheduler
Node 1
Worker
Scheduler
A1
Lease manager
Obj Task Loc
X B() N2
B
3
2 (b)
Node 1 Node 2
Worker
Obj Task Loc
X B() N2 Worker
A
SchedulerScheduler
1 2
3
4 5
B6 (c)
Node 1 Node 2
Worker
Obj Task Loc
Y C(X) N2 Worker
SchedulerScheduler
C2
A
1 (d)
Node L. RTTs R. RTTs
Master L. 1 1
R. 1 1
Leases L. 1 0
R. 1 0.5
Owner- L. 0.5 or 1.5 0
ship R. 0 or 1 0.5 or 1.5
(e)
Figure 8: Task scheduling and the method of recording a task’s location for the program in Figure 6a.(a) Centralized master. (b) Distributed
leases. (c) Scheduling with ownership. (1-2) Local scheduler redirects owner to node 2. (3) Update task location. (4-5) Remote scheduler grants
worker lease. (6) Task dispatch. (d) Direct scheduling by the owner, using the worker and resources leased from node 2 in (c). (e) Length of
critical path of local (L) and remote (R) task execution, in terms of local (L) and remote (R) RTTs.
in resource requests and returns the ID of a node where the
resources should be allocated. The policy may also update its
decision, e.g., due to changes in resource availability.
Figure 8c shows the protocol to dispatch a task. Upon task
invocation, the caller, i.e. the owner of the returnedDFut, first
requests resources from its local scheduler2. The request is
a tuple of the task’s required resources (e.g., {"CPU": 1} )
and arguments in distributed memory. If the policy chooses
the local node, the scheduler accepts the request: it fetches
the arguments, allocates the resources, then leases a local
worker to the owner. Else, the scheduler rejects the request
and redirects the owner to the node chosen by the policy.
In both cases, the scheduler responds to the owner with the
new location: either the ID of the leased worker or the ID of
another node. The owner stores this new location in its local
ownership table before dispatching the task to that location.
If the request was granted, the owner sends the task directly
to the leased worker for execution; otherwise, it repeats the
protocol at the next scheduler.
Thus, the owner always dispatches the task to its next lo-
cation, ensuring that the task’s pending Location (Table 2)
is synchronously updated. This also allows the owner to by-
pass the scheduler by dispatching a task directly to an already
leased worker, if the task’s resource requirements are met. For
example, in Figure 8d, worker 1 reuses the resources leased
from node 2 in Figure 8c to execute C. The owner returns the
lease after a configurable expiration time, or when it has no
more tasks to dispatch. We currently do not reuse resources for
tasks with different distributed memory dependencies, since
these are fetched by the scheduler. We leave other policies for
lease revocation and worker reuse for future work.
The worst-case number of RTTs before a task executes is
higher than in previous solutions because each policy decision
is returned to the owner (Figure 8e). However, the throughput
of previous solutions is limited (Figure 12) because they can-
not support direct worker-to-worker scheduling (Figure 8d).
This is because workers do not store system state, and thus all
tasks must be routed through the master or per-node scheduler
to update the task location (Figures 8a and 8b).
Actor scheduling. The system schedules actor constructor
tasks much like normal tasks. After completion, however, the
2The owner can also choose a remote scheduler, e.g., for data locality.
owner holds the worker’s lease until the actor is no longer
referenced (Section 4.2) and the worker can only execute
actor tasks submitted through a corresponding ARef.
A caller requests the actor’s location from the owner us-
ing the ARef’s Owner field. The location can be cached and
requested again if the actor restarts (Section 4.3). The caller
can then dispatch tasks directly to the actor, as in Figure 8d,
since the resources are leased for the actor’s lifetime. For a
given caller, the actor executes tasks in the order submitted.
4.2 Memory management
Allocation. The distributed memory layer consists of a set
of object store nodes, with locations stored at the owner (Fig-
ures 9b to 9d). It exposes a key-value interface (Figure 9a).
The object store may replicate objects for efficiency but is not
required to handle recovery: if there are no copies of an object,
a Get call will block until a client (i.e. a worker) Creates the
object.
Small objects may be faster to copy than to pass through
distributed memory, which requires updating the object di-
rectory, fetching the object from a remote node, etc. Thus, at
object creation time, the system transparently chooses based
on size whether to pass by value or by reference.
Objects over a configurable threshold are stored in the
distributed object store (step 1, Figure 9b) and returned by
reference to the owner (step 2). This reduces the total number
of copies, at the cost of requiring at least one IPC to the
distributed object store for Get (steps 4-5, Figure 9c). Small
objects are returned by value to the owner (step 6, Figure 9c),
and each reference holder is given its own copy. This produces
more copies in return for faster dereferencing.
The initial copy of a large object is known as the primary.
This copy is pinned (step 1, Figure 9b) until the owner releases
the object (step 8, Figure 9d) or fails. This allows the object
store to treat additional capacity as an LRU cache without
having to consult the owners about which objects are safe
to evict. For example, the secondary copy of X created on
node 3 in Figure 9c is cached to reduce Get and recovery
time (Section 4.3) but can be evicted under memory pressure.
Dereferencing. The system dereferences a task’s DFut argu-
ments before execution. The task’s caller first waits for the
Value field in its local ownership table to be populated (Fig-

Operation Semantics
Create(ObjID Store an object.
o, Value v)
Pin(ObjID o, NodeID
loc)→ bool
Pin o on loc until released.
Returns false if loc failed.
Release(ObjID o) Object o is safe to evict.
Get(ObjID o)→
Value
Get the object value. May
fetch copy from remote node.
(a)
Node 2
Object
Store
X
Worker
Node 1
Worker
Obj Val Loc
X N2
Y
A BX: N2
2
1. Create(X)
   Pin(X) (b)
Node 1
Worker
Obj Val Loc
X *X N2
Y N3
A
Node 2
Object
Store
X
Worker
Node 3
Worker
Object
Store
Obj O. Val
X W1 *X
X
C
3
5
Y6
4. Get(X) (c)
Node 1
Worker
Obj Val Loc
X *X N2,
N3
Y ...
A
Node 3
Object
Store
Worker
X
Node 2
Object
Store
Worker
X
8. Release(X)
7 (d)
Figure 9: (a) Distributed memory store API, and (b-d) Memory management for the program in Figure 6a. (1-2) B returns a large object X in
distributed memory. The primary copy is pinned until all references have been deleted. (3) Worker 1 dispatchesC once X is available. (4-5) Get
the value from distributed memory (location lookup not shown). (6) C returns a small object Y directly to the owner. (7-8) Object reclamation.
ure 9b), then copies theValue into the dispatched task descrip-
tion. The executing worker then copies the received Value
into its local table (Figure 9c). For large objects, the sent value
is a pointer to distributed memory, so the worker must also
call Get to retrieve the actual value (step 4, Figure 9c).
If the task’s caller is also the owner of its DFut arguments,
the above protocol is sufficient. If the task’s caller is bor-
rowing an argument, then it must populate the Value field
through a protocol with the owner. Upon receiving aDFut, the
borrower sends the associated Owner a request for the Value.
The owner replies with the Value (either the inlined value or
a pointer) once populated. The borrower populates its local
Value field by copying the reply.
Reclamation. The owner reclaims the object memory once
there are no more reference holders (Figure 9d) by deleting its
local Value field (step 7) and, if necessary, callingRelease
on the distributed object store (step 8). An object’s reference
holders are tracked with a distributed reference count main-
tained by the owner and borrowers.
Each process with a DFut instance keeps a local count of
submitted tasks ( References, Table 2). The task count is
incremented each time the process invokes a dependent task
and decremented when the task completes. Each process also
keeps a local set of the worker IDs of any borrowers that
it created, by passing the DFut as a first-class value. This
forms a tree of borrowers with the owner at the root (see
Appendix A). The owner releases the object once there are no
more submitted tasks or borrowers anywhere in the cluster.
Actors. Actors are reference-counted with the same protocol
used to track borrowers of a DFut. Once the set of reference
holders is empty, the owner of the actor reclaims the actor
resources by returning the worker lease (Section 4.1).
4.3 Failure recovery
The system guarantees that any reference holder will eventu-
ally be able to resolve the value in the presence of failures.
Failure detection. Failure notifications containing a worker
or node ID are published to all workers. Workers do not
exchange heartbeats; a worker failure is published by its local
scheduler. Node failure is detected by exchanging heartbeats
between nodes, and all workers fate-share with their node.
Upon receiving a node or worker failure notification, each
worker scans its local ownership table to detect aDFut failure.
A DFut is considered failed in two cases: 1) loss of an owned
object (Figure 10a), by comparing the Location field, or
2) loss of an owner (Figure 11a), by comparing the Owner
field. We discuss the handling for these two cases next, using
lineage reconstruction and fate sharing, respectively.
Note that a non-owner does not need to detect the loss of
an object. For example, in Figure 10a, node 2 fails just as
worker 3 receives C. When worker 3 looks up X at the owner,
it may not find any locations. From worker 3’s perspective,
this means that either node 2’s write to the directory was
delayed, or node 2 failed. Worker 3 does not need to decide
which it is; it simply waits for X’s owner to handle the failure.
Object recovery. The owner recovers a lost value through
lineage reconstruction. During execution, the owner records
the object’s lineage by storing each invoked Task in its own-
ership table (Table 2). Then, upon detecting a DFut failure,
the owner resubmits the corresponding task (Figure 10b). The
task’s arguments are recursively reconstructed, if needed.
Like previous systems [34, 37, 56], we can avoid lineage
reconstruction if other copies of a required object still exist.
Thus, when reconstructing an object, the owner will first try
to locate and designate a secondary copy as the new primary.
To increase the odds of finding a secondary copy, object recla-
mation (Section 4.2) is done lazily: the owner releases the
primary copy once there are no more reference holders, but
the copy is not evicted until there is memory pressure.
Often, the owner of an object will also own the objects in
its lineage (Section 5.2). Thus, upon failure, the owner can
locally determine the set of tasks to resubmit, with a recursive
lookup of the Task fields. In some cases, an object’s lineage
may also contain borrowed references. Then, the borrower
requests reconstruction from the owner.
The owner can delete the Task field once the task has
finished and all objects returned by reference will never be
reconstructed again. When a worker returns an object by
value, the owner can immediately delete the corresponding
Task field. This is safe because objects passed by value do
not require reconstruction (Section 3.1).
For an object passed by reference, the owner keeps alineage
reference count to determine when to collect the Task. The

Node 1
Worker A
Node 2
Object
Store
X
Worker
Node 3
Worker
Object
Store
Obj O. Val
X W1 *X
X
C
Obj Task Val Loc
X B() *X N2
Y C(X) N3
(a) Failure detection.
Node 1
Worker A
Node 4
Object
Store
X
Worker
Obj Task Val Loc
X B() N4
Y C(X) N3
B
1
2
X: N43 (b) Lineage reconstruction.
Figure 10: Object recovery.
Node 1
Worker A
Node 2
Object
Store
X
Worker
Node 3
Worker
Object
Store
Obj O. Val
X W1 *X
X
C
Obj Task Val Loc
X B() *X N2
Y C(X) N3
...
(a) Failure detection.
Node 2
Object
Store
X
Worker
Node 3
Worker
Object
Store
Obj O. Val
X W1 *X
CA... (b) Fate sharing.
Figure 11: Owner recovery.
count is incremented each time the DFut is passed to another
task and decremented when that Task is itself collected. The
owner collects a record after collecting both the Task and
Value (Section 4.2) fields. We also plan to support object
checkpointing to allow the lineage to be collected early.
Owner recovery. An owner failure can result in a “dangling
pointer”: a DFut that cannot be dereferenced. This can happen
if the object is simultaneously lost from distributed memory.
For example, C in Figure 11a will hang if node 2 also fails.
We use fate sharing to ensure that the system can make
progress upon an owner’s failure. First, all resources held by
the owner and any reference holders are reclaimed. Specifi-
cally, upon notification of the owner’s failure, either the dis-
tributed object store frees the object (if it exists) or the schedul-
ing layer reclaims the worker lease (if the object is pending),
shown in Figure 11b. All reference holders, i.e. borrowers
and dependent tasks, also fate-share with the owner.
Then, to recover the fate-shared state, we rely on lineage
reconstruction. In particular, the task or actor that was exe-
cuting on the failed owner must itself have been owned by
another process. That process will eventually resubmit the
failed task. As the new owner re-executes, it will recreate its
previous state, with no system intervention needed. For ex-
ample, the owner of A in Figure 11a will eventually resubmit
A (Figure 11b), which will again submit B and C.
For correctness, we show that all previous reference holders
are recreated, with the address of the new owner. Consider task
T that computes the value of a DFut x. T initially executes
on worker W and re-executes on W′ during recovery. The
API (Section 2) gives three ways to create another reference
to x: (1) pass x as a task argument, (2) castx to a SharedDFut
then pass as a task argument, and (3) return x from T .
In the two former cases, the new reference holder must be
a child task of T . In case (2), when x is passed as a first-class
value, the child task can create additional reference holders
by passing x again. All such reference holders are therefore
descendants of T . Then, when T re-executes on W′, W′ will
recreate T ’s descendants.
T can also return x, which can be useful for returning a
child task’s result without dereferencing withget. Suppose T
returns x to its parent task P. Then, P’s worker becomes a bor-
rower and will fate-share with W. In this case, P is recovered
by its owner, and again submits T and receives x.
Thus, because any borrower ofx must be a child or ancestor
of T , fate-sharing and re-execution guarantees that the bor-
rower will be recreated with W′ as the new owner. Note that
for actors, this requires that an actor not store borrowedDFuts
in its local state. Of course, this is only required for transparent
recovery; the application may also choose to handle failures
manually and rely on the system for failure detection only.
While fate-sharing and lineage reconstruction add minimal
run-time overhead, it is not suitable for all applications. In
particular, the application will fate-share with the driver. In
fact, this is the same failure model offered by some BSP
systems [3], which can be written as a distributed futures
program in which the driver submits all tasks. As shown by
these systems, this approach can be extended to reduce the re-
execution needed during recovery. We leave such extensions,
including application-level checkpointing (Section 5.2), and
persistence of the ownership table, for future work.
Actor recovery. Actor recovery is handled through the same
protocols. If an actor fails, its owner restarts the actor through
lineage reconstruction, i.e. resubmitting the constructor task.
If the owner fails, the actor and any ARef holders fate-share.
Unlike functions, actors have local state that may require
recovery. This is out of scope for this work, but is an interest-
ing future direction. Ownership provides the infrastructure to
manage and restart actors, while other methods can be layered
on top for transparent recovery of local state [17, 34, 52].
5 Evaluation
We study the following questions:
1. Under what scenarios is distributed futures beneficial
compared to pass-by-value RPC?
2. How does the ownership architecture compare against
existing solutions for distributed futures, in terms of
throughput, latency, and recovery time?
3. What benefits does ownership provide for applications
with dynamic, fine-grained parallelism?
We compare against three baselines: (1) a pass-by-value
model with futures but no distributed memory, similar to Fig-
ure 2c, (2) a decentralized lease-based system for distributed
futures (Ray v0.7), and (3) a centralized master for distributed
futures (Ray v0.7 modified to write to a centralized master
before task execution). All distributed futures systems use
sharded, unreplicated Redis for the global metadata store,
with asynchronous requests. All systems use the Ray dis-
tributed scheduler and (where applicable) distributed object
store. Ownership and pass-by-value use gRPC [2] for worker-

0.05
 0.00 0.05
0.050
0.025
0.000
0.025
0.050 Pass by
value
Leases
Centralized
Ownership
20 40 60 80 100
Worker nodes
0
50k
100k
150k
200k
Throughput
(tasks/s)
(a) Small objects,
colocated.
20 40 60 80 100
Worker nodes
0
50k
100k
150k
200k
Throughput
(tasks/s)
(b) Small objects,
spread.
20 40 60 80 100
Worker nodes
0
50k
100k
150k
200k
Throughput
(tasks/s)
(c) Large objects,
colocated.
20 40 60 80 100
Worker nodes
0
50k
100k
150k
200k
Throughput
(tasks/s)
(d) Large objects,
spread.
0 1 2 4 8
# nested tasks
0
10k
20k
30k
Throughput
(tasks/s)
(e) Single node,
nested tasks.
Figure 12: Throughput and scalability.(a-d) Task submission is divided across multiple intermediate drivers, either colocated on the m5.8xlarge
head node or spread with one m5.8xlarge node per driver. 1 intermediate driver is added per 5 worker nodes. Each task returns either a small
(short binary string) or large (1MB blob) object. (e) Scaling task submission using nested tasks and first-class distributed futures.
to-worker communication. All benchmarks schedule tasks to
predetermined nodes to reduce scheduling variation.
All experiments are run on AWS EC2. Global system meta-
data, such as an object directory, is hosted on the same node
as the driver, where applicable. Unless stated otherwise, this
“head node” is an m5.16xlarge instance. Other node configu-
ration is listed inline. All benchmark code is available at [53].
5.1 Microbenchmarks
Throughput and scalability. The driver submits one nested
task for every 5 worker nodes (m5.8xlarge). Each interme-
diate “driver” submits no-op tasks to its 5 worker nodes.
We report the total throughput of the leaf tasks, which re-
turn either a short string (Figures 12a and 12b) or a 1MB
blob (Figures 12c and 12d). The drivers are either colo-
cated (Figures 12a and 12c) on the same m5.8xlarge node
as the root driver, or spread (Figures 12b and 12d), each on
its own m5.8xlarge node. We could not produce stable re-
sults for pass-by-value with large objects due to the lack of
backpressure in our implementation.
At <60 nodes, the centralized and lease-based architectures
achieve about the same throughput because the centralized
master is not yet a bottleneck. In general, ownership achieves
better throughput than either because it distributes some sys-
tem operations to the workers. In contrast, the baselines han-
dle all system operations in the global or per-node processes.
The gap between ownership and the baselines is more sig-
nificant with small return values (Figures 12a and 12b). For
these, ownership matches pass-by-value because small ob-
jects are returned directly to their owner. The baseline sys-
tems could implement a similar optimization, e.g., by inlining
small objects in the object directory (Section 4.2), but this
would still require at minimum one RPC per read.
When the drivers are spread (Figures 12b and 12d), owner-
ship and leases both scale linearly. Ownership scales better
than leases in Figure 12b because more work is offloaded
onto the worker processes. Ownership and leases achieve sim-
ilar throughput in Figure 12d, but the ownership system also
includes memory safety (Section 4.2). The centralized design
(2 shards) scales linearly to∼60 nodes. Adding more shards
would raise this threshold, but only by a constant amount.
When the drivers are colocated (Figures 12a and 12c), both
baselines flatline because of a centralized bottleneck: the
scheduler on the drivers’ node. Ownership also shows this,
but there is less scheduler load overall because the drivers
reuse resources for multiple tasks (Section 4.1). A comparable
optimization for the baselines would require each driver to
batch task submission, at the cost of latency. Throughput for
ownership is lower in Figure 12c than in Figure 12a due to
the overhead of garbage collection.
Thus, because ownership decentralizes system state among
the workers, it can achieve vertical (Figures 12a and 12c) and
horizontal (Figures 12b and 12d) scalability. Also, it matches
the performance of pass-by-value RPC while enabling new
workloads through distributed memory (Section 2.2).
Scaling through borrowing. We show how first-class fu-
tures enable delegation. Figure 12e shows the task throughput
for an application that submits 100K no-op tasks that each
depend on the same 1MB object created by the driver. The
tasks are submitted either by the driver (x=0) or by a number
of nested tasks that each borrow a reference to the driver’s
object. All workers are colocated on an m5.16xlarge node.
For all systems, the throughput with a single borrower (x=1)
is about the same as when the driver submits all tasks directly
(x=0). Distributing task submission across multiple borrowers
results in a 2× improvement for ownership and negligible
improvement for the baselines. Thus, with ownership, an ap-
plication can scale past the task dispatch throughput of a
single worker by delegating to nested tasks. This is due to (1)
support for first-class distributed futures, and (2) the hierarchi-
cal distributed reference counting protocol, which distributes
an object’s reference count among its borrowers instead of
centralizing it at the owner (Section 4.2). In contrast, the
baselines would require additional nodes to scale.
Latency. Figure 13 measures task latency with a single
worker, hosted either on the same node as the driver (“lo-
cal”), or on a separate m5.16xlarge node (“remote”). The
driver submits 3k tasks that each take the same 1MB object as
an argument and that immediately returns a short string. We
report the average duration before each task starts execution.
First, distributed memory achieves better latency than pass-
by-value in all cases because these systems avoid unnecessary
copies of the task argument from the driver to the worker.
Second, compared to centralized and leases, ownership
achieves on average 1.6× lower latency. This is due to (1)

local
actors
local
tasks
remote
actors
remote
tasks
0
1
2
3
4Latency (ms)
Pass by
value
Centralized
Leases
Ownership
Figure 13: Task latency. Local means that the worker and driver are
on the same node. Error bars for standard deviation (across 3k tasks).
0 500 1000
Task duration (ms)
1
5
10Relative time (log)
(a) Small objects.
0 500 1000
Task duration (ms)
1
5
10Relative time (log)
Pass by value
Pass by value;
failure
Leases
Leases;
failure
Ownership;
failure (b) Large objects.
Figure 14: Total run time (log-scale), relative to ownership without
failures. The application is a chain of dependent tasks that execute
on one node. Each task sleeps for the duration on the x-axis (total
10s) and returns either (a) a short binary string, or (b) a 10MB blob.
the ability to write metadata locally at the owner instead of a
remote process, and (2) the ability to reuse leased resources,
in many cases bypassing the scheduling layer (Section 4.1).
Recovery. This benchmark submits a chain of tasks that
execute on a remote m5.xlarge node. Each task depends on
the previous, sleeps for the time on the x-axis (total duration
10s), and returns either a short binary string (Figure 14a) or a
10MB blob (Figure 14b). We report the run time relative to
ownership without failures. To test recovery, the worker node
is killed and restarted 5s into the job (1s heartbeat timeout).
We do not include centralized due to implementation effort.
Normal run time for leases is up to 1.18× faster than own-
ership, but recovery time is more than double, worse than
restarting the application. This is because a task’s lease must
expire before it can be re-executed, adding delay for short
tasks. The recovery delay for longer tasks is also high because
the implementation (Ray v0.7) repeatedly doubles a lease’s
expiration time to reduce renewal overhead. A shorter lease
interval would reduce recovery delay but can be unstable.
Ownership recovers within 2× the normal run time. Re-
covery time is the same as pass-by-value for small objects
because only in-flight tasks are re-executed (Figure 14a). For
large objects (Figure 14b), ownership achieves better normal
run time than pass-by-value because arguments are passed by
reference; the gap decreases as task execution dominates.
Thus, ownership can achieve the same or better normal
run-time performance as leases and pass-by-value, while also
guaranteeing timely recovery through lineage reconstruction.
5.2 End-to-end applications
Model serving. We implement Figure 3a. Figure 15a shows
the latency on 4 p3.16xlarge nodes, each with 1Router and 8
ResNet-50 [23] Models. We use a GPU batch size of 16 and
generate 2300 requests/s. Ownership and centralized achieve
the same median latency (54ms), but the tail latency for cen-
tralized is 9× higher (1s vs. 108ms). We also show the utility
of first-class distributed futures: in “-borrow”, the Router re-
ceives the image values and must copy these to the Model. As
expected, the Router is a bottleneck (p50=80ms, p100=3.2s).
Online video processing. We implement Figure 3b with 60
concurrent videos. The tasks for each stream are executed on
an m5.xlarge “worker” node (1 per stream) and submitted by a
driver task on a separate m5.xlarge “owner” node. Each owner
node hosts 4 drivers. Each video source uses an actor to hold
frame-to-frame decoder state. However, tasks are idempotent:
a previous frame may be reread with some latency penalty.
We use a YouTube video with a frame rate of 29 frames/s and
a radius of 1s for the moving average.
Figure 15b shows latency without failures. All systems
achieve similar median latency (∼65ms), but leases and cen-
tralized have a long tail (1208ms and 1923ms, respectively).
Figure 15c shows latency during an injected failure, 5s after
the start, of the Decoder actor (Figure 3b). Lease-based re-
covery is slow because the decoder actor must replay all tasks,
and each task accumulates overhead from lease expiration.
Checkpointing the actor was infeasible because the leases
implementation does not safely garbage-collect lineage.
Figure 15c also shows different failure scenarios for own-
ership, with a failure after 10s. The owner uses lineage re-
construction to recover quickly from a worker failure (1.9s in
O;WF). Owner recovery is slower because the failed owner
must re-execute from the beginning (8.8s in O;OF). To bound
re-execution, we use application-level checkpoints (O+CP,
checkpoints to a remote Redis instance once per second).
Each checkpoint includes all intermediate state needed to
transform the given frame, such as the cumulative sum so
far (Figure 3b). When the sink receives the transformed frame,
it “commits” the checkpoint by writing the frame’s index to
Redis. This results in negligible overhead (O vs. O+CP) and
faster recovery (1.1s in O+CP;OF).
6 Related Work
Distributed futures. Several systems [4, 34, 37, 45, 48, 52]
have implemented a distributed futures model. Most [37, 45]
use a centralized master (Section 3.2). In contrast, ownership
is a decentralized design that stores system state directly in the
workers that invoke the tasks. Ray [34] shards the centralized
state, but must still write to the centralized store before task
execution and does not support automatic memory manage-
ment. Lineage stash [52] is a complementary technique for
recovering nondeterministic execution; ownership provides
infrastructure for failure detection and memory management.
Other dataflow systems. Distributed data-parallel systems
provide high-throughput batch computation and transparent
data recovery [15,25,54,56]. Many of our techniques build on

0 1000 2000 3000
Latency (ms)
0.0
0.5
1.0CDF
Ownership -borrow
Centralized
Ownership
0 1000 2000 3000
Latency (ms)
0.95
0.96
0.97
0.98
0.99
1.00CDF
(a)
0 1000 2000
Latency (ms)
0.0
0.5
1.0CDF
Leases
Centralized
Ownership (b)
0 10 20
Latency (s)
0.90
0.95
1.00CDF
L; WF
O
O; WF
O; OF
O+CP
O+CP; OF (c)
Figure 15: End-to-end benchmarks. (a) Image classification latency (right is p95-p100).(b) Online video stabilization latency. (c) Online video
stabilization latency with failures (starting at p90). L=leases; O=ownership; CP=checkpointing; WF=worker failure; OF=owner failure.
these systems, in particular the use of distributed memory [25,
56] and lineage re-execution [15, 25, 54, 56]. Indeed, a data-
parallel program is equivalent to a distributed futures program
with no nested functions.
Most distributed data-parallel systems [15, 25, 54, 56] em-
ploy some form of centralized master, a bottleneck for appli-
cations with fine-grained tasks [32,44,51]. Naiad [35,36] and
Canary [44] support fine-grained tasks but, like other data-
parallel systems, implement a static task graph, i.e. all tasks
must be specified upfront. In contrast, distributed futures are
an extension of RPC, which allows tasks to be dynamically in-
voked. Nimbus [32] supports both fine-grainedand dynamic
tasks with a centralized controller by leveraging execution
templates for iterative computations. In contrast, ownership
distributes the control plane and schedules tasks one at a time.
These approaches are complementary; an interesting future
direction is to apply execution templates to distributed futures.
Actor systems. Distributed futures are compatible with the
actor model [7, 24]. Other actor frameworks [1, 12] already
use futures for asynchrony, but with pass-by-value semantics,
making it expensive to process large data. Actors can be
extended with distributed memory to enable pass-by-reference
semantics. Since distributed memory is immutable, it does
not violate the condition of no shared state.
Our fault tolerance model is inspired by supervision in
actor systems [7]. In this model, a supervisor actor delegates
work to its children actors and is responsible for handling
any failures among its children. By default, an actor also fate-
shares with its supervisor. Our contribution is in extending
the supervision model to objects and object recovery.
Parallel programming systems. MPI [18] exposes a low-
level pass-by-value interface. In contrast, distributed futures
supports pass-by-reference and heterogenerous processes.
Distributed futures are more similar in interface to other
parallel programming runtimes [10, 14, 21, 31, 47]: the user
annotates a sequential program to designate procedures that
can be executed in parallel. Out of these systems, ownership
is perhaps most similar to Legion [10], in that the developer
specifies a task hierarchy that dictates system behavior. Our
contribution is in identifying and addressing the challenges
of failure detection and recovery for distributed futures.
Distributed memory. Distributed shared memory [40] pro-
vides the illusion of a single globally shared and mutable
address space across a physically distributed system. Trans-
parency has historically been difficult to achieve without
adding exorbitant runtime overhead. Mutability makes con-
sistency a major problem [11, 26, 28, 40], and fault tolerance
has never been satisfactorily addressed [40].
More recent distributed memory systems [6, 9, 16, 27, 41]
implement a higher-level key-value store interface. Most tar-
get a combination of performance, consistency, and durabil-
ity. Similar to our use of distributed memory (Section 4.2),
in-memory data replicas are used to improve durability and
recovery time. Indeed, many of these systems could likely be
used in place of our distributed memory subsystem.
However, the requirements of our distributed memory sub-
system are minimal compared to previous work, e.g., dura-
bility is only an optimization. This is because we target an
even higher-level interface that integrates directly with the
programming language: unlike a key, a DFut can be used to
express rich application semantics to the system, such as an
RPC’s data dependencies. Also, like previous data processing
systems [15, 37, 56], data is immutable. Thus, fine-grained
mutations are expensive, but consistency is not a problem.
7 Discussion
Ownership is the basis of the Ray architecture in v1.0+ [5], im-
plemented in∼14k C++ LoC. Previously, Ray used a sharded
global metadata store [34]. There were two problems with
this approach: (1) latency, and (2) worker nodes still had to
coordinate for operations such as failure detection. Ray v0.7
introduced leases (Section 3.2), which solved the latency prob-
lem but not coordination. It became impractical to introduce
distributed protocols involving multiple objects, such as for
garbage collection. We designed ownership for this purpose.
While transparent recovery is an explicit goal of this paper,
it is not the only benefit of ownership. Anecdotally, the two
main benefits of ownership for Ray users are performance and
reliability. In particular, reliability includes correct and timely
failure detection and garbage collection. Notably, ownership-
based transparent recovery is not yet widely used.
We believe that this is due to: (1) applications having cus-
tom recovery requirements that cannot be met with lineage
reconstruction alone, and (2) the cost of transparent recovery.
Thus, one design goal was to ensure that only applications
that needed transparent recovery would have to pay the cost.
Ownership is a first step towards this: it provides reliability to
all applications and transparent object recovery as an option.
In the future, we hope to extend this work to support a
spectrum of application recovery requirements. For example,
we could extend ownership with options to recover actor state.

Acknowledgements
We thank our anonymous reviewers and our shepherd Ryan
Huang for their insightful feedback. We also thank Alvin
Cheung, Michael Whittaker, Joe Hellerstein, and many others
at the RISELab for their helpful discussion and comments.
In addition to NSF CISE Expeditions Award CCF-1730628,
this research is supported by gifts from Alibaba Group, Ama-
zon Web Services, Ant Group, CapitalOne, Ericsson, Face-
book, Futurewei, Google, Intel, Microsoft, Nvidia, Scotiabank,
Splunk, and VMware.
References
[1] Akka. https://akka.io/.
[2] gRPC. https://grpc.io.
[3] Improved Fault-tolerance and Zero Data Loss in Apache
Spark Streaming. https://databricks.com/blog/
2015/01/15/improved-driver-fault-tolerance-
and-zero-data-loss-in-spark-streaming .html.
[4] PyTorch - Remote Reference Protocol. https://
pytorch.org/docs/stable/notes/rref.html.
[5] Ray v1.0. https://github.com/ray-project/ray/
releases/tag/ray-1.0.0.
[6] David G Andersen, Jason Franklin, Michael Kaminsky,
Amar Phanishayee, Lawrence Tan, and Vijay Vasudevan.
Fawn: A fast array of wimpy nodes. In Proceedings
of the ACM SIGOPS 22nd symposium on Operating
systems principles, pages 1–14, 2009.
[7] Joe Armstrong. Making reliable distributed systems in
the presence of software errors. PhD thesis, Mikroelek-
tronik och informationsteknik, 2003.
[8] Henry C Baker Jr and Carl Hewitt. The incremental
garbage collection of processes. ACM SIGART Bulletin,
(64):55–59, 1977.
[9] Mahesh Balakrishnan, Dahlia Malkhi, Vijayan Prab-
hakaran, Ted Wobbler, Michael Wei, and John D Davis.
{CORFU}: A shared log design for flash clusters. In
Presented as part of the 9th {USENIX} Symposium
on Networked Systems Design and Implementation
({NSDI} 12), pages 1–14, 2012.
[10] Michael Bauer, Sean Treichler, Elliott Slaughter, and
Alex Aiken. Legion: Expressing locality and indepen-
dence with logical regions. InSC’12: Proceedings of the
International Conference on High Performance Com-
puting, Networking, Storage and Analysis, pages 1–11.
IEEE, 2012.
[11] John K Bennett, John B Carter, and Willy Zwaenepoel.
Munin: Distributed shared memory based on type-
specific memory coherence. In Proceedings of the sec-
ond ACM SIGPLAN symposium on Principles & prac-
tice of parallel programming, pages 168–176, 1990.
[12] Phil Bernstein, Sergey Bykov, Alan Geller, Gabriel Kliot,
and Jorgen Thelin. Orleans: Distributed virtual actors
for programmability and scalability. Technical Report
MSR-TR-2014-41, March 2014.
[13] Andrew D Birrell and Bruce Jay Nelson. Implementing
remote procedure calls. ACM Transactions on Computer
Systems (TOCS), 2(1):39–59, 1984.
[14] Robert D Blumofe, Christopher F Joerg, Bradley C Kusz-
maul, Charles E Leiserson, Keith H Randall, and Yuli
Zhou. Cilk: An efficient multithreaded runtime system.
Journal of parallel and distributed computing, 37(1):55–
69, 1996.
[15] Jeffrey Dean and Sanjay Ghemawat. MapReduce: Sim-
plified data processing on large clusters.Commun. ACM,
51(1):107–113, January 2008.
[16] Aleksandar Dragojevi´c, Dushyanth Narayanan, Miguel
Castro, and Orion Hodson. Farm: Fast remote memory.
In 11th{USENIX} Symposium on Networked Systems
Design and Implementation ({NSDI} 14), pages 401–
414, 2014.
[17] Elmootazbellah Nabil Elnozahy, Lorenzo Alvisi, Yi-Min
Wang, and David B Johnson. A survey of rollback-
recovery protocols in message-passing systems. ACM
Computing Surveys (CSUR), 34(3):375–408, 2002.
[18] Edgar Gabriel, Graham E. Fagg, George Bosilca,
Thara Angskun, Jack J. Dongarra, Jeffrey M. Squyres,
Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, An-
drew Lumsdaine, Ralph H. Castain, David J. Daniel,
Richard L. Graham, and Timothy S. Woodall. Open
MPI: Goals, concept, and design of a next generation
MPI implementation. In Proceedings, 11th European
PVM/MPI Users’ Group Meeting , pages 97–104, Bu-
dapest, Hungary, September 2004.
[19] Cary Gray and David Cheriton. Leases: An efficient
fault-tolerant mechanism for distributed file cache con-
sistency. ACM SIGOPS Operating Systems Review ,
23(5):202–210, 1989.
[20] Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao,
Antoine Kaufmann, Ymir Vigfusson, and Jonathan
Mace. Serving dnns like clockwork: Performance pre-
dictability from the bottom up. In 14th{USENIX}
Symposium on Operating Systems Design and Imple-
mentation ({OSDI} 20), pages 443–462, 2020.

[21] Robert H Halstead Jr. Multilisp: A language for concur-
rent symbolic computation. ACM Transactions on Pro-
gramming Languages and Systems (TOPLAS), 7(4):501–
538, 1985.
[22] Brandon Haynes, Amrita Mazumdar, Armin Alaghi,
Magdalena Balazinska, Luis Ceze, and Alvin Cheung.
Lightdb: A DBMS for virtual reality video. Proc. VLDB
Endow., 11(10):1192–1205, 2018.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 770–778, 2016.
[24] Carl Hewitt, Peter Bishop, and Richard Steiger. A uni-
versal modular actor formalism for artificial intelligence.
In Proceedings of the 3rd International Joint Conference
on Artificial Intelligence, IJCAI’73, page 235–245, San
Francisco, CA, USA, 1973. Morgan Kaufmann Publish-
ers Inc.
[25] Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell,
and Dennis Fetterly. Dryad: Distributed data-parallel
programs from sequential building blocks. In Proceed-
ings of the 2nd ACM SIGOPS/EuroSys European Con-
ference on Computer Systems 2007, EuroSys ’07, pages
59–72, New York, NY , USA, 2007. ACM.
[26] Pete Keleher, Alan L Cox, Sandhya Dwarkadas, and
Willy Zwaenepoel. Treadmarks: Distributed shared
memory on standard workstations and operating sys-
tems. Distributed Shared Memory: Concepts and Sys-
tems, pages 211–227, 1994.
[27] Avinash Lakshman and Prashant Malik. Cassandra: a
decentralized structured storage system. ACM SIGOPS
Operating Systems Review, 44(2):35–40, 2010.
[28] Kai Li. Ivy: A shared virtual memory system for parallel
computing. ICPP (2), 88:94, 1988.
[29] Barbara Liskov and Liuba Shrira. Promises: Linguistic
support for efficient asynchronous procedure calls in
distributed systems. ACM SIGPLAN Notices, 23(7):260–
267, 1988.
[30] Nirmesh Malviya, Ariel Weisberg, Samuel Madden, and
Michael Stonebraker. Rethinking main memory oltp
recovery. In 2014 IEEE 30th International Conference
on Data Engineering, pages 604–615. IEEE, 2014.
[31] Simon Marlow. Parallel and concurrent programming
in Haskell: Techniques for multicore and multithreaded
programming. " O’Reilly Media, Inc.", 2013.
[32] Omid Mashayekhi, Hang Qu, Chinmayee Shah, and
Philip Levis. Execution templates: Caching con-
trol plane decisions for strong scaling of data analyt-
ics. In 2017{USENIX} Annual Technical Conference
({USENIX}{ATC} 17), pages 513–526, 2017.
[33] Luc Moreau. Hierarchical distributed reference count-
ing. In Proceedings of the 1st international symposium
on Memory management, pages 57–67, 1998.
[34] Philipp Moritz, Robert Nishihara, Stephanie Wang,
Alexey Tumanov, Richard Liaw, Eric Liang, Melih Eli-
bol, Zongheng Yang, William Paul, Michael I. Jordan,
and Ion Stoica. Ray: A distributed framework for emerg-
ing AI applications. In 13th USENIX Symposium on
Operating Systems Design and Implementation (OSDI
18), Carlsbad, CA, 2018. USENIX Association.
[35] Derek G. Murray, Frank McSherry, Rebecca Isaacs,
Michael Isard, Paul Barham, and Martín Abadi. Na-
iad: A timely dataflow system. In Proceedings of the
Twenty-Fourth ACM Symposium on Operating Systems
Principles, SOSP ’13, pages 439–455, New York, NY ,
USA, 2013. ACM.
[36] Derek G. Murray, Frank McSherry, Michael Isard, Re-
becca Isaacs, Paul Barham, and Martin Abadi. Incre-
mental, iterative data processing with timely dataflow.
Commun. ACM, 59(10):75–83, September 2016.
[37] Derek G. Murray, Malte Schwarzkopf, Christopher
Smowton, Steven Smith, Anil Madhavapeddy, and
Steven Hand. CIEL: A universal execution engine for
distributed data-flow computing. In Proceedings of the
8th USENIX Conference on Networked Systems Design
and Implementation, NSDI’11, pages 113–126, Berke-
ley, CA, USA, 2011. USENIX Association.
[38] D.G. Murray. A Distributed Execution Engine Sup-
porting Data-dependent Control Flow. University of
Cambridge, 2012.
[39] Robert Nishihara, Philipp Moritz, Stephanie Wang,
Alexey Tumanov, William Paul, Johann Schleier-Smith,
Richard Liaw, Mehrdad Niknami, Michael I. Jordan,
and Ion Stoica. Real-time machine learning: The miss-
ing pieces. In Workshop on Hot Topics in Operating
Systems, 2017.
[40] B. Nitzberg and V . Lo. Distributed shared memory: a
survey of issues and algorithms.Computer, 24(8):52–60,
1991.
[41] John Ousterhout, Arjun Gopalan, Ashish Gupta, Ankita
Kejriwal, Collin Lee, Behnam Montazeri, Diego On-
garo, Seo Jin Park, Henry Qin, Mendel Rosenblum, et al.
The RAMCloud storage system. ACM Transactions on
Computer Systems (TOCS), 33(3):7, 2015.

[42] David Plainfossé and Marc Shapiro. A survey of dis-
tributed garbage collection techniques. In International
Workshop on Memory Management , pages 211–249.
Springer, 1995.
[43] Alex Poms, Will Crichton, Pat Hanrahan, and Kayvon
Fatahalian. Scanner: Efficient video analysis at scale.
ACM Trans. Graph., 37(4):138:1–138:13, July 2018.
[44] Hang Qu, Omid Mashayekhi, Chinmayee Shah, and
Philip Levis. Decoupling the control plane from pro-
gram control flow for flexibility and performance in
cloud computing. In Proceedings of the Thirteenth Eu-
roSys Conference, EuroSys ’18, New York, NY , USA,
2018. Association for Computing Machinery.
[45] Matthew Rocklin. Dask: Parallel computation with
blocked algorithms and task scheduling. In Kathryn
Huff and James Bergstra, editors, Proceedings of the
14th Python in Science Conference , pages 130 – 136,
2015.
[46] Danilo Sato, Arif Wider, and Windheuser Christoph.
Continuous delivery for machine learning, Sep 2019.
[47] Elliott Slaughter, Wonchan Lee, Sean Treichler, Michael
Bauer, and Alex Aiken. Regent: a high-productivity pro-
gramming language for hpc with logical regions. In
Proceedings of the International Conference for High
Performance Computing, Networking, Storage and Anal-
ysis, pages 1–12, 2015.
[48] Vikram Sreekanti, Chenggang Wu Xiayue Charles
Lin, Jose M Faleiro, Joseph E Gonzalez, Joseph M
Hellerstein, and Alexey Tumanov. Cloudburst:
Stateful functions-as-a-service. arXiv preprint
arXiv:2001.04592, 2020.
[49] Vikram Sreekanti, Harikaran Subbaraj, Chenggang Wu,
Joseph E Gonzalez, and Joseph M Hellerstein. Op-
timizing prediction serving on low-latency serverless
dataflow. arXiv preprint arXiv:2007.05832, 2020.
[50] Andrew S Tanenbaum and Maarten Van Steen. Dis-
tributed systems: principles and paradigms. Prentice-
Hall, 2007.
[51] Shivaram Venkataraman, Aurojit Panda, Kay Ouster-
hout, Ali Ghodsi, Michael Armbrust, Benjamin Recht,
Michael Franklin, and Ion Stoica. Drizzle: Fast and
adaptable stream processing at scale. In Proceedings
of the Twenty-Sixth ACM Symposium on Operating Sys-
tems Principles, SOSP ’17. ACM, 2017.
[52] Stephanie Wang, John Liagouris, Robert Nishihara,
Philipp Moritz, Ujval Misra, Alexey Tumanov, and Ion
Stoica. Lineage stash: fault tolerance off the critical
path. In Proceedings of the 27th ACM Symposium on
Operating Systems Principles, pages 338–352, 2019.
[53] Stephanie Wang, Edward Oakes, and Frank Luan.
Ownership nsdi’21 artifact. https://github.com/
stephanie-wang/ownership-nsdi2021-artifact .
[54] Tom White. Hadoop: The Definitive Guide. O’Reilly
Media, Inc., 2012.
[55] Yuan Yu, Panagiotis Manolios, and Leslie Lamport.
Model checking tla+ specifications. In In Correct Hard-
ware Design and Verification Methods (CHARME ’99),
Laurence Pierre and Thomas Kropf editors. Lecture
Notes in Computer Science, Springer-Verlag., volume
1703, pages 54–66, June 1999.
[56] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das,
Ankur Dave, Justin Ma, Murphy McCauley, Michael J
Franklin, Scott Shenker, and Ion Stoica. Resilient dis-
tributed datasets: A fault-tolerant abstraction for in-
memory cluster computing. In Proceedings of the 9th
USENIX conference on Networked Systems Design and
Implementation, pages 2–2. USENIX Association, 2012.

A Distributed Reference Counting
Type Description
Local refer-
ence
A flag indicating whether the DFut has gone out of
the process’s scope.
Submitted
task count
Number of tasks that depend on the object that were
submitted by this process and that have not yet com-
pleted execution.
Borrowers The set of worker IDs of the borrowers created by this
process, by passing the DFut as a first-class value.
Nested DFuts The set of DFuts that are in scope and whose values
contain this DFut.
Lineage count Number of Tasks that depend on this DFut that may
get re-executed. This count only determines when the
lineage (the Task field) should be released; thevalue
can be released even when this count is nonzero.
Table 3: Full description of the References field in Table 2. Every
process with an instance of the DFut (either the owner or a borrower)
maintains these fields.
If a DFut never leaves the scope of its owner, it does not
require a distributed reference count. This is because the
owner always has full information about which pending tasks
require the object. However, since our API allows passing
DFuts to other tasks as first-class values, we use a distributed
reference count to decide when the object is out of scope.
Our reference counting protocol is similar to existing so-
lutions [33, 42]. As explained in Section 4.2, the reference
count is maintained with a tree of processes. Each process
keeps a local set of borrower worker IDs, i.e. its children
nodes in the tree. Most of the messages needed to maintain
the tree are piggy-backed on existing protocols, such as for
task scheduling.
A borrower is created when a task returns aSharedDFut to
its parent task, or passes aSharedDFut to a child task. In both
cases, the process executing the task adds the ID of the worker
that executes the parent or child task to its local borrower set.
In many cases, a child task will finish borrowing theDFut
by the time it has finished execution. Concretely, this means
that the worker executing the child task will no longer have a
local reference to the DFut, nor will it have any pending de-
pendent tasks. Thus, when the worker returns the task’s result
to its owner, the owner can remove the worker from its local
set of borrowers, with no additional messages needed. This
optimization is important for distributing load imposed by ref-
erence counting among the borrowers, rather than requiring
all reference holders to be tracked by the owner.
However, in some cases, the worker may borrow the DFut
past the duration of the child task. There are two cases: (1) the
worker passed the DFut as an argument to a task that is still
pending execution, or (2) the worker is an actor and stored
the DFut in its local state. In these cases, the worker notifies
the owner that it is still borrowing the DFut when replying
with the task’s return value.
Eventually, the owner must collect all of the borrowers in
its local set. It does this by sending a request to each borrower
to reply once the borrower’s reference count has gone to
zero. Borrowers themselves never delete from their local set
of borrowers. Once a borrower no longer has a reference or
any pending dependent tasks, it replies to the owner with its
accumulated local borrower set. The owner then removes the
borrower, merges the received borrowers into its local set
and repeats the same process with any new borrowers. If a
borrower dies before it can be removed, the owner removes it
upon being notified of the borrower’s death.
When a DFut is returned by a task, it results in a nested
DFut. Nested DFuts can be automatically flattened, e.g., when
submitting a dependent task, but we must still account for
nesting during reference counting. We do this by keeping a
set of DFuts whose values contain the DFut in question in the
ownership table (Table 2). The DFut’s value is pinned if its
nested set is non-empty.
B Formal Specification
We developed a formal specification for the ownership-based
system architecture [53]. It models the system state transitions
of the ownership table for task scheduling, garbage collection,
and worker failures. The goal is to check the correctness
of the system design, which is manifested in the following
properties:
• Safety: A future’s lineage information is preserved as
long as a task exists that depends on the value of the
future. This is defined recursively: at any time, either the
value of a future is stored inline (thus cannot be lost),
or all futures that this future depends on for computing
its value must be safe. Formally, it means the following
invariant holds at any given time:∀x,
LineageInScope(x) ≜
∨ x = INLINE_VALUE
∨∀ arg∈ x.args : LineageInScope(arg)
• Liveness: The system will eventually execute all tasks
and resolve all future values, even in case of failures, i.e.,
all Get calls eventually return.
• No Resource Leakage: The system will eventually clean
up all task states and future values, after the all references
to futures become out-of-scope.
We checked the model using the TLA+Model Checker [55]
for up to 3 levels of recursive remote function calls, where
each function creates up to 3 futures, and verified that the
safety and liveness properties hold in more than 44 million
distinct states. Currently, the model does not include first-class
futures or actors; we plan to include these and open-source
the full TLA+specification in the future.

6.5840 Spanner FAQ

Q: This is  a complex paper; what should we focus on?

A: For us, the key ideas are 1) running two-phase commit on
Paxos-replicated entities so that failure doesn't cause blocking with
locks held; 2) use of snapshot isolation to allow read-only
transactions without locks; and 3) synchronized time to ensure that
read-only transactions see the latest writes (external consistency).

Q: How does TrueTime select intervals in a way that's guaranteed to
contain the correct time?

A: Here a simple example of the kind of reasoning it uses.

Suppose master time server S1 has the correct time (from GPS or an
atomic clock). S2 sends a request to S1 asking for the time, and gets a
response. The response says "10:00:00 AM" and it arrives two seconds
after S2 sent the request (it's reasonable to assume that S2 can time
how long things take even though it doesn't know the absolute time).
Since the whole request/response took two seconds, S2 can conclude
that the network might have delayed the request by up to two seconds;
or delayed the response by up to two seconds; but no more than that.
So S2 can conclude that, at the moment it receives the response, that
the correct time must be between 10:00:00 and 10:00:02.

Q: How does external consistency relate to linearizability and
serializability?

A: Spanner's external consistency is the same as strict
serializability. And strict serializability is linearizability, but
applied to multi-read/write transactions rather than single reads and
writes.

Q: Why is external consistency desirable?

A: Suppose Hatshepsut changes the password on an account shared by her
workgroup, via a web server in a datacenter in San Jose. She whispers
the new password over the cubicle wall to her colleague Cassandra.
Cassandra logs into the account via a web server in a different
datacenter, in San Mateo. External consistency guarantees that
Cassandra will observe the change to the password, and not, for
example, see a stale replica.

Q: Could Spanner use Raft rather than Paxos?

A: Yes. At the level of this paper there is no difference. At the time
Spanner was being built, Raft didn't exist, and Google already had a
high-performance and robust Paxos implementation. Have a look at the
paper Paxos Made Live by Chandra et al.

Q: Why does read-only transaction T2 having a greater time-stamp than
read-write transaction T1 mean that T2 will see T1's writes?

A: Spanner keeps multiple versions of each record, one for each recent
write of that record, and remembers the time-stamp of the transaction
that wrote each version (this is the timestamp:int64 at the start of
Section 2.1). When read-only transaction T2 reads a record, the
version it reads is the one with the highest time-stamp that's less
than T2's time-stamp. Thus a read-only transaction will see writes by
transactions with lower time-stamps, and will not see writes by
transactions with higher time-stamps. This technique of read-only
transactions seeing a version of the database as of a specific instant
in time is often called Snapshot Isolation.

Q: What is the purpose of the t_safe "safe time" machinery in Section
4.1.3?

A: A read-only transaction with time-stamp T2 is required to see the
writes of any read-write transaction T1 if T1's time-stamp is less
than T2's time-stamp. But T2 reads only Paxos replicas in its own
data-center; if T1 executed at a distant data-center, the replicas in
T2's data-center might not yet have received T1's writes. The safe
time mechanism causes T2's local Paxos replicas to delay T2's reads
until it's guaranteed that the replica has seen all writes with
timestamp < T2.

Q: What is the purpose of Spanner's commit wait?

A: Commit wait ensures that a read/write transaction does not complete
(release locks and reply to the client) until the time in its
timestamp is guaranteed to have passed. That means that a read/only
transaction that starts after the read/write transaction completes is
guaranteed to have a higher timestamp, and thus to see the read/write
transaction's writes. This helps fulfill the guarantee of external
consistency: if T1 completes before T2 starts, T2 will come after T1
in the equivalent serial order (i.e. T2 will see T1's writes).

Commit wait is needed because clocks on different computers aren't
perfectly synchronized. If they were guaranteed to be perfectly
synchronized, then the TrueTime interval could always be zero, and
commit wait would never have to wait because TT.after(TT.now().latest)
would always be true. Another way to look at this is that there's
value in making TrueTime more accurate (i.e. shrinking the interval),
because that allows read/write transactions to commit faster.

Q: Why does a read/only transaction use TT.now().latest as
its timestamp?

A: Suppose a read-only transaction T2 is starting. External
consistency demands that T2 see writes by any read-write transaction
T1 that finished before T2 started. Spanner's Snapshot Isolation means
that it's sufficient for T2 to choose a time-stamp that's greater than
T1's time-stamp: then T2 will see T1's writes.

T1 chose TT.now().latest as its time-stamp; then commit-wait caused T1
to wait until that time-stamp was guaranteed to be in the past; and
only then did T1 commit and finish. So after T1 has finished, its
time-stamp is guaranteed to be in the past.

Since T2 starts after T1 finishes, and T1's time-stamp is guaranteed
to have passed, and TT.now().latest is guaranteed not to have passed
yet, TT.now().latest is guaranteed to be larger than T1's time-stamp.
And thus T2 using TT.now().latest means that T2 will see T1's writes.

Q: Why does a read/write transaction use TT.now().latest as its
timestamp?

A: If there are two read/write transactions T1 and T2, and T1 ends
before T2 starts, TS2 needs to be greater than TS1 so that any r/o
transactions that see T2's writes will also see T1's writes. Just as
with r/o transactions, T2 chooses TS2 = TT.now().latest in order to
guarantee that TS2 is larger than TS1.

Q: What are the steps in committing a read-write transaction?

A:
0. A read-write transaction causes read locks to be taken as it
   executes on all the records it uses (including for writes).
1. Once the client has finished all its reads and writes, and is ready
   to ask Spanner to try to commit, the client chooses a two-phase
   commit coordinator.
2. The client sends each of its writes to the leader of the
   relevant shard, along with the identity of the chosen coordinator.
3. Each shard leader upgrades locks to write (perhaps waiting), chooses
   a "prepare timestamp", logs this information in Paxos, and sends a
   "prepared" message to the coordinator containing the prepare
   timestamp. The shard leader chooses the prepare timestamp to be
   larger than the timestamp of any transaction it knows about.
4. The coordinator waits for all the prepared messages. It chooses a
   timestamp TS for the transaction that's >= TT.now().latest and
   greater than any participant's prepared timestamp. The coordinator
   waits until TS is guaranteed to have passed (the commit-wait). Then
   the coordinator sends a reply to the client, and sends each
   participant a message with TS telling the participant to commit.
5. Each participant performs the writes for its shard, and then
   releases locks.

Q: What's the point of the "prepare timestamp"?

A: Each shard leader must apply writes in timestamp order, so that the
"safe time" mechanism in 4.1.3 can guarantee that if a replica has
seen a write with timestamp X, it has also seen every write with
timestamp less than X. So a participant needs to be able to tell the
two-phase-commit coordinator "please don't assign this transaction a
timestamp less than X", where X is the highest timestamp the
participant has already seen.

Q: What is wound-wait (Section 4.2.1)?

A: Wound-wait is a technique for avoiding deadlock. When transaction
T1 needs a lock held by T2, and T1 is older (smaller time-stamp) than
T2, wound-wait will abort T2 and allow T1 to immediately have the lock
it needs. Otherwise (if T1 is younger than T2) T1 will wait for the
lock. So the waits-for graph will be a DAG, and thus have a no cycles
and no deadlocks.

On the plus side, wound-wait avoids deadlocks, and is relatively
simple. If deadlocks were allowed to develop, the options for
detecting and resolving them in a distributed database are none of
them attractive. On the minus side, wound-wait may sometimes generate
aborts (and consequent re-tries) in situations where waiting for the
lock would be safe and more efficient.

Q: What is a schema change (as in Section 4.2.3)?

A: "Schema" refers to the set of tables, columns, types, and indices
provided by a database deployment, plus perhaps the way these objects
are laid out over physical servers and datacenters. A schema change
involves adding or deleting or changing some of these objects.

The challenge during a schema change is to allow concurrent reads and
writes, and also to ensure that all writes are reflected in the final
data state. Every concurrent write must either logically occur before
the schema change (which will then copy the write to the new DB
state), or after the schema change (in which case the write will
naturally affect the post-change DB state).

I cannot tell, after reading 4.2.3, whether Spanner's schema change
transactions are ordinary read/write transactions, so that their nice
features fall out naturally from Spanner's design, or whether Spanner
handles schema change transactions specially. The paper's text does
not say whether "they must block behind the schema-change transaction"
means that the blocking is caused by locks acquired by schema change,
or whether the blocking is caused by use of time-stamps. My guess is
the latter.

Q: In what ways is Spanner better than GFS?

A: GFS provides just huge files. GFS has few consistency guarantees.
For example, if I write to a file, and then you read it, GFS does not
guarantee that you will see my write. GFS has no notion of
transactions.

Spanner provides most of the nice features and properties of an ACID
database. Spanner has tables with rows and columns, and relational
queries over those tables (as in SQL). Spanner has atomic distributed
transactions, and guarantees very strong consistency (linearizability
and strict serializability).

Q: What is an atomic clock?

A: A very stable oscillator. There are two main technologies that go
by the name "atomic clock": rubidium clocks and cesium clocks. Both
exploit changes in the state of the outer electron, which involve
specific quanta of energy and thus wavelength. One can tune a signal
generator to precisely that wavelength by watching how excited the
electrons are. An atomic clock is just the oscillator part of a clock:
it produces a frequency that can cause a clock to tick at exactly the
right rate, but does not by itself know what time it is. To provide
time, an atomic clock must initially be synchronized with the time,
often via GPS (which itself is fed the time by a bunch of atomic
clocks).

Q: What kind of atomic clock does Spanner use?

A: Sadly the paper doesn't say. Rubidium clocks are typically a few
thousand dollars (e.g. https://thinksrs.com/products/fs725.html).
Rubidium clocks drift by perhaps a few microseconds per week, so they
need to be re-synchronized to UTC (typically by GPS) every once in a
while. Cesium clocks cost perhaps $50,000; the HP 5071A is a good
example. A working cesium clock doesn't drift. Of course, any one
clock might fail or suffer a power failure, so even with cesium clocks
you still need more than one and the ability to synchronize to UTC. A
picture on this page includes a 5071A:
https://sookocheff.com/post/time/truetime/

Q: Does anyone use Spanner?

A: It's said that hundreds of Google services depend on Spanner. The
paper talks about its use by Google's advertising system. Google's
Zanzibar Authorization system uses Spanner. It's offered as a service
to Google's cloud customers in the form of Cloud Spanner. The
CockroachDB open-source database is based on the Spanner design.

Spanner: Google’s Globally-Distributed Database
James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, JJ Furman,
Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser , Peter Hochschild, Wilson Hsieh,
Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura,
David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak,
Christopher Taylor , Ruth Wang, Dale Woodford
Google, Inc.
Abstract
Spanner is Google’s scalable, multi-version, globally-
distributed, and synchronously-replicated database. It is
the first system to distribute data at global scale and sup-
port externally-consistent distributed transactions. This
paper describes how Spanner is structured, its feature set,
the rationale underlying various design decisions, and a
novel time API that exposes clock uncertainty. This API
and its implementation are critical to supporting exter-
nal consistency and a variety of powerful features: non-
blocking reads in the past, lock-free read-only transac-
tions, and atomic schema changes, across all of Spanner.
1 Introduction
Spanner is a scalable, globally-distributed database de-
signed, built, and deployed at Google. At the high-
est level of abstraction, it is a database that shards data
across many sets of Paxos [21] state machines in data-
centers spread all over the world. Replication is used for
global availability and geographic locality; clients auto-
matically failover between replicas. Spanner automati-
cally reshards data across machines as the amount of data
or the number of servers changes, and it automatically
migrates data across machines (even across datacenters)
to balance load and in response to failures. Spanner is
designed to scale up to millions of machines across hun-
dreds of datacenters and trillions of database rows.
Applications can use Spanner for high availability,
even in the face of wide-area natural disasters, by repli-
cating their data within or even across continents. Our
initial customer was F1 [35], a rewrite of Google’s ad-
vertising backend. F1 uses five replicas spread across
the United States. Most other applications will probably
replicate their data across 3 to 5 datacenters in one ge-
ographic region, but with relatively independent failure
modes. That is, most applications will choose lower la-
tency over higher availability, as long as they can survive
1 or 2 datacenter failures.
Spanner’s main focus is managing cross-datacenter
replicated data, but we have also spent a great deal of
time in designing and implementing important database
features on top of our distributed-systems infrastructure.
Even though many projects happily use Bigtable [9], we
have also consistently received complaints from users
that Bigtable can be difficult to use for some kinds of ap-
plications: those that have complex, evolving schemas,
or those that want strong consistency in the presence of
wide-area replication. (Similar claims have been made
by other authors [37].) Many applications at Google
have chosen to use Megastore [5] because of its semi-
relational data model and support for synchronous repli-
cation, despite its relatively poor write throughput. As a
consequence, Spanner has evolved from a Bigtable-like
versioned key-value store into a temporal multi-version
database. Data is stored in schematized semi-relational
tables; data is versioned, and each version is automati-
cally timestamped with its commit time; old versions of
data are subject to configurable garbage-collection poli-
cies; and applications can read data at old timestamps.
Spanner supports general-purpose transactions, and pro-
vides a SQL-based query language.
As a globally-distributed database, Spanner provides
several interesting features. First, the replication con-
figurations for data can be dynamically controlled at a
fine grain by applications. Applications can specify con-
straints to control which datacenters contain which data,
how far data is from its users (to control read latency),
how far replicas are from each other (to control write la-
tency), and how many replicas are maintained (to con-
trol durability, availability, and read performance). Data
can also be dynamically and transparently moved be-
tween datacenters by the system to balance resource us-
age across datacenters. Second, Spanner has two features
that are difficult to implement in a distributed database: it
Published in the Proceedings of OSDI 2012 1

provides externally consistent [16] reads and writes, and
globally-consistent reads across the database at a time-
stamp. These features enable Spanner to support con-
sistent backups, consistent MapReduce executions [12],
and atomic schema updates, all at global scale, and even
in the presence of ongoing transactions.
These features are enabled by the fact that Spanner as-
signs globally-meaningful commit timestamps to trans-
actions, even though transactions may be distributed.
The timestamps reflect serialization order. In addition,
the serialization order satisfies external consistency (or
equivalently, linearizability [20]): if a transaction T1
commits before another transaction T2 starts, then T1’s
commit timestamp is smaller than T2’s. Spanner is the
first system to provide such guarantees at global scale.
The key enabler of these properties is a new TrueTime
API and its implementation. The API directly exposes
clock uncertainty, and the guarantees on Spanner’s times-
tamps depend on the bounds that the implementation pro-
vides. If the uncertainty is large, Spanner slows down to
wait out that uncertainty. Google’s cluster-management
software provides an implementation of the TrueTime
API. This implementation keeps uncertainty small (gen-
erally less than 10ms) by using multiple modern clock
references (GPS and atomic clocks).
Section 2 describes the structure of Spanner’s imple-
mentation, its feature set, and the engineering decisions
that went into their design. Section 3 describes our new
TrueTime API and sketches its implementation. Sec-
tion 4 describes how Spanner uses TrueTime to imple-
ment externally-consistent distributed transactions, lock-
free read-only transactions, and atomic schema updates.
Section 5 provides some benchmarks on Spanner’s per-
formance and TrueTime behavior, and discusses the ex-
periences of F1. Sections 6, 7, and 8 describe related and
future work, and summarize our conclusions.
2 Implementation
This section describes the structure of and rationale un-
derlying Spanner’s implementation. It then describes the
directory abstraction, which is used to manage replica-
tion and locality, and is the unit of data movement. Fi-
nally, it describes our data model, why Spanner looks
like a relational database instead of a key-value store, and
how applications can control data locality.
A Spanner deployment is called a universe. Given
that Spanner manages data globally, there will be only
a handful of running universes. We currently run a
test/playground universe, a development/production uni-
verse, and a production-only universe.
Spanner is organized as a set of zones, where each
zone is the rough analog of a deployment of Bigtable
Figure 1: Spanner server organization.
servers [9]. Zones are the unit of administrative deploy-
ment. The set of zones is also the set of locations across
which data can be replicated. Zones can be added to or
removed from a running system as new datacenters are
brought into service and old ones are turned off, respec-
tively. Zones are also the unit of physical isolation: there
may be one or more zones in a datacenter, for example,
if different applications’ data must be partitioned across
different sets of servers in the same datacenter.
Figure 1 illustrates the servers in a Spanner universe.
A zone has one zonemaster and between one hundred
and several thousand spanservers. The former assigns
data to spanservers; the latter serve data to clients. The
per-zone location proxies are used by clients to locate
the spanservers assigned to serve their data. The uni-
verse master and the placement driver are currently sin-
gletons. The universe master is primarily a console that
displays status information about all the zones for inter-
active debugging. The placement driver handles auto-
mated movement of data across zones on the timescale
of minutes. The placement driver periodically commu-
nicates with the spanservers to find data that needs to be
moved, either to meet updated replication constraints or
to balance load. For space reasons, we will only describe
the spanserver in any detail.
2.1 Spanserver Software Stack
This section focuses on the spanserver implementation
to illustrate how replication and distributed transactions
have been layered onto our Bigtable-based implementa-
tion. The software stack is shown in Figure 2. At the
bottom, each spanserver is responsible for between 100
and 1000 instances of a data structure called a tablet. A
tablet is similar to Bigtable’s tablet abstraction, in that it
implements a bag of the following mappings:
(key:string, timestamp:int64)→ string
Unlike Bigtable, Spanner assigns timestamps to data,
which is an important way in which Spanner is more
like a multi-version database than a key-value store. A
Published in the Proceedings of OSDI 2012 2

Figure 2: Spanserver software stack.
tablet’s state is stored in set of B-tree-like files and a
write-ahead log, all on a distributed file system called
Colossus (the successor to the Google File System [15]).
To support replication, each spanserver implements a
single Paxos state machine on top of each tablet. (An
early Spanner incarnation supported multiple Paxos state
machines per tablet, which allowed for more flexible
replication configurations. The complexity of that de-
sign led us to abandon it.) Each state machine stores
its metadata and log in its corresponding tablet. Our
Paxos implementation supports long-lived leaders with
time-based leader leases, whose length defaults to 10
seconds. The current Spanner implementation logs ev-
ery Paxos write twice: once in the tablet’s log, and once
in the Paxos log. This choice was made out of expedi-
ency, and we are likely to remedy this eventually. Our
implementation of Paxos is pipelined, so as to improve
Spanner’s throughput in the presence of W AN latencies;
but writes are applied by Paxos in order (a fact on which
we will depend in Section 4).
The Paxos state machines are used to implement a
consistently replicated bag of mappings. The key-value
mapping state of each replica is stored in its correspond-
ing tablet. Writes must initiate the Paxos protocol at the
leader; reads access state directly from the underlying
tablet at any replica that is sufficiently up-to-date. The
set of replicas is collectively a Paxos group.
At every replica that is a leader, each spanserver im-
plements a lock table to implement concurrency control.
The lock table contains the state for two-phase lock-
ing: it maps ranges of keys to lock states. (Note that
having a long-lived Paxos leader is critical to efficiently
managing the lock table.) In both Bigtable and Span-
ner, we designed for long-lived transactions (for exam-
ple, for report generation, which might take on the order
of minutes), which perform poorly under optimistic con-
currency control in the presence of conflicts. Operations
Figure 3: Directories are the unit of data movement between
Paxos groups.
that require synchronization, such as transactional reads,
acquire locks in the lock table; other operations bypass
the lock table.
At every replica that is a leader, each spanserver also
implements a transaction manager to support distributed
transactions. The transaction manager is used to imple-
ment a participant leader; the other replicas in the group
will be referred to as participant slaves . If a transac-
tion involves only one Paxos group (as is the case for
most transactions), it can bypass the transaction manager,
since the lock table and Paxos together provide transac-
tionality. If a transaction involves more than one Paxos
group, those groups’ leaders coordinate to perform two-
phase commit. One of the participant groups is chosen as
the coordinator: the participant leader of that group will
be referred to as thecoordinator leader, and the slaves of
that group as coordinator slaves. The state of each trans-
action manager is stored in the underlying Paxos group
(and therefore is replicated).
2.2 Directories and Placement
On top of the bag of key-value mappings, the Spanner
implementation supports a bucketing abstraction called a
directory, which is a set of contiguous keys that share a
common prefix. (The choice of the term directory is a
historical accident; a better term might be bucket.) We
will explain the source of that prefix in Section 2.3. Sup-
porting directories allows applications to control the lo-
cality of their data by choosing keys carefully.
A directory is the unit of data placement. All data in
a directory has the same replication configuration. When
data is moved between Paxos groups, it is moved direc-
tory by directory, as shown in Figure 3. Spanner might
move a directory to shed load from a Paxos group; to put
directories that are frequently accessed together into the
same group; or to move a directory into a group that is
closer to its accessors. Directories can be moved while
client operations are ongoing. One could expect that a
50MB directory can be moved in a few seconds.
The fact that a Paxos group may contain multiple di-
rectories implies that a Spanner tablet is different from
Published in the Proceedings of OSDI 2012 3

a Bigtable tablet: the former is not necessarily a single
lexicographically contiguous partition of the row space.
Instead, a Spanner tablet is a container that may encap-
sulate multiple partitions of the row space. We made this
decision so that it would be possible to colocate multiple
directories that are frequently accessed together.
Movedir is the background task used to move direc-
tories between Paxos groups [14]. Movedir is also used
to add or remove replicas to Paxos groups [25], be-
cause Spanner does not yet support in-Paxos configura-
tion changes. Movedir is not implemented as a single
transaction, so as to avoid blocking ongoing reads and
writes on a bulky data move. Instead, movedir registers
the fact that it is starting to move data and moves the data
in the background. When it has moved all but a nominal
amount of the data, it uses a transaction to atomically
move that nominal amount and update the metadata for
the two Paxos groups.
A directory is also the smallest unit whose geographic-
replication properties (or placement, for short) can
be specified by an application. The design of our
placement-specification language separates responsibil-
ities for managing replication configurations. Adminis-
trators control two dimensions: the number and types of
replicas, and the geographic placement of those replicas.
They create a menu of named options in these two di-
mensions (e.g., North America, replicated 5 ways with
1 witness ). An application controls how data is repli-
cated, by tagging each database and/or individual direc-
tories with a combination of those options. For example,
an application might store each end-user’s data in its own
directory, which would enable userA’s data to have three
replicas in Europe, and user B’s data to have five replicas
in North America.
For expository clarity we have over-simplified. In fact,
Spanner will shard a directory into multiple fragments
if it grows too large. Fragments may be served from
different Paxos groups (and therefore different servers).
Movedir actually moves fragments, and not whole direc-
tories, between groups.
2.3 Data Model
Spanner exposes the following set of data features
to applications: a data model based on schematized
semi-relational tables, a query language, and general-
purpose transactions. The move towards support-
ing these features was driven by many factors. The
need to support schematized semi-relational tables and
synchronous replication is supported by the popular-
ity of Megastore [5]. At least 300 applications within
Google use Megastore (despite its relatively low per-
formance) because its data model is simpler to man-
age than Bigtable’s, and because of its support for syn-
chronous replication across datacenters. (Bigtable only
supports eventually-consistent replication across data-
centers.) Examples of well-known Google applications
that use Megastore are Gmail, Picasa, Calendar, Android
Market, and AppEngine. The need to support a SQL-
like query language in Spanner was also clear, given
the popularity of Dremel [28] as an interactive data-
analysis tool. Finally, the lack of cross-row transactions
in Bigtable led to frequent complaints; Percolator [32]
was in part built to address this failing. Some authors
have claimed that general two-phase commit is too ex-
pensive to support, because of the performance or avail-
ability problems that it brings [9, 10, 19]. We believe it
is better to have application programmers deal with per-
formance problems due to overuse of transactions as bot-
tlenecks arise, rather than always coding around the lack
of transactions. Running two-phase commit over Paxos
mitigates the availability problems.
The application data model is layered on top of the
directory-bucketed key-value mappings supported by the
implementation. An application creates one or more
databases in a universe. Each database can contain an
unlimited number of schematized tables. Tables look
like relational-database tables, with rows, columns, and
versioned values. We will not go into detail about the
query language for Spanner. It looks like SQL with some
extensions to support protocol-buffer-valued fields.
Spanner’s data model is not purely relational, in that
rows must have names. More precisely, every table is re-
quired to have an ordered set of one or more primary-key
columns. This requirement is where Spanner still looks
like a key-value store: the primary keys form the name
for a row, and each table defines a mapping from the
primary-key columns to the non-primary-key columns.
A row has existence only if some value (even if it is
NULL) is defined for the row’s keys. Imposing this struc-
ture is useful because it lets applications control data lo-
cality through their choices of keys.
Figure 4 contains an example Spanner schema for stor-
ing photo metadata on a per-user, per-album basis. The
schema language is similar to Megastore’s, with the ad-
ditional requirement that every Spanner database must
be partitioned by clients into one or more hierarchies
of tables. Client applications declare the hierarchies in
database schemas via the INTERLEAVE IN declara-
tions. The table at the top of a hierarchy is a directory
table. Each row in a directory table with key K, together
with all of the rows in descendant tables that start withK
in lexicographic order, forms a directory. ON DELETE
CASCADE says that deleting a row in the directory table
deletes any associated child rows. The figure also illus-
trates the interleaved layout for the example database: for
Published in the Proceedings of OSDI 2012 4

CREATE TABLE Users {
uid INT64 NOT NULL, email STRING
} PRIMARY KEY (uid), DIRECTORY;
CREATE TABLE Albums {
uid INT64 NOT NULL, aid INT64 NOT NULL,
name STRING
} PRIMARY KEY (uid, aid),
INTERLEAVE IN PARENT Users ON DELETE CASCADE;
Figure 4: Example Spanner schema for photo metadata, and
the interleaving implied by INTERLEAVE IN .
example, Albums(2,1) represents the row from the
Albums table for user id 2, album id 1 . This
interleaving of tables to form directories is significant
because it allows clients to describe the locality relation-
ships that exist between multiple tables, which is nec-
essary for good performance in a sharded, distributed
database. Without it, Spanner would not know the most
important locality relationships.
3 TrueTime
Method Returns
TT.now() TTinterval: [earliest, latest]
TT.after(t) true ift has definitely passed
TT.before(t) true ift has definitely not arrived
Table 1: TrueTime API. The argumentt is of type TTstamp.
This section describes the TrueTime API and sketches
its implementation. We leave most of the details for an-
other paper: our goal is to demonstrate the power of
having such an API. Table 1 lists the methods of the
API. TrueTime explicitly represents time as aTTinterval,
which is an interval with bounded time uncertainty (un-
like standard time interfaces that give clients no notion
of uncertainty). The endpoints of a TTinterval are of
type TTstamp. The TT.now() method returns aTTinterval
that is guaranteed to contain the absolute time during
which TT.now() was invoked. The time epoch is anal-
ogous to UNIX time with leap-second smearing. De-
fine the instantaneous error bound as ϵ, which is half of
the interval’s width, and the average error bound as ϵ.
The TT.after() and TT.before() methods are convenience
wrappers around TT.now().
Denote the absolute time of an event e by the func-
tion tabs(e). In more formal terms, TrueTime guaran-
tees that for an invocation tt = TT.now(), tt.earliest≤
tabs(enow)≤ tt.latest, whereenow is the invocation event.
The underlying time references used by TrueTime
are GPS and atomic clocks. TrueTime uses two forms
of time reference because they have different failure
modes. GPS reference-source vulnerabilities include an-
tenna and receiver failures, local radio interference, cor-
related failures (e.g., design faults such as incorrect leap-
second handling and spoofing), and GPS system outages.
Atomic clocks can fail in ways uncorrelated to GPS and
each other, and over long periods of time can drift signif-
icantly due to frequency error.
TrueTime is implemented by a set of time master ma-
chines per datacenter and a timeslave daemon per ma-
chine. The majority of masters have GPS receivers with
dedicated antennas; these masters are separated physi-
cally to reduce the effects of antenna failures, radio in-
terference, and spoofing. The remaining masters (which
we refer to as Armageddon masters) are equipped with
atomic clocks. An atomic clock is not that expensive:
the cost of an Armageddon master is of the same order
as that of a GPS master. All masters’ time references
are regularly compared against each other. Each mas-
ter also cross-checks the rate at which its reference ad-
vances time against its own local clock, and evicts itself
if there is substantial divergence. Between synchroniza-
tions, Armageddon masters advertise a slowly increasing
time uncertainty that is derived from conservatively ap-
plied worst-case clock drift. GPS masters advertise un-
certainty that is typically close to zero.
Every daemon polls a variety of masters [29] to re-
duce vulnerability to errors from any one master. Some
are GPS masters chosen from nearby datacenters; the
rest are GPS masters from farther datacenters, as well
as some Armageddon masters. Daemons apply a variant
of Marzullo’s algorithm [27] to detect and reject liars,
and synchronize the local machine clocks to the non-
liars. To protect against broken local clocks, machines
that exhibit frequency excursions larger than the worst-
case bound derived from component specifications and
operating environment are evicted.
Between synchronizations, a daemon advertises a
slowly increasing time uncertainty. ϵ is derived from
conservatively applied worst-case local clock drift.ϵ also
depends on time-master uncertainty and communication
delay to the time masters. In our production environ-
ment,ϵ is typically a sawtooth function of time, varying
from about 1 to 7 ms over each poll interval. ϵ is there-
fore 4 ms most of the time. The daemon’s poll interval is
currently 30 seconds, and the current applied drift rate is
set at 200 microseconds/second, which together account
Published in the Proceedings of OSDI 2012 5

Timestamp Concurrency
Operation Discussion Control Replica Required
Read-Write Transaction § 4.1.2 pessimistic leader
Read-Only Transaction § 4.1.4 lock-free leader for timestamp; any for
read, subject to§ 4.1.3
Snapshot Read, client-provided timestamp — lock-free any, subject to§ 4.1.3
Snapshot Read, client-provided bound § 4.1.3 lock-free any, subject to§ 4.1.3
Table 2: Types of reads and writes in Spanner, and how they compare.
for the sawtooth bounds from 0 to 6 ms. The remain-
ing 1 ms comes from the communication delay to the
time masters. Excursions from this sawtooth are possi-
ble in the presence of failures. For example, occasional
time-master unavailability can cause datacenter-wide in-
creases inϵ. Similarly, overloaded machines and network
links can result in occasional localizedϵ spikes.
4 Concurrency Control
This section describes how TrueTime is used to guaran-
tee the correctness properties around concurrency con-
trol, and how those properties are used to implement
features such as externally consistent transactions, lock-
free read-only transactions, and non-blocking reads in
the past. These features enable, for example, the guar-
antee that a whole-database audit read at a timestamp t
will see exactly the effects of every transaction that has
committed as of t.
Going forward, it will be important to distinguish
writes as seen by Paxos (which we will refer to as Paxos
writes unless the context is clear) from Spanner client
writes. For example, two-phase commit generates a
Paxos write for the prepare phase that has no correspond-
ing Spanner client write.
4.1 Timestamp Management
Table 2 lists the types of operations that Spanner sup-
ports. The Spanner implementation supports read-
write transactions , read-only transactions (predeclared
snapshot-isolation transactions), and snapshot reads .
Standalone writes are implemented as read-write trans-
actions; non-snapshot standalone reads are implemented
as read-only transactions. Both are internally retried
(clients need not write their own retry loops).
A read-only transaction is a kind of transaction that
has the performance benefits of snapshot isolation [6].
A read-only transaction must be predeclared as not hav-
ing any writes; it is not simply a read-write transaction
without any writes. Reads in a read-only transaction ex-
ecute at a system-chosen timestamp without locking, so
that incoming writes are not blocked. The execution of
the reads in a read-only transaction can proceed on any
replica that is sufficiently up-to-date (Section 4.1.3).
A snapshot read is a read in the past that executes with-
out locking. A client can either specify a timestamp for a
snapshot read, or provide an upper bound on the desired
timestamp’s staleness and let Spanner choose a time-
stamp. In either case, the execution of a snapshot read
proceeds at any replica that is sufficiently up-to-date.
For both read-only transactions and snapshot reads,
commit is inevitable once a timestamp has been cho-
sen, unless the data at that timestamp has been garbage-
collected. As a result, clients can avoid buffering results
inside a retry loop. When a server fails, clients can inter-
nally continue the query on a different server by repeat-
ing the timestamp and the current read position.
4.1.1 Paxos Leader Leases
Spanner’s Paxos implementation uses timed leases to
make leadership long-lived (10 seconds by default). A
potential leader sends requests for timed lease votes ;
upon receiving a quorum of lease votes the leader knows
it has a lease. A replica extends its lease vote implicitly
on a successful write, and the leader requests lease-vote
extensions if they are near expiration. Define a leader’s
lease interval as starting when it discovers it has a quo-
rum of lease votes, and as ending when it no longer has
a quorum of lease votes (because some have expired).
Spanner depends on the following disjointness invariant:
for each Paxos group, each Paxos leader’s lease interval
is disjoint from every other leader’s. Appendix A de-
scribes how this invariant is enforced.
The Spanner implementation permits a Paxos leader
to abdicate by releasing its slaves from their lease votes.
To preserve the disjointness invariant, Spanner constrains
when abdication is permissible. Define smax to be the
maximum timestamp used by a leader. Subsequent sec-
tions will describe when smax is advanced. Before abdi-
cating, a leader must wait until TT.after(smax) is true.
4.1.2 Assigning Timestamps to RW Transactions
Transactional reads and writes use two-phase locking.
As a result, they can be assigned timestamps at any time
Published in the Proceedings of OSDI 2012 6

when all locks have been acquired, but before any locks
have been released. For a given transaction, Spanner as-
signs it the timestamp that Paxos assigns to the Paxos
write that represents the transaction commit.
Spanner depends on the following monotonicity in-
variant: within each Paxos group, Spanner assigns times-
tamps to Paxos writes in monotonically increasing or-
der, even across leaders. A single leader replica can triv-
ially assign timestamps in monotonically increasing or-
der. This invariant is enforced across leaders by making
use of the disjointness invariant: a leader must only as-
sign timestamps within the interval of its leader lease.
Note that whenever a timestamp s is assigned, smax is
advanced tos to preserve disjointness.
Spanner also enforces the following external-
consistency invariant: if the start of a transaction T2
occurs after the commit of a transaction T1, then the
commit timestamp of T2 must be greater than the
commit timestamp of T1. Define the start and commit
events for a transaction Ti by estart
i and ecommit
i ; and
the commit timestamp of a transaction Ti by si. The
invariant becomestabs(ecommit
1 )<t abs(estart
2 )⇒s1 <s 2.
The protocol for executing transactions and assigning
timestamps obeys two rules, which together guarantee
this invariant, as shown below. Define the arrival event
of the commit request at the coordinator leader for a
writeTi to beeserver
i .
Start The coordinator leader for a write Ti assigns
a commit timestamp si no less than the value of
TT.now().latest, computed after eserver
i . Note that the
participant leaders do not matter here; Section 4.2.1 de-
scribes how they are involved in the implementation of
the next rule.
Commit Wait The coordinator leader ensures that
clients cannot see any data committed by Ti until
TT.after(si) is true. Commit wait ensures that si is
less than the absolute commit time of Ti, or si <
tabs(ecommit
i ). The implementation of commit wait is de-
scribed in Section 4.2.1. Proof:
s1 <t abs(ecommit
1 ) (commit wait)
tabs(ecommit
1 )<t abs(estart
2 ) (assumption)
tabs(estart
2 )≤tabs(eserver
2 ) (causality)
tabs(eserver
2 )≤s2 (start)
s1 <s 2 (transitivity)
4.1.3 Serving Reads at a Timestamp
The monotonicity invariant described in Section 4.1.2 al-
lows Spanner to correctly determine whether a replica’s
state is sufficiently up-to-date to satisfy a read. Every
replica tracks a value called safe time tsafe which is the
maximum timestamp at which a replica is up-to-date. A
replica can satisfy a read at a timestampt ift< =tsafe.
Define tsafe = min(tPaxos
safe ,t TM
safe), where each Paxos
state machine has a safe time tPaxos
safe and each transac-
tion manager has a safe time tTM
safe. tPaxos
safe is simpler: it
is the timestamp of the highest-applied Paxos write. Be-
cause timestamps increase monotonically and writes are
applied in order, writes will no longer occur at or below
tPaxos
safe with respect to Paxos.
tTM
safe is∞ at a replica if there are zero prepared (but
not committed) transactions—that is, transactions in be-
tween the two phases of two-phase commit. (For a par-
ticipant slave,tTM
safe actually refers to the replica’s leader’s
transaction manager, whose state the slave can infer
through metadata passed on Paxos writes.) If there are
any such transactions, then the state affected by those
transactions is indeterminate: a participant replica does
not know yet whether such transactions will commit. As
we discuss in Section 4.2.1, the commit protocol ensures
that every participant knows a lower bound on a pre-
pared transaction’s timestamp. Every participant leader
(for a group g) for a transaction Ti assigns a prepare
timestampsprepare
i,g to its prepare record. The coordinator
leader ensures that the transaction’s commit timestamp
si >= sprepare
i,g over all participant groups g. Therefore,
for every replica in a groupg, over all transactionsTi pre-
pared atg,tTM
safe = mini(sprepare
i,g )− 1 over all transactions
prepared atg.
4.1.4 Assigning Timestamps to RO Transactions
A read-only transaction executes in two phases: assign
a timestamp sread [8], and then execute the transaction’s
reads as snapshot reads at sread. The snapshot reads can
execute at any replicas that are sufficiently up-to-date.
The simple assignment of sread = TT.now().latest, at
any time after a transaction starts, preserves external con-
sistency by an argument analogous to that presented for
writes in Section 4.1.2. However, such a timestamp may
require the execution of the data reads at sread to block
if tsafe has not advanced sufficiently. (In addition, note
that choosing a value of sread may also advance smax to
preserve disjointness.) To reduce the chances of block-
ing, Spanner should assign the oldest timestamp that pre-
serves external consistency. Section 4.2.2 explains how
such a timestamp can be chosen.
4.2 Details
This section explains some of the practical details of
read-write transactions and read-only transactions elided
earlier, as well as the implementation of a special trans-
action type used to implement atomic schema changes.
Published in the Proceedings of OSDI 2012 7

It then describes some refinements of the basic schemes
as described.
4.2.1 Read-Write Transactions
Like Bigtable, writes that occur in a transaction are
buffered at the client until commit. As a result, reads
in a transaction do not see the effects of the transaction’s
writes. This design works well in Spanner because a read
returns the timestamps of any data read, and uncommit-
ted writes have not yet been assigned timestamps.
Reads within read-write transactions use wound-
wait [33] to avoid deadlocks. The client issues reads
to the leader replica of the appropriate group, which
acquires read locks and then reads the most recent
data. While a client transaction remains open, it sends
keepalive messages to prevent participant leaders from
timing out its transaction. When a client has completed
all reads and buffered all writes, it begins two-phase
commit. The client chooses a coordinator group and
sends a commit message to each participant’s leader with
the identity of the coordinator and any buffered writes.
Having the client drive two-phase commit avoids send-
ing data twice across wide-area links.
A non-coordinator-participant leader first acquires
write locks. It then chooses a prepare timestamp that
must be larger than any timestamps it has assigned to pre-
vious transactions (to preserve monotonicity), and logs a
prepare record through Paxos. Each participant then no-
tifies the coordinator of its prepare timestamp.
The coordinator leader also first acquires write locks,
but skips the prepare phase. It chooses a timestamp for
the entire transaction after hearing from all other partici-
pant leaders. The commit timestamp s must be greater or
equal to all prepare timestamps (to satisfy the constraints
discussed in Section 4.1.3), greater than TT.now().latest
at the time the coordinator received its commit message,
and greater than any timestamps the leader has assigned
to previous transactions (again, to preserve monotonic-
ity). The coordinator leader then logs a commit record
through Paxos (or an abort if it timed out while waiting
on the other participants).
Before allowing any coordinator replica to apply
the commit record, the coordinator leader waits until
TT.after(s), so as to obey the commit-wait rule described
in Section 4.1.2. Because the coordinator leader chose s
based on TT.now().latest, and now waits until that time-
stamp is guaranteed to be in the past, the expected wait
is at least 2∗ϵ. This wait is typically overlapped with
Paxos communication. After commit wait, the coordi-
nator sends the commit timestamp to the client and all
other participant leaders. Each participant leader logs the
transaction’s outcome through Paxos. All participants
apply at the same timestamp and then release locks.
4.2.2 Read-Only Transactions
Assigning a timestamp requires a negotiation phase be-
tween all of the Paxos groups that are involved in the
reads. As a result, Spanner requires a scope expression
for every read-only transaction, which is an expression
that summarizes the keys that will be read by the entire
transaction. Spanner automatically infers the scope for
standalone queries.
If the scope’s values are served by a single Paxos
group, then the client issues the read-only transaction to
that group’s leader. (The current Spanner implementa-
tion only chooses a timestamp for a read-only transac-
tion at a Paxos leader.) That leader assigns sread and ex-
ecutes the read. For a single-site read, Spanner gener-
ally does better than TT.now().latest. Define LastTS() to
be the timestamp of the last committed write at a Paxos
group. If there are no prepared transactions, the assign-
mentsread = LastTS() trivially satisfies external consis-
tency: the transaction will see the result of the last write,
and therefore be ordered after it.
If the scope’s values are served by multiple Paxos
groups, there are several options. The most complicated
option is to do a round of communication with all of
the groups’s leaders to negotiatesread based on LastTS().
Spanner currently implements a simpler choice. The
client avoids a negotiation round, and just has its reads
execute at sread = TT.now().latest (which may wait for
safe time to advance). All reads in the transaction can be
sent to replicas that are sufficiently up-to-date.
4.2.3 Schema-Change Transactions
TrueTime enables Spanner to support atomic schema
changes. It would be infeasible to use a standard transac-
tion, because the number of participants (the number of
groups in a database) could be in the millions. Bigtable
supports atomic schema changes in one datacenter, but
its schema changes block all operations.
A Spanner schema-change transaction is a generally
non-blocking variant of a standard transaction. First, it
is explicitly assigned a timestamp in the future, which
is registered in the prepare phase. As a result, schema
changes across thousands of servers can complete with
minimal disruption to other concurrent activity. Sec-
ond, reads and writes, which implicitly depend on the
schema, synchronize with any registered schema-change
timestamp at time t: they may proceed if their times-
tamps precede t, but they must block behind the schema-
change transaction if their timestamps are after t. With-
out TrueTime, defining the schema change to happen att
would be meaningless.
Published in the Proceedings of OSDI 2012 8

latency (ms) throughput (Kops/sec)
replicas write read-only transaction snapshot read write read-only transaction snapshot read
1D 9.4±.6 — — 4.0±.3 — —
1 14.4±1.0 1.4±.1 1.3±.1 4.1±.05 10.9±.4 13.5±.1
3 13.9±.6 1.3±.1 1.2±.1 2.2±.5 13.8±3.2 38.5±.3
5 14.4±.4 1.4±.05 1.3±.04 2.8±.3 25.3±5.2 50.0±1.1
Table 3: Operation microbenchmarks. Mean and standard deviation over 10 runs. 1D means one replica with commit wait disabled.
4.2.4 Refinements
tTM
safe as defined above has a weakness, in that a single
prepared transaction prevents tsafe from advancing. As
a result, no reads can occur at later timestamps, even
if the reads do not conflict with the transaction. Such
false conflicts can be removed by augmenting tTM
safe with
a fine-grained mapping from key ranges to prepared-
transaction timestamps. This information can be stored
in the lock table, which already maps key ranges to
lock metadata. When a read arrives, it only needs to be
checked against the fine-grained safe time for key ranges
with which the read conflicts.
LastTS() as defined above has a similar weakness: if
a transaction has just committed, a non-conflicting read-
only transaction must still be assigned sread so as to fol-
low that transaction. As a result, the execution of the read
could be delayed. This weakness can be remedied sim-
ilarly by augmenting LastTS() with a fine-grained map-
ping from key ranges to commit timestamps in the lock
table. (We have not yet implemented this optimization.)
When a read-only transaction arrives, its timestamp can
be assigned by taking the maximum value of LastTS()
for the key ranges with which the transaction conflicts,
unless there is a conflicting prepared transaction (which
can be determined from fine-grained safe time).
tPaxos
safe as defined above has a weakness in that it cannot
advance in the absence of Paxos writes. That is, a snap-
shot read att cannot execute at Paxos groups whose last
write happened beforet. Spanner addresses this problem
by taking advantage of the disjointness of leader-lease
intervals. Each Paxos leader advances tPaxos
safe by keeping
a threshold above which future writes’ timestamps will
occur: it maintains a mapping MinNextTS(n) from Paxos
sequence numbern to the minimum timestamp that may
be assigned to Paxos sequence number n + 1. A replica
can advancetPaxos
safe to MinNextTS(n)− 1 when it has ap-
plied throughn.
A single leader can enforce its MinNextTS()
promises easily. Because the timestamps promised
by MinNextTS() lie within a leader’s lease, the disjoint-
ness invariant enforces MinNextTS() promises across
leaders. If a leader wishes to advance MinNextTS()
beyond the end of its leader lease, it must first extend its
lease. Note that smax is always advanced to the highest
value in MinNextTS() to preserve disjointness.
A leader by default advances MinNextTS() values ev-
ery 8 seconds. Thus, in the absence of prepared trans-
actions, healthy slaves in an idle Paxos group can serve
reads at timestamps greater than 8 seconds old in the
worst case. A leader may also advance MinNextTS() val-
ues on demand from slaves.
5 Evaluation
We first measure Spanner’s performance with respect to
replication, transactions, and availability. We then pro-
vide some data on TrueTime behavior, and a case study
of our first client, F1.
5.1 Microbenchmarks
Table 3 presents some microbenchmarks for Spanner.
These measurements were taken on timeshared ma-
chines: each spanserver ran on scheduling units of 4GB
RAM and 4 cores (AMD Barcelona 2200MHz). Clients
were run on separate machines. Each zone contained one
spanserver. Clients and zones were placed in a set of dat-
acenters with network distance of less than 1ms. (Such a
layout should be commonplace: most applications do not
need to distribute all of their data worldwide.) The test
database was created with 50 Paxos groups with 2500 di-
rectories. Operations were standalone reads and writes of
4KB. All reads were served out of memory after a com-
paction, so that we are only measuring the overhead of
Spanner’s call stack. In addition, one unmeasured round
of reads was done first to warm any location caches.
For the latency experiments, clients issued sufficiently
few operations so as to avoid queuing at the servers.
From the 1-replica experiments, commit wait is about
5ms, and Paxos latency is about 9ms. As the number
of replicas increases, the latency stays roughly constant
with less standard deviation because Paxos executes in
parallel at a group’s replicas. As the number of replicas
increases, the latency to achieve a quorum becomes less
sensitive to slowness at one slave replica.
For the throughput experiments, clients issued suffi-
ciently many operations so as to saturate the servers’
Published in the Proceedings of OSDI 2012 9

latency (ms)
participants mean 99th percentile
1 17.0±1.4 75.0±34.9
2 24.5±2.5 87.6±35.9
5 31.5±6.2 104.5±52.2
10 30.0±3.7 95.6±25.4
25 35.5±5.6 100.4±42.7
50 42.7±4.1 93.7±22.9
100 71.4±7.6 131.2±17.6
200 150.5±11.0 320.3±35.1
Table 4: Two-phase commit scalability. Mean and standard
deviations over 10 runs.
CPUs. Snapshot reads can execute at any up-to-date
replicas, so their throughput increases almost linearly
with the number of replicas. Single-read read-only trans-
actions only execute at leaders because timestamp as-
signment must happen at leaders. Read-only-transaction
throughput increases with the number of replicas because
the number of effective spanservers increases: in the
experimental setup, the number of spanservers equaled
the number of replicas, and leaders were randomly dis-
tributed among the zones. Write throughput benefits
from the same experimental artifact (which explains the
increase in throughput from 3 to 5 replicas), but that ben-
efit is outweighed by the linear increase in the amount of
work performed per write, as the number of replicas in-
creases.
Table 4 demonstrates that two-phase commit can scale
to a reasonable number of participants: it summarizes
a set of experiments run across 3 zones, each with 25
spanservers. Scaling up to 50 participants is reasonable
in both mean and 99th-percentile, and latencies start to
rise noticeably at 100 participants.
5.2 Availability
Figure 5 illustrates the availability benefits of running
Spanner in multiple datacenters. It shows the results of
three experiments on throughput in the presence of dat-
acenter failure, all of which are overlaid onto the same
time scale. The test universe consisted of 5 zones Zi,
each of which had 25 spanservers. The test database was
sharded into 1250 Paxos groups, and 100 test clients con-
stantly issued non-snapshot reads at an aggregrate rate
of 50K reads/second. All of the leaders were explic-
itly placed in Z1. Five seconds into each test, all of
the servers in one zone were killed: non-leader killsZ2;
leader-hard killsZ1; leader-soft killsZ1, but it gives no-
tifications to all of the servers that they should handoff
leadership first.
Killing Z2 has no effect on read throughput. Killing
Z1 while giving the leaders time to handoff leadership to
0 5 10 15 20
Time in seconds
200K
400K
600K
800K
1M
1.2M
1.4MCumulative reads completed
non-leader
leader-soft
leader-hard
Figure 5: Effect of killing servers on throughput.
a different zone has a minor effect: the throughput drop
is not visible in the graph, but is around 3-4%. On the
other hand, killing Z1 with no warning has a severe ef-
fect: the rate of completion drops almost to 0. As leaders
get re-elected, though, the throughput of the system rises
to approximately 100K reads/second because of two ar-
tifacts of our experiment: there is extra capacity in the
system, and operations are queued while the leader is un-
available. As a result, the throughput of the system rises
before leveling off again at its steady-state rate.
We can also see the effect of the fact that Paxos leader
leases are set to 10 seconds. When we kill the zone,
the leader-lease expiration times for the groups should
be evenly distributed over the next 10 seconds. Soon af-
ter each lease from a dead leader expires, a new leader is
elected. Approximately 10 seconds after the kill time, all
of the groups have leaders and throughput has recovered.
Shorter lease times would reduce the effect of server
deaths on availability, but would require greater amounts
of lease-renewal network traffic. We are in the process of
designing and implementing a mechanism that will cause
slaves to release Paxos leader leases upon leader failure.
5.3 TrueTime
Two questions must be answered with respect to True-
Time: is ϵ truly a bound on clock uncertainty, and how
bad does ϵ get? For the former, the most serious prob-
lem would be if a local clock’s drift were greater than
200us/sec: that would break assumptions made by True-
Time. Our machine statistics show that bad CPUs are 6
times more likely than bad clocks. That is, clock issues
are extremely infrequent, relative to much more serious
hardware problems. As a result, we believe that True-
Time’s implementation is as trustworthy as any other
piece of software upon which Spanner depends.
Figure 6 presents TrueTime data taken at several thou-
sand spanserver machines across datacenters up to 2200
Published in the Proceedings of OSDI 2012 10

Mar 29 Mar 30 Mar 31 Apr 1
Date
2
4
6
8
10
Epsilon (ms)
99.9
99
90
6AM 8AM 10AM 12PM
Date (April 13)
1
2
3
4
5
6
Figure 6: Distribution of TrueTime ϵ values, sampled right
after timeslave daemon polls the time masters. 90th, 99th, and
99.9th percentiles are graphed.
km apart. It plots the 90th, 99th, and 99.9th percentiles
of ϵ, sampled at timeslave daemons immediately after
polling the time masters. This sampling elides the saw-
tooth in ϵ due to local-clock uncertainty, and therefore
measures time-master uncertainty (which is generally 0)
plus communication delay to the time masters.
The data shows that these two factors in determining
the base value of ϵ are generally not a problem. How-
ever, there can be significant tail-latency issues that cause
higher values ofϵ. The reduction in tail latencies begin-
ning on March 30 were due to networking improvements
that reduced transient network-link congestion. The in-
crease inϵ on April 13, approximately one hour in dura-
tion, resulted from the shutdown of 2 time masters at a
datacenter for routine maintenance. We continue to in-
vestigate and remove causes of TrueTime spikes.
5.4 F1
Spanner started being experimentally evaluated under
production workloads in early 2011, as part of a rewrite
of Google’s advertising backend called F1 [35]. This
backend was originally based on a MySQL database that
was manually sharded many ways. The uncompressed
dataset is tens of terabytes, which is small compared to
many NoSQL instances, but was large enough to cause
difficulties with sharded MySQL. The MySQL sharding
scheme assigned each customer and all related data to a
fixed shard. This layout enabled the use of indexes and
complex query processing on a per-customer basis, but
required some knowledge of the sharding in application
business logic. Resharding this revenue-critical database
as it grew in the number of customers and their data was
extremely costly. The last resharding took over two years
of intense effort, and involved coordination and testing
across dozens of teams to minimize risk. This operation
was too complex to do regularly: as a result, the team had
to limit growth on the MySQL database by storing some
# fragments # directories
1 >100M
2–4 341
5–9 5336
10–14 232
15–99 34
100–500 7
Table 5: Distribution of directory-fragment counts in F1.
data in external Bigtables, which compromised transac-
tional behavior and the ability to query across all data.
The F1 team chose to use Spanner for several rea-
sons. First, Spanner removes the need to manually re-
shard. Second, Spanner provides synchronous replica-
tion and automatic failover. With MySQL master-slave
replication, failover was difficult, and risked data loss
and downtime. Third, F1 requires strong transactional
semantics, which made using other NoSQL systems im-
practical. Application semantics requires transactions
across arbitrary data, and consistent reads. The F1 team
also needed secondary indexes on their data (since Span-
ner does not yet provide automatic support for secondary
indexes), and was able to implement their own consistent
global indexes using Spanner transactions.
All application writes are now by default sent through
F1 to Spanner, instead of the MySQL-based application
stack. F1 has 2 replicas on the west coast of the US, and
3 on the east coast. This choice of replica sites was made
to cope with outages due to potential major natural disas-
ters, and also the choice of their frontend sites. Anecdo-
tally, Spanner’s automatic failover has been nearly invisi-
ble to them. Although there have been unplanned cluster
failures in the last few months, the most that the F1 team
has had to do is update their database’s schema to tell
Spanner where to preferentially place Paxos leaders, so
as to keep them close to where their frontends moved.
Spanner’s timestamp semantics made it efficient for
F1 to maintain in-memory data structures computed from
the database state. F1 maintains a logical history log of
all changes, which is written into Spanner itself as part
of every transaction. F1 takes full snapshots of data at a
timestamp to initialize its data structures, and then reads
incremental changes to update them.
Table 5 illustrates the distribution of the number of
fragments per directory in F1. Each directory typically
corresponds to a customer in the application stack above
F1. The vast majority of directories (and therefore cus-
tomers) consist of only 1 fragment, which means that
reads and writes to those customers’ data are guaranteed
to occur on only a single server. The directories with
more than 100 fragments are all tables that contain F1
secondary indexes: writes to more than a few fragments
Published in the Proceedings of OSDI 2012 11

latency (ms)
operation mean std dev count
all reads 8.7 376.4 21.5B
single-site commit 72.3 112.8 31.2M
multi-site commit 103.0 52.2 32.1M
Table 6: F1-perceived operation latencies measured over the
course of 24 hours.
of such tables are extremely uncommon. The F1 team
has only seen such behavior when they do untuned bulk
data loads as transactions.
Table 6 presents Spanner operation latencies as mea-
sured from F1 servers. Replicas in the east-coast data
centers are given higher priority in choosing Paxos lead-
ers. The data in the table is measured from F1 servers
in those data centers. The large standard deviation in
write latencies is caused by a pretty fat tail due to lock
conflicts. The even larger standard deviation in read la-
tencies is partially due to the fact that Paxos leaders are
spread across two data centers, only one of which has
machines with SSDs. In addition, the measurement in-
cludes every read in the system from two datacenters:
the mean and standard deviation of the bytes read were
roughly 1.6KB and 119KB, respectively.
6 Related Work
Consistent replication across datacenters as a storage
service has been provided by Megastore [5] and Dy-
namoDB [3]. DynamoDB presents a key-value interface,
and only replicates within a region. Spanner follows
Megastore in providing a semi-relational data model,
and even a similar schema language. Megastore does
not achieve high performance. It is layered on top of
Bigtable, which imposes high communication costs. It
also does not support long-lived leaders: multiple repli-
cas may initiate writes. All writes from different repli-
cas necessarily conflict in the Paxos protocol, even if
they do not logically conflict: throughput collapses on
a Paxos group at several writes per second. Spanner pro-
vides higher performance, general-purpose transactions,
and external consistency.
Pavlo et al. [31] have compared the performance of
databases and MapReduce [12]. They point to several
other efforts that have been made to explore database
functionality layered on distributed key-value stores [1,
4, 7, 41] as evidence that the two worlds are converging.
We agree with the conclusion, but demonstrate that in-
tegrating multiple layers has its advantages: integrating
concurrency control with replication reduces the cost of
commit wait in Spanner, for example.
The notion of layering transactions on top of a repli-
cated store dates at least as far back as Gifford’s disser-
tation [16]. Scatter [17] is a recent DHT-based key-value
store that layers transactions on top of consistent repli-
cation. Spanner focuses on providing a higher-level in-
terface than Scatter does. Gray and Lamport [18] de-
scribe a non-blocking commit protocol based on Paxos.
Their protocol incurs more messaging costs than two-
phase commit, which would aggravate the cost of com-
mit over widely distributed groups. Walter [36] provides
a variant of snapshot isolation that works within, but not
across datacenters. In contrast, our read-only transac-
tions provide a more natural semantics, because we sup-
port external consistency over all operations.
There has been a spate of recent work on reducing
or eliminating locking overheads. Calvin [40] elimi-
nates concurrency control: it pre-assigns timestamps and
then executes the transactions in timestamp order. H-
Store [39] and Granola [11] each supported their own
classification of transaction types, some of which could
avoid locking. None of these systems provides external
consistency. Spanner addresses the contention issue by
providing support for snapshot isolation.
V oltDB [42] is a sharded in-memory database that
supports master-slave replication over the wide area for
disaster recovery, but not more general replication con-
figurations. It is an example of what has been called
NewSQL, which is a marketplace push to support scal-
able SQL [38]. A number of commercial databases im-
plement reads in the past, such as MarkLogic [26] and
Oracle’s Total Recall [30]. Lomet and Li [24] describe an
implementation strategy for such a temporal database.
Farsite derived bounds on clock uncertainty (much
looser than TrueTime’s) relative to a trusted clock refer-
ence [13]: server leases in Farsite were maintained in the
same way that Spanner maintains Paxos leases. Loosely
synchronized clocks have been used for concurrency-
control purposes in prior work [2, 23]. We have shown
that TrueTime lets one reason about global time across
sets of Paxos state machines.
7 Future Work
We have spent most of the last year working with the
F1 team to transition Google’s advertising backend from
MySQL to Spanner. We are actively improving its mon-
itoring and support tools, as well as tuning its perfor-
mance. In addition, we have been working on improving
the functionality and performance of our backup/restore
system. We are currently implementing the Spanner
schema language, automatic maintenance of secondary
indices, and automatic load-based resharding. Longer
term, there are a couple of features that we plan to in-
Published in the Proceedings of OSDI 2012 12

vestigate. Optimistically doing reads in parallel may be
a valuable strategy to pursue, but initial experiments have
indicated that the right implementation is non-trivial. In
addition, we plan to eventually support direct changes of
Paxos configurations [22, 34].
Given that we expect many applications to replicate
their data across datacenters that are relatively close to
each other, TrueTime ϵ may noticeably affect perfor-
mance. We see no insurmountable obstacle to reduc-
ing ϵ below 1ms. Time-master-query intervals can be
reduced, and better clock crystals are relatively cheap.
Time-master query latency could be reduced with im-
proved networking technology, or possibly even avoided
through alternate time-distribution technology.
Finally, there are obvious areas for improvement. Al-
though Spanner is scalable in the number of nodes, the
node-local data structures have relatively poor perfor-
mance on complex SQL queries, because they were de-
signed for simple key-value accesses. Algorithms and
data structures from DB literature could improve single-
node performance a great deal. Second, moving data au-
tomatically between datacenters in response to changes
in client load has long been a goal of ours, but to make
that goal effective, we would also need the ability to
move client-application processes between datacenters in
an automated, coordinated fashion. Moving processes
raises the even more difficult problem of managing re-
source acquisition and allocation between datacenters.
8 Conclusions
To summarize, Spanner combines and extends on ideas
from two research communities: from the database com-
munity, a familiar, easy-to-use, semi-relational interface,
transactions, and an SQL-based query language; from
the systems community, scalability, automatic sharding,
fault tolerance, consistent replication, external consis-
tency, and wide-area distribution. Since Spanner’s in-
ception, we have taken more than 5 years to iterate to the
current design and implementation. Part of this long it-
eration phase was due to a slow realization that Spanner
should do more than tackle the problem of a globally-
replicated namespace, and should also focus on database
features that Bigtable was missing.
One aspect of our design stands out: the linchpin of
Spanner’s feature set is TrueTime. We have shown that
reifying clock uncertainty in the time API makes it possi-
ble to build distributed systems with much stronger time
semantics. In addition, as the underlying system en-
forces tighter bounds on clock uncertainty, the overhead
of the stronger semantics decreases. As a community, we
should no longer depend on loosely synchronized clocks
and weak time APIs in designing distributed algorithms.
Acknowledgements
Many people have helped to improve this paper: our
shepherd Jon Howell, who went above and beyond
his responsibilities; the anonymous referees; and many
Googlers: Atul Adya, Fay Chang, Frank Dabek, Sean
Dorward, Bob Gruber, David Held, Nick Kline, Alex
Thomson, and Joel Wein. Our management has been
very supportive of both our work and of publishing this
paper: Aristotle Balogh, Bill Coughran, Urs H ¨olzle,
Doron Meyer, Cos Nicolaou, Kathy Polizzi, Sridhar Ra-
maswany, and Shivakumar Venkataraman.
We have built upon the work of the Bigtable and
Megastore teams. The F1 team, and Jeff Shute in particu-
lar, worked closely with us in developing our data model
and helped immensely in tracking down performance and
correctness bugs. The Platforms team, and Luiz Barroso
and Bob Felderman in particular, helped to make True-
Time happen. Finally, a lot of Googlers used to be on our
team: Ken Ashcraft, Paul Cychosz, Krzysztof Ostrowski,
Amir V oskoboynik, Matthew Weaver, Theo Vassilakis,
and Eric Veach; or have joined our team recently: Nathan
Bales, Adam Beberg, Vadim Borisov, Ken Chen, Brian
Cooper, Cian Cullinan, Robert-Jan Huijsman, Milind
Joshi, Andrey Khorlin, Dawid Kuroczko, Laramie Leav-
itt, Eric Li, Mike Mammarella, Sunil Mushran, Simon
Nielsen, Ovidiu Platon, Ananth Shrinivas, Vadim Su-
vorov, and Marcel van der Holst.
References
[1] Azza Abouzeid et al. “HadoopDB: an architectural hybrid of
MapReduce and DBMS technologies for analytical workloads”.
Proc. of VLDB. 2009, pp. 922–933.
[2] A. Adya et al. “Efficient optimistic concurrency control using
loosely synchronized clocks”. Proc. of SIGMOD. 1995, pp. 23–
34.
[3] Amazon. Amazon DynamoDB. 2012.
[4] Michael Armbrust et al. “PIQL: Success-Tolerant Query Pro-
cessing in the Cloud”. Proc. of VLDB. 2011, pp. 181–192.
[5] Jason Baker et al. “Megastore: Providing Scalable, Highly
Available Storage for Interactive Services”. Proc. of CIDR .
2011, pp. 223–234.
[6] Hal Berenson et al. “A critique of ANSI SQL isolation levels”.
Proc. of SIGMOD. 1995, pp. 1–10.
[7] Matthias Brantner et al. “Building a database on S3”. Proc. of
SIGMOD. 2008, pp. 251–264.
[8] A. Chan and R. Gray. “Implementing Distributed Read-Only
Transactions”. IEEE TOSE SE-11.2 (Feb. 1985), pp. 205–212.
[9] Fay Chang et al. “Bigtable: A Distributed Storage System for
Structured Data”. ACM TOCS 26.2 (June 2008), 4:1–4:26.
[10] Brian F. Cooper et al. “PNUTS: Yahoo!’s hosted data serving
platform”. Proc. of VLDB. 2008, pp. 1277–1288.
[11] James Cowling and Barbara Liskov. “Granola: Low-Overhead
Distributed Transaction Coordination”. Proc. of USENIX ATC .
2012, pp. 223–236.
Published in the Proceedings of OSDI 2012 13

[12] Jeffrey Dean and Sanjay Ghemawat. “MapReduce: a flexible
data processing tool”. CACM 53.1 (Jan. 2010), pp. 72–77.
[13] John Douceur and Jon Howell. Scalable Byzantine-Fault-
Quantifying Clock Synchronization. Tech. rep. MSR-TR-2003-
67. MS Research, 2003.
[14] John R. Douceur and Jon Howell. “Distributed directory service
in the Farsite file system”. Proc. of OSDI. 2006, pp. 321–334.
[15] Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. “The
Google file system”. Proc. of SOSP. Dec. 2003, pp. 29–43.
[16] David K. Gifford. Information Storage in a Decentralized Com-
puter System . Tech. rep. CSL-81-8. PhD dissertation. Xerox
PARC, July 1982.
[17] Lisa Glendenning et al. “Scalable consistency in Scatter”. Proc.
of SOSP. 2011.
[18] Jim Gray and Leslie Lamport. “Consensus on transaction com-
mit”. ACM TODS 31.1 (Mar. 2006), pp. 133–160.
[19] Pat Helland. “Life beyond Distributed Transactions: an Apos-
tate’s Opinion”. Proc. of CIDR. 2007, pp. 132–141.
[20] Maurice P. Herlihy and Jeannette M. Wing. “Linearizability: a
correctness condition for concurrent objects”. ACM TOPLAS
12.3 (July 1990), pp. 463–492.
[21] Leslie Lamport. “The part-time parliament”. ACM TOCS 16.2
(May 1998), pp. 133–169.
[22] Leslie Lamport, Dahlia Malkhi, and Lidong Zhou. “Reconfigur-
ing a state machine”. SIGACT News 41.1 (Mar. 2010), pp. 63–
73.
[23] Barbara Liskov. “Practical uses of synchronized clocks in dis-
tributed systems”. Distrib. Comput. 6.4 (July 1993), pp. 211–
219.
[24] David B. Lomet and Feifei Li. “Improving Transaction-Time
DBMS Performance and Functionality”. Proc. of ICDE (2009),
pp. 581–591.
[25] Jacob R. Lorch et al. “The SMART way to migrate replicated
stateful services”. Proc. of EuroSys. 2006, pp. 103–115.
[26] MarkLogic. MarkLogic 5 Product Documentation. 2012.
[27] Keith Marzullo and Susan Owicki. “Maintaining the time in a
distributed system”. Proc. of PODC. 1983, pp. 295–305.
[28] Sergey Melnik et al. “Dremel: Interactive Analysis of Web-
Scale Datasets”. Proc. of VLDB. 2010, pp. 330–339.
[29] D.L. Mills. Time synchronization in DCNET hosts . Internet
Project Report IEN–173. COMSAT Laboratories, Feb. 1981.
[30] Oracle. Oracle Total Recall. 2012.
[31] Andrew Pavlo et al. “A comparison of approaches to large-scale
data analysis”. Proc. of SIGMOD. 2009, pp. 165–178.
[32] Daniel Peng and Frank Dabek. “Large-scale incremental pro-
cessing using distributed transactions and notifications”. Proc.
of OSDI. 2010, pp. 1–15.
[33] Daniel J. Rosenkrantz, Richard E. Stearns, and Philip M. Lewis
II. “System level concurrency control for distributed database
systems”. ACM TODS 3.2 (June 1978), pp. 178–198.
[34] Alexander Shraer et al. “Dynamic Reconfiguration of Pri-
mary/Backup Clusters”. Proc. of USENIX ATC. 2012, pp. 425–
438.
[35] Jeff Shute et al. “F1 — The Fault-Tolerant Distributed RDBMS
Supporting Google’s Ad Business”. Proc. of SIGMOD . May
2012, pp. 777–778.
[36] Yair Sovran et al. “Transactional storage for geo-replicated sys-
tems”. Proc. of SOSP. 2011, pp. 385–400.
[37] Michael Stonebraker. Why Enterprises Are Uninterested in
NoSQL. 2010.
[38] Michael Stonebraker. Six SQL Urban Myths . 2010.
[39] Michael Stonebraker et al. “The end of an architectural era: (it’s
time for a complete rewrite)”. Proc. of VLDB. 2007, pp. 1150–
1160.
[40] Alexander Thomson et al. “Calvin: Fast Distributed Transac-
tions for Partitioned Database Systems”. Proc. of SIGMOD .
2012, pp. 1–12.
[41] Ashish Thusoo et al. “Hive — A Petabyte Scale Data Ware-
house Using Hadoop”. Proc. of ICDE. 2010, pp. 996–1005.
[42] V oltDB. V oltDB Resources. 2012.
A Paxos Leader-Lease Management
The simplest means to ensure the disjointness of Paxos-
leader-lease intervals would be for a leader to issue a syn-
chronous Paxos write of the lease interval, whenever it
would be extended. A subsequent leader would read the
interval and wait until that interval has passed.
TrueTime can be used to ensure disjointness without
these extra log writes. The potential ith leader keeps a
lower bound on the start of a lease vote from replicar as
vleader
i,r = TT.now().earliest, computed before esend
i,r (de-
fined as when the lease request is sent by the leader).
Each replica r grants a lease at lease egrant
i,r , which hap-
pens after ereceive
i,r (when the replica receives a lease re-
quest); the lease ends at tend
i,r = TT.now().latest + 10,
computed after ereceive
i,r . A replica r obeys the single-
vote rule: it will not grant another lease vote until
TT.after(tend
i,r ) is true. To enforce this rule across different
incarnations ofr, Spanner logs a lease vote at the grant-
ing replica before granting the lease; this log write can
be piggybacked upon existing Paxos-protocol log writes.
When the ith leader receives a quorum of votes
(event equorum
i ), it computes its lease interval as
leasei = [ TT.now().latest, minr(vleader
i,r ) + 10] . The
lease is deemed to have expired at the leader when
TT.before(minr(vleader
i,r ) + 10) is false. To prove disjoint-
ness, we make use of the fact that the ith and (i + 1)th
leaders must have one replica in common in their quo-
rums. Call that replicar0. Proof:
leasei.end = minr(vleader
i,r ) + 10 (by definition)
minr(vleader
i,r ) + 10≤vleader
i,r0 + 10 (min)
vleader
i,r0 + 10≤tabs(esend
i,r0) + 10 (by definition)
tabs(esend
i,r0) + 10≤tabs(ereceive
i,r0 ) + 10 (causality)
tabs(ereceive
i,r0 ) + 10≤tend
i,r0 (by definition)
tend
i,r0 <t abs(egrant
i+1,r0) (single-vote)
tabs(egrant
i+1,r0)≤tabs(equorum
i+1 ) (causality)
tabs(equorum
i+1 )≤ leasei+1.start (by definition)
Published in the Proceedings of OSDI 2012 14

FAQ Secure Untrusted Data Repository (SUNDR) (2004)

Q: SUNDR provides a means to verify whether or not files were tampered
with. But does it hide the actual files themselves from a malicious
server / do any sort of encryption on them?

A: No. SUNDR provides integrity, but not confidentiality. Users could
encrypt file content for confidentiality.

Q: How slow/fast is the computation of an i-handle in a fairly large
file system?

A: Computing SHA-1s is quite fast (~10 cycles per byte). Furthermore, if
you change one block, you just recompute the hash on that one block, and then
hash all the blocks to get you back up to the root. The signing part is
the expensive part. The sha overhead is definitely not neglible,
however, and the paper proposes to avoid the cost of recomputing hash
trees over several operations by allowing an i-handle to store the
hash and small log of changes that have been made to the i-table. This
also speeds up the checking for clients that have checked a recent
version; they can just apply the several operations in the log.

Q: In Serialized SUNDR, why does each user have a separate i-table?
Why not a single i-table, since it's a single file-system?

A: One of SUNDR's goals is to enforce permissions on users. This means that
a client (a user's workstation) can't be allowed to modify arbitrary
parts of the file-system -- a user should only be allowed to modify
their own files. And SUNDR would like to enforce this restriction even
if the server is malicious and is conspiring with a malicious user. This
rules out a single data-structure (such as a single i-table) with a
single signed root hash, since presumably all users would have to have
the private key required to update the signed root hash.

So SUNDR gives each user their own file-system tree, each with a root
hash signed by the owning user's private key. When a user looks at the
file system, the SUNDR client effectively merges the different users'
trees.

It seems likely that each file is owned by a particular user, and that
a file can only be written by its owner.

Q: What are fork and fetch-modify consistency?

A: A fork is an attack in which a malicious server presents different file
system contents to different users. Such attacks are possible in SUNDR.
For example, if user X updates file F, and later Y reads F, but the
server shows Y the old content of F (from before X updated it), that's a
fork attack.

Fork consistency means that, if the server ever conceals any operation
of user X from user Y, the server cannot show X and Y any of each
other's subsequent operations. Once the server has concealed an
operation, it must present two different version of the file system,
one reflecting just X's operations, and the other just Y's.

Fetch-modify consistency is what you get when the server is acting
correctly: every user sees the effects of every other user's operations.

Q: What does the paper mean when it says that users can detect a fork
if they communicate?

A: By communication the authors mean any out-of-band communication
between users (e.g., email, chat, etc.).  For example, if user A tells
user B please look at file "x", and B doesn't see "x", then they know
the server forked the file system (or user A is lying to user B).
Other than out-of-band communication, the authors also suggest using a
time-stamp box.  In bitcoin, there is yet another method to resolve
forks, which we will discuss next week.

Q: What are the current applications of SUNDR in products or internal
systems?

A: As far as I know, none.  But the ideas are powerful and you see them in
other decentralized systems such as git, ledgers, etc.  The one
commercial system that I know off that is directly influenced by SUNDR
is keybase (which was acquired by zoom).

Q: In what scenarios would you choose to store data on untrusted servers?

A: You would not store your data on a server you knew to be already
corrupt, but you might use a server you assumed was trustworthy but
later was compromised. The attacker may change files on your server
and now you have a problem, because recovering from such an attack is
difficult. If this server stores something widely relied on like the
source of Debian Linux, as in the attack mentioned in the paper, then
the problem is very serious. Similar attacks happen: Canonical was
compromised in 2019.

Q: Why is fork consistency good? It seems to allow forks -- aren't
forks bad?

A: Indeed, a fork is not at all desirable. However, with ordinary
assumptions, it seems to be impossible to prevent malicious servers
from forking their clients -- from hiding some updates from some
clients. So we're forced to live with the possibility of forks.

What SUNDR does is limit how much freedom the server has to pick and
choose which operations it hides and reveals. In particular, SUNDR
restricts the server so that it can only fork -- it cannot join. This
means that the server can reveal all of X's updates to Y before a
certain point (the fork), and hide all of X's updates from Y after
that point, but the server cannot hide one of X's updates but reveal a
later one of X's updates. Once the server has forked two clients, the
server can never allow the clients to see subsequent updates from each
other without revealing that the server forked them.

The larger point is that the consequences of a fork are likely to be
pretty obvious (cooperating users no longer see each other's updates),
and thus people are likely to fairly quickly realize they've been
forked. Then they will stop trusting their storage service, and switch
to a different and hopefully more trustworthy service.

Q: If a client recognizes a fork attack, what should it do?

A: I don't think there's any very good answer to this. Certainly you
would want to switch to a different storage service provider. And you
would have to inspect the various forks and merge (or thoughtfully
discard) any conflicting writes. What this means depends on the
applications and file formats involved; there's no general technique.
Most troubling, you'd have to consider the bad effects that the fork
(and hiding of user updates) might have on the outside world -- on any
decisions that users might have made based on forked (incorrect) data.

Q: The paper says "Fork consistency is the strongest notion of
integrity possible without online trusted parties." Why is that?

A: It seems like a reasonable claim if you assume that the clients are
often off-line, and can't directly communicate. If only one client is
online at a time, or clients can only talk to the SUNDR server, there
doesn't seem to be much a client can do to detect that the server is
hiding other clients' recent updates.

But if clients are on-line all the time, and can directly communicate
over the Internet, the situation may be different. Clients can
directly exchange messages alerting each other about recent updates,
or recent hashes of data, or log entries, or even file data itself.
Which would seem to eliminate the possibility of successful fork
attacks by the SUNDR server. But perhaps in such a design the
Internet, or the clients themselves, serve as the online trusted
party, which would make the original claim technically correct. And
one might find that a malicious Internet (or time synchronization
service) could execute fork attacks.

A larger point of the paper is that once you have fork consistency,
it's relatively easy to add external measures in order to achieve
something close to linearizability. This is what Section 3.2 is about.

Go FAQ

Q: Why does 6.5840 use Go for the labs?

A: Until a few years ago 6.5840 used C++, which worked well. Go works a
little better for 6.5840 labs for a couple of reasons. Go is garbage
collected and type-safe, which eliminates some common classes of bugs.
Go has good support for threads (goroutines), and a nice RPC package,
which are directly useful in 6.5840. Threads and garbage collection
work particularly well together, since garbage collection can
eliminate programmer effort to decide when the last thread using an
object has stopped using it. There are other languages with these
features that would probably work fine for 6.5840 labs, such as Java.

Q: are there any tips/tricks for building an intuition of how to build
effective Go code?

A: Get experience by writing Go code and reading other's people go
code.  This page has many useful tips: https://go.dev/doc/effective_go

Q: Do goroutines run in parallel? Can you use them to increase
performance?

A: Go's goroutines are the same as threads in other languages. The Go
runtime executes goroutines on all available cores, in parallel. If
there are fewer cores than runnable goroutines, the runtime will
pre-emptively time-share the cores among goroutines.

Q: How do Go channels work? How does Go make sure they are
synchronized between the many possible goroutines?

A: You can see the source at https://golang.org/src/runtime/chan.go,
though it is not easy to follow.

At a high level, a chan is a struct holding a buffer and a lock.
Sending on a channel involves acquiring the lock, waiting (perhaps
releasing the CPU) until some thread is receiving, and handing off the
message. Receiving involves acquiring the lock and waiting for a
sender. You could implement your own channels with Go sync.Mutex and
sync.Cond.

Q: I'm using a channel to wake up another goroutine, by sending a
dummy bool on the channel. But if that other goroutine is already
running (and thus not receiving on the channel), the sending goroutine
blocks. What should I do?

A: Try condition variables (Go's sync.Cond) rather than channels.
Condition variables work well to alert goroutines that may (or may
not) be waiting for something. Channels, because they are synchronous,
are awkward if you're not sure if there will be a goroutine waiting at
the other end of the channel.

Q: How can I have a goroutine wait for input from any one of a number
of different channels? Trying to receive on any one channel blocks if
there's nothing to read, preventing the goroutine from checking other
channels.

A: Try creating a separate goroutine for each channel, and have each
goroutine block on its channel. That's not always possible, but when
it works it's often the simplest approach.

Otherwise try Go's select.

Q: When should we use sync.WaitGroup instead of channels? and vice versa?

A: WaitGroup is fairly special-purpose; it's only useful when waiting
for a bunch of activities to complete. Channels are more
general-purpose; for example, you can communicate values over
channels. You can wait for multiple goroutines using channels, though it
takes a few more lines of code than with WaitGroup.

Q: I need my code to perform a task once per second. What's the
easiest way to do that?

A: Create a goroutine dedicated to that periodic task. It should have
a loop that uses time.Sleep() to pause for a second, and then do the
task, and then loop around to the time.Sleep().

Q: How do we know when the overhead of spawning goroutines exceeds
the concurrency we gain from them?

A: It depends! If your machine has 16 cores, and you are looking for
CPU parallelism, you should have roughly 16 executable goroutines. If
it takes 0.1 second of real time to fetch a web page, and your network
is capable of transmitting 100 web pages per second, you probably need
about 10 goroutines concurrently fetching in order to use all of the
network capacity. Experimentally, as you increase the number of
goroutines, for a while you'll see increased throughput, and then
you'll stop getting more throughput; at that point you have enough
goroutines from the point of view of performance.

Q: How would one create a Go channel that connects over the Internet?
How would one specify the protocol to use to send messages?

A: A Go channel only works within a single program; channels cannot be
used to talk to other programs or other computers.

Have a look at Go's RPC package, which lets you talk to other Go
programs over the Internet:

  https://golang.org/pkg/net/rpc/

Q: What are some important/useful Go-specific concurrency patterns to know?

A: Here's a slide deck on this topic, from a Go expert:

https://talks.golang.org/2012/concurrency.slide

Q: How are slices implemented?

A: A slice is an object that contains a pointer to an array and a start and
end index into that array. This arrangement allows multiple slices to
share an underlying array, with each slice perhaps exposing a different
range of array elements.

Here's a more extended discussion:

  https://blog.golang.org/go-slices-usage-and-internals

I use slices often, and arrays never. A Go slice is more flexible than
a Go array since an array's size is part of its type, whereas a
function that takes a slice as argument can take a slice of any
length.

Q: What are common debugging tools people use for Go?

A: fmt.Printf()

As far as I know there's not a great debugger for Go, though gdb can be
made to work:

https://golang.org/doc/gdb

In any case, for most bugs I've found fmt.Printf() to be an extremely
effective debugging tool.

Q: When is it right to use a synchronous RPC call and when is it right to
use an asynchronous RPC call?

A: Most code needs the RPC reply before it can proceed; in that case it
makes sense to use synchronous RPC.

But sometimes a client wants to launch many concurrent RPCs; in that
case async may be better. Or the client wants to do other work while it
waits for the RPC to complete, perhaps because the server is far away
(so speed-of-light time is high) or because the server might not be
reachable so that the RPC suffers a long timeout period.

I have never used async RPC in Go. When I want to send an RPC but not
have to wait for the result, I create a goroutine, and have the
goroutine make a synchronous Call().

Q: Is Go used in industry?

A: Yes. You can see an estimate of how much different programming
languages are used here:

https://www.tiobe.com/tiobe-index/

Q: What are common problems that developers face when starting with Go?

A:  Here are a few:

- Not protecting maps with locks when there is concurrent access. Use
  Go's race detector!

- Deadlocks with channels.

- Not capturing a variable when creating a goroutine.

- Leaking goroutines.

Q: Does Go support inheritance? (In the Java/C++ kind of "extends" way?)

A: Go doesn't support C++ style inheritance but has generics,
interfaces, and embedded structs, which allow you to do many things
for which you would use inheritance in C++.

Q: The thing I found most confusing about the Go tutorial was that
goroutines don't continue executing after the main thread has
completed. I don't think this was mentioned explicitly anywhere in the
tutorial; I figured it out through debuging the crawler exercise.

A: Yes, I don't think it is in the tutorial, but the language spec is
explicit about this: https://golang.org/ref/spec (see Program
execution).

Q: I'm still a little confused about when to choose value or pointer
receivers. Can you provide any concrete/real-world examples of when we
would choose one over the other?

A: When you want to modify the state of the receiver, you have to use
pointer receivers.  If the struct is very big, you probably want to
use a pointer receiver because value receivers operate on a copy.  If
neither applies, you can use a value receiver.  However, be careful
with value receivers; e.g., if you have a mutex in a struct, you
cannot make it a value receiver, because the mutex would be copied,
defeating its purpose.

ZooKeeper FAQ

Q: What's the main take-away from this paper?

A: ZooKeeper's main academic contribution lies in the detailed design
of a storage system specialized to fault-tolerant high-performance
configuration management: watches, sessions, the choice of
consistency, the specific semantics of the operations. It builds on
existing work such as Chubby and Paxos.

A lot of what's interesting for us is the idea that one can obtain
fault tolerance by keeping the critical state in fault-tolerant
storage (ZooKeeper), and running the computation in non-fault-tolerant
servers. For example, a MapReduce coordinator might keep state about
jobs, task status, workers, location of intermediate output, &c, in
ZooKeeper. If the coordinator fails, a new computer can be selected to
run the MapReduce coordinator software, and it can load its state from
ZooKeeper. This provides fault-tolerance for the coordinator without
the complexity of state-machine replication (e.g. without having to
write the MapReduce coordinator using a Raft library). You can think
of this as providing fault-tolerance by making the state alone
fault-tolerant, whereas use of Raft makes the entire computation
fault-tolerant. This general pattern isn't new -- for example it's how
database-backed web sites work -- but ZooKeeper is a good fit if your
main concern is managing fault-tolerant services.

Q: What's the point of sessions?

A: A session consists of some state maintained by the client and
ZooKeeper. The client tags each request with its session ID. If
ZooKeeper doesn't hear from a client for a while because the client
has failed or there's a network problem, the ZooKeeper leader will
expire (destroy) the session.

One role of sessions is to manage the state required to guarantee FIFO
client order, and to keep track of each client's watches.

More importantly, sessions are involved in the way ephemeral znodes
work. When a client creates an ephemeral znode, ZooKeeper remembers
which client session created the znode. If the ZooKeeper leader expires
a session, then it will also delete all ephemeral znodes created by
that session. In addition, and atomically with deleting the ephemeral
znodes, ZooKeeper guarantees to ignore any further client requests from
the expired session.

This arrangement works particularly well when applications implement
elections with ZooKeeper, e.g. to elect a GFS or MapReduce
coordinator. A typical election scheme: the candidates all try to
create the same ephemeral znode; ZooKeeper allows only one of the
creates to succeed (create is "exclusive"); the candidate whose create
succeeded is the winner. Use of an ephemeral znode means that if
ZooKeeper decides the winner has failed, ZooKeeper will automatically
expire its session and delete the znode, so that there can be a new
election. The ephemeral znode and session act as a lease.

But what if the original winner is actually alive, and continues to
act as e.g. GFS coordinator even after ZooKeeper deletes its ephemeral
znode and a new election is held? Because ZooKeeper starts ignoring a
session's requests at the moment ZooKeeper expires the session and
deletes the ephemeral znode, the original winner will automatically be
prevented from changing anything stored in ZooKeeper. Assuming the
application store all state in ZooKeeper, this prevents the deposed
winner from interfering with the new winner. This idea is sometimes
called "fencing."

Q: The paper's Section 2.4 describes locks. What if one used these
locks to protect a group of updates to a set of znodes, and the lock
holder crashed while only halfway through those updates? How could the
application recover?

A: One part of the answer is that ZooKeeper's ephemeral znode
mechanism would cause the lock file to disappear, so that another
client could acquire the lock.

The new lock-holder would be faced with partially-updated data, and
would need a strategy for recovering. This situation is similar to
crash recovery in databases, and the possible solutions are similar at
a high level. For example, the updates could take the form of writes
to newly created files, so that the if the new lock holder doesn't see
a complete set of new files, it would know to fall back on the
previous set. The section 2.3 "ready" znode scheme works like this.

Q: How does A-linearizability differ from linearizability?

A: A ZooKeeper client can send lots of "asynchronous" requests, without
waiting for each to finish before sending the next. By the rules of
ordinary linearizability, these requests are concurrent (they overlap in
time), and therefor can be executed in any order. In contrast, ZooKeeper
guarantees to execute them in the order that the client sent them. The
paper calls this A-linearizability.

Q: Why are only update requests A-linearizable? Why not reads as well?

A: The authors want high total read throughput, so they want ZooKeeper
replicas to be able to satisfy client reads and maintain watches
without involving the leader. A given replica may not know about a
committed write (if it's not in the majority that the leader waited
for), or may know about a write but not yet know if it is committed.
Thus a replica's state may lag behind the leader and other replicas.
Thus serving reads from replicas can yield data that doesn't reflect
recent writes -- that is, reads can return stale results.

Q: How does linearizability differ from serializability?

A: The usual definition of serializability is much like
linearizability, but without the requirement that operations respect
real-time ordering. Have a look at this explanation:
http://www.bailis.org/blog/linearizability-versus-serializability/

Section 2.3 of the ZooKeeper paper uses "serializable" to indicate
that the system behaves as if writes (from all clients combined) were
executed one by one in some order. The "FIFO client order" property
means that reads occur at specific points in the order of writes, and
that a given client's successive reads never move backwards in that
order. Note that the guarantees for writes and reads are different.

Q: Why is it OK for ZooKeeper to respond to read requests with
out-of-date data?

A: ZooKeeper is likely to yield data that is only slightly out of
date: the leader tries hard to keep all the followers up to date, much
as in Raft. So a follower may be a few writes (or batches of writes)
behind the leader, but rarely much more than that.

Most uses of ZooKeeper have no problem with slightly-out-of-date read
results. After all, even if ZooKeeper guaranteed to provide fresh
results as of the time ZooKeeper executed the read, those results could
easily be out of date by the time the reply arrived at the client.
Because some other client might send a write request to ZooKeeper just
after ZooKeeper replied to the read.

A typical use of ZooKeeper is for a MapReduce worker to register
itself and look for work, perhaps in a ZooKeeper directory in which
the MR coordinator writes task assignments. Suppose the worker checks
for work every 10 seconds, or uses a ZooKeeper watch to be notified of
any change in work assignments. In both cases, the worker is likely to
hear about a new assignment somewhat after the assignment was made.
But a little delay does not matter much (as long as each item of work
is relatively long).

One situation where ZooKeeper's read semantics might cause trouble is
if different clients compare notes. Client C1 might perform a read and
see a new value; and after that, client C2 might perform a read and
see an older value. This cannot happen in a linearizable system, but
it can happen with ZooKeeper.

Q: What is pipelining?

A: There are two things going on here. First, the ZooKeeper leader
(really the leader's Zab layer) batches together multiple client
operations in order to send them efficiently over the network, and in
order to efficiently write them to disk. For both network and disk,
it's often far more efficient to send a batch of N small items all at
once than it is to send or write them one at a time. This kind of
batching is only effective if the leader sees many client requests at
the same time; so it depends on there being lots of active clients.

The second aspect of pipelining is that ZooKeeper makes it easy for
each client to keep many write requests outstanding at a time, by
supporting asynchronous operations. From the client's point of view,
it can send lots of write requests without having to wait for the
responses (which arrive later, as notifications after the writes
commit). From the leader's point of view, that client behavior gives
the leader lots of requests to accumulate into big efficient batches.

A worry with pipelining is that operations that are in flight might be
re-ordered, which would cause the problem that the authors discuss
2.3. If the leader has many write operations in flight followed by the
creation of "ready", you don't want those operations to be re-ordered,
because then other clients may observe "ready" before the preceding
writes have been applied. To ensure that this cannot happen, Zookeeper
guarantees FIFO order for a client's operations; that is, ZooKeeper
applies operations in the order that the client issued them. This
guarantee is per-client; there's no guarantee about the order of
concurrent operations from different clients.

Q: How does the leader know the order in which a client wants a bunch
of asynchronous updates to be performed?

A: The paper doesn't say. The answer is likely to involve the client's
ZooKeeper library numbering its asynchronous requests, and the leader
tracking for each client (really session) what number it should next
expect. This information would have to be preserved when a leader
fails and another server takes over, so the client sequence numbers
are likely passed along in replicated log entries.

Q: What does wait-free mean?

A: The precise definition: A wait-free implementation of a concurrent
data object is one that guarantees that any process can complete any
operation in a finite number of steps, regardless of the execution
speeds of the other processes. This definition was introduced in the
following paper by Herlihy:
https://cs.brown.edu/~mph/Herlihy91/p124-herlihy.pdf

Zookeeper is wait-free because it processes one client's requests
without needing to wait for other clients to take action. This is
partially a consequence of the API: despite being designed to support
client/client coordination and synchronization, no ZooKeeper API call
is defined in a way that would require one client to wait for another.
In contrast, a system that provided a lock acquire operation that
waited for the current lock holder to release the lock would not be
wait-free.

Ultimately, however, ZooKeeper clients often need to wait for each
other, with watches or polling. The main effect of wait-freedom on the
API is that watches are factored out from other operations. The
combination of atomic test-and-set updates (e.g. file creation and
writes conditional on version) with watches allows clients to synthesize
more complex blocking abstractions (e.g. Section 2.4's locks and
barriers).

Q: What does a client do if it doesn't get a reply for a request? Does
it re-send, in case the network lost a request or reply, or the leader
crashed before committing? How does ZooKeeper avoid re-sends leading
to duplicate executions?

A: The paper doesn't say. Probably the leader tracks what request
numbers from each session it has received and committed, so that it
can filter out duplicate requests. If a client sends a request at
about the same time that ZooKeeper decides to expire the client's
session, the client may not be able to tell if the request actually
executed.

Q: If a client submits an asynchronous write, and immediately
afterwards does a read, will the read see the effect of the write?

A: The paper doesn't explicitly say, but the implication of the "FIFO
client order" property of Section 2.3 is that the read will see the
write. That implies that a ZK follower will block a read until the
follower has received (from the ZK leader) all of the client's
preceding writes. The follower is in a position to do this because a
client session sends all requests (read and write) to the same
follower, which therefor will be aware that the client has issued a
write that hasn't yet appeared in the stream of committed operations
from the leader.

Q: What is the reason for 'fuzzy snapshots'?

A: ZooKeeper needs to write its state to disk so that it can recover
from a power failure (by reading the data from the disk). It does this
by appending every write operation to a log on the disk, and (to
prevent that log from growing too long) it periodically writes a
"snapshot" of its entire state (all the data) to disk and truncates
the log. Thus the most most recent snapshot plus the log since that
snapshot contain all the data.

A precise snapshot would correspond to a specific point in the log:
the snapshot would include every write before that point, and no
writes after that point; and it would be clear exactly where to start
replay of log entries after a reboot to bring the snapshot up to date.
However, creation of a precise snapshot requires a way to prevent any
writes from happening while the snapshot is being created and written
to disk. Blocking writes for the duration of snapshot creation might
decrease performance a lot.

The "fuzzy" refers to the fact that ZooKeeper creates the snapshot
from its in-memory database while allowing writes to the database.
This means that a snapshot does not correspond to a particular point
in the log -- a snapshot includes a more or less random subset of the
writes that were concurrent with snapshot creation. After reboot,
ZooKeeper constructs a consistent snapshot by replaying all log
entries from the point at which the snapshot started, in order.
Because logged updates in Zookeeper are idempotent and describe the
state resulting from the client operation, the application-state will
be correct after reboot and replay---some messages may be applied
twice (once to the state before recovery and once after recovery) but
that is OK, because they are idempotent. The replay fixes the fuzzy
snapshot to be a consistent snapshot of the application state.

The Zookeeper leader turns the operations in the client API into
idempotent transactions. For example, if a client issues a conditional
setData and the version number in the request matches, the Zookeeper
leader creates a setDataTXN that contains the new data, the new
version number, and updated time stamps. This transaction (TXN) is
idempotent: Zookeeper can execute it twice and it will result in the
same state.

Q: What's an example of a situation where it's important that
ZooKeeper transform client operations to an idempotent form?

A: create(path, data, flags=sequential) isn't idempotent because
executing it twice produces a different result than executing
it once: executing it twice produces two znodes, with different
numbers in their names.

The ZooKeeper leader transforms operations to make them idempotent by
computing the outcome; for sequential create, that includes the number
for the created znode. It's these transformed operations that are put in
the log and sent to followers.

Idempotence is important for the fuzzy snapshot scheme. A fuzzy snapshot
does not capture an exact prefix of the log; instead, some operations
towards the end of the log might be included in the snapshot, and some
not. Suppose that while ZK was creating a fuzzy snapshot, this client
request arrived:

  create("x", "a", flags=sequential)

Assume that the "flags=sequential" caused the ZK leader to assign number
7 to the znode, so that it creates znode "x7". And that the newly
created znode "x7" exists both in the fuzzy snapshot and (as an
idempotent operation) in the log.

If the ZK server then rebooted, and restored its state by loading the
snapshot from disk and replaying the log, we don't want the replayed
create to create a new znode with the next highest number, which would
be "x8". We want ZK to notice that the create() is already reflected in
the server's state, and not repeat it. The transformation to idempotent
operations accomplishes this.

Q: How is the ZooKeeper leader chosen?

A: Zookeeper uses ZAB, an atomic broadcast system, which has leader
election built in, much like Raft. Here's a paper about Zab:
http://dl.acm.org/citation.cfm?id=2056409

Q: How does Zookeeper's performance compare to other systems
such as Paxos?

A: Zookeeper has impressive performance (in particular throughput).
Three Zookeeper servers process 21,000 writes per second. Typical
6.5840 Rafts with 3 servers commit on the order of tens of operations
per second (assuming a magnetic disk for storage) and maybe hundreds
per second with SSDs.

Q: How big is the ZooKeeper database? It seems like the server must
have a lot of memory.

Q: It depends on the application, and, unfortunately, the paper doesn't
report the authors' experience in this area. Since Zookeeper is
intended for configuration and coordination, and not as a
general-purpose data store, an in-memory database seems reasonable.
For example, you could imagine using Zookeeper for GFS's coordinator and
that amount of data should fit in the memory of a well-equipped
server, as it did for GFS.

Q: What's a universal object?

A: It is a theoretical statement of how good the API of Zookeeper is
based on a theory of concurrent objects that Herlihy introduced:
https://cs.brown.edu/~mph/Herlihy91/p124-herlihy.pdf. We won't
spend any time on this statement and theory, but if you care there is
a gentle introduction on this Wikipedia page:
https://en.wikipedia.org/wiki/Non-blocking_algorithm.

The authors appeal to this concurrent-object theory in order to show
that Zookeeper's API is general-purpose: that the API includes enough
features to implement any coordination scheme you'd want.

Q: When might one use a double barrier?

A: Here's a use for a single barrier: suppose you have a big
computation that proceeds in two phases. All of the results for phase
1 must be available before phase 2 can begin. Suppose you run each
phase in parallel on many workers (like MapReduce's Map, and then
Reduce). A barrier can be used to enforce "all workers must finish
phase 1 before any can start phase 2". If you have many phases, or a
loop in which each iteration is a phase, you might want a double
barrier.

Q: How does a client know when to leave one of the paper's double barriers?

A: Leaving the barrier involves each client watching the znodes for
all other clients participating in the barrier. Each client waits for
all of these znodes to disappear. Once they are all gone, the clients
all leave the barrier and continue computing.

Q: Is it possible to add more servers to an existing ZooKeeper service
without taking the service down?

A: It is -- although when the original paper was published, cluster
membership was static. Nowadays, ZooKeeper supports "dynamic
reconfiguration":

https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html

... and there is a paper describing the mechanism:

https://www.usenix.org/system/files/conference/atc12/atc12-final74.pdf

How do you think this compares to Raft's dynamic configuration change
via overlapping consensus, which appeared two years later?

Q: How are watches implemented in the client library?

A: The client library probably registers a callback function that will
be invoked when the watch triggers.

For example, a Go client for ZooKeeper implements it by passing a
channel into "GetW()" (get with watch); when the watch triggers, an
"Event" structure is sent through the channel. The application can
check the channel in a select clause.

See https://godoc.org/github.com/samuel/go-zookeeper/zk#Conn.GetW.

Q: Why does the read lock on page 6 in the code jump to line 3 instead
of line 2 like the write lock does?

A: Good catch; I believe it is a bug. The correct recipe is here:
https://zookeeper.apache.org/doc/r3.1.2/recipes.html#Shared+Locks

Q: Section 5.2 says the request latency for a write is 1.2
milliseconds. Isn't that less than the time required for a single hard
disk write?

A: I don't know what's going on here. Section 4 says each new log
entry is forced to disk, and seems to imply that ZooKeeper waits for
the data to be on disk before proceeding. The wait seems necessary for
crash recovery -- it would be bad to tell a client that a write had
succeeded, only to forget about it in a power failure. How long might
a disk write take? Section 5 says they use mechanical hard drives, and
appending an entry to a log file on disk takes at least half a
rotation on average. A typical disk spins at 7200 RPM, which means a
half rotation takes about 4 milliseconds, which is longer than the
reported overall latency.

If the paper were published today, and said it used SSDs instead of
hard drives, the 1.2 millisecond latency would make more sense, since
one can write an SSD in much less than a millisecond. Another way to
get fast writes is to interpose a battery-backed cache between the
software and the disk -- the point of the battery is to preserve
recent writes despite power failures, so they can be written to disk
after the power is restored.

Q: Is ZooKeeper widely used in real life?

A: Yes; have a look here for a partial list of projects and companies
that use ZooKeeper:

https://zookeeper.apache.org/doc/r3.8.4/zookeeperUseCases.html

Q: ZooKeeper includes many detailed design decisions about the API and
its semantics; have these turned out to be the right ones?

A: ZooKeeper has spurred the development of newer systems with
arguably better detailed designs. One example is etcd, which is aimed
at the same use cases as ZooKeeper, but differs in a bunch of ways
explained here:

  https://etcd.io/docs/v3.3/learning/why/

Q: Why did the authors choose the name ZooKeeper?

A: The apache zookeeper web site says this: "ZooKeeper: Because
Coordinating Distributed Systems is a Zoo"

ZooKeeper: Wait-free coordination for Internet-scale systems
Patrick Hunt and Mahadev Konar
Yahoo! Grid
{phunt,mahadev}@yahoo-inc.com
Flavio P. Junqueira and Benjamin Reed
Yahoo! Research
{fpj,breed}@yahoo-inc.com
Abstract
In this paper, we describe ZooKeeper, a service for co-
ordinating processes of distributed applications. Since
ZooKeeper is part of critical infrastructure, ZooKeeper
aims to provide a simple and high performance kernel
for building more complex coordination primitives at the
client. It incorporates elements from group messaging,
shared registers, and distributed lock services in a repli-
cated, centralized service. The interface exposed by Zoo-
Keeper has the wait-free aspects of shared registers with
an event-driven mechanism similar to cache invalidations
of distributed file systems to provide a simple, yet pow-
erful coordination service.
The ZooKeeper interface enables a high-performance
service implementation. In addition to the wait-free
property, ZooKeeper provides a per client guarantee of
FIFO execution of requests and linearizability for all re-
quests that change the ZooKeeper state. These design de-
cisions enable the implementation of a high performance
processing pipeline with read requests being satisfied by
local servers. We show for the target workloads, 2:1
to 100:1 read to write ratio, that ZooKeeper can handle
tens to hundreds of thousands of transactions per second.
This performance allows ZooKeeper to be used exten-
sively by client applications.
1 Introduction
Large-scale distributed applications require different
forms of coordination. Configuration is one of the most
basic forms of coordination. In its simplest form, con-
figuration is just a list of operational parameters for the
system processes, whereas more sophisticated systems
have dynamic configuration parameters. Group member-
ship and leader election are also common in distributed
systems: often processes need to know which other pro-
cesses are alive and what those processes are in charge
of. Locks constitute a powerful coordination primitive
that implement mutually exclusive access to critical re-
sources.
One approach to coordination is to develop services
for each of the different coordination needs. For exam-
ple, Amazon Simple Queue Service [3] focuses specif-
ically on queuing. Other services have been devel-
oped specifically for leader election [25] and configura-
tion [27]. Services that implement more powerful prim-
itives can be used to implement less powerful ones. For
example, Chubby [6] is a locking service with strong
synchronization guarantees. Locks can then be used to
implement leader election, group membership, etc.
When designing our coordination service, we moved
away from implementing specific primitives on the
server side, and instead we opted for exposing an API
that enables application developers to implement their
own primitives. Such a choice led to the implementa-
tion of a coordination kernel that enables new primitives
without requiring changes to the service core. This ap-
proach enables multiple forms of coordination adapted to
the requirements of applications, instead of constraining
developers to a fixed set of primitives.
When designing the API of ZooKeeper, we moved
away from blocking primitives, such as locks. Blocking
primitives for a coordination service can cause, among
other problems, slow or faulty clients to impact nega-
tively the performance of faster clients. The implemen-
tation of the service itself becomes more complicated
if processing requests depends on responses and fail-
ure detection of other clients. Our system, Zookeeper,
hence implements an API that manipulates simple wait-
free data objects organized hierarchically as in file sys-
tems. In fact, the ZooKeeper API resembles the one of
any other file system, and looking at just the API signa-
tures, ZooKeeper seems to be Chubby without the lock
methods, open, and close. Implementing wait-free data
objects, however, differentiates ZooKeeper significantly
from systems based on blocking primitives such as locks.
Although the wait-free property is important for per-
1

formance and fault tolerance, it is not sufficient for co-
ordination. We have also to provide order guarantees for
operations. In particular, we have found that guarantee-
ing both FIFO client ordering of all operations and lin-
earizable writes enables an efficient implementation of
the service and it is sufficient to implement coordination
primitives of interest to our applications. In fact, we can
implement consensus for any number of processes with
our API, and according to the hierarchy of Herlihy, Zoo-
Keeper implements a universal object [14].
The ZooKeeper service comprises an ensemble of
servers that use replication to achieve high availability
and performance. Its high performance enables appli-
cations comprising a large number of processes to use
such a coordination kernel to manage all aspects of co-
ordination. We were able to implement ZooKeeper us-
ing a simple pipelined architecture that allows us to have
hundreds or thousands of requests outstanding while still
achieving low latency. Such a pipeline naturally enables
the execution of operations from a single client in FIFO
order. Guaranteeing FIFO client order enables clients to
submit operations asynchronously. With asynchronous
operations, a client is able to have multiple outstanding
operations at a time. This feature is desirable, for exam-
ple, when a new client becomes a leader and it has to ma-
nipulate metadata and update it accordingly. Without the
possibility of multiple outstanding operations, the time
of initialization can be of the order of seconds instead of
sub-second.
To guarantee that update operations satisfy lineariz-
ability, we implement a leader-based atomic broadcast
protocol [23], called Zab [24]. A typical workload
of a ZooKeeper application, however, is dominated by
read operations and it becomes desirable to scale read
throughput. In ZooKeeper, servers process read opera-
tions locally, and we do not use Zab to totally order them.
Caching data on the client side is an important tech-
nique to increase the performance of reads. For example,
it is useful for a process to cache the identifier of the
current leader instead of probing ZooKeeper every time
it needs to know the leader. ZooKeeper uses a watch
mechanism to enable clients to cache data without man-
aging the client cache directly. With this mechanism,
a client can watch for an update to a given data object,
and receive a notification upon an update. Chubby man-
ages the client cache directly. It blocks updates to in-
validate the caches of all clients caching the data being
changed. Under this design, if any of these clients is
slow or faulty, the update is delayed. Chubby uses leases
to prevent a faulty client from blocking the system indef-
initely. Leases, however, only bound the impact of slow
or faulty clients, whereas ZooKeeper watches avoid the
problem altogether.
In this paper we discuss our design and implementa-
tion of ZooKeeper. With ZooKeeper, we are able to im-
plement all coordination primitives that our applications
require, even though only writes are linearizable. To val-
idate our approach we show how we implement some
coordination primitives with ZooKeeper.
To summarize, in this paper our main contributions are:
Coordination kernel: We propose a wait-free coordi-
nation service with relaxed consistency guarantees
for use in distributed systems. In particular, we de-
scribe our design and implementation of a coordi-
nation kernel , which we have used in many criti-
cal applications to implement various coordination
techniques.
Coordination recipes: We show how ZooKeeper can
be used to build higher level coordination primi-
tives, even blocking and strongly consistent primi-
tives, that are often used in distributed applications.
Experience with Coordination: We share some of the
ways that we use ZooKeeper and evaluate its per-
formance.
2 The ZooKeeper service
Clients submit requests to ZooKeeper through a client
API using a ZooKeeper client library. In addition to ex-
posing the ZooKeeper service interface through the client
API, the client library also manages the network connec-
tions between the client and ZooKeeper servers.
In this section, we first provide a high-level view of the
ZooKeeper service. We then discuss the API that clients
use to interact with ZooKeeper.
Terminology. In this paper, we use client to denote a
user of the ZooKeeper service,server to denote a process
providing the ZooKeeper service, and znode to denote
an in-memory data node in the ZooKeeper data, which
is organized in a hierarchical namespace referred to as
the data tree. We also use the terms update and write to
refer to any operation that modifies the state of the data
tree. Clients establish a session when they connect to
ZooKeeper and obtain a session handle through which
they issue requests.
2.1 Service overview
ZooKeeper provides to its clients the abstraction of a set
of data nodes (znodes), organized according to a hierar-
chical name space. The znodes in this hierarchy are data
objects that clients manipulate through the ZooKeeper
API. Hierarchical name spaces are commonly used in file
systems. It is a desirable way of organizing data objects,
since users are used to this abstraction and it enables bet-
ter organization of application meta-data. To refer to a
2

given znode, we use the standard UNIX notation for file
system paths. For example, we use /A/B/C to denote
the path to znode C, where C has B as its parent and B
has A as its parent. All znodes store data, and all znodes,
except for ephemeral znodes, can have children.
/
/app1 /app2
/app1/p_1 /app1/p_2 /app1/p_3
Figure 1: Illustration of ZooKeeper hierarchical name
space.
There are two types of znodes that a client can create:
Regular: Clients manipulate regular znodes by creating
and deleting them explicitly;
Ephemeral: Clients create such znodes, and they ei-
ther delete them explicitly, or let the system remove
them automatically when the session that creates
them terminates (deliberately or due to a failure).
Additionally, when creating a new znode, a client can
set a sequential flag. Nodes created with the sequen-
tial flag set have the value of a monotonically increas-
ing counter appended to its name. If n is the new znode
and p is the parent znode, then the sequence value of n
is never smaller than the value in the name of any other
sequential znode ever created under p.
ZooKeeper implements watches to allow clients to
receive timely notifications of changes without requir-
ing polling. When a client issues a read operation
with a watch flag set, the operation completes as nor-
mal except that the server promises to notify the client
when the information returned has changed. Watches
are one-time triggers associated with a session; they
are unregistered once triggered or the session closes.
Watches indicate that a change has happened, but do
not provide the change. For example, if a client is-
sues a getData(‘‘/foo’’, true) before “/foo”
is changed twice, the client will get one watch event
telling the client that data for “/foo” has changed. Ses-
sion events, such as connection loss events, are also sent
to watch callbacks so that clients know that watch events
may be delayed.
Data model. The data model of ZooKeeper is essen-
tially a file system with a simplified API and only full
data reads and writes, or a key/value table with hierar-
chical keys. The hierarchal namespace is useful for al-
locating subtrees for the namespace of different applica-
tions and for setting access rights to those subtrees. We
also exploit the concept of directories on the client side to
build higher level primitives as we will see in section 2.4.
Unlike files in file systems, znodes are not designed
for general data storage. Instead, znodes map to abstrac-
tions of the client application, typically corresponding
to meta-data used for coordination purposes. To illus-
trate, in Figure 1 we have two subtrees, one for Applica-
tion 1 (/app1) and another for Application 2 (/app2).
The subtree for Application 1 implements a simple group
membership protocol: each client process pi creates a
znode p i under /app1, which persists as long as the
process is running.
Although znodes have not been designed for general
data storage, ZooKeeper does allow clients to store some
information that can be used for meta-data or configu-
ration in a distributed computation. For example, in a
leader-based application, it is useful for an application
server that is just starting to learn which other server is
currently the leader. To accomplish this goal, we can
have the current leader write this information in a known
location in the znode space. Znodes also have associated
meta-data with time stamps and version counters, which
allow clients to track changes to znodes and execute con-
ditional updates based on the version of the znode.
Sessions. A client connects to ZooKeeper and initiates
a session. Sessions have an associated timeout. Zoo-
Keeper considers a client faulty if it does not receive any-
thing from its session for more than that timeout. A ses-
sion ends when clients explicitly close a session handle
or ZooKeeper detects that a clients is faulty. Within a ses-
sion, a client observes a succession of state changes that
reflect the execution of its operations. Sessions enable a
client to move transparently from one server to another
within a ZooKeeper ensemble, and hence persist across
ZooKeeper servers.
2.2 Client API
We present below a relevant subset of the ZooKeeper
API, and discuss the semantics of each request.
create(path, data, flags) : Creates a znode
with path name path, stores data[] in it, and
returns the name of the new znode. flags en-
ables a client to select the type of znode: regular,
ephemeral, and set the sequential flag;
delete(path, version) : Deletes the znode
path if that znode is at the expected version;
exists(path, watch) : Returns true if the znode
with path name path exists, and returns false oth-
erwise. The watch flag enables a client to set a
3

watch on the znode;
getData(path, watch) : Returns the data and
meta-data, such as version information, associated
with the znode. The watch flag works in the same
way as it does for exists(), except that Zoo-
Keeper does not set the watch if the znode does not
exist;
setData(path, data, version) : Writes
data[] to znode path if the version number is
the current version of the znode;
getChildren(path, watch) : Returns the set of
names of the children of a znode;
sync(path): Waits for all updates pending at the start
of the operation to propagate to the server that the
client is connected to. The path is currently ignored.
All methods have both a synchronous and an asyn-
chronous version available through the API. An applica-
tion uses the synchronous API when it needs to execute
a single ZooKeeper operation and it has no concurrent
tasks to execute, so it makes the necessary ZooKeeper
call and blocks. The asynchronous API, however, en-
ables an application to have both multiple outstanding
ZooKeeper operations and other tasks executed in par-
allel. The ZooKeeper client guarantees that the corre-
sponding callbacks for each operation are invoked in or-
der.
Note that ZooKeeper does not use handles to access
znodes. Each request instead includes the full path of
the znode being operated on. Not only does this choice
simplifies the API (no open() or close() methods),
but it also eliminates extra state that the server would
need to maintain.
Each of the update methods take an expected ver-
sion number, which enables the implementation of con-
ditional updates. If the actual version number of the zn-
ode does not match the expected version number the up-
date fails with an unexpected version error. If the version
number is−1, it does not perform version checking.
2.3 ZooKeeper guarantees
ZooKeeper has two basic ordering guarantees:
Linearizable writes: all requests that update the state
of ZooKeeper are serializable and respect prece-
dence;
FIFO client order: all requests from a given client are
executed in the order that they were sent by the
client.
Note that our definition of linearizability is different
from the one originally proposed by Herlihy [15], and
we call it A-linearizability (asynchronous linearizabil-
ity). In the original definition of linearizability by Her-
lihy, a client is only able to have one outstanding opera-
tion at a time (a client is one thread). In ours, we allow a
client to have multiple outstanding operations, and con-
sequently we can choose to guarantee no specific order
for outstanding operations of the same client or to guar-
antee FIFO order. We choose the latter for our property.
It is important to observe that all results that hold for
linearizable objects also hold for A-linearizable objects
because a system that satisfies A-linearizability also sat-
isfies linearizability. Because only update requests are A-
linearizable, ZooKeeper processes read requests locally
at each replica. This allows the service to scale linearly
as servers are added to the system.
To see how these two guarantees interact, consider the
following scenario. A system comprising a number of
processes elects a leader to command worker processes.
When a new leader takes charge of the system, it must
change a large number of configuration parameters and
notify the other processes once it finishes. We then have
two important requirements:
• As the new leader starts making changes, we do not
want other processes to start using the configuration
that is being changed;
• If the new leader dies before the configuration has
been fully updated, we do not want the processes to
use this partial configuration.
Observe that distributed locks, such as the locks pro-
vided by Chubby, would help with the first requirement
but are insufficient for the second. With ZooKeeper,
the new leader can designate a path as the ready znode;
other processes will only use the configuration when that
znode exists. The new leader makes the configuration
change by deleting ready, updating the various configu-
ration znodes, and creating ready. All of these changes
can be pipelined and issued asynchronously to quickly
update the configuration state. Although the latency of a
change operation is of the order of 2 milliseconds, a new
leader that must update 5000 different znodes will take
10 seconds if the requests are issued one after the other;
by issuing the requests asynchronously the requests will
take less than a second. Because of the ordering guaran-
tees, if a process sees the ready znode, it must also see
all the configuration changes made by the new leader. If
the new leader dies before theready znode is created, the
other processes know that the configuration has not been
finalized and do not use it.
The above scheme still has a problem: what happens
if a process sees that ready exists before the new leader
starts to make a change and then starts reading the con-
figuration while the change is in progress. This problem
is solved by the ordering guarantee for the notifications:
if a client is watching for a change, the client will see
the notification event before it sees the new state of the
system after the change is made. Consequently, if the
process that reads the ready znode requests to be notified
of changes to that znode, it will see a notification inform-
4

ing the client of the change before it can read any of the
new configuration.
Another problem can arise when clients have their own
communication channels in addition to ZooKeeper. For
example, consider two clientsA and B that have a shared
configuration in ZooKeeper and communicate through a
shared communication channel. If A changes the shared
configuration in ZooKeeper and tells B of the change
through the shared communication channel,B would ex-
pect to see the change when it re-reads the configuration.
If B’s ZooKeeper replica is slightly behind A’s, it may
not see the new configuration. Using the above guar-
antees B can make sure that it sees the most up-to-date
information by issuing a write before re-reading the con-
figuration. To handle this scenario more efficiently Zoo-
Keeper provides the sync request: when followed by
a read, constitutes a slow read . sync causes a server
to apply all pending write requests before processing the
read without the overhead of a full write. This primitive
is similar in idea to the flush primitive of ISIS [5].
ZooKeeper also has the following two liveness and
durability guarantees: if a majority of ZooKeeper servers
are active and communicating the service will be avail-
able; and if the ZooKeeper service responds successfully
to a change request, that change persists across any num-
ber of failures as long as a quorum of servers is eventu-
ally able to recover.
2.4 Examples of primitives
In this section, we show how to use the ZooKeeper API
to implement more powerful primitives. The ZooKeeper
service knows nothing about these more powerful primi-
tives since they are entirely implemented at the client us-
ing the ZooKeeper client API. Some common primitives
such as group membership and configuration manage-
ment are also wait-free. For others, such as rendezvous,
clients need to wait for an event. Even though ZooKeeper
is wait-free, we can implement efficient blocking primi-
tives with ZooKeeper. ZooKeeper’s ordering guarantees
allow efficient reasoning about system state, and watches
allow for efficient waiting.
Configuration Management ZooKeeper can be used
to implement dynamic configuration in a distributed ap-
plication. In its simplest form configuration is stored in
a znode, zc. Processes start up with the full pathname
of zc. Starting processes obtain their configuration by
reading zc with the watch flag set to true. If the config-
uration in zc is ever updated, the processes are notified
and read the new configuration, again setting the watch
flag to true.
Note that in this scheme, as in most others that use
watches, watches are used to make sure that a process has
the most recent information. For example, if a process
watching zc is notified of a change to zc and before it
can issue a read for zc there are three more changes to
zc, the process does not receive three more notification
events. This does not affect the behavior of the process,
since those three events would have simply notified the
process of something it already knows: the information
it has for zc is stale.
Rendezvous Sometimes in distributed systems, it is
not always clear a priori what the final system config-
uration will look like. For example, a client may want to
start a master process and several worker processes, but
the starting processes is done by a scheduler, so the client
does not know ahead of time information such as ad-
dresses and ports that it can give the worker processes to
connect to the master. We handle this scenario with Zoo-
Keeper using a rendezvous znode, zr, which is an node
created by the client. The client passes the full pathname
of zr as a startup parameter of the master and worker
processes. When the master starts it fills in zr with in-
formation about addresses and ports it is using. When
workers start, they read zr with watch set to true. If zr
has not been filled in yet, the worker waits to be notified
when zr is updated. If zr is an ephemeral node, master
and worker processes can watch for zr to be deleted and
clean themselves up when the client ends.
Group Membership We take advantage of ephemeral
nodes to implement group membership. Specifically, we
use the fact that ephemeral nodes allow us to see the state
of the session that created the node. We start by designat-
ing a znode, zg to represent the group. When a process
member of the group starts, it creates an ephemeral child
znode under zg. If each process has a unique name or
identifier, then that name is used as the name of the child
znode; otherwise, the process creates the znode with the
SEQUENTIAL flag to obtain a unique name assignment.
Processes may put process information in the data of the
child znode, addresses and ports used by the process, for
example.
After the child znode is created under zg the process
starts normally. It does not need to do anything else. If
the process fails or ends, the znode that represents it un-
der zg is automatically removed.
Processes can obtain group information by simply list-
ing the children of zg. If a process wants to monitor
changes in group membership, the process can set the
watch flag to true and refresh the group information (al-
ways setting the watch flag to true) when change notifi-
cations are received.
5

Simple Locks Although ZooKeeper is not a lock ser-
vice, it can be used to implement locks. Applications
using ZooKeeper usually use synchronization primitives
tailored to their needs, such as those shown above. Here
we show how to implement locks with ZooKeeper to
show that it can implement a wide variety of general syn-
chronization primitives.
The simplest lock implementation uses “lock files”.
The lock is represented by a znode. To acquire a lock,
a client tries to create the designated znode with the
EPHEMERAL flag. If the create succeeds, the client
holds the lock. Otherwise, the client can read the zn-
ode with the watch flag set to be notified if the current
leader dies. A client releases the lock when it dies or ex-
plicitly deletes the znode. Other clients that are waiting
for a lock try again to acquire a lock once they observe
the znode being deleted.
While this simple locking protocol works, it does have
some problems. First, it suffers from the herd effect. If
there are many clients waiting to acquire a lock, they will
all vie for the lock when it is released even though only
one client can acquire the lock. Second, it only imple-
ments exclusive locking. The following two primitives
show how both of these problems can be overcome.
Simple Locks without Herd Effect We define a lock
znode l to implement such locks. Intuitively we line up
all the clients requesting the lock and each client obtains
the lock in order of request arrival. Thus, clients wishing
to obtain the lock do the following:
Lock
1 n = create(l + “/lock-”, EPHEMERAL|SEQUENTIAL)
2 C = getChildren(l, false)
3 if n is lowest znode in C, exit
4 p = znode in C ordered just before n
5 if exists(p, true) wait for watch event
6 goto 2
Unlock
1 delete(n)
The use of the SEQUENTIAL flag in line 1 of Lock
orders the client’s attempt to acquire the lock with re-
spect to all other attempts. If the client’s znode has the
lowest sequence number at line 3, the client holds the
lock. Otherwise, the client waits for deletion of the zn-
ode that either has the lock or will receive the lock be-
fore this client’s znode. By only watching the znode
that precedes the client’s znode, we avoid the herd effect
by only waking up one process when a lock is released
or a lock request is abandoned. Once the znode being
watched by the client goes away, the client must check
if it now holds the lock. (The previous lock request may
have been abandoned and there is a znode with a lower
sequence number still waiting for or holding the lock.)
Releasing a lock is as simple as deleting the zn-
ode n that represents the lock request. By using the
EPHEMERAL flag on creation, processes that crash will
automatically cleanup any lock requests or release any
locks that they may have.
In summary, this locking scheme has the following ad-
vantages:
1. The removal of a znode only causes one client to
wake up, since each znode is watched by exactly
one other client, so we do not have the herd effect;
2. There is no polling or timeouts;
3. Because of the way we have implemented locking,
we can see by browsing the ZooKeeper data the
amount of lock contention, break locks, and debug
locking problems.
Read/Write Locks To implement read/write locks we
change the lock procedure slightly and have separate
read lock and write lock procedures. The unlock pro-
cedure is the same as the global lock case.
Write Lock
1 n = create(l + “/write-”, EPHEMERAL|SEQUENTIAL)
2 C = getChildren(l, false)
3 if n is lowest znode in C, exit
4 p = znode in C ordered just before n
5 if exists(p, true) wait for event
6 goto 2
Read Lock
1 n = create(l + “/read-”, EPHEMERAL|SEQUENTIAL)
2 C = getChildren(l, false)
3 if no write znodes lower than n in C, exit
4 p = write znode in C ordered just before n
5 if exists(p, true) wait for event
6 goto 3
This lock procedure varies slightly from the previous
locks. Write locks differ only in naming. Since read
locks may be shared, lines 3 and 4 vary slightly because
only earlier write lock znodes prevent the client from ob-
taining a read lock. It may appear that we have a “herd
effect” when there are several clients waiting for a read
lock and get notified when the “write-” znode with the
lower sequence number is deleted; in fact, this is a de-
sired behavior, all those read clients should be released
since they may now have the lock.
Double Barrier Double barriers enable clients to syn-
chronize the beginning and the end of a computation.
When enough processes, defined by the barrier thresh-
old, have joined the barrier, processes start their compu-
tation and leave the barrier once they have finished. We
represent a barrier in ZooKeeper with a znode, referred
to as b. Every process p registers with b – by creating
a znode as a child of b – on entry, and unregisters – re-
moves the child – when it is ready to leave. Processes
can enter the barrier when the number of child znodes
of b exceeds the barrier threshold. Processes can leave
the barrier when all of the processes have removed their
children. We use watches to efficiently wait for enter and
6

exit conditions to be satisfied. To enter, processes watch
for the existence of a ready child of b that will be cre-
ated by the process that causes the number of children to
exceed the barrier threshold. To leave, processes watch
for a particular child to disappear and only check the exit
condition once that znode has been removed.
3 ZooKeeper Applications
We now describe some applications that use ZooKeeper,
and explain briefly how they use it. We show the primi-
tives of each example in bold.
The Fetching Service Crawling is an important part of
a search engine, and Yahoo! crawls billions of Web doc-
uments. The Fetching Service (FS) is part of the Yahoo!
crawler and it is currently in production. Essentially, it
has master processes that command page-fetching pro-
cesses. The master provides the fetchers with configura-
tion, and the fetchers write back informing of their status
and health. The main advantages of using ZooKeeper
for FS are recovering from failures of masters, guaran-
teeing availability despite failures, and decoupling the
clients from the servers, allowing them to direct their re-
quest to healthy servers by just reading their status from
ZooKeeper. Thus, FS uses ZooKeeper mainly to man-
age configuration metadata, although it also uses Zoo-
Keeper to elect masters (leader election).
 0
 500
 1000
 1500
 2000
66h60h54h48h42h36h30h24h18h12h6h0h
Number of operations
Time in seconds
read
write
Figure 2: Workload for one ZK server with the Fetching
Service. Each point represents a one-second sample.
Figure 2 shows the read and write traffic for a Zoo-
Keeper server used by FS through a period of three days.
To generate this graph, we count the number of opera-
tions for every second during the period, and each point
corresponds to the number of operations in that second.
We observe that the read traffic is much higher compared
to the write traffic. During periods in which the rate is
higher than 1, 000 operations per second, the read:write
ratio varies between 10:1 and 100:1. The read operations
in this workload are getData(), getChildren(),
and exists(), in increasing order of prevalence.
Katta Katta [17] is a distributed indexer that uses Zoo-
Keeper for coordination, and it is an example of a non-
Yahoo! application. Katta divides the work of indexing
using shards. A master server assigns shards to slaves
and tracks progress. Slaves can fail, so the master must
redistribute load as slaves come and go. The master can
also fail, so other servers must be ready to take over in
case of failure. Katta uses ZooKeeper to track the status
of slave servers and the master ( group membership ),
and to handle master failover ( leader election ). Katta
also uses ZooKeeper to track and propagate the assign-
ments of shards to slaves (configuration management).
Yahoo! Message Broker Yahoo! Message Broker
(YMB) is a distributed publish-subscribe system. The
system manages thousands of topics that clients can pub-
lish messages to and receive messages from. The topics
are distributed among a set of servers to provide scala-
bility. Each topic is replicated using a primary-backup
scheme that ensures messages are replicated to two ma-
chines to ensure reliable message delivery. The servers
that makeup YMB use a shared-nothing distributed ar-
chitecture which makes coordination essential for correct
operation. YMB uses ZooKeeper to manage the distribu-
tion of topics ( configuration metadata), deal with fail-
ures of machines in the system ( failure detection and
group membership), and control system operation.
broker domain
broker_disabledtopicsnodesshutdown migration_prohibited
<hostname><hostname> <hostname>    .....
load
# of topics
<topic> <topic> <topic> ....
primary backup
hostname
Figure 3: The layout of Yahoo! Message Broker (YMB)
structures in ZooKeeper
Figure 3 shows part of the znode data layout for YMB.
Each broker domain has a znode called nodes that has
an ephemeral znode for each of the active servers that
compose the YMB service. Each YMB server creates
an ephemeral znode under nodes with load and sta-
tus information providing both group membership and
status information through ZooKeeper. Nodes such as
shutdown and migration prohibited are mon-
itored by all of the servers that make up the service and
allow centralized control of YMB. The topics direc-
tory has a child znode for each topic managed by YMB.
These topic znodes have child znodes that indicate the
7

primary and backup server for each topic along with the
subscribers of that topic. The primary and backup
server znodes not only allow servers to discover the
servers in charge of a topic, but they also manage leader
election and server crashes.
Request
Processor
Atomic
Broadcast
Replicated
Database
Write
Request
Response
ZooKeeper Service
txn
txn
Read
Request
Figure 4: The components of the ZooKeeper service.
4 ZooKeeper Implementation
ZooKeeper provides high availability by replicating the
ZooKeeper data on each server that composes the ser-
vice. We assume that servers fail by crashing, and such
faulty servers may later recover. Figure 4 shows the high-
level components of the ZooKeeper service. Upon re-
ceiving a request, a server prepares it for execution (re-
quest processor). If such a request requires coordina-
tion among the servers (write requests), then they use an
agreement protocol (an implementation of atomic broad-
cast), and finally servers commit changes to the Zoo-
Keeper database fully replicated across all servers of the
ensemble. In the case of read requests, a server simply
reads the state of the local database and generates a re-
sponse to the request.
The replicated database is anin-memory database con-
taining the entire data tree. Each znode in the tree stores a
maximum of 1MB of data by default, but this maximum
value is a configuration parameter that can be changed in
specific cases. For recoverability, we efficiently log up-
dates to disk, and we force writes to be on the disk media
before they are applied to the in-memory database. In
fact, as Chubby [8], we keep a replay log (a write-ahead
log, in our case) of committed operations and generate
periodic snapshots of the in-memory database.
Every ZooKeeper server services clients. Clients con-
nect to exactly one server to submit its requests. As we
noted earlier, read requests are serviced from the local
replica of each server database. Requests that change the
state of the service, write requests, are processed by an
agreement protocol.
As part of the agreement protocol write requests are
forwarded to a single server, called the leader1. The
rest of the ZooKeeper servers, called followers, receive
1Details of leaders and followers, as part of the agreement protocol,
are out of the scope of this paper.
message proposals consisting of state changes from the
leader and agree upon state changes.
4.1 Request Processor
Since the messaging layer is atomic, we guarantee that
the local replicas never diverge, although at any point in
time some servers may have applied more transactions
than others. Unlike the requests sent from clients, the
transactions are idempotent. When the leader receives
a write request, it calculates what the state of the sys-
tem will be when the write is applied and transforms it
into a transaction that captures this new state. The fu-
ture state must be calculated because there may be out-
standing transactions that have not yet been applied to
the database. For example, if a client does a conditional
setData and the version number in the request matches
the future version number of the znode being updated,
the service generates a setDataTXN that contains the
new data, the new version number, and updated time
stamps. If an error occurs, such as mismatched version
numbers or the znode to be updated does not exist, an
errorTXN is generated instead.
4.2 Atomic Broadcast
All requests that update ZooKeeper state are forwarded
to the leader. The leader executes the request and
broadcasts the change to the ZooKeeper state through
Zab [24], an atomic broadcast protocol. The server that
receives the client request responds to the client when it
delivers the corresponding state change. Zab uses by de-
fault simple majority quorums to decide on a proposal,
so Zab and thus ZooKeeper can only work if a majority
of servers are correct ( i.e., with 2f + 1 server we can
tolerate f failures).
To achieve high throughput, ZooKeeper tries to keep
the request processing pipeline full. It may have thou-
sands of requests in different parts of the processing
pipeline. Because state changes depend on the appli-
cation of previous state changes, Zab provides stronger
order guarantees than regular atomic broadcast. More
specifically, Zab guarantees that changes broadcast by a
leader are delivered in the order they were sent and all
changes from previous leaders are delivered to an estab-
lished leader before it broadcasts its own changes.
There are a few implementation details that simplify
our implementation and give us excellent performance.
We use TCP for our transport so message order is main-
tained by the network, which allows us to simplify our
implementation. We use the leader chosen by Zab as
the ZooKeeper leader, so that the same process that cre-
ates transactions also proposes them. We use the log to
keep track of proposals as the write-ahead log for the in-
8

memory database, so that we do not have to write mes-
sages twice to disk.
During normal operation Zab does deliver all mes-
sages in order and exactly once, but since Zab does not
persistently record the id of every message delivered,
Zab may redeliver a message during recovery. Because
we use idempotent transactions, multiple delivery is ac-
ceptable as long as they are delivered in order. In fact,
ZooKeeper requires Zab to redeliver at least all messages
that were delivered after the start of the last snapshot.
4.3 Replicated Database
Each replica has a copy in memory of the ZooKeeper
state. When a ZooKeeper server recovers from a crash, it
needs to recover this internal state. Replaying all deliv-
ered messages to recover state would take prohibitively
long after running the server for a while, so ZooKeeper
uses periodic snapshots and only requires redelivery of
messages since the start of the snapshot. We call Zoo-
Keeper snapshots fuzzy snapshots since we do not lock
the ZooKeeper state to take the snapshot; instead, we do
a depth first scan of the tree atomically reading each zn-
ode’s data and meta-data and writing them to disk. Since
the resulting fuzzy snapshot may have applied some sub-
set of the state changes delivered during the generation of
the snapshot, the result may not correspond to the state
of ZooKeeper at any point in time. However, since state
changes are idempotent, we can apply them twice as long
as we apply the state changes in order.
For example, assume that in a ZooKeeper data tree two
nodes /foo and /goo have values f1 and g1 respec-
tively and both are at version 1 when the fuzzy snap-
shot begins, and the following stream of state changes
arrive having the form⟨transactionType, path,
value, new-version⟩:
⟨SetDataTXN, /foo, f2, 2 ⟩
⟨SetDataTXN, /goo, g2, 2 ⟩
⟨SetDataTXN, /foo, f3, 3 ⟩
After processing these state changes, /foo and /goo
have values f3 and g2 with versions 3 and 2 respec-
tively. However, the fuzzy snapshot may have recorded
that /foo and /goo have values f3 and g1 with ver-
sions 3 and 1 respectively, which was not a valid state
of the ZooKeeper data tree. If the server crashes and
recovers with this snapshot and Zab redelivers the state
changes, the resulting state corresponds to the state of the
service before the crash.
4.4 Client-Server Interactions
When a server processes a write request, it also sends out
and clears notifications relative to any watch that corre-
sponds to that update. Servers process writes in order
and do not process other writes or reads concurrently.
This ensures strict succession of notifications. Note that
servers handle notifications locally. Only the server that
a client is connected to tracks and triggers notifications
for that client.
Read requests are handled locally at each server. Each
read request is processed and tagged with azxid that cor-
responds to the last transaction seen by the server. This
zxid defines the partial order of the read requests with re-
spect to the write requests. By processing reads locally,
we obtain excellent read performance because it is just an
in-memory operation on the local server, and there is no
disk activity or agreement protocol to run. This design
choice is key to achieving our goal of excellent perfor-
mance with read-dominant workloads.
One drawback of using fast reads is not guaranteeing
precedence order for read operations. That is, a read op-
eration may return a stale value, even though a more
recent update to the same znode has been committed.
Not all of our applications require precedence order, but
for applications that do require it, we have implemented
sync. This primitive executes asynchronously and is
ordered by the leader after all pending writes to its lo-
cal replica. To guarantee that a given read operation re-
turns the latest updated value, a client calls sync fol-
lowed by the read operation. The FIFO order guarantee
of client operations together with the global guarantee of
sync enables the result of the read operation to reflect
any changes that happened before the sync was issued.
In our implementation, we do not need to atomically
broadcast sync as we use a leader-based algorithm, and
we simply place the sync operation at the end of the
queue of requests between the leader and the server ex-
ecuting the call to sync. In order for this to work, the
follower must be sure that the leader is still the leader.
If there are pending transactions that commit, then the
server does not suspect the leader. If the pending queue
is empty, the leader needs to issue a null transaction to
commit and orders the sync after that transaction. This
has the nice property that when the leader is under load,
no extra broadcast traffic is generated. In our implemen-
tation, timeouts are set such that leaders realize they are
not leaders before followers abandon them, so we do not
issue the null transaction.
ZooKeeper servers process requests from clients in
FIFO order. Responses include the zxid that the response
is relative to. Even heartbeat messages during intervals
of no activity include the last zxid seen by the server that
the client is connected to. If the client connects to a new
server, that new server ensures that its view of the Zoo-
Keeper data is at least as recent as the view of the client
by checking the last zxid of the client against its lastzxid.
If the client has a more recent view than the server, the
9

server does not reestablish the session with the client un-
til the server has caught up. The client is guaranteed to
be able to find another server that has a recent view of the
system since the client only sees changes that have been
replicated to a majority of the ZooKeeper servers. This
behavior is important to guarantee durability.
To detect client session failures, ZooKeeper uses time-
outs. The leader determines that there has been a failure
if no other server receives anything from a client ses-
sion within the session timeout. If the client sends re-
quests frequently enough, then there is no need to send
any other message. Otherwise, the client sends heartbeat
messages during periods of low activity. If the client
cannot communicate with a server to send a request or
heartbeat, it connects to a different ZooKeeper server to
re-establish its session. To prevent the session from tim-
ing out, the ZooKeeper client library sends a heartbeat
after the session has been idle for s/3 ms and switch to a
new server if it has not heard from a server for 2s/3 ms,
where s is the session timeout in milliseconds.
5 Evaluation
We performed all of our evaluation on a cluster of 50
servers. Each server has one Xeon dual-core 2.1GHz
processor, 4GB of RAM, gigabit ethernet, and two SATA
hard drives. We split the following discussion into two
parts: throughput and latency of requests.
5.1 Throughput
To evaluate our system, we benchmark throughput when
the system is saturated and the changes in throughput
for various injected failures. We varied the number of
servers that make up the ZooKeeper service, but always
kept the number of clients the same. To simulate a large
number of clients, we used 35 machines to simulate 250
simultaneous clients.
We have a Java implementation of the ZooKeeper
server, and both Java and C clients 2. For these experi-
ments, we used the Java server configured to log to one
dedicated disk and take snapshots on another. Our bench-
mark client uses the asynchronous Java client API, and
each client has at least 100 requests outstanding. Each
request consists of a read or write of 1K of data. We
do not show benchmarks for other operations since the
performance of all the operations that modify state are
approximately the same, and the performance of non-
state modifying operations, excludingsync, are approx-
imately the same. (The performance of sync approxi-
mates that of a light-weight write, since the request must
2The implementation is publicly available at http://hadoop.
apache.org/zookeeper.
go to the leader, but does not get broadcast.) Clients
send counts of the number of completed operations ev-
ery 300ms and we sample every 6s. To prevent memory
overflows, servers throttle the number of concurrent re-
quests in the system. ZooKeeper uses request throttling
to keep servers from being overwhelmed. For these ex-
periments, we configured the ZooKeeper servers to have
a maximum of 2, 000 total requests in process.
 0
 10000
 20000
 30000
 40000
 50000
 60000
 70000
 80000
 90000
 0  20  40  60  80  100
Operations per second
Percentage of read requests
Throughput of saturated system
3 servers
5 servers
7 servers
9 servers
13 servers
Figure 5: The throughput performance of a saturated sys-
tem as the ratio of reads to writes vary.
Servers 100% Reads 0% Reads
13 460k 8k
9 296k 12k
7 257k 14k
5 165k 18k
3 87k 21k
Table 1: The throughput performance of the extremes of
a saturated system.
In Figure 5, we show throughput as we vary the ratio
of read to write requests, and each curve corresponds to
a different number of servers providing the ZooKeeper
service. Table 1 shows the numbers at the extremes of
the read loads. Read throughput is higher than write
throughput because reads do not use atomic broadcast.
The graph also shows that the number of servers also has
a negative impact on the performance of the broadcast
protocol. From these graphs, we observe that the number
of servers in the system does not only impact the num-
ber of failures that the service can handle, but also the
workload the service can handle. Note that the curve for
three servers crosses the others around 60%. This situ-
ation is not exclusive of the three-server configuration,
and happens for all configurations due to the parallelism
local reads enable. It is not observable for other config-
urations in the figure, however, because we have capped
the maximum y-axis throughput for readability.
There are two reasons for write requests taking longer
than read requests. First, write requests must go through
atomic broadcast, which requires some extra processing
10

and adds latency to requests. The other reason for longer
processing of write requests is that servers must ensure
that transactions are logged to non-volatile store before
sending acknowledgments back to the leader. In prin-
ciple, this requirement is excessive, but for our produc-
tion systems we trade performance for reliability since
ZooKeeper constitutes application ground truth. We use
more servers to tolerate more faults. We increase write
throughput by partitioning the ZooKeeper data into mul-
tiple ZooKeeper ensembles. This performance trade off
between replication and partitioning has been previously
observed by Gray et al. [12].
 0
 10000
 20000
 30000
 40000
 50000
 60000
 70000
 80000
 90000
 0  20  40  60  80  100
Operations per second
Percentage of read requests
Throughput of saturated system (all requests to leader)
3 servers
5 servers
7 servers
9 servers
13 servers
Figure 6: Throughput of a saturated system, varying the
ratio of reads to writes when all clients connect to the
leader.
ZooKeeper is able to achieve such high throughput by
distributing load across the servers that makeup the ser-
vice. We can distribute the load because of our relaxed
consistency guarantees. Chubby clients instead direct all
requests to the leader. Figure 6 shows what happens if
we do not take advantage of this relaxation and forced
the clients to only connect to the leader. As expected the
throughput is much lower for read-dominant workloads,
but even for write-dominant workloads the throughput is
lower. The extra CPU and network load caused by ser-
vicing clients impacts the ability of the leader to coor-
dinate the broadcast of the proposals, which in turn ad-
versely impacts the overall write performance.
The atomic broadcast protocol does most of the work
of the system and thus limits the performance of Zoo-
Keeper more than any other component. Figure 7 shows
the throughput of the atomic broadcast component. To
benchmark its performance we simulate clients by gen-
erating the transactions directly at the leader, so there is
no client connections or client requests and replies. At
maximum throughput the atomic broadcast component
becomes CPU bound. In theory the performance of Fig-
ure 7 would match the performance of ZooKeeper with
100% writes. However, the ZooKeeper client commu-
nication, ACL checks, and request to transaction con-
 0
 10000
 20000
 30000
 40000
 50000
 60000
 70000
 2  4  6  8  10  12  14
Requests per second
Size of ensemble
Atomic Broadcast Throughput
Figure 7: Average throughput of the atomic broadcast
component in isolation. Error bars denote the minimum
and maximum values.
versions all require CPU. The contention for CPU low-
ers ZooKeeper throughput to substantially less than the
atomic broadcast component in isolation. Because Zoo-
Keeper is a critical production component, up to now our
development focus for ZooKeeper has been correctness
and robustness. There are plenty of opportunities for im-
proving performance significantly by eliminating things
like extra copies, multiple serializations of the same ob-
ject, more efficient internal data structures, etc.
 0
 10000
 20000
 30000
 40000
 50000
 60000
 70000
 0  50  100  150  200  250  300
dnoces rep snoitarepO
Seconds since start of series
Time series with failures
Throughput
1 2
3
4a
5
64b
4c
Figure 8: Throughput upon failures.
To show the behavior of the system over time as fail-
ures are injected we ran a ZooKeeper service made up
of 5 machines. We ran the same saturation benchmark
as before, but this time we kept the write percentage at
a constant 30%, which is a conservative ratio of our ex-
pected workloads. Periodically we killed some of the
server processes. Figure 8 shows the system throughput
as it changes over time. The events marked in the figure
are the following:
1. Failure and recovery of a follower;
2. Failure and recovery of a different follower;
3. Failure of the leader;
4. Failure of two followers (a, b) in the first two marks,
and recovery at the third mark (c);
5. Failure of the leader.
11

6. Recovery of the leader.
There are a few important observations from this
graph. First, if followers fail and recover quickly, then
ZooKeeper is able to sustain a high throughput despite
the failure. The failure of a single follower does not pre-
vent servers from forming a quorum, and only reduces
throughput roughly by the share of read requests that the
server was processing before failing. Second, our leader
election algorithm is able to recover fast enough to pre-
vent throughput from dropping substantially. In our ob-
servations, ZooKeeper takes less than 200ms to elect a
new leader. Thus, although servers stop serving requests
for a fraction of second, we do not observe a throughput
of zero due to our sampling period, which is on the order
of seconds. Third, even if followers take more time to re-
cover, ZooKeeper is able to raise throughput again once
they start processing requests. One reason that we do
not recover to the full throughput level after events 1, 2,
and 4 is that the clients only switch followers when their
connection to the follower is broken. Thus, after event 4
the clients do not redistribute themselves until the leader
fails at events 3 and 5. In practice such imbalances work
themselves out over time as clients come and go.
5.2 Latency of requests
To assess the latency of requests, we created a bench-
mark modeled after the Chubby benchmark [6]. We cre-
ate a worker process that simply sends a create, waits
for it to finish, sends an asynchronous delete of the new
node, and then starts the next create. We vary the number
of workers accordingly, and for each run, we have each
worker create 50,000 nodes. We calculate the throughput
by dividing the number of create requests completed by
the total time it took for all the workers to complete.
Number of servers
Workers 3 5 7 9
1 776 748 758 711
10 2074 1832 1572 1540
20 2740 2336 1934 1890
Table 2: Create requests processed per second.
Table 2 show the results of our benchmark. The cre-
ate requests include 1K of data, rather than 5 bytes in
the Chubby benchmark, to better coincide with our ex-
pected use. Even with these larger requests, the through-
put of ZooKeeper is more than 3 times higher than the
published throughput of Chubby. The throughput of the
single ZooKeeper worker benchmark indicates that the
average request latency is 1.2ms for three servers and
1.4ms for 9 servers.
# of clients
# of barriers 50 100 200
200 9.4 19.8 41.0
400 16.4 34.1 62.0
800 28.9 55.9 112.1
1600 54.0 102.7 234.4
Table 3: Barrier experiment with time in seconds. Each
point is the average of the time for each client to finish
over five runs.
5.3 Performance of barriers
In this experiment, we execute a number of barriers se-
quentially to assess the performance of primitives imple-
mented with ZooKeeper. For a given number of barriers
b, each client first enters all b barriers, and then it leaves
all b barriers in succession. As we use the double-barrier
algorithm of Section 2.4, a client first waits for all other
clients to execute the enter() procedure before mov-
ing to next call (similarly for leave()).
We report the results of our experiments in Table 3.
In this experiment, we have 50, 100, and 200 clients
entering a number b of barriers in succession, b ∈
{200, 400, 800, 1600}. Although an application can have
thousands of ZooKeeper clients, quite often a much
smaller subset participates in each coordination oper-
ation as clients are often grouped according to the
specifics of the application.
Two interesting observations from this experiment are
that the time to process all barriers increase roughly lin-
early with the number of barriers, showing that concur-
rent access to the same part of the data tree did not pro-
duce any unexpected delay, and that latency increases
proportionally to the number of clients. This is a con-
sequence of not saturating the ZooKeeper service. In
fact, we observe that even with clients proceeding in
lock-step, the throughput of barrier operations (enter and
leave) is between 1,950 and 3,100 operations per second
in all cases. In ZooKeeper operations, this corresponds
to throughput values between 10,700 and 17,000 opera-
tions per second. As in our implementation we have a
ratio of reads to writes of 4:1 (80% of read operations),
the throughput our benchmark code uses is much lower
compared to the raw throughput ZooKeeper can achieve
(over 40,000 according to Figure 5). This is due to clients
waiting on other clients.
6 Related work
ZooKeeper has the goal of providing a service that mit-
igates the problem of coordinating processes in dis-
tributed applications. To achieve this goal, its design uses
ideas from previous coordination services, fault tolerant
systems, distributed algorithms, and file systems.
12

We are not the first to propose a system for the coor-
dination of distributed applications. Some early systems
propose a distributed lock service for transactional ap-
plications [13], and for sharing information in clusters
of computers [19]. More recently, Chubby proposes a
system to manage advisory locks for distributed appli-
cations [6]. Chubby shares several of the goals of Zoo-
Keeper. It also has a file-system-like interface, and it uses
an agreement protocol to guarantee the consistency of the
replicas. However, ZooKeeper is not a lock service. It
can be used by clients to implement locks, but there are
no lock operations in its API. Unlike Chubby, ZooKeeper
allows clients to connect to any ZooKeeper server, not
just the leader. ZooKeeper clients can use their local
replicas to serve data and manage watches since its con-
sistency model is much more relaxed than Chubby. This
enables ZooKeeper to provide higher performance than
Chubby, allowing applications to make more extensive
use of ZooKeeper.
There have been fault-tolerant systems proposed in
the literature with the goal of mitigating the problem of
building fault-tolerant distributed applications. One early
system is ISIS [5]. The ISIS system transforms abstract
type specifications into fault-tolerant distributed objects,
thus making fault-tolerance mechanisms transparent to
users. Horus [30] and Ensemble [31] are systems that
evolved from ISIS. ZooKeeper embraces the notion of
virtual synchrony of ISIS. Finally, Totem guarantees total
order of message delivery in an architecture that exploits
hardware broadcasts of local area networks [22]. Zoo-
Keeper works with a wide variety of network topologies
which motivated us to rely on TCP connections between
server processes and not assume any special topology or
hardware features. We also do not expose any of the en-
semble communication used internally in ZooKeeper.
One important technique for building fault-tolerant
services is state-machine replication [26], and Paxos [20]
is an algorithm that enables efficient implementations
of replicated state-machines for asynchronous systems.
We use an algorithm that shares some of the character-
istics of Paxos, but that combines transaction logging
needed for consensus with write-ahead logging needed
for data tree recovery to enable an efficient implementa-
tion. There have been proposals of protocols for practical
implementations of Byzantine-tolerant replicated state-
machines [7, 10, 18, 1, 28]. ZooKeeper does not assume
that servers can be Byzantine, but we do employ mech-
anisms such as checksums and sanity checks to catch
non-malicious Byzantine faults. Clement et al. dis-
cuss an approach to make ZooKeeper fully Byzantine
fault-tolerant without modifying the current server code
base [9]. To date, we have not observed faults in produc-
tion that would have been prevented using a fully Byzan-
tine fault-tolerant protocol. [29].
Boxwood [21] is a system that uses distributed lock
servers. Boxwood provides higher-level abstractions to
applications, and it relies upon a distributed lock service
based on Paxos. Like Boxwood, ZooKeeper is a com-
ponent used to build distributed systems. ZooKeeper,
however, has high-performance requirements and is used
more extensively in client applications. ZooKeeper ex-
poses lower-level primitives that applications use to im-
plement higher-level primitives.
ZooKeeper resembles a small file system, but it only
provides a small subset of the file system operations
and adds functionality not present in most file systems
such as ordering guarantees and conditional writes. Zoo-
Keeper watches, however, are similar in spirit to the
cache callbacks of AFS [16].
Sinfonia [2] introduces mini-transactions, a new
paradigm for building scalable distributed systems. Sin-
fonia has been designed to store application data,
whereas ZooKeeper stores application metadata. Zoo-
Keeper keeps its state fully replicated and in memory for
high performance and consistent latency. Our use of file
system like operations and ordering enables functionality
similar to mini-transactions. The znode is a convenient
abstraction upon which we add watches, a functionality
missing in Sinfonia. Dynamo [11] allows clients to get
and put relatively small (less than 1M) amounts of data in
a distributed key-value store. Unlike ZooKeeper, the key
space in Dynamo is not hierarchal. Dynamo also does
not provide strong durability and consistency guarantees
for writes, but instead resolves conflicts on reads.
DepSpace [4] uses a tuple space to provide a Byzan-
tine fault-tolerant service. Like ZooKeeper DepSpace
uses a simple server interface to implement strong syn-
chronization primitives at the client. While DepSpace’s
performance is much lower than ZooKeeper, it provides
stronger fault tolerance and confidentiality guarantees.
7 Conclusions
ZooKeeper takes a wait-free approach to the problem of
coordinating processes in distributed systems, by expos-
ing wait-free objects to clients. We have found Zoo-
Keeper to be useful for several applications inside and
outside Yahoo!. ZooKeeper achieves throughput val-
ues of hundreds of thousands of operations per second
for read-dominant workloads by using fast reads with
watches, both of which served by local replicas. Al-
though our consistency guarantees for reads and watches
appear to be weak, we have shown with our use cases that
this combination allows us to implement efficient and
sophisticated coordination protocols at the client even
though reads are not precedence-ordered and the imple-
mentation of data objects is wait-free. The wait-free
property has proved to be essential for high performance.
13

Although we have described only a few applications,
there are many others using ZooKeeper. We believe such
a success is due to its simple interface and the powerful
abstractions that one can implement through this inter-
face. Further, because of the high-throughput of Zoo-
Keeper, applications can make extensive use of it, not
only course-grained locking.
Acknowledgements
We would like to thank Andrew Kornev and Runping Qi
for their contributions to ZooKeeper; Zeke Huang and
Mark Marchukov for valuable feedback; Brian Cooper
and Laurence Ramontianu for their early contributions
to ZooKeeper; Brian Bershad and Geoff V oelker made
important comments on the presentation.
References
[1] M. Abd-El-Malek, G. R. Ganger, G. R. Goodson, M. K. Reiter,
and J. J. Wylie. Fault-scalable byzantine fault-tolerant services.
In SOSP ’05: Proceedings of the twentieth ACM symposium on
Operating systems principles, pages 59–74, New York, NY , USA,
2005. ACM.
[2] M. Aguilera, A. Merchant, M. Shah, A. Veitch, and C. Karamano-
lis. Sinfonia: A new paradigm for building scalable distributed
systems. In SOSP ’07: Proceedings of the 21st ACM symposium
on Operating systems principles , New York, NY , 2007.
[3] Amazon. Amazon simple queue service. http://aws.
amazon.com/sqs/, 2008.
[4] A. N. Bessani, E. P. Alchieri, M. Correia, and J. da Silva Fraga.
Depspace: A byzantine fault-tolerant coordination service. In
Proceedings of the 3rd ACM SIGOPS/EuroSys European Systems
Conference - EuroSys 2008, Apr. 2008.
[5] K. P. Birman. Replication and fault-tolerance in the ISIS system.
In SOSP ’85: Proceedings of the 10th ACM symposium on Oper-
ating systems principles, New York, USA, 1985. ACM Press.
[6] M. Burrows. The Chubby lock service for loosely-coupled dis-
tributed systems. InProceedings of the 7th ACM/USENIX Sympo-
sium on Operating Systems Design and Implementation (OSDI) ,
2006.
[7] M. Castro and B. Liskov. Practical byzantine fault tolerance and
proactive recovery. ACM Transactions on Computer Systems ,
20(4), 2002.
[8] T. Chandra, R. Griesemer, and J. Redstone. Paxos made live: An
engineering perspective. In Proceedings of the 26th annual ACM
symposium on Principles of distributed computing (PODC), Aug.
2007.
[9] A. Clement, M. Kapritsos, S. Lee, Y . Wang, L. Alvisi, M. Dahlin,
and T. Riche. UpRight cluster services. In Proceedings of the 22
nd ACM Symposium on Operating Systems Principles (SOSP) ,
Oct. 2009.
[10] J. Cowling, D. Myers, B. Liskov, R. Rodrigues, and L. Shira. Hq
replication: A hybrid quorum protocol for byzantine fault toler-
ance. In SOSP ’07: Proceedings of the 21st ACM symposium on
Operating systems principles, New York, NY , USA, 2007.
[11] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lak-
shman, A. Pilchin, S. Sivasubramanian, P. V osshall, and W. V o-
gels. Dynamo: Amazons highly available key-value store. In
SOSP ’07: Proceedings of the 21st ACM symposium on Operat-
ing systems principles, New York, NY , USA, 2007. ACM Press.
[12] J. Gray, P. Helland, P. O’Neil, and D. Shasha. The dangers of
replication and a solution. In Proceedings of SIGMOD ’96, pages
173–182, New York, NY , USA, 1996. ACM.
[13] A. Hastings. Distributed lock management in a transaction pro-
cessing environment. In Proceedings of IEEE 9th Symposium on
Reliable Distributed Systems, Oct. 1990.
[14] M. Herlihy. Wait-free synchronization. ACM Transactions on
Programming Languages and Systems, 13(1), 1991.
[15] M. Herlihy and J. Wing. Linearizability: A correctness condi-
tion for concurrent objects. ACM Transactions on Programming
Languages and Systems, 12(3), July 1990.
[16] J. H. Howard, M. L. Kazar, S. G. Menees, D. A. Nichols,
M. Satyanarayanan, R. N. Sidebotham, and M. J. West. Scale
and performance in a distributed file system. ACM Trans. Com-
put. Syst., 6(1), 1988.
[17] Katta. Katta - distribute lucene indexes in a grid. http://
katta.wiki.sourceforge.net/, 2008.
[18] R. Kotla, L. Alvisi, M. Dahlin, A. Clement, and E. Wong.
Zyzzyva: speculative byzantine fault tolerance. SIGOPS Oper .
Syst. Rev., 41(6):45–58, 2007.
[19] N. P. Kronenberg, H. M. Levy, and W. D. Strecker. Vaxclus-
ters (extended abstract): a closely-coupled distributed system.
SIGOPS Oper . Syst. Rev., 19(5), 1985.
[20] L. Lamport. The part-time parliament. ACM Transactions on
Computer Systems, 16(2), May 1998.
[21] J. MacCormick, N. Murphy, M. Najork, C. A. Thekkath, and
L. Zhou. Boxwood: Abstractions as the foundation for storage
infrastructure. In Proceedings of the 6th ACM/USENIX Sympo-
sium on Operating Systems Design and Implementation (OSDI) ,
2004.
[22] L. Moser, P. Melliar-Smith, D. Agarwal, R. Budhia, C. Lingley-
Papadopoulos, and T. Archambault. The totem system. In Pro-
ceedings of the 25th International Symposium on Fault-Tolerant
Computing, June 1995.
[23] S. Mullender, editor. Distributed Systems, 2nd edition . ACM
Press, New York, NY , USA, 1993.
[24] B. Reed and F. P. Junqueira. A simple totally ordered broad-
cast protocol. In LADIS ’08: Proceedings of the 2nd Workshop
on Large-Scale Distributed Systems and Middleware , pages 1–6,
New York, NY , USA, 2008. ACM.
[25] N. Schiper and S. Toueg. A robust and lightweight stable leader
election service for dynamic systems. In DSN, 2008.
[26] F. B. Schneider. Implementing fault-tolerant services using the
state machine approach: A tutorial. ACM Computing Surveys ,
22(4), 1990.
[27] A. Sherman, P. A. Lisiecki, A. Berkheimer, and J. Wein. ACMS:
The Akamai configuration management system. In NSDI, 2005.
[28] A. Singh, P. Fonseca, P. Kuznetsov, R. Rodrigues, and P. Ma-
niatis. Zeno: eventually consistent byzantine-fault tolerance.
In NSDI’09: Proceedings of the 6th USENIX symposium on
Networked systems design and implementation , pages 169–184,
Berkeley, CA, USA, 2009. USENIX Association.
[29] Y . J. Song, F. Junqueira, and B. Reed. BFT for the
skeptics. http://www.net.t-labs.tu-berlin.de/
˜petr/BFTW3/abstracts/talk-abstract.pdf.
[30] R. van Renesse and K. Birman. Horus, a flexible group com-
munication systems. Communications of the ACM , 39(16), Apr.
1996.
[31] R. van Renesse, K. Birman, M. Hayden, A. Vaysburd, and
D. Karr. Building adaptive systems using ensemble. Software
- Practice and Experience , 28(5), July 1998.
14

6.5840 Project

6.5840 - Spring 2026

6.5840 Project

Proposals due:
    Mar 20 23:59

Code and write-up due:
    May 8 23:59

Presentations:
    May 12 in class

Introduction

You can either do a final project based on your own ideas,
or Lab 5.

If you want to do a project, you must get our approval for your idea
in advance. You must form a group of 2 or 3 6.5840 students to
collaborate on the project. At the end of the term you'll turn in your
code and a short write-up describing the design and implementation of
your project, and make a short in-class presentation about your work.
We'll post the project write-ups and code on the course web site.

Your project should be something interesting and challenging that's
closely related to 6.5840 core topics, such as fault tolerance.
The project must involve at least as much effort as Lab 5.
Below
you'll find some half-baked ideas that we think could turn into
interesting projects, but we haven't given them too much thought.

Deliverables

There are four concrete steps to the final project, as follows:

Form a group and decide on the project you would like to work on.
Feel free to use Piazza to find group members and discuss ideas.
Course staff will be happy to discuss project ideas via e-mail or in
person.

Flesh out the exact problem you
will be addressing and how you will go about solving it.
By the proposal deadline, you must
submit a proposal (less than a page) describing: your group members
list, the problem you want to address, how you plan to address it,
and what are you proposing to specifically design and implement.
Submit your proposal to Gradescope (to the assignment called "Final Project Proposal")
We'll tell you whether we approve, or not, and give you feedback.

Execute your project: design and build something neat!

Write a document describing the design and implementation of your project,
and turn it in along with your project's code by the final deadline.
The document should be about 3 pages of text that helps us understand what
problem you solved, and what your code does.  You can either send the code to
the staff list or provide a link to an repository (e.g., on GitHub) in your
writeup.  The code and writeups will be posted online after the end of the
semester.

Prepare a short presentation about the work that you have done for
your final project, and deliver it during the last class meeting.

Half-baked project ideas

You should feel free to propose your own project idea. If you'd like
some starting points, here are some topics that might (or might not)
be worth thinking about.

 Re-implement one of the systems described in the papers discussed in 6.5840.

 Build a high-performance Raft implementation, changing
the design as needed.

 Build a distributed, decentralized, fault-tolerant Reddit.

 Build a system for making Node.js applications fault-tolerant,
perhaps using some form of replicated execution.

 Add cross-shard atomic transactions to Lab 5, using two-phase commit
and/or snapshots.

 Build a data-flow processing system in the style of Google FlumeJava
or Spark or Naiad.

 Build a system with asynchronous replication (like Dynamo or
Ficus or Bayou). Perhaps add stronger consistency (as in COPS
or Walter or Lynx).

 Build a file synchronizer (like
  Unison or
  Tra).

 Build a coherent caching system for use by web sites (a bit
like memcached), perhaps along the lines of
TxCache.

 Build a distributed cooperative web cache, perhaps along
the lines of
Firecoral or
Maygh.

 Build a collaborative editor like EtherPad, using
eventually-consistent or CRDT primitives.

 Use a block-chain to build something other than a crypto-currency.

 Build a fault-tolerant and/or sharded file service.

 Build a
distributed shared memory (DSM) system, to make it possible to run existing
parallel code intended for a single multi-core machine, but on a cluster of
machines.

 Build a distributed block store in the style of Amazon EBS or FAB.
Maybe you can get standard operating systems
to talk to you network virtual disk using iSCSI or
Linux's NBD (network block device).

 Build a geo-replicated storage system, like Dynamo or COPS,
perhaps providing something useful and/or efficient in the the way
of transactions or consistency.

 Use modern high-speed NIC features (e.g. RDMA or DPDK) to build a
high-speed service, perhaps with replication or transactions.

 Use modern fast non-volatile storage (e.g. Intel Optane) to
simplify the design of a fault-tolerant system.

 Build a fault-tolerance framework that's easier than Raft to
layer service code on top of.

 Figure how to say something useful about whether applications
really need strictly consistent storage, or what the cost in application
complexity is of having to use storage with weak consistency.

 Build a data-processing system that is good at both big data
(like MapReduce and Spark) and on-line processing (like a key/value
store or SQL database).

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Spring 2026 Paper Questions

6.5840 Spring 2026 Paper Questions

  For each paper, your assignment is two-fold.  Before the start of
  lecture discussing the paper:

Read the paper, and
Submit your answer for each lecture's paper question.

If you have a question about a paper, you may also (optionally)
submit it; we may answer and/or adjust the lecture to answer your question.

Put your answer in a.txt and question in q.txt.
Submit both files via
Gradescope.


   The assigned reading for today is not a paper, but
   the Online Go tutorial.  The assigned
   "question" is the
   Crawler exercise in the
   tutorial.  Also, take a look at
   Go's RPC package, which you
   will use in lab 1.

  The GFS paper is a "classic" paper that describes one of the first
    distributed file systems for data-center applications such as large
    MapReduce jobs.  It touches on many themes of 6.5840: parallel
    performance, fault tolerance, replication, and consistency.  It is
    good systems paper with details from apps all the way to network
    successful.

   GFS has been replaced by something called
    Colossus,
    with the same overall goals, but improvements in master performance
    and fault-tolerance. In addition, many applications within Google have
    switched to more database-like storage systems such as BigTable and
    Spanner. However, much of the GFS design lives on in HDFS, the storage
    system for the Hadoop open-source MapReduce.

  Question:
    Describe a sequence of events that would result in a client reading stale
    data from the Google File System.





    How does VM FT handle network partitions?  That
    is, is it possible that if the primary and the backup end up in different
    network partitions that the backup will become a primary too and the system
    will run with two primaries?



    Consider the following code from the "incorrect synchronization" examples:



var a string
var done bool

func setup() {
    a = "hello, world"
    done = true
}

func main() {
    go setup()
    for !done {
    }
    print(a)
}

  Using the synchronization mechanisms of your choice, fix this code so it
  is guaranteed to have the intended behavior according to the Go language
  specification. Explain why your modification works in terms of the
  happens-before relation.

Paxos Made Simple
Suppose that the acceptors are A, B,
and C. A and B are also
proposers.
How does Paxos
ensure that the following sequence of events can't happen?
What actually happens, and which value is ultimately chosen?

 A sends prepare requests with proposal number 1, and gets positive responses
  from A, B, and C.
 A sends accept(1, "X")
  to A and C and gets positive responses from both.
  Because a majority accepted, A thinks
  that "X" has been chosen.
  However, A crashes before sending an accept to B.
B sends prepare messages with proposal number 2, and gets positive responses
  from B and C.
B sends accept(2, "Y") messages
  to B and C and gets positive responses
  from both, so B thinks that "Y" has been chosen.

  The Raft paper describes a consensus algorithm, including many
    details that are needed to build replication-state machine
    applications. The paper is also the topic of several of the 6.5840
    labs. The important sections are 2, 5, 7, and 8.

  The paper positions itself as a
    better Paxos,
    but another way to look at Raft is that it solves a bigger problem
    than Paxos. To build a real-world replicated service, the replicas
    need to agree on an indefinite sequence of values (the client
    commands), and they need ways to efficiently recover when servers
    crash and restart or miss messages. People have built such systems
    with Paxos as the starting point (e.g., Google's Chubby and Paxos
    Made Live papers, and ZooKeeper/ZAB). There is also a protocol
    called Viewstamped Replication; it's a good design, and similar to
    Raft, but the paper about it is hard to understand.

  These real-world protocols are complex, and (before Raft) there
    was not a good introductory paper describing how they work. The Raft
    paper, in contrast, is relatively easy to read and fairly detailed.

  Question:
    Suppose we have the scenario shown in the Raft paper's Figure 7: a cluster of seven
    servers, with the log contents shown. The first server crashes (the one
    at the top of the figure), and
    cannot be contacted. A leader election ensues.
    For each of the servers marked (a), (d), and (f), could that server
    be elected?
    If yes, which servers would vote for it? If no, what specific
    Raft mechanism(s) would prevent it from being elected?

The lecture today covers effective approaches to debugging the 6.5840 labs.
Submit a question about the testing and debugging process of your lab 1
solution: for example, something you struggled with, something you aren't
confident about, or something you're curious about.

Could a received InstallSnapshot RPC cause the state machine to go
backwards in time? That is, could step 8 in Figure 13 cause the state
machine to be reset so that it reflects fewer executed operations? If
yes, explain how this could happen. If no, explain why it can't happen.


    Please read the paper's Appendices. In Spinnaker a leader to responds to a client request after the leader
    and one follower have written a log record for the request on
    persistent storage. Why is this sufficient to guarantee strong
    consistency even after the leader or the one follower fail?



   (This paper relies on Zookeeper, which we will read later.)



    The lecture today is a Q&A session about Lab 3A/3B.
    Submit a question about the lab: for
    example, something you wondered about while doing the lab,
    something you didn't understand, a memorable bug, or just anything.


    With a linearizable key/value storage system, could two
    clients who issue get() requests for the same key
    at the same time receive different values? Explain why
    not, or how it could occur.



    One use of Zookeeper is as a fault-tolerant
    lock service (see the section "Simple locks" on page 6).  Why isn't possible
    for two clients to acquire the same lock?  In particular, how does
    Zookeeper decide if a client has failed and it can give the client's locks to
    other clients?



    Item 4 in Section 2.3 says that, if a client read request arrives
    and the latest version is dirty, the node should ask the tail for
    the latest committed version. Suppose, instead, that the node
    replied with its most recent clean version (ignoring any dirty
    version and not sending a version query to the tail). This change
    would cause reads to reflect the most recent committed write that
    the node is aware of. Explain how this could lead to violations of
    linearizability -- or violations of the paper's goal of strong
    consistency.



  Suppose Chain Replication replied to update requests
  from the head, as soon as the next chain server said it received the
  forwarded update, instead of responding from the tail. Explain how
  that could cause Chain Replication to produce results that
  are not linearizable.



The second paragraph of Section 4.1 says "The runtime state maintained
by the database lets us use single segment reads rather than quorum
reads..." What runtime state does the database need to maintain in
order to avoid having to read from a quorum?


Russ Cox is one of the leads on the Go project. What do you like best about Go?
Why?  Would you want to change anything in the language? If so, what and why?

6.033 Book.
Read just these parts of Chapter 9: 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3. The
last two sections (on two-phase locking and distributed two-phase
commit) are the most important.
The Question: describe a situation where Two-Phase Locking yields higher performance than Simple Locking.

Spanner
Suppose a Spanner server's TT.now() returns correct
information, but the uncertainty is large. For example,
suppose the absolute time is 10:15:30, and TT.now()
returns the interval [10:15:20,10:15:40]. That interval
is correct in that it contains the absolute time, but the
error bound is 10 seconds.
See Section 3 for an explanation of TT.now().
What bad effect will a large error bound have on Spanner's operation?
Give a specific example.

At the end of Section 6.2, the
Chardonnay
paper says that a snapshot read transaction can be made
linearizable by waiting for the epoch to advance by one.
By "linearizable" the paper means that the transaction
observes the results of all transactions that completed
before it started.
Why does waiting make the transaction linearizable?

No compromises: distributed
transactions with consistency, availability, and performance:
Suppose there are two FaRM transactions that both increment the same
object. They start at the same time and see the same initial value for
the object. One transaction completely finishes committing (see
Section 4 and Figure 4). Then the second transaction starts to commit.
There are no failures. What is the evidence that FaRM will use to
realize that it must abort the second transaction? At what point in
the Section 4 / Figure 4 protocol will FaRM realize that it must
abort?


Amazon DynamoDB: A scalable,
  predictably performant, and fully managed NoSQL Database service
  To avoid interference, DynamoDB may throttle a customer's
  application if it exceeds its read or write capacity, which can
  cause the application to become unavailable. How does DynamoDB avoid
  that a single hot partition of a database table can cause an
  application to be unavailable?



Distributed Transactions at
Scale in Amazon DynamoDB.
Section 3.3 says, in the third bullet,
"The transaction's timestamp is greater than the
item's timestamp indicating when it was last written."
What's an example of an incorrect outcome that this check helps rule out?


MDCC
Come back later for this year's question.

Ownership: A Distributed Futures System for
  Fine-Grained Tasks.
If C() in figure 6(a) is as follows:

  def C(x):
    z = D(X)
    return Get(z)   # return value of future z

Suppose the node than runs D fails before finishing, which node would
initiates the re-execution of D()?

Resilient Distributed Datasets: A
  Fault-Tolerant Abstraction for In-Memory Cluster Computing
What applications can Spark support well that MapReduce/Hadoop cannot support?

Naiad: A Timely Dataflow System:
Consider the data-flow graph in Figure 3, and assume that the vertices are
instantiated as follows:

  A is a distinct-by-key operator, which for inputs of (key,
      value) only emits the first (key, value) record
      with each distinct key.
  B is a multiply operator, which multiplies the value by two
      (i.e., for (key, value), it emits (key, value * 2).
  C is a conditional operator, which sends its output to
      E if the input value is greater than 10, and sends it
      to F otherwise.
  D is an aggregation operator that sums all values irrespective
      of key.

Now assume that we introduce two records in epoch e = 1 at the
In vertex: (a, 2) and (b, 6); and one record in
e = 2: (a, 5).

Write down the timestamp changes that the records experience as they flow
through the graph, e.g., (a, 2): at A, t = (1, []); at B, t = ....
You may omit vertices that do not modify the timestamp.

Informally explain when E.OnNotify((1, [])) will be called.
You do not need to step through the details of the pointstamp protocol.

Parameter Server:
The parameter server model was developed for running machine-learning
algorithms like sparse logistic regression (§ 5.1).

What do you expect the bottleneck – that is, the most loaded –
resource (e.g., CPU, memory, network) to be at the workers and at
the servers?

What resource utilization would you likely observe if using Spark's
logistic regression (see §3.2.1 in the Spark paper) instead
of the parameter server for the experiment in §5.1?

Frangipani: A Scalable
Distributed File System: Suppose a server modifies an i-node,
appends the modification to its log, then another server modifies the
same i-node, and then the first server crashes. The recovery system
will see the i-node modification in the crashed server's log, but
should not apply that log entry to the i-node, because that would
un-do the second server's change. How does Frangipani avoid or cope
with this situation?

Memcache at Facebook.
Section 3.3 implies that a client that writes data does not
delete the corresponding key from the Gutter servers, even
though the client does try to delete the key from the ordinary
Memcached servers (Figure 1). Explain why it would be a bad
idea for writing clients to delete keys from Gutter servers.

COPS. The last sentence in Section 4.3
says a client clears its context after a put, replacing the context
with just the put. The text observes "This put depends on all previous
key-version pairs and thus is nearer than them." Why does clearing the
context and replacing it with just the put make sense? You might think
that the client's subsequent puts would need to carry along the
dependency information about previous gets. What entity ultimately
uses the context information, and why does it not need the information
about gets before the last put?

Explain why it's important that all viewers of a Certificate
Transparency log see the same content. What attacks would be possible
if a malicious log server was able to cause some readers of its log to
see one sequence of records, and other readers to see a different set
of records? What mechanisms ensure that all viewers see the same log
content?

Why is it important that Blockstack names be unique, human-readable,
and decentralized? Why is providing all three properties hard?

Managing Update Conflicts in Bayou
  Suppose we build a distributed filesystem using Bayou, and the
  system has a copy operation. Initially, file A contains "foo"
  and file B contains "bar". On one node, a user copies file A
  to file B, overwriting the old contents of B. On another node, a
  user copies file B to file A. After both operations are committed, we
  want both files to contain "foo" or for both files to contain
  "bar". Sketch a dependency check and merge procedure for the
  copy operation that makes this work. How does Bayou ensure that
  all the nodes agree about whether A and B contain "foo" or
  "bar"?

For the design described in Chord: a scalable
  peer-to-peer lookup service for Internet applications, if two Chord nodes
  x and y are located nearby each other in the Internet (e.g., they are both in
  the same datacenter), is the number of Chord hops small (say 1 or 2 instead of
  log N) when x is looking up a key that is stored on node y?

Dynamo
Suppose Dynamo server S1 is perfectly healthy with a working network
connection. By mistake, an administrator instructs server S2 to remove
S1 using the mechanisms described in 4.8.1 and 4.9. It takes a while
for the membership change to propagate from S2 to the rest of the
system (including S1), so for a while some clients and servers will
think that S1 is still part of the system. Will Dynamo operate
correctly in this situation? Why, or why not?

The Zanzibar paper says,
in the fourth paragraph of Section 4, that Zanzibar serves
a total of roughly 10 million client requests per second.
Section 4.4 says that the aclservers perform roughly 200 million
in-memory lookups per second. What could be the reason for
the order of magnitude difference?

The paper mentions that decryption increases latency for fetching a
chunk in Section 5.1.  Why encrypt chunks with convergent encryption
in the first place?

Boki ensures that functions appear to execute exactly-once even though
the function may be re-executed because of a machine failure. One
challenge is that if the function updates an external database (e.g.,
updating a row in DynamoDB), this update must happen only once.  How
does Boki achieve this goal in Figure 6(a)?

Secure Untrusted Data Repository (SUNDR).
You only need to read through the end of 3.3.2.
In the simple straw-man, both fetch and modify operations are placed
in the log and signed. Suppose an alternate design that only signs and
logs modify operations. Does this allow a malicious server to break
fetch-modify consistency or fork consistency? Why or why not?

Suppose that we eliminated the pre-prepare phase from the Practical
BFT protocol. Instead, the primary multicasts a PREPARE,v,n,m message
to the replicas, and the replicas multicast COMMIT messages and reply
to the client as before. What could go wrong with this protocol? Give
a short example, e.g., ``The primary sends foo, replicas 1 and 2 reply
with bar...''

Bitcoin
Try to buy something with Bitcoin. It may help to cooperate
with some 6.5840 class-mates, and it may help to start a
few days early.
If you decide to give up, that's OK. Briefly describe your experience.

Ethereum Whitepaper.
Use an Ethereum decentralized application (dapp) and send a transaction to a smart contract.

    You will need an Ethereum wallet. For the purpose of this assignment, MetaMask is convenient and has integration with many dapps.

    You can either acquire "real" (mainnet) Ethereum (e.g. by buying some using Coinbase) or acquire some free Sepolia testnet Ethereum using a "faucet" (you can use Google to find faucet websites). Note: not all dapps will support testnet Ethereum.

    Find a dapp to interact with. You can find lists online, such as the dapp list and DappRadar.

If you decide to give up, that's OK. Briefly describe your experience.

  IronFleet: Consider a Raft
  implementation (written in Dafny) which does not implement the paper's
  election restriction (i.e. followers do not check if the candidate's log is
  up-to-date when voting). Where in Figure 3 could a developer get stuck while
  applying the IronFleet methodology to this implementation?

Experiences with a Distributed,
 Scalable, Methodological File System: AnalogicFS. In many ways,
 this experiences paper raises more questions than it answers. Please
 answer one of the following questions, taking into consideration the
 rich history of AnalogicFS and the spirit in which the paper was
 written:

a) The analysis of A* search shown in Figure 1 claims to be an
introspective visualization of the AnalogicFS methodology; however,
not all decisions are depicted in the figure. In particular, if I
<= P, what should be the next node explored such that all assumptions
in Section 2 still hold? Show your work.

b) Despite the authors' claims in the introduction that AnalogicFS was
developed to study SCSI disks (and their interaction with lambda
calculus), the experimental setup detailed in Section 4.1 involves
decommissioned Gameboys instead, which use cartridge-based, Flash-like
memory. If the authors had used actual SCSI disks during the
experiments, how exactly might have their results changed
quantitatively?

c) AnalogicFS shows rather unstable multicast algorithm popularity
(Figure 5), especially compared with some of the previous systems
we've read about in 6.5840. Give an example of another system that
would have a more steady measurement of popularity pages, especially
in the range of 0.1-0.4 decibels of bandwidth.

d) For his 6.5840 project, Ben Bitdiddle chose to build a
variant of Lab 5
that faithfully emulates the constant expected seek time across
LISP machines, as AnalogicFS does. Upon implementation, however, he
immediately ran into the need to cap the value size to 400 nm, rather
than 676 nm. Explain what assumptions made for the AnalogicFS
implementation do not hold true for Lab 5, and why that changes the
maximum value size.

Questions or comments regarding 6.5840? Send e-mail to 6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Exams

6.5840 - Spring 2026

Past Exams

Here are some exams from past years. The course
number was 6.824 in 2022 and earlier.

 2025: Q1
           (with answers)
           Q2
           (with answers)

 2024: Q1
           (with answers)
           Q2
           (with answers)

 2023: Q1
           (with answers)
           Q2
           (with answers)

 2022: Q1
           (with answers)
           Q2
           (with answers)

 2021: Q1
           (answers)
           Q2 (w. answers)

 2020: Q1 (with answers) |
  Q2 (with answers)
  Q2 (without)

 2018: Q1 (with answers)
           (without) |
           Q2 (with answers)
           (without)

 2017: Q1 (with answers)
           (without) |
           Q2 (with answers)
           (without)

 2016: Q1 (with answers)
           (without) |
           Q2 (with answers)
           (without)

 2015: Q1 (with answers) |
           Q2 (with answers)

 2014: Q1 (with answers) |
           Q2 (with answers)

 2013: Q1 (with answers) |
           Q2 (with answers)

 2012: Q1 (with answers) |
           Q2 (with answers)

 2011: Q1 (with answers) |
           Q2 (with answers)

 2010: Q1 (with answers) |
           Q2 (with answers)

 2009: Q1 (with answers) |
           Q2 (with answers)

 2007: Q1 (with answers) |
           Q2 (with answers)

 2006: Q1 |
           Q2

 2005: Q1 (with answers) |
           Q2 (with answers)

 2004: Q1 (with answers) |
           Q2 (with answers)

 2002: Q1 (with answers)

Questions or comments regarding 6.5840?  Send e-mail to
6824-staff@lists.csail.mit.edu.

Top //
6.824 home //

6.5840 Schedule: Spring 2026

6.5840 Schedule: Spring 2026

54-100, TR1-2:30

Here is the tentative schedule of lectures and due dates.  The lecture
notes and paper questions for future dates are copies from previous
years, and may change.



   MondayTuesday
   WednesdayThursday
   Friday





  feb 2

    First day of classes
  feb 3

    LEC 1 (rtm): Introduction

    Preparation:  Read MapReduce (2004)

    Assigned: Lab 1: MapReduce
  feb 4
  feb 5

    LEC 2 (fk): RPC and Threads, crawler.go, kv.go, vote examples

    Preparation:  Do Online Go tutorial  (FAQ) (Question)
  feb 6


  feb 9
  feb 10

    LEC 3 (fk): GFS

    Preparation:  Read GFS (2003) (FAQ) (Question)

    Assigned: Lab 2: Key/Value server
  feb 11
  feb 12

    LEC 4 (rtm): Paxos pseudo-code FAQ

    Preparation:  Read Paxos (Question)
  feb 13

    DUE: Lab 1. All labs are due at 11:59pm.


  feb 16

    President's day
  feb 17

    Assigned: Lab 3: Raft

    Monday schedule
  feb 18
  feb 19

    LEC 5 (guest lecture): (Russ Cox of Google/Go) Go patterns

    Preparation:  Read The Go Programming Language and Environment (FAQ) (Question)
  feb 20

    DUE: Lab 2


  feb 23
  feb 24

    Snow day, no lecture
  feb 25
  feb 26

    LEC 6 (fk): Fault Tolerance: Raft (1)

    Preparation:  Read Raft (extended) (2014), to end of Section 5  (FAQ) (Question)
  feb 27

    DUE: Lab 3A


  mar 2
  mar 3

    LEC 7 (fk): Fault Tolerance: Raft (2)

    Preparation:  Read Raft (extended) (2014), Section 7 to end (but not Section 6)  (FAQ) (Question)

    Assigned: Final Project
  mar 4
  mar 5

    LEC 8 (rtm): Consistency and Linearizability

    Preparation:  Linearizability (only through Section 3.1) (FAQ) (Question)
  mar 6

    DUE: Lab 3B

    ADD DATE


  mar 9
  mar 10

    LEC 9 (fk): Zookeeper

    Preparation:  Read ZooKeeper (2010)  (FAQ) (Question)

    Assigned: Lab 4: KV Raft
  mar 11
  mar 12

    LEC 10 (fk): Q&A Lab 3A+B

    Preparation:  (Question)
  mar 13

    DUE: Lab 3C


  mar 16
  mar 17

    LEC 11 (rtm): Distributed Transactions

    Preparation:  Read 6.033 Chapter 9, just 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3 (FAQ) (Question)
  mar 18
  mar 19

    Mid-term Exam, Walker (50-340), 1:00 to 2:30

    Materials: Open book, notes, laptop.

    Scope: Lectures 1 through 11, Labs 1, 2, and 3A-C

    Old Exams
  mar 20

    DUE: Project proposals (if you are doing a project)


  mar 23

    Spring break
  mar 24

    Spring break
  mar 25

    Spring break
  mar 26

    Spring break
  mar 27

    Spring break


  mar 30
  mar 31

    LEC 12 (rtm): Spanner

    Preparation:  Read Spanner (2012) (FAQ) (Question)
  apr 1
  apr 2

    LEC 13 (fk): Chain Replication

    Preparation:  Read CR (2004) (FAQ) (Question)
  apr 3

    DUE: Lab 3D


  apr 6
  apr 7

    LEC 14 (fk): Optimistic Concurrency Control

    Preparation:  Read FaRM (2015)   (FAQ) (Question)

    Assigned: Lab 5: Sharded KV
  apr 8
  apr 9

    LEC 15 (Upamanyu Sharma): Verification of distributed systems

    Preparation:  Read IronFleet (2015) (Question)
  apr 10

    DUE: Lab 4A


  apr 13
  apr 14

    Hacking day, no lecture
  apr 15
  apr 16

    LEC 16 (rtm): Cache Consistency: Memcached at Facebook

    Preparation:  Read Memcached at Facebook (2013)  (FAQ) (Question)
  apr 17

    DUE: Lab 4B+C


  apr 20

    Patriot's day
  apr 21

    LEC 17 (Marc Brooker, on zoom): AWS Lambda

    Preparation:  Read On-demand Container Loading (2023) (Question)

    DROP DATE
  apr 22
  apr 23

    LEC 18 (fk): Ray

    Preparation:  Read Ray (2021) (FAQ) (Question)
  apr 24


  apr 27
  apr 28

    LEC 19 (rtm): Fork Consistency, SUNDR

    Preparation:  Read SUNDR (2004) (through the end of Section 3.3.2) (FAQ) (Question)
  apr 29

    DUE: Lab 5A
  apr 30

    LEC 20 (rtm): Peer-to-peer: Bitcoin

    Preparation:  Read Bitcoin (2008), and summary (FAQ) (Question)
  may 1


  may 4
  may 5

    Hacking day, no lecture
  may 6
  may 7

    LEC 21 (Derek Leung): Byzantine Fault Tolerance, slides

    Preparation:  Read Practical BFT (1999)  (FAQ) (Question)
  may 8

    DUE: Lab 5B+C+D

    DUE: Project reports and code


  may 11
  may 12

    LEC 22: Project demos

    Last day of classes
  may 13
  may 14
  may 15

    Final exam (9a-11am, 34-101)

    Materials: Open book, notes, laptop.

    Scope: Lectures 12 through 21, Labs 3D, 4A-C

    Old Exams

For questions or comments, email 6824-staff@lists.csail.mit.edu.

Back to 6.824 home.