边界、依赖与验收
构建支持动态配置、跨组迁移与分布式事务的分片容错键值服务。
发布 Lab 5;OCC 与重配置并发
用不变量组织 controller 恢复
分片缓存拓扑与热点
基础分片与迁移
恢复、并发 controller 与扩展
实验或 final project:分片 KV 总览
本页可离线完成Handout 完整本土化
你可以选择自拟 final project,或完成本实验。本实验把 key 分片到多组由 Raft 复制的 KV server group(shardgrp)。shard 是 KV 子集,例如首字母 a/b 的 key 可分属不同 shard。每个 shardgrp 只处理若干 shard,多个组并行,使总 Put/Get 吞吐随组数扩展。
客户端通过总体 shardkv Clerk 调用 Get/Put。Clerk 向 Lab 2 的 kvsrv 查询 configuration,得到 shard→group 以及 group→server 列表,再联系正确 group。管理员/测试器通过 controller 调 ChangeConfigTo 添加/删除组并改变归属;controller 对 shardgrp 发 FreezeShard、InstallShard、DeleteShard,并更新 kvsrv 中配置。
挑战是重配置期间仍保持线性一致:任何时刻每 shard 最多一个 group 服务;controller 中途崩溃/分区时,部分 shard 可能搬到一半,新 controller 必须完成旧工作。
这里 configuration 指 shard 分配,不是 Raft membership change。每个 server 永远只属于一个 shardgrp,每组成员集合固定。客户端/服务器间只能用 RPC,不得共享 Go 变量/文件。
Part A 实现基本 controller、shardgrp 与搬迁;B 处理 controller 失败;C 允许并发 controller;D 自选扩展。Lab 5 使用 Lab 2 kvsrv 与 Lab 4 rsm/Raft,Lab 4/5 必须用同一实现。A 可用 late hours,B–D 不可用。
代码布局与启动测试
本页可离线完成Handout 完整本土化
先 git pull。骨架/测试在 src/shardkv1:client.go 是总体 Clerk;shardcfg 计算配置;shardgrp 包含 group clerk/server;shardctrler 包含 ChangeConfigTo 与 Query。
启动:
cd ~/6.5840
git pull
cd src
make RUN="-run 5A" shardkv命令构建 main/kvsrv1d、main/shardgrp1d,并以 15 分钟 timeout、verbose、race 运行 shardkv1 5A。初始从 TestInitQuery5A 开始。
本实验借鉴 FDS、BigTable、Spanner、FAWN、HBase、Rosebud、Spinnaker 的总体形状,但简化了固定 Raft 成员、数据/查询模型等。
Part 5A(一):InitConfig 与 Query
本页可离线完成Handout 完整本土化
shardcfg.ShardConfig 含唯一 Num、shard→gid 映射、gid→复制该组的 server 列表。通常 shard 多于 group,以便细粒度迁移负载。
在 shardctrler/shardctrler.go 实现 InitConfig 与 Query。InitConfig 接收 tester 给的首个配置并存入 Lab 2 kvsrv;Query 从 kvsrv 读取当前配置。
使用 ShardCtrler.IKVClerk 的 Get/Put;ShardConfig.String() 编码成可存字符串,shardcfg.FromString() 解码。
运行:
make RUN="-run TestInitQuery5A" shardkv此步无需 shardgrp,必须通过 TestInitQuery5A。
Part 5A(二):静态 shardgrp 与总体 Clerk
本页可离线完成Handout 完整本土化
从 Lab 4 kvraft 复制所需代码,在 shardkv1/shardgrp/server.go、client.go 实现初版 group。总体 shardkv1/client.go Clerk 用 Query 找 key 的 group,再联系它。
用 shardcfg.Key2Shard(key) 计算 shard;MakeClerk 已收到 ShardCtrler。用配置 GidServers() 得到该 shard 的 group/server,调用 shardgrp.MakeClerk(servers, ck.clnt) 创建 group clerk。
总体 Clerk Put 必须传播 inner shardgrp Put 的 ErrMaybe。首个 group shardcfg.Gid1 创建时初始化拥有全部 shard。
运行:
make RUN="-run TestStaticOneShardGroup5A" shardkv必须通过单 group static test。
Part 5A(三):Freeze→Install→Delete→Publish
本页可离线完成Handout 完整本土化
实现 ChangeConfigTo(old→new),让每组实际 shard 与新配置一致。推荐顺序:源组 Freeze,拒绝该 shard 的 Put;把数据 Install 到目标组;从源删除 frozen shard;最后发布新 current config,让客户端找到新位置。未受影响 shard 继续服务,group 之间无需直接通信。
每配置有唯一 Num。Part A 顺序调用,new Num=old Num+1。网络可延迟/乱序,所以 Freeze/Install/Delete RPC 都携带 Num,shardgrp 为每 shard 记住已见最大 Num,拒绝旧请求。
在 shardctrler 和 shardgrp client/server 实现三 RPC,协议类型在 shardgrp/shardrpc。所有三操作与 Put/Get 一样必须通过 rsm 复制。group 不负责某 shard 时,客户端 Put/Get 返回 ErrWrongGroup,总体 Clerk 重新读配置重试。
可以在 RPC 传整个 map 简化搬迁,但 reply 不能直接引用 server state map:RPC encoder 在 handler 返回后读 map,服务器可能并发修改。必须在锁内复制 map放进 reply。
先通过 JoinBasic/DeleteBasic(添加 group),再支持旧配置有、new 没有的 leaving group,通过 TestJoinLeaveBasic5A。Freeze 后源保存不可变数据直到安全 Delete;目标安装完整数据与保持 Put 至多一次所需的 version/metadata。
Part 5A(四):全套重配置验收
本页可离线完成Handout 完整本土化
运行:
make RUN="-run 5A" shardkv测试覆盖 Init/Query、单 group、加入、删除、基本/大量 join-leave、可靠/不可靠网络、shutdown、部分 shard offline 时其他 shard 继续 progress、一个/多个并发 Clerk 与重配置并发下的线性一致性。
官方示例全套约 243 秒。必须继续服务不受当前配置变更影响的 shard;不能用全局“正在重配置”开关停止整个系统。shardgrp snapshot/restart 后要恢复每 shard 数据、状态与最大 Num。
ChangeConfigTo Part A 无故障应总能完成;ErrWrongGroup 是客户端刷新路由信号,不是返回给最终用户的稳定结果。
Part 5B:controller 失败后的可恢复重配置
本页可离线完成Handout 完整本土化
controller 是管理员启动的短命令,移动完成后退出;它可能中途失败或失联。tester 分区旧 controller 后启动新 controller,并调用 InitController;新实例必须完成中断的 ChangeConfigTo。
推荐在 kvsrv 保存 current 与 next 两个配置。开始重配置先耐久写 next;所有 shard move 完成后,才把 next 提升为 current。InitController 启动时若发现 next.Num>current.Num,就恢复并完成必要搬迁。
恢复 controller 会重复 Freeze/Install/Delete;shardgrp 使用每 shard Num 识别重复/旧 RPC,使操作幂等。不能依赖旧 controller 本地内存知道进度;current/next 与 group 状态是恢复证据。
运行:
make RUN="-run 5B" shardkv必须通过 group down 时 join/leave 与 recover controller。恢复逻辑可放 shardctrler/shardctrler.go 的 InitController。
Part 5C:并发 controller 与条件发布 next
本页可离线完成Handout 完整本土化
旧 controller 分区时 tester 会启动新实例,因此多个 controller 可能并发向 shardgrp 和 kvsrv 发 RPC。Part A 已用 Num fence group 操作;多个恢复者重复同一配置通常无害。
剩余难点是同一 current.Num 上,两个 controller 各算出不同但 Num 相同的 next 配置。必须保证只有一个能发布 next。利用 kvsrv key version 与 versioned Put 做条件更新;赢家开始 ChangeConfigTo,其他调用发现版本冲突后直接返回,不得覆盖 next。
运行并发可靠测试:
make RUN="-run TestConcurrentReliable5C" shardkv以及不可靠锁竞争:
make RUN="-run TestAcquireLockConcurrentUnreliable5C" shardkvtester 的 concurCtrler 展示并发方式。随后把 Part B recovery 与新 controller 结合;若旧实例在 ChangeConfigTo 中恢复,所有更新已按 Num fence,通常无需额外 group 代码。
运行:
make RUN="-run Partition" shardkv覆盖 join 中分区、leased leadership 的可靠/不可靠和高负载情形。通过后你完成了可扩展、多 group、可重配置、controller 容错的高可用分片 KV。最后回跑所有测试;Gradescope 除 5C 外还会重跑 Lab 3A–D 与 4A–C:
make raft1
make kvraft1
make shardkvPart 5D:自选扩展(完整选题列表)
本页可离线完成Handout 完整本土化
最后一部分可任意扩展,并为扩展自写测试。实现下列之一或自拟;在 extension.md 用一段文字描述并上传 Gradescope。较难开放题可与一位同学合作。
- 让 tester 的 controller 配置存储从 kvsrv 换成 kvraft;测试一个 kvraft peer down 时仍可 Query/更新。tester 代码分布于
src/kvtest1、src/shardkv1、src/tester1。 - 为 kvsrv 实现 exactly-once Put/Get(参考往年 Lab 2 dropped messages),可移植 2024 测试;kvraft 也实现 exactly-once。
- 为 kvsrv 增加
Range(low,high);不要只遍历 map,使用 B-tree 等支持 range 的结构,并写能区分两者的测试。 - 优化 kvraft leader,不让 Get 进入 rsm;实现 Raft 论文 Section 8 的 lease 线性一致读。保留现有测试,并加入 RPC 数/速度对比与 term 切换必须等 lease 到期的测试。
- 在 kvraft 支持多个 Put/Get 原子事务;事务可取代 versioned Put。可参考 etcd transaction API并写测试。
- 在 shardkv 支持跨 shard 事务,需要 2PC 与 2PL,并写测试。
这些是扩展而非必做评分路径;先冻结稳定主线。
最终提交检查
本页可离线完成Handout 完整本土化
提交前最后运行全部测试,并再次确认:
make raft1
make kvraft1
make shardkvPart D 另上传 extension.md。遵守课程协作与私有代码政策;B–D 不可使用 late hours。
完整官方 Handout 与配套资料
以下是本讲对应官方材料的可搜索离线文本。中文精读负责解释;资料附录保留原始细节、例子、问答与代码,不以摘要替代原文。
网页讲义labs/lab-shard1.html785 行 · 4,053 词 · 完整收录
6.5840 Lab 5: Sharded Key/Value Service
6.5840 - Spring 2026
6.5840 Lab 5: Sharded Key/Value Service
Collaboration policy //
Submit lab //
Setup Go //
Guidance //
Piazza
Introduction
You can either do a final project based
on your own ideas, or this lab.
In this lab you'll build a key/value storage system
that "shards," or partitions, the keys over a set of
Raft-replicated key/value server groups
(shardgrps). A shard is a subset of the
key/value pairs; for example, all the keys starting
with "a" might be one shard, all the keys starting
with "b" another, etc. The reason for sharding is
performance. Each shardgrp
handles puts and gets for just a few of
the shards, and the shardgrps operate in parallel;
thus total system throughput (puts and gets per unit
time) increases in proportion to the number of
shardgrps.
The sharded key/value service has the components shown
above. Shardgrps (shown with blue squares) store
shards with keys: shardgrp 1 holds a shard storing key
"a", and shardgrp 2 holds a shard storing key
"b". Clients of the sharded key/value service interact
with the service through a clerk (shown with a green
circle), which implements Get
and Put methods. To find the shardgrp for a
key passed to Put/Get, the clerk
gets the configuration from the kvsrv (shown with a
black square), which you implemented in Lab 2. The
configuration (not shown) describes the mapping from
shards to shardgrps (e.g., shard 1 is served by
shardgrp 3).
An administrator (i.e., the tester) uses another
client, the controller (shown with a purple circle),
to add/remove shardgrps from the cluster and
update which shardgrp should serve a shard. The
controller has one main
method: ChangeConfigTo, which takes as
argument a new configuration and changes the system
from the current configuration to the new
configuration; this involves moving shards to new
shardgrps that are joining the system and moving
shards away from shardgrps that are leaving the system. To
do so the controller 1) makes RPCs
(FreezeShard, InstallShard,
and DeleteShard) to shardgrps, and 2) updates the
configuration stored in kvsrv.
The reason for the controller is that a sharded
storage system must be able to shift shards among
shardgrps. One reason is that some shardgrps may
become more loaded than others, so that shards need to
be moved to balance the load. Another reason is that
shardgrps may join and leave the system: new shardgrps
may be added to increase capacity, or existing
shardgrps may be taken offline for repair or
retirement.
The main challenges in this lab will be ensuring
linearizability of
Get/Put operations while handling 1)
changes in the assignment of shards to shardgrps,
and 2) recovering from a controller that fails or is partitioned
during ChangeConfigTo.
ChangeConfigTo moves shards from one
shardgrp to another. A risk is that some clients might
use the old shardgrp while other clients use the new
shardgrp, which could break linearizability. You will
need to ensure that at most one shardgrp is serving
requests for each shard at any one time.
If ChangeConfigTo fails while
reconfiguring, some shards may be inaccessible if they
have started but not completed moving from one
shardgrp to another. To make forward progress, the
tester starts a new controller, and your job is to
ensure that the new one completes the reconfiguration
that the previous controller started.
This lab uses "configuration" to refer to the
assignment of shards to shardgrps. This is not the
same as Raft cluster membership changes. You don't
have to implement Raft cluster membership changes.
A shardgrp server is a member of only
a single shardgrp. The set of servers in a
given shardgrp will never change.
Only RPC may be used for interaction among clients and servers.
For example, different instances of your server are not allowed
to share Go variables or files.
In Part A, you will implement a working shardctrler,
which will store and retrieve configurations in a kvsrv.
You will also implement the shardgrp, replicated
with your Raft rsm package, and a corresponding
shardgrp clerk. The shardctrler talks
to the shardgrp clerks to move shards between
different groups.
In Part B, you will modify your shardctrler to handle
failures and partitions during config changes. In Part C,
you will extend your shardctrler to allow for concurrent
controllers without interfering with each other. Finally, in
Part D, you will have the opportunity to extend your solution
in any way you like.
This lab's sharded key/value service follows the same
general design as Flat Datacenter Storage, BigTable,
Spanner, FAWN, Apache HBase, Rosebud, Spinnaker, and
many others. These systems differ in many details from
this lab, though, and are also typically more
sophisticated and capable. For example, the lab
doesn't evolve the sets of peers in each Raft group;
its data and query models are simple; and so on.
Lab 5 will use your kvsrv from Lab 2, and your
rsm and Raft from Lab 4. Your Lab 5
and Lab 4 must use the same rsm and Raft
implementations.
You may use late hours for Part A, but you may not use
late hours for Parts B-D.
Getting Started
Do a git pull to get the latest lab software.
We supply you with tests and skeleton code
in src/shardkv1:
client.go for the shardkv clerk
shardcfg package for computing shard
configurations
shardgrp package: for the shardgrp
clerk and server.
shardctrler
package, which contains shardctrler.go with
methods for the controller to change a configuration
(ChangeConfigTo) and to get a
configuration (Query)
To get up and running, execute the following commands:
$ cd ~/6.5840
$ git pull
...
$ cd src
$ make RUN="-run 5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5A
=== RUN TestInitQuery5A
Test (5A): Init and Query ... (reliable network)...
...
Part A: Moving shards
Your first job is to implement shardgrps and the
InitConfig, Query,
and ChangeConfigTo methods
when there are no failures.
We have given you the code for describing a
configuration, in
shardkv1/shardcfg.
Each shardcfg.ShardConfig has a unique
identifying number, a mapping from shard number to
group number, and a mapping from group number to the
list of servers replicating that group.
There will usually be more shards than groups
(so that each group serves more than one shard), in
order that load can be shifted at a fairly fine
granularity.
Implement these two methods in
shardctrler/shardctrler.go:
The InitConfig method receives the first
configuration, passed to it by the tester as
a shardcfg.ShardConfig.
InitConfig should store the configuration in
an instance of Lab 2's kvsrv.
The Query method returns the current
configuration; it should read the configuration from
kvsrv, previously stored there by InitConfig.
Implement InitConfig and Query, and
store the configuration in kvsrv.
You're done when your code passes the
first test. Note this task doesn't require any shardgrps.
$ cd ~/6.5840/src
$ make RUN="-run TestInitQuery5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestInitQuery5A
=== RUN TestInitQuery5A
Test (5A): Init and Query ... (reliable network)...
... Passed -- time 0.0s #peers 1 #RPCs 3 #Ops 0
--- PASS: TestInitQuery5A (0.13s)
PASS
ok 6.5840/shardkv1 1.143s
Implement InitConfig and Query
by storing and reading the initial
configuration from kvsrv:
use the Get/Put methods of
ShardCtrler.IKVClerk to talk to kvsrv,
use the String method
of ShardConfig to turn
a ShardConfig into a string that you
can pass to Put,
and use the shardcfg.FromString() function to
turn a string into a ShardConfig.
Implement an initial version of shardgrp
in shardkv1/shardgrp/server.go and a corresponding
clerk in shardkv1/shardgrp/client.go by
copying code from your Lab 4 kvraft solution.
Implement a clerk in shardkv1/client.go
that uses the Query method to find the
shardgrp for a key, and then talks to that
shardgrp. You're done when your code passes the
Static test.
$ cd ~/6.5840/src
$ make RUN="-run TestStatiOneShardGroup5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestStaticOneShardGroup5A
=== RUN TestStaticOneShardGroup5A
Test (5A): one shard group ... (reliable network)...
... Passed -- time 9.4s #peers 1 #RPCs 822 #Ops 420
--- PASS: TestStaticOneShardGroup5A (9.52s)
PASS
ok 6.5840/shardkv1 10.542s
Copy code from your kvraft client.go and
server.go for Put and Get,
and any other code you need from kvraft.
The code in shardkv1/client.go provides
the Put/Get clerk for the overall system: it finds out
which shardgrp holds the desired key's shard by
invoking the Query method,
and then talks to the shardgrp that holds
that shard.
Implement
shardkv1/client.go, including its Put/Get methods.
Use shardcfg.Key2Shard()
to find the shard number for a key.
The tester passes a
ShardCtrler object to
MakeClerk in shardkv1/client.go.
Retrieve the current configuration using the
Query method.
To put/get a key from a shardgrp, the shardkv
clerk should create a shardgrp clerk for the
shardgrp by calling shardgrp.MakeClerk,
passing in the servers found in the
configuration and the shardkv
clerk's ck.clnt. Use the
GidServers() method from ShardConfig to get the group for
a shard.
shardkv1/client.go's Put must
return ErrMaybe when the reply was maybe lost, but
this Put invokes shardgrp's Put to talk
a particular shardgrp. The inner Put can signal
this with an error.
Upon creation, the first shardgrp (shardcfg.Gid1)
should initialize itself to own all shards.
Now you should support movement of shards among groups by
implementing
the ChangeConfigTo method, which
changes from an old configuration to a new configuration. The new
configuration may include new shardgrps that are not present in the
old configuration, and may exclude shardgrps that were present
in the old configuration. The controller should move shards
(the key/value data) so that the set of shards stored by each
shardgrp matches the new configuration.
The approach we suggest for
moving a shard is for ChangeConfigTo
to first "freeze" the shard at the source
shardgrp, causing that shardgrp to
reject Put's for keys in the moving
shard. Then, copy (install) the shard to the destination
shardgrp; then delete the frozen shard. Finally,
post a new configuration so that clients can find the
moved shard. A nice property of this approach is that it
avoids any direct interactions among the shardgrps.
It also supports serving shards that are not affected
by an ongoing configuration change.
To be able to order changes to the configuration,
each configuration has a unique
number Num
(see shardcfg/shardcfg.go). The tester
in Part A invokes ChangeConfigTo
sequentially, and the configuration passed
to ChangeConfigTo will have
a Num one larger than the previous one;
thus, a configuration with a higher
Num is newer than one with a
lower Num.
The network may delay RPCs, and RPCs may arrive
out of order at the shardgrps. To reject old
FreezeShard, InstallShard,
and DeleteShard RPCs, they should
include Num
(see shardgrp/shardrpc/shardrpc.go), and
shardgrps must remember the largest Num
they have seen for each shard.
Implement ChangeConfigTo
(in shardctrler/shardctrler.go) and
extend shardgrp to support freeze,
install, and delete. ChangeConfigTo
should always succeed in Part A because the tester
doesn't induce failures in this part. You will
need to implement
FreezeShard, InstallShard, and DeleteShard
in shardgrp/client.go
and shardgrp/server.go using the RPCs in
the shardgrp/shardrpc package, and reject old RPCs
based on Num. You will also need modify the
shardkv clerk in shardkv1/client.go to
handle ErrWrongGroup, which a shardgrp should
return if it isn't reponsible for the shard.
You have completed this task when
you pass the JoinBasic and DeleteBasic tests. These
tests focus on adding shardgrps; you don't have to
worry about shardgrps leaving just yet.
A shardgrp should respond with
an ErrWrongGroup error to a
client Put/Get with a key that the
shardgrp isn't responsible for (i.e., for a key
whose shard is not assigned to the shardgrp). You
will have to modify shardkv1/client.go to
reread the configuration and retry
the Put/Get.
Note that you will have to run FreezeShard,
InstallShard, and DeleteShard through
your rsm package, just like Put
and Get.
You can send an entire map as your state
in an RPC request or reply, which may help
keep the code for shard transfer simple.
If one of your RPC handlers includes in its reply
a map (e.g. a key/value map) that's part of your
server's state, you may get bugs due to races. The
RPC system has to read the map in order to send it
to the caller, but it isn't holding a lock that
covers the map. Your server, however, may proceed to
modify the same map while the RPC system is reading
it. The solution is for the RPC handler to include a
copy of the map in the reply.
Extend ChangeConfigTo to
handle shard groups that leave; i.e., shardgrps
that are present in the current configuration but
not in the new one. Your solution should
pass TestJoinLeaveBasic5A now. (You
may have handled this scenario already in the
previous task, but the previous tests didn't test
for shardgrps leaving.)
Make your solution pass all Part A
tests, which check that your sharded key/value
service supports many groups joining and leaving,
shardgrps restarting from snapshots, processing Gets
while some shards are offline or involved in a
configuration change, and linearizability when many
clients interact with the service while the tester
concurrently invokes the controller's ChangeConfigTo to
rebalance shards.
$ cd ~/6.5840/src
$ make RUN="-run 5A" shardkv
go build -race -o main/kvsrv1d main/kvsrv1d.go
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5A
Test (5A): Init and Query ... (reliable network)...
... Passed -- time 0.0s #peers 1 #RPCs 3 #Ops 0
Test (5A): one shard group ... (reliable network)...
... Passed -- time 5.1s #peers 1 #RPCs 792 #Ops 180
Test (5A): a group joins... (reliable network)...
... Passed -- time 12.9s #peers 1 #RPCs 6300 #Ops 180
Test (5A): delete ... (reliable network)...
... Passed -- time 8.4s #peers 1 #RPCs 1533 #Ops 360
Test (5A): basic groups join/leave ... (reliable network)...
... Passed -- time 13.7s #peers 1 #RPCs 5676 #Ops 240
Test (5A): many groups join/leave ... (reliable network)...
... Passed -- time 22.1s #peers 1 #RPCs 3529 #Ops 180
Test (5A): many groups join/leave ... (unreliable network)...
... Passed -- time 54.8s #peers 1 #RPCs 5055 #Ops 180
Test (5A): shutdown ... (reliable network)...
... Passed -- time 11.7s #peers 1 #RPCs 2807 #Ops 180
Test (5A): progress ... (reliable network)...
... Passed -- time 8.8s #peers 1 #RPCs 974 #Ops 82
Test (5A): progress ... (reliable network)...
... Passed -- time 13.9s #peers 1 #RPCs 2443 #Ops 390
Test (5A): one concurrent clerk reliable... (reliable network)...
... Passed -- time 20.0s #peers 1 #RPCs 5326 #Ops 1248
Test (5A): many concurrent clerks reliable... (reliable network)...
... Passed -- time 20.4s #peers 1 #RPCs 21688 #Ops 10500
Test (5A): one concurrent clerk unreliable ... (unreliable network)...
... Passed -- time 25.8s #peers 1 #RPCs 2654 #Ops 176
Test (5A): many concurrent clerks unreliable... (unreliable network)...
... Passed -- time 25.3s #peers 1 #RPCs 7553 #Ops 1896
PASS
ok 6.5840/shardkv1 243.115s
$
Your solution must continue serving shards
that are not affected by an ongoing configuration
change.
Part B: Handling a failed controller
The controller is a short-lived command, which an
administrator invokes: it moves shards and then
exits. But, it may fail or lose network
connectivity while moving shards. The main task in
this part of the lab is recovering from a controller
that fails to complete
ChangeConfigTo. The tester starts a new
controller and invokes its ChangeConfigTo
after partitioning the first controller; you have to
modify the controller so that the new one finishes
the reconfiguration.
The tester calls InitController when
starting a controller; you can modify that function
to check whether an interrupted configuration
change needs to be completed.
A good approach to allowing a controller to
finish a reconfiguration that a previous one started
is to keep two configurations: a current one and a
next one, both stored in the controller's kvsrv.
When a controller starts a reconfiguration, it
stores the next configuration.
Once a controller completes the reconfiguration, it
makes the next configuration the current one.
Modify InitController
to first check if there is a stored next
configuration with a higher configuration number than
the current one, and if so, complete the shard moves
necessary to reconfigure to the next one.
Modify shardctrler to implement the above approach.
A controller that picks up the work from a
failed controller may
repeat FreezeShard, InstallShard,
and Delete RPCs; shardgrps can use Num to
detect duplicates and reject them. You have completed
this task if your solution passes the Part B tests.
$ cd ~/6.5840/src
$ make RUN="-run 5B" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run 5B
Test (5B): Join/leave while a shardgrp is down... (reliable network)...
... Passed -- time 9.2s #peers 1 #RPCs 899 #Ops 120
Test (5B): recover controller ... (reliable network)...
... Passed -- time 26.4s #peers 1 #RPCs 3724 #Ops 360
PASS
ok 6.5840/shardkv1 35.805s
$
The tester calls InitController
when starting a controller; you can implement
recovery in that method
in shardctrler/shardctrler.go.
Part C: Concurrent configuration changes
In this part of the lab you will modify the
controller to allow for concurrent controllers. When a
controller crashes or is partitioned, the tester will
start a new controller, which must finish any work
that the old controller might have in progress (i.e.,
finishing moving shards like in Part B). This means
that several controllers may run concurrently and
send RPCs to the shardgrps and the kvsrv that stores
configurations.
The main challenge is to ensure these controllers
don't step on each other. In Part A you already
fenced all the shardgrp RPCs with Num so that
old RPCs are rejected. Even if several controllers
pick up the work of an old controller concurrently,
one of them succeeds and the others repeat all the
RPCs, the shardgrps will ignore them.
Thus the challenging case left is to ensure that
only one controller updates the next configuration
to avoid that two controllers (e.g., a partitioned
one and a new one) put different configurations in
the next one. To stress this scenario, the tester runs
several controllers concurrently and each one
computes the next configuration by reading the
current configuration and updating it for a shardgrp
that left or joined, and then the tester
invokes ChangeConfigTo; thus multiple
controllers may invoke ChangeConfigTo with
different configuration with the same Num.
You can use the version number of a key and
versioned Puts to ensure that only one
controller updates the next configuration and that
the other invocations return without doing anything.
Modify your controller so that only one controller
can post a next configuration for a
configuration Num. The tester will start
many controllers but only one should
start ChangeConfigTo for a new
configuation.
You have completed this task
if you pass the concurrent tests of
Part C:
$ cd ~/6.5840/src
$ make RUN="-run TestConcurrentReliable5C" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestConcurrentReliable5C
Test (5C): Concurrent ctrlers ... (reliable network)...
... Passed -- time 8.2s #peers 1 #RPCs 1753 #Ops 120
PASS
ok 6.5840/shardkv1 8.364s
$ make RUN="-run TestAcquireLockConcurrentUnreliable5C" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run TestAcquireLockConcurrentUnreliable5C
Test (5C): Concurrent ctrlers ... (unreliable network)...
... Passed -- time 23.8s #peers 1 #RPCs 1850 #Ops 120
PASS
ok 6.5840/shardkv1 24.008s
$
See concurCtrler in test.go
to see how the tester runs controllers concurrently.
In this exercise you will put recovery of an old
controller together with a new controller: a new
controller should perform recovery from Part B. If
the old controller was partitioned
during ChangeConfigTo, you will have to make
sure that the old controller doesn't interfere with the
new controller. If all the controller's updates are
already properly fenced with Num checks from Part B,
you don't have to write extra code. You have completed
this task if you pass the
Partition tests.
$ cd ~/6.5840/src
$ make RUN="-run Partition" shardkv
go build -race -o main/kvsrv1d main/k
go build -race -o main/shardgrp1d main/shardgrp1d.go
cd shardkv1; go test -timeout 15m -v -race -run Partition
Test (5C): partition controller in join... (reliable network)...
... Passed -- time 7.8s #peers 1 #RPCs 876 #Ops 120
Test (5C): controllers with leased leadership ... (reliable network)...
... Passed -- time 36.8s #peers 1 #RPCs 3981 #Ops 360
Test (5C): controllers with leased leadership ... (unreliable network)...
... Passed -- time 52.4s #peers 1 #RPCs 2901 #Ops 240
Test (5C): controllers with leased leadership ... (reliable network)...
... Passed -- time 60.2s #peers 1 #RPCs 27415 #Ops 11182
Test (5C): controllers with leased leadership ... (unreliable network)...
... Passed -- time 60.5s #peers 1 #RPCs 11422 #Ops 2336
PASS
ok 6.5840/shardkv1 217.779s
$
You have completed implementing a highly-available
sharded key/value service with many shard groups for
scalability, reconfiguration to handle changes in
load, and with a fault-tolerant controller; congrats!
Rerun all tests to check that your
recent changes to the controller haven't broken earlier
tests.
Gradescope will rerun the Lab 3A-D and Lab 4A-C tests
on your submission, in addition to the 5C tests.
Before submitting, double check that your solution works:
$ make raft1
$ make kvraft1
$ make shardkv
Part D: Extend your solution
In this final part of the lab, you get to extend
your solution in any way you like. You will have to
write your own tests for whatever extensions you choose
to implement.
Implement one of the ideas below or
come up with your own idea. Write a paragraph in a
file extension.md describing your extension,
and upload extension.md to Gradescope.
If you would like to do one of the harder, open-ended
extensions, feel free to partner up with another
student in the class.
Here are some ideas for possible extension (the first
few are easy and the later ones are more open ended):
Change the tester
to use kvraft instead of kvsrv
(i.e., replace the kvsrv.StartKVServer
in MakeTestMaxRaft in test.go
with kvraft.StartKVServer) so that the
controller uses your kvraft to store its
configuration. Write a test that checks that the
controller can query and update the configuration
while one of the kvraft peers is down. The
existing code for the tester is distributed across
src/kvtest1, src/shardkv1, and
src/tester1.
Change kvsrv to implement
exactly-once semantics for Put/Get as in
Lab 2
from last year (see the dropped messages part).
You may be able to port over some tests from
2024 instead of having to write your own from
scratch. Implement exactly-once also in your kvraft.
Change kvsrv to support a
Range function, which returns all keys in
the range low key to high key.
The lazy way to implement Range is to
iterate through the key/value map that the server
maintains; a better way is to use a data structure
that supports range searches (e.g.,
B-tree). Include a test that fails the lazy
solution but passes on the better solution.
Modify your kvraft
implementation to allow the leader to
serve Gets without running
the Get through rsm. That is,
implement the optimization described at the end
of section 8 of the raft paper, including
leases, to ensure that kvraft maintains
linearizability. Your implementation should pass
the existing kvraft tests. You should also add
a test that checks that your optimized
implementation is faster (e.g., by comparing the
number of RPCs) and a test that checks that term
switches are slower because a new leader must wait
until the lease has expired.
Support transactions
in kvraft so that a developer can
perform several Put and Gets
atomically. Once you have transaction, you don't
need version Puts anymore; transactions subsume
versioned Puts. Look at
etcd's
transactions for an example interface. Write
tests to demonstrate your extension works.
Modify shardkv to support transactions
so that a developer can perform
several Puts and Gets
atomically across shards. This requires
implementing two-phase-commit and two-phase
locking. Write tests to demonstrate that your
extension works.
Handin procedure
Before submitting, please run all the tests
one final time.
Before submitting, double check that your solution
works with:
$ make raft1
$ make kvraft1
$ make shardkv