Raft 实验问答与调试

6.5840 Lecture 10: lab 3 A+B Q&A

Two different structures of Raft library
  multi-threaded with locks
  single-threaded state-machine
  neither one is strictly better than the other
  other plans are possible

Correctness in distributed systems:
 safety: never returning an incorrect result
 liveness: eventually returning a result
 Q: give an example of a safety and liveness bug in your solution

Getting started on a lab
  make first test case work
   fill out VoteRequest structs
   do an RPC
  then much easier to get the lay of the land
  other strategy: read all guides, read test code, etc.

Debugging
  Run with race detector
  Log all action/messages in easy searchable way
   standard format: src, dst, opcode, raft state,..
  Run test case
    if ok:
      next test case
    if fail:
      repeat:
        study test case
        formulate hypothesis about what might be wrong
        study log and figure 2, run with race detector
        modify code and try test again

Multi-threaded Raft library
  many threads (Start() thread, thread reading from applych)
  many RPC threads to talk to peers in parallel
  many RPC handler threads (started by RPC package)
  raft state with lock
  one thread writing to applych
    use condvar to signal it

Locking
  Raft lock serializes operations
  RPC handlers hold Raft lock so are atomic
    little parallelism
  Threads don't hold lock during RPC
    risk: deadlock if locks are held during RPC

Single-threaded state-machine
  observation: little parallelism anyway
    main use: sending RPCs in parallel
  one thread runs Raft protocol
    receives input events (e.g., tick, RPC request, reply)
    process event, updating raft structure, without locks
      collecting output events (persist, send RPCs)
    performs output events
      first persist, then RPC requests
  one thread for the applier
    need to separate state between applier and state-machine
    protect shared state with lock (e.g., the log)

Code tour
  Raft struct
  Ticker
  Election timeout
  Start election
  VoteRequest handling
  becomeLeader
  send appends
  AppendEntries request handling
  AppendEntries reply handling
  commit
  applier
  Start()

Lab guidance


Lab guidance

Hardness of assignments


Each lab task is tagged to indicate
roughly how long we expect the task to take:

    Easy: A few hours.

    Moderate: ~ 6 hours (per week).

    Hard: More than 6 hours (per week). If
    you start late, your solution is unlikely to pass all tests.



Most of the labs require only a modest amount of code
(perhaps a
few hundred lines per lab part), but can be conceptually difficult
and may require a good deal of thought and debugging.
Some of the tests are difficult to pass.

Don't start a lab the night before it is due; it's more
efficient to do the labs in several sessions spread over multiple
days. Tracking down bugs in distributed systems is difficult,
because of concurrency, crashes, and an unreliable network.

Tips


Do the Online Go tutorial and
  consult
  Effective Go.
    See Editors to
    set up your editor for Go.

The lab Makefiles are set up to use
Go's race detector.
Fix any races it reports.

Advice on locking in labs.

Advice on structuring your Raft lab.

This Diagram of Raft interactions may
help you understand code flow
between different parts of the system.

Learn about Go's Printfformat strings:
Go format strings.

 To learn more about git, look at the
Pro Git book or the
git user's manual.

Debugging

Efficient debugging takes experience. It
helps to be systematic: form a hypothesis about a possible cause of the
problem; collect evidence that might be relevant; think about the
information you've gathered; repeat as needed. For extended debugging
sessions it helps to keep notes, both to accumulate evidence and to
remind yourself why you've discarded specific earlier hypotheses.

The most effective debugging technique is often to add print
statements to your code, run the test that is failing and collect the
print output in a file, and then look through the output file to
identify the point at which things start to go wrong. You may need to
iterate, adding more print statements as you learn more about what is
going wrong.

Concurrency among different peers and among the threads in a single
peer can cause actions to be interleaved in unexpected ways. For
example, it's quite possible for a Raft peer to be elected leader
while the previous leader still thinks it is the leader, or for a
leader to send an RPC but receive the reply after it has lost
leadership. Adding print statements may help you spot such situations.

Feel free to examine the test code (mr/mt_test.go,
raft1/raft_test.go, &c) to understand what the tests are
exploring. You can add print statements to the tests to help you
understand what they are doing and why they are failing, but be sure
you code passes with the original test code before submitting.

The Raft paper's Figure 2 must be followed fairly exactly. It is easy
to miss a condition that Figure 2 says must be checked, or a state
change that it says must be made. If you have a bug, re-check that all
of your code adheres closely to Figure 2.

As you're writing code (i.e., before you have a bug), it may be worth
adding explicit checks for conditions that the code assumes to be
true, perhaps using Go's
panic. Such checks may
help detect situations where later code unwittingly violates the
assumptions.

The TAs are happy to help you think about your code during office
hours, but you're likely to get the most mileage out of limited office
hour time if you've already dug as deep as you can into the situation.

Porcupine





        Clients

        Time



        Valid LP

        Invalid LP
        [ jump to first error ]