Antiquity: Exploiting a Secure

Log for Wide-Area Distributed

Storage
Hakim Weatherspoon, Patrick Eaton, Byung-Gon Chun,
and John Kubiatowicz

Slides by: Noah Watkins

Friday, May 7, 2010

 Overview

• Distributed storage service

• Log structured
• Security
• Durability
• Consistency

Friday, May 7, 2010

 Distributed, Log-Structured Storage
Service

• Building block system service

• Utilized by applications such as file systems
• Manages many logs on behalf of many
owners
• Log-structure simplifies design
• Semantics and API
Friday, May 7, 2010
 Related Work
• Log-based storage: Zebra, SUNDR
• BFT Services: OceanStore, FarSite
• Distributed Storage: CFS, Ivy, PAST, Venti
• Replicated Systems: GFS, XFS
• Digital Library: LOCKSS (read-only,
preservation)

Friday, May 7, 2010

 Distributed, Log-Structured Storage
Service

Friday, May 7, 2010

 Security and Integrity

• Only the owner of a log can make

modifications
• Public-key certificates used to verify log
operations

Friday, May 7, 2010

 Durability and
Consistency
• Temporary and permanent failures are
common
• Logs must remain available (replication)
• And data in all log replicas must remain
consistent (quorums)

Friday, May 7, 2010

 The Secure Log

• Append-only
• Each log owned by a single principal
• Identified by hash of owner’s public key
• Log and components are self-verifying

Friday, May 7, 2010

 Structure of Log
• A log is a series of data blocks grouped into
extents
• Extents and blocks are addressed by hashes
of their contents
• Two-level naming requires (extent, block)
tuple to access a particular block
• Extents decouple client and infrastructure
data-unit management

Friday, May 7, 2010

 Log Structure
• All extents that compose a log are
immutable...(hash-verified)
• ...except for the head which accepts new
appends (key-verified)

Friday, May 7, 2010

 Extent Verifier

• Used to...
• verify the content of an extent
• verify containment in a specific log
• unambiguously ties an extent to a log
position

Friday, May 7, 2010

 Extent Verifier

Friday, May 7, 2010

 Antiquity Interface

Friday, May 7, 2010

 Append(key, cert, pred, data)

• Key: H(PK) identifies the log

• Certificate verifies requester possesses
private key
• Predicate is a client’s current knowledge of
log (i.e. current verifier)
• Breaks ties. Retries occur by first
refreshing state
• Data: duh -- written to head extent
Friday, May 7, 2010
 Snapshot/Truncate(key, cert,
predicate)

• Snapshot: convert head extent to hash-

verified extent
• Truncate: Establish new, empty head extent
that precedes previous snapshot extent
• Append/Snapshot/Truncate

Friday, May 7, 2010

 File System Example
• Build a file system on top of the secure log

Friday, May 7, 2010

 File System Example
• Represented as tree of block addresses

Friday, May 7, 2010

 File System Example
• Realized within the log

Friday, May 7, 2010

 Consistency
• Replicated logs must be consistent in the
presence of conflicting updates and
arbitrary failures
• Byzantine-fault tolerant quorum agreement
• Required configuration: n > 3f
• Quorum for progress: q = n - f

Friday, May 7, 2010

 Consistency

• Soundness: client receives POS responses

from threshold of servers (durable)
• Unsound: Receives NEG responses such
that a sound result is not possible (failure)
• Undefined: Cannot determine soundness or
unsoundness from replies (needs repair)

Friday, May 7, 2010

 Quorum Repair
• Restore log replicas to latest sound write
• Used when log replicas are in inconsistent
state
• And/or a quorum is not available for the
current configuration

Friday, May 7, 2010

 Quorum Repair
• A particular configuration does not have an
available quorum
• Administrator creates a new configuration
• Admin needs 2f + 1 repair requests
• Servers in new config fetch data from old
config
• But, a quorum is still needed, and not
available
Friday, May 7, 2010
 Quorum Repair
• Sound operations store a soundness proof
• Proof that an operation was sound
• Repair requests include latest soundness
proof
• Administrator decides on the latest sound
state that the new configuration must
consistently replicate

Friday, May 7, 2010

 Evaluation

• Two environments
• High-performance cluster of 60 nodes
• 400+ PlanetLab nodes

Friday, May 7, 2010

 Latency

Friday, May 7, 2010

 Throughput

Friday, May 7, 2010

 Availability (Read test)

Friday, May 7, 2010

 Availability (Server traces)

Friday, May 7, 2010

 Quorum Repair

Friday, May 7, 2010

 Conclusion

• Antiquity implements a secure, distributed,

fault-tolerant log
• It is made durable and maintains
consistency using replication and a
Byzantine fault-tolerant quorum protocol

Friday, May 7, 2010