Deep Store: An Archival Storage System Architecture

Abstract

We present the Deep Store archival storage architecture, a large-scale storage system that stores immutable data effi- ciently and reliably for long periods of time. Archived data is stored across a cluster of nodes and recorded to hard disk. The design differentiates itself from traditional file systems by eliminating redundancy within and acrossfiles, distribut- ing content for scalability, associating rich metadata with content, and using variable levels of replication based on the importance or degree of dependency of each piece of stored data. We evaluate the foundations of our design, including PRESIDIO, a virtual content-addressable storage frame- work with multiple methods for inter-file and intra-file com- pression that effectively addresses the data-dependent vari- ability of data compression. We measure content and meta- data storage efficiency, demonstrate the need for a variable- degree replication model, and provide preliminary results for storage performance.