TH-DPMS
- 1 October 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Storage
- Vol. 16 (4), 1-31
- https://doi.org/10.1145/3412852
Abstract
The rapidly increasing data in recent years requires the datacenter infrastructure to store and process data with extremely high throughput and low latency. Fortunately, persistent memory (PM) and RDMA technologies bring new opportunities towards this goal. Both of them are capable of delivering more than 10 GB/s of bandwidth and sub-microsecond latency. However, our past experiences and recent studies show that it is non-trivial to build an efficient and distributed storage system with such new hardware. In this article, we design and implement TH-DPMS (TsingHua Distributed Persistent Memory System) based on persistent memory and RDMA, which unifies the memory, file system, and key-value interface in a single system. TH-DPMS is designed based on a unified distributed persistent memory abstract, pDSM. pDSM acts as a generic layer to connect the PMs of different storage nodes via high-speed RDMA network and organizes them into a global shared address space. It provides the fundamental functionalities, including global address management, space management, fault tolerance, and crash consistency guarantees. Applications are enabled to access pDSM with a group of flexible and easy-to-use APIs by using either raw read/write interfaces or the transactional ones with ACID guarantees. Based on pDSM, we implement a distributed file system and a key-value store named pDFS and pDKVS, respectively. Together, they uphold TH-DPMS with high-performance, low-latency, and fault-tolerant data storage. We evaluate TH-DPMS with both micro-benchmarks and real-world memory-intensive workloads. Experimental results show that TH-DPMS is capable of delivering an aggregated bandwidth of 120 GB/s with 6 nodes. When processing memory-intensive workloads such as YCSB and Graph500, TH-DPMS improves the performance by one order of magnitude compared to existing systems and keeps consistent high efficiency when the workload size grows to multiple terabytes.Keywords
Funding Information
- The National Natural Science Foundation of China (61772300, 61832011)
- National Key Research and Development Program of China (2018YFB1003301)
This publication has 47 references indexed in Scilit:
- DAREPublished by Association for Computing Machinery (ACM) ,2015
- Let's Talk About Storage & Recovery Methods for Non-Volatile Memory Database SystemsPublished by Association for Computing Machinery (ACM) ,2015
- Blurred persistence in transactional persistent memoryPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Persistent B + -trees in non-volatile main memoryProceedings of the VLDB Endowment, 2015
- An Astrocyte Regenerative Response from Vimentin-Containing Cells in the Spinal Cord of Amyotrophic Lateral Sclerosis's Disease-Like Transgenic (G93A SOD1) MiceNeurodegenerative Diseases, 2015
- HEAPOACM Transactions on Storage, 2014
- Using RDMA efficiently for key-value servicesACM SIGCOMM Computer Communication Review, 2014
- SAP HANA databaseACM SIGMOD Record, 2012
- NV-HeapsACM SIGPLAN Notices, 2011
- Techniques for reducing consistency-related communication in distributed shared-memory systemsACM Transactions on Computer Systems, 1995