Can MPI Benefit Hadoop and MapReduce Applications?
- 1 September 2011
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 371-379
- https://doi.org/10.1109/icppw.2011.56
Abstract
The Message Passing Interface (MPI) standard and its implementations (such as MPICH and OpenMPI) have been widely used in the high-performance computing area to provide an efficient communication infrastructure. This paper investigates whether MPI can be adapted to the data intensive computing area to substantially speed up Hadoop and MapReduce applications, by reducing communication overheads. Three specific issues are studied. First, is the potential for reducing communication overheads significant, if MPI is used? Second, what are the main technical challenges to adapt MPI to Hadoop? Third, what are the minimal extensions to the MPI standard that can help alleviate the challenges while promise to significantly improve performance? To answer the first question, we identify important and basic communication primitives in both MPI and Hadoop, and make fair comparisons of their performance through experiments. The results show that the potential for improvement could be high. To answer the second and the third questions, we analyze the Hadoop code base to identify communication related programmers' needs. Furthermore, we propose a minimal interface extension to the MPI standard (only one pair of library calls are added), which capture the key-value pair nature commonly found in data intensive computing. This extension is implemented in a prototype library called MPI-D. Benchmark tests based on simulation show that Hadoop augmented with MPI-D could significantly speed up MapReduce application performance.Keywords
This publication has 8 references indexed in Scilit:
- RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systemsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- MapReduce in MPI for Large-scale graph algorithmsParallel Computing, 2011
- TwisterPublished by Association for Computing Machinery (ACM) ,2010
- Towards Efficient MapReduce Using MPILecture Notes in Computer Science, 2009
- MapReduceCommunications of the ACM, 2008
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Open MPI: Goals, Concept, and Design of a Next Generation MPI ImplementationLecture Notes in Computer Science, 2004
- A high-performance, portable implementation of the MPI message passing interface standardParallel Computing, 1996