A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds
- 1 December 2017
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
Containerization, or OS-level virtualization has taken root within the computing industry. However, container utilization and its impact on performance and functionality within High Performance Computing (HPC) is still relatively undefined. This paper investigates the use of containers with advanced supercomputing and HPC system software. With this, we define a model for parallel MPI application DevOps and deployment using containers to enhance development effort and provide container portability from laptop to clouds or supercomputers. In this endeavor, we extend the use of Sin- gularity containers to a Cray XC-series supercomputer. We use the HPCG and IMB benchmarks to investigate potential points of overhead and scalability with containers on a Cray XC30 testbed system. Furthermore, we also deploy the same containers with Docker on Amazon's Elastic Compute Cloud (EC2), and compare against our Cray supercomputer testbed. Our results indicate that Singularity containers operate at native performance when dynamically linking Cray's MPI libraries on a Cray supercomputer testbed, and that while Amazon EC2 may be useful for initial DevOps and testing, scaling HPC applications better fits supercomputing resources like a Cray.Keywords
This publication has 17 references indexed in Scilit:
- Singularity: Scientific containers for mobility of computePLOS ONE, 2017
- Scaling Spark on HPC SystemsPublished by Association for Computing Machinery (ACM) ,2016
- Container and Microservice Driven Design for Cloud Infrastructure DevOpsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Microservices Architecture Enables DevOps: Migration to a Cloud-Native ArchitectureIEEE Software, 2016
- Supporting High Performance Molecular Dynamics in Virtualized Clusters using IOMMU, SR-IOV, and GPUDirectPublished by Association for Computing Machinery (ACM) ,2015
- An updated performance comparison of virtual machines and Linux containersPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Containers and Cloud: From LXC to Docker to KubernetesIEEE Cloud Computing, 2014
- GPU Passthrough Performance: A Comparison of KVM, Xen, VMWare ESXi, and LXC for CUDA and OpenCL ApplicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2014
- Minimal-overhead virtualization of a large scale supercomputerACM SIGPLAN Notices, 2011
- An overview of the Trilinos projectACM Transactions on Mathematical Software, 2005