An Efficient Approach to Consolidating Job Schedulers in Traditional Independent Scientific Workflows
Open Access
- 20 February 2020
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 10 (4), 1455
- https://doi.org/10.3390/app10041455
Abstract
The current research paradigm is one of data-driven research. Researchers are beginning to deploy computer facilities to produce and analyze large amounts of data. As requirements for computing power grow, data processing in traditional workstations is always under pressure for efficient resource management. In such an environment, a tremendous amount of data is being processed using parallel computing for efficient and effective research results. HTCondor, as an example, provides computing power for data analysis for researchers. Although such a system works well in a traditional computing cluster environment, we need an efficient methodology to meet the ever-increasing demands of computing using limited resources. In this paper, we propose an approach to integrating clusters that can share their computing power on the basis of a priority policy. Our approach makes it possible to share worker nodes while maintaining the resources allocated to each group. In addition, we have utilized the historical data of user usage in order to analyze problems that have occurred during job execution due to resource sharing and the actual operating results. Our findings can provide a reasonable guideline for limited computing powers shared by multiple scientific groups.Funding Information
- National Research Foundation of Korea (NRF-2010-0018156)
This publication has 8 references indexed in Scilit:
- A Roadmap for HEP Software and Computing R&D for the 2020sComputing and Software for Big Science, 2019
- Data Analytics ImplicationsIEEE Potentials, 2018
- Experience on HTCondor batch system for HEP and other research fields at KISTI-GSDCJournal of Physics: Conference Series, 2017
- Singularity: Scientific containers for mobility of computePLOS ONE, 2017
- Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials scienceAPL Materials, 2016
- Trends in Computation, Communication and Storage and the Consequences for Data-intensive SciencePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2012
- THE EXTENSION OF TORQUE SCHEDULER ALLOWING THE USE OF PLANNING AND OPTIMIZATION IN GRIDSComputer Science, 2012
- Distributed computing in practice: the Condor experienceConcurrency and Computation: Practice and Experience, 2005