A Near Metal Platform for Intensive Big Data Processing Using A Novel Approach
- 28 May 2020
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM) in Proceedings of the 2020 5th International Conference on Big Data and Computing
Abstract
In this paper we present a new Golang based framework for distributed intensive data processing and also micro batching. It uses a novel approach, the persistent distributed channels, based on the concept of Share memory by communicating, and inspired from Resilient distributed datasets of Apache Spark. The architecture of our proposed system is considered as near-metal platform for Big Data operations in order to enhance the speed of massive data processing.Keywords
This publication has 9 references indexed in Scilit:
- An adaptive and real-time based architecture for financial data integrationJournal of Big Data, 2019
- Deep Differential Testing of JVM ImplementationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2019
- Understanding Real-World Concurrency Bugs in GoPublished by Association for Computing Machinery (ACM) ,2019
- The big data system, components, tools, and technologies: a surveyKnowledge and Information Systems, 2018
- Cloud-based parallel power flow calculation using resilient distributed datasets and directed acyclic graphJournal of Modern Power Systems and Clean Energy, 2018
- JVMPublished by Springer Science and Business Media LLC ,2017
- Thrill: High-performance algorithmic distributed batch data processing with C++Published by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Big data analytics on Apache SparkInternational Journal of Data Science and Analytics, 2016
- Communicating sequential processesCommunications of the ACM, 1978