<abstract xmlns="http://www.w3.org/1999/xhtml"><p>The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes.</p></abstract>

The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes.

Performance Optimization System for Hadoop and Spark Frameworks

Université Fédérale Toulouse Midi-Pyrénées

National Polytechnic University of Armenia

Cybernetics and Information Technologies

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"Performance Optimization System for Hadoop and Spark Frameworks"}

The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows...

Performance Optimization System for Hadoop and Spark Frameworks

Published Online: Dec 31, 2020

Page range: 5 - 17

Received: Jul 06, 2020

Accepted: Sep 25, 2020

DOI: https://doi.org/10.2478/cait-2020-0056

Keywords
Hadoop, Spark, data compression, CPU/IO tradeoff, performance optimization

© 2020 Hrachya Astsatryan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Performance Optimization System for Hadoop and Spark Frameworks

Published Online: Dec 31, 2020

Page range: 5 - 17

Received: Jul 06, 2020

Accepted: Sep 25, 2020

DOI: https://doi.org/10.2478/cait-2020-0056

KeywordsHadoop, Spark, data compression, CPU/IO tradeoff, performance optimization

© 2020 Hrachya Astsatryan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
Hadoop, Spark, data compression, CPU/IO tradeoff, performance optimization