%0 Conference Proceedings
%T Self-Balancing Job Parallelism and Throughput in Hadoop
%+ Université de Lille, Sciences et Technologies
%+ Self-adaptation for distributed services and large software systems (SPIRALS)
%+ Faculty of Information Technology [Prague] (FIT CTU)
%+ Institut universitaire de France (IUF)
%A Zhang, Bo
%A Křikava, Filip
%A Rouvoy, Romain
%A Seinturier, Lionel
%Z PIA Datalyse
%< avec comité de lecture
%( Lecture Notes in Computer Science
%B 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS)
%C Heraklion, Crete, Greece
%Y Márk Jelasity
%Y Evangelia Kalyvianaki
%I Springer
%3 Distributed Applications and Interoperable Systems
%V LNCS-9687
%P 129-143
%8 2016-06-06
%D 2016
%R 10.1007/978-3-319-39577-7_11
%K Hadoop
%K Map Reduce
%K Self-optimisation
%Z Computer Science [cs]/Software Engineering [cs.SE]
%Z Computer Science [cs]/Ubiquitous Computing
%Z Computer Science [cs]/Mobile Computing
%Z Computer Science [cs]/Web
%Z Computer Science [cs]/Operating Systems [cs.OS]Conference papers
%X In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.
%G English
%Z TC 6
%Z WG 6.1
%2 https://inria.hal.science/hal-01294834/document
%2 https://inria.hal.science/hal-01294834/file/zhang-dais16.pdf
%L hal-01294834
%U https://inria.hal.science/hal-01294834
%~ CNRS
%~ INRIA
%~ INRIA-LILLE
%~ INRIA_TEST
%~ GRID5000
%~ TESTALAIN1
%~ IFIP-LNCS
%~ IFIP
%~ CRISTAL
%~ INRIA2
%~ IFIP-TC
%~ IFIP-WG
%~ CRISTAL-SPIRALS
%~ IFIP-TC6
%~ IFIP-WG6-1
%~ IFIP-DAIS
%~ IFIP-LNCS-9687
%~ UNIV-LILLE
%~ SILECS