%0 Conference Proceedings
%T RDMA-Based Apache Storm for High-Performance Stream Data Processing
%+ University of Science and Technology of China [Hefei] (USTC)
%+ East China University of Science and Technology
%A Zhang, Ziyu
%A Liu, Zitan
%A Jiang, Qingcai
%A Wu, Zheng
%A Chen, Junshi
%A An, Hong
%Z Part 5: Big Data and Cloud
%< avec comité de lecture
%( Lecture Notes in Computer Science
%B 17th IFIP International Conference on Network and Parallel Computing (NPC)
%C Zhengzhou, China
%Y Xin He
%Y En Shao
%Y Guangming Tan
%I Springer International Publishing
%3 Network and Parallel Computing
%V LNCS-12639
%P 276-287
%8 2020-09-28
%D 2020
%R 10.1007/978-3-030-79478-1_24
%K Apache Storm
%K RDMA
%K InfiniBand
%K Stream-processing framework
%K Cloud computing
%K Communication optimization
%Z Computer Science [cs]Conference papers
%X Apache Storm is a scalable fault-tolerant distributed real-time stream-processing framework widely used in big data applications. For distributed data-sensitive applications, low-latency, high-throughput communication modules have a critical impact on overall system performance. Apache Storm currently uses Netty as its communication component, an asynchronous server/client framework based on TCP/IP protocol stack. The TCP/IP protocol stack has inherent performance flaws due to frequent memory copying and context switching. The Netty component not only limits the performance of the Storm but also increases the CPU load in the IPoIB (IP over InfiniBand) communication mode. In this paper, we introduce two new implementations for Apache Storm communication components with the help of RDMA technology. The performance evaluation on Mellanox QDR Cards (40 Gbps) shows that our implementations can achieve speedup up to 5$$\times $$× compared with IPoIB and 10$$\times $$× with 1 Gigabit Ethernet. Our implementations also significantly reduce the CPU load and increase the throughput of the system.
%G English
%Z TC 10
%Z WG 10.3
%2 https://inria.hal.science/hal-03768735/document
%2 https://inria.hal.science/hal-03768735/file/511910_1_En_24_Chapter.pdf
%L hal-03768735
%U https://inria.hal.science/hal-03768735
%~ IFIP-LNCS
%~ IFIP
%~ IFIP-TC
%~ IFIP-TC10
%~ IFIP-NPC
%~ IFIP-WG10-3
%~ IFIP-LNCS-12639