%0 Conference Proceedings
%T DDP-B: A Distributed Dynamic Parallel Framework for Meta-genomics Binary Similarity
%+ University of Science and Technology of China [Hefei] (USTC)
%+ East China University of Science and Technology
%A Chi, Mengxian
%A Jin, Xu
%A Li, Feng
%A An, Hong
%Z Part 4: Big Data+Cloud
%< avec comité de lecture
%( Lecture Notes in Computer Science
%B 16th IFIP International Conference on Network and Parallel Computing (NPC)
%C Hohhot, China
%Y Xiaoxin Tang
%Y Quan Chen
%Y Pradip Bose
%Y Weiming Zheng
%Y Jean-Luc Gaudiot
%I Springer International Publishing
%3 Network and Parallel Computing
%V LNCS-11783
%P 143-155
%8 2019-08-23
%D 2019
%R 10.1007/978-3-030-30709-7_12
%K Meta-genomics
%K Big data
%K Parallel computing
%K Binary distance
%K Dynamic scheduling
%K Distributed scalability
%Z Computer Science [cs]Conference papers
%X Great efforts have been made on meta-genomics in the field of new species exploration in the past decades. With the development of next-generation sequencing technology, meta-genomics datasets have been produced as large as dozens of hundreds of gigabytes or even several terabytes, which brings a severe challenge to data analysis. Besides, conventional meta-genomics comparing algorithms may not take full advantage of powerful computing capacity from parallel computing techniques due to lack of parallelism. In this paper, we propose DDP-B, a distributed dynamic parallel framework for meta-genomics binary similarity analysis, to overcome these limitations. In this framework, we introduce a binary distance algorithm for meta-genomics similarity measurement and develop different levels of parallel granularity of the algorithm utilizing MPI, OpenMP, and SIMD techniques. Moreover, we establish a dynamic scheduling method to deliver asynchronous parallel computing tasks and design a distributed cluster to deploy the dynamic parallel system, which completes 2.97K pairs of meta-genomics vectors comparison per second and achieves an 134.79x speedup versus the baseline in the optimal condition. Our framework shows stable scalability when assigned larger workloads.
%G English
%Z TC 10
%Z WG 10.3
%2 https://inria.hal.science/hal-03770527/document
%2 https://inria.hal.science/hal-03770527/file/486810_1_En_12_Chapter.pdf
%L hal-03770527
%U https://inria.hal.science/hal-03770527
%~ IFIP-LNCS
%~ IFIP
%~ IFIP-TC
%~ IFIP-TC10
%~ IFIP-NPC
%~ IFIP-WG10-3
%~ IFIP-LNCS-11783