Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province - Computer and Computing Technologies in Agriculture XI
Conference Papers Year : 2019

Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province

Siwei Fu
  • Function : Author
  • PersonId : 1046181
Guifen Chen
  • Function : Author
  • PersonId : 972131

Abstract

Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes; and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.
Fichier principal
Vignette du fichier
478293_1_En_1_Chapter.pdf (604.74 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-02111532 , version 1 (26-04-2019)

Licence

Identifiers

Cite

Siwei Fu, Guifen Chen, Shan Zhao, Enze Xiao. Research and Application of Spark Platform on Big Data Processing in Intelligent Agriculture of Jilin Province. 11th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Aug 2017, Jilin, China. pp.1-12, ⟨10.1007/978-3-030-06179-1_1⟩. ⟨hal-02111532⟩
73 View
63 Download

Altmetric

Share

More