Design and Implementation of Real Time Data Processing System Based on Spark Streaming

  • Huang He Guangdong Innovative Techical College
DOI: https://doi.org/10.18282/l-e.v10i3.2451
Keywords: Spark Streaming, Real-time data processing, The system design

Abstract

With the rapid development of Internet technology, people have more abundant ways to obtain information through the Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more complicated. In order to implement data implementation, the industry is generally combined with different data types and business scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost, and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data analysis and processing system based on Spark Streaming.

References

[1] Shi Wei. Design and Implementation of Real Time Data Processing System Based on Spark Stream [J]. Modern Information Technology, 2020, 4 (20): 10-12.

[2] Li Tianxi. Research and Implementation of Experimental Data Processing System based on Spark Streaming [D]. Shaanxi: Xidian University,2015.

[3] Xu Yifeng, Feng Dajun, ZHANG Hanwen, et al. Application of electronic technique,2017,43(9):98-100,105.

[4] Zhang Muwei, Dong Feng. Design of real-time data processing system for infrared camera based on FPGA [J]. Infrared,2016,37(4):1-6,32.

Published
2021-11-07
Issue
Vol. 10 No. 3 (2021)

Authors contributing to this journal agree to publish their articles under the  Creative Commons Attribution-Noncommercial 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit, that the work is not used for commercial purposes, and that in the event of reuse or distribution, the terms of this license are made clear. With this license, the authors hold the copyright without restrictions and are allowed to retain publishing rights without restrictions as long as this journal is the original publisher of the articles.