DIsCO: DynamIc Data COmpression in Distributed Stream Processing Systems - Distributed Applications and Interoperable Systems (DAIS 2017) Access content directly
Conference Papers Year : 2017

DIsCO: DynamIc Data COmpression in Distributed Stream Processing Systems

Abstract

Supporting high throughput in Distributed Stream Processing Systems (DSPSs) has been an important goal in recent years. Current works either focus on automatically increasing the system resources whenever the current setup is inadequate or apply load shedding techniques discarding some of the incoming data. However, both approaches have significant shortcomings as they require on the fly application reconfiguration where the application needs to be stopped and re-uploaded in the cluster with the new configurations, and can lead to significant information loss. One approach that has not yet been considered for improving the throughput of DSPSs is exploiting compression algorithms to minimize the communication overhead between components especially in cases where we have large-sized data like live CCTV camera reports. This work is the first that provides a novel framework, built on top of Apache Storm, which enables dynamic compression of incoming streaming data. Our approach uses a profiling algorithm to automatically determine the compression algorithm that should be applied and supports both lossless and lossy compression techniques. Furthermore, we propose a novel algorithm for determining when profiling should be applied. Finally, our detailed experimental evaluation with commonly used stream processing applications, indicates a clear improvement on the applications’ throughput when our proposed techniques are applied.
Fichier principal
Vignette du fichier
450046_1_En_2_Chapter.pdf (502.99 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01800129 , version 1 (25-05-2018)

Licence

Attribution

Identifiers

Cite

Nikos Zacheilas, Vana Kalogeraki. DIsCO: DynamIc Data COmpression in Distributed Stream Processing Systems. 17th IFIP International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2017, Neuchâtel, Switzerland. pp.19-33, ⟨10.1007/978-3-319-59665-5_2⟩. ⟨hal-01800129⟩
460 View
79 Download

Altmetric

Share

Gmail Facebook X LinkedIn More