Reference : AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterat...

	Document type :	Scientific congresses, symposiums and conference proceedings : Paper published in a book
	Discipline(s) :	Engineering, computing & technology : Computer science
	To cite this reference:	http://hdl.handle.net/10993/45326

Title :	AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing
Language :	English
Author, co-author :	Ellampallil Venugopal, Vinu [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
	Theobald, Martin [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
	Chaychi, Samira [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
	Tawakuli, Amal [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
Publication date :	1-Sep-2020
Main document title :	AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing
Publisher :	IEEE
Pages :	51-58
Peer reviewed :	Yes
On invitation :	Yes
Audience :	International
ISBN :	978-1-7281-9924-5
Event name :	32nd International Symposium on Computer Architecture and High Performance Computing
Event date :	from 08-09-2020 to 11-09-2020
Keywords :	[en] Stream data processing ; Big Data ; sustainable-throughput
Abstract :	[en] Distributed Stream Processing Engines (DSPEs) are currently among the most emerging topics in data management, with applications ranging from real-time event monitoring to processing complex dataflow programs and big data analytics. In this paper, we describe the architecture of our AIR engine, which is designed from scratch in C++ using the Message Passing Interface (MPI), pthreads for multithreading, and is directly deployed on top of a common HPC workload manager such as SLURM. AIR implements a light-weight, dynamic sharding protocol (referred to as “Asynchronous Iterative Routing”), which facilitates a direct and asynchronous communication among all worker nodes and thereby completely avoids any additional communication overhead with a dedicated master node. With its unique design, AIR fills the gap between the prevalent scale-out (but Java-based) architectures like Apache Spark and Flink, on one hand, and recent scale-up (and C++ based) prototypes such as StreamBox and PiCo, on the other hand. Our experiments over various benchmark settings confirm that AIR performs as good as the best scale-up SPEs on a single-node setup, while it outperforms existing scale-out DSPEs in terms of processing latency and sustainable throughput by a factor of up to 15 in a distributed setting.
Funders :	University of Luxembourg - UL
Permalink :	http://hdl.handle.net/10993/45326
Other URL :	https://conferences.computer.org/sbacpad/pdfs/SBAC-PAD2020-2fQ2vSYuhExkpkZ9tActSv/992400a051/992400a051.pdf

	File(s) associated to this reference
	Fulltext file(s): File Commentary Version Size Access Limited access sbac_cr.pdf Author postprint 573.75 kB Request a copy

All documents in ORBi^lu are protected by a user license.

 

University of Luxembourg Library |  Feedback |  Legal notices Site Map