Architecture

Current trends towards the use of big data technologies in the context of smart cities suggest the need of developing novel software development ecosystems upon which advanced mobility functionalities can be developed. CLASS is creating a novel software architecture that allows users to develop and execute advanced big-data applications in an efficient way. The goal of this new software infrastructure is to allow collecting, storing, analysing and processing vast amount of geographically-distributed data, in order to transform it into valuable knowledge for public sector, private companies and citizens.

Cameras to collect real-time mobility data for data analytics

Applying big-data technologies to smart cities field applications entails many challenges: from processing data across the compute continuum (from edge to cloud), to predicting real-time responses, and employing a programming model that can mix different application program interfaces (APIs) and models. The CLASS platform is facing these needs by integrating technologies from different computing domains into a single development framework. This allows to efficiently combine data-in-motion and data-at-rest analytics along the compute continuum, while providing real time guarantees.

The compute continuum addressed by the CLASS project

The CLASS software architecture

CLASS software architecture components

The CLASS software architecture components can be found in the dedicated CLASS github channel and they include:

Data Analytics Platform

This layer exposes the interfaces for the deployment of big-data analytics methods. Analytics workloads in CLASS are intended to be wrapped as a set of OpenWhisk actions (functions), which are pieces of logic that can be explicitly invoked or executed in response to events.

EXtended PREdictability ServerlesS (EXPRESS) is a serverless cloud platform, based on Apache OpenWhisk, that executes functions (called actions) in response to events, at scale, and with enhancements for predictable computation.

Link: https://github.com/class-euproject/express

This layer also exposes a serverless-based map/reduce engine, called PyWren, that executes user’s Python code and its dependencies on serverless computing platforms. Map/Reduce is a generic API, originally from Google and later evolved publically, for processing a large amount of data where each data element or record is processed in a similar way, such as applying a mathematical operation. Naturally, map/reduce can scale out, and efficiently apply parallel processing techniques for performing its computation on a data set. In CLASS, both PyWren and COMPSs support map/reduce, while augmenting it with latency awareness for real-time capabilities.

Link: https://github.com/class-euproject/pywren-ibm-cloud

Moreover, the data analytics tooling includes evaluation capabilities. Owperf is used as a performance evaluation tool for OpenWhisk-based deployments. Combined with an extended monitoring infrastructure, it can provide deep feedback about performance of complex applications.

Link: https://github.com/class-euproject/owperf

 

Computation Coordination and Distribution Framework

In CLASS, data analytics operations are encapsulated within COMPSs tasks, that will be distributed across the continuum with the objective of minimizing the response time of the overall workflow, while providing real-time guarantees.

COMP Superscalar (COMPSs) is a portable programming environment based on a tasking execution model, whose main objective is to facilitate the parallelization of sequential source code (written in Java, C/C++ or Python programming languages) in a distributed and heterogeneous computing environment.

Link: https://github.com/class-euproject/compss

dataClay is used as the CLASS shared data storage backbone among the different components of the platform. dataClay, developed at BSC, is a distributed data store that enables applications to store and access objects in the same format they have in memory, and executes object methods within the data store.

Link: https://github.com/class-euproject/dataclay-class

Cloud Layer

Rotterdam is a Container-as-a-Service (CaaS) layer, developed by ATOS, that abstracts the cloud infrastructure details away from the data analytics service developers. It provides not only service life cycle management but also guaranteed performance of workloads.

Link: https://github.com/class-euproject/Rotterdam

Edge Layer

tkDNN is a deep neural network library and toolkit for high performance inference on NVIDIA platforms. Concretely, a fully convolutional neural network deep learning method is used within the CLASS project to perform object detection and classification.

Link: https://github.com/class-euproject/tkDNN

Data Analytics Methods

The CLASS software architecture also includes a set of advanced data-analytics methods. Interestingly, all these data analytics engines are optimized to execute at both, the edge and the cloud side, providing the required flexibility needed to distribute the computation of complex data analytics workflows composed of different analytics frameworks across the compute continuum.

The software component that manages data analytics is a modified version of a state-of-the-art Deep Neural Network (DNN) called YOLOv3. The DNN has been retrained using different datasets with respect to the original implementation to obtain a higher precision of the output. YOLO is implemented with tkDNN.

Link: https://github.com/class-euproject/tkDNN

A Tracker, based on Extended Kalman Filter, is a fast method to track objects from a camera. After the DNN detects the bounding-boxes of the objects, the central bottom point of the bounding box is taken as a reference of that object, and on that the tracker is instantiated.

Link: https://github.com/class-euproject/tracker_CLASS

A parallel Complex Event Processor (pCEP) is a parallelized version of Atos’ Complex Event Processing (CEP), for state and event inference, i.e., identifying relevant data among the multitude of information arriving to the system (e.g., movement of objects such as vehicles or persons, and traffic information). The pCEP engine, based in the Dolce language, receives primitive events (e.g., obstacle detected) and, depending on the Dolce rules, produces complex events (e.g., probability of collision with the obstacle detected).

Link: https://github.com/class-euproject/pCEP