Data Ingestion and Inspection for Smart City Applications

Smart cities are distributed heterogeneous systems of systems connected to each other via a variety of heterogeneous data streams involving multiple stakeholders and organizations. This complexity is reflected also in the dataset that have to be managed to provide a concrete and useful real time service to the citizens. The data ingestion phase is critical for the whole services, since it has to preserve the information and connect the new data with old data and establish right connections with city entities. This paper describes data ingestion and inspection in the Snap4City open source scalable Smart aNalytic APplication builder, with a specific focus on how heterogeneous data is represented, how its quality is inspected, and how to develop ingestion procedures in an efficient manner. The Snap4City ingestion processes are based on a semantic and unified data ingestion model, capable of aggregating different types of data. Performance comparison of different data ingestion modalities is presented.