Um sistema para processamento de dados em tempo real aplicado à análise e monitoramento de rede

Abstract

As the number of devices connected to the network grows exponentially, more and more data is generated and transmitted in the form of continuous streams over the network. In this context, proper monitoring of computer networks becomes increasingly paramount. Late detection of anomalies often causes or substantially increases the risk of irreparable damage and makes a defense attempt unfeasible, what makes crucial to obtain up-to-date real-time information about the flow of data generated by network traffic. In this context, Big Data architectures for real time processing, such as Lambda and Kappa, have been highlighted in recent years. The present work aims to develop a system based on Lambda architecture, applied to the monitoring and processing of network traffic data flow, integrating different open source tools to perform from data collection to data storage. An experimental analysis of the system takes place simulating over 320,000 sudden connections on the monitored local network, which the system took an average of 1 minute and 36 seconds to process and save the results on the database.