Project Name

Data Collection Using SNMP Protocol with Apache NiFi

Industry

Telecommunication

Technology

BigData, Java, SNMP, Cassandra, ZooKeeper

Overview

Our client was a leading service provider in the telecommunications industry who works on managing the data and entire data collection. They were facing issue in managing massive number of modems on the production system and need a supportive system to rescans the single and multiple modems.

data-collection-with-apache-nifi-banner-sider

Challenges

The main challenge is that there were millions of modems on the production system for which the client needs data to be collected at least 2 times a day.
The client was searching for support system that helps them for on-demand rescans on single and multiple modems.
Issue in collecting the data for the optimal usage of the primary memory on the servers.

Our Solution

We have provided a robust solution to our client that includes:

With leveraging Apache NiFi, our team had created a multiple custom processors for handling data collection from a several type of devices including CMTS or modems and for moving the data directly through the pipeline.
Through Apache NiFi, it becomes possible for us to keep the usage of primary memory below the critical threshold as the NiFi stores the data on the secondary storage and loads it onto the memory once it is needed by the pipeline.
Utilizing the Apache Cassandra, our team gained the ability to save the data for millions of modems with quick read access, fault tolerance, and high data availability.
Even, we had used snmp4j library with the custom processors for direct communication with the modems and CMTS devices for data polling.
Our team had used Zookeper with custom changes for handling the Apache NiFi cluster that contains multiple NiFi nodes and works on Job orchestration, inventory management, and load balancing.

Data Flow Diagram

Conclusion

Hence, we had successfully implemented Apache NiFi with SNMP usage. It becomes possible for us to collect the data for millions of modem devices multiple times during the day while also supporting on demand rescans and simultaneously done transformation and processing on polled data by implementing custom processors in NiFi while offloading the data into Cassandra tables for quick read access and high data availability for the users.