Integrating ODL with PNDA open source data analytics platform
Blog originally posted on PNDAemic: The PNDA.io blog.
So you want to integrate OpenDaylight with PNDA!
It’s really easy to aggregate data into a PNDA instance. But it takes a little more work to stream data out of an OpenDaylight instance. There are three steps to set this up:
- Pick a data source, i.e. an OpenDaylight application that can supply data to an OpenDaylight event topic.
- Configure the OpenDaylight event aggregator to collect the desired data and publish it on a topic.
- Deploy the OpenDaylight Kafka Plugin and configure it to forward events to the Kafka message bus of your PNDA instance.
THE DATA SOURCE
If you have NECONF enabled devices that send notifications then the odl-netconf-connector might be sufficient to get started. Alternatively you can write a new OpenDaylight application to collect your desired dataset. I have chosen to write an application that uses SNMP to collect IF-MIB:ifTable data. You can get the application from github here: https://github.com/donaldh/if-table-collector.
Once built, the if-table-collector can be started as a standalone karaf instance:
$ cd if-table-collector
$ ./karaf/target/assembly/bin/karaf
Check that the if-table-collector is running:
opendaylight-user@root>bundle:list | grep if-table
270 | Active | 80 | 0.1.0.SNAPSHOT | if-table-collector-api
274 | Active | 80 | 0.1.0.SNAPSHOT | if-table-collector-impl
Now you can tell the application to collect ifTable data from a device. The application uses an augmentation of topology-netconf to enable SNMP collection for a device. Here’s an example:
POST http://localhost:8181/restconf/config/network-topology:network-topology/topology/topology-netconf
Content-Type: application/xml
<node xmlns=”urn:TBD:params:xml:ns:yang:network-topology”>
<node-id>{node-name}</node-id>
<host xmlns=”urn:opendaylight:netconf-node-topology”>{ip-address}</host>
<port xmlns=”urn:opendaylight:netconf-node-topology”>{netconf-port}</port>
<username xmlns=”urn:opendaylight:netconf-node-topology”>{username}</username>
<password xmlns=”urn:opendaylight:netconf-node-topology”>{password}</password>
<tcp-only xmlns=”urn:opendaylight:netconf-node-topology”>false</tcp-only>
<keepalive-delay xmlns=”urn:opendaylight:netconf-node-topology”>0</keepalive-delay>
<snmp-community xmlns=”urn:net:donaldh:if-table-collector”>{snmp-community}</snmp-community>
<poll-interval xmlns=”urn:net:donaldh:if-table-collector”>60</poll-interval>
</node>
You should see the device being polled every 60 seconds:
2017-02-22 16:48:24,079 | INFO | Executing poll cycle for 10.1.1.105 …
2017-02-22 16:48:24,460 | INFO | Polled 6 rows.
CONFIGURING THE EVENT AGGREGATOR
The OpenDaylight event aggregator is responsible for gathering events from the desired data sources within OpenDaylight and publishing them on an specific topic on the OpenDaylight message bus. To get started, we will use a simple topic configuration which matches all notification names for all nodes:
POST http://localhost:8181/restconf/operations/event-aggregator:create-topic
{ “event-aggregator:input”:
{ “notification-pattern”: “*”, “node-id-pattern”: “.*” }
}
DEPLOYING THE KAFKA PLUGIN
The odl-kafka-plugin can be built and added to any OpenDaylight installation. I have a fork of it here that builds for OpenDaylight Boron-SR2: https://github.com/donaldh/odl-kafka-plugin. Once you have built the kafka-agent plugin, you can copy it to the karaf deploy directory to have it auto-deploy into the running karaf instance.
The odl-kafka-plugin simply registers for messages on the OpenDaylight message bus and sends them on to the kafka message bus. It can be configured to listen for specific topics or, by default, listen for any topic. I used the simplest configuration which listens for any topic:
PUT http://localhost:8181/restconf/config/kafka-agent:kafka-producer-config
{
kafka-producer-config: {
kafka-broker-list: “{kafka-host}:9092”,
kafka-topic: “odl”,
compression-type: “none”,
message-serialization: “avro”,
avro-schema-namespace:”com.example.project”
}
}
VERIFYING THE END RESULTS
You can use the Kafka console consumer to verify that messages are being received by the Kafka bus. The messages are AVRO encoded so the textual representation looks a bit weird:
% ./bin/kafka-console-consumer.sh –bootstrap-server $(KAFKA_HOST):9092 –topic odl
?????V$if-table-collector172.16.1.105?
<?xml version=”1.0″ encoding=”UTF-8″?><payload xmlns=”urn:cisco:params:xml:ns:yang:messagebus:eventaggregator”><source>172.16.1.105</source><message><IfEntryBuilder><IfAdminStatus>Down</IfAdminStatus><IfInDiscards>0</IfInDiscards><IfInErrors>0</IfInErrors><IfInOctets>0</IfInOctets><IfInUcastPkts>0</IfInUcastPkts><IfInUnknownProtos>0</IfInUnknownProtos><IfIndex>6</IfIndex><IfMtu>1514</IfMtu><IfOperStatus>Down</IfOperStatus><IfOutDiscards>0</IfOutDiscards><IfOutErrors>0</IfOutErrors><IfOutOctets>373</IfOutOctets><IfOutUcastPkts>2</IfOutUcastPkts><IfPhysAddress/><IfSpeed>4294967295</IfSpeed><IfType>Eon</IfType></IfEntryBuilder></message></payload>
…
Assuming you have already configured the PNDA instance to accept kafka messages on the “odl” topic, you will start to see ifTable data in the HDFS datastore.