Skip to main content
All Posts By


How Performance Testing Improved the Nitrogen Release

By Blog

From Sai Sindhur Malleni

One of my responsibilities as a performance engineer working on OpenStack is to make sure that OpenStack scales, no matter what the use case is or the backend technologies being used. The flexibility to use multiple open source and proprietary backends to support the various services in OpenStack such as Neutron, Cinder and Glance is what makes OpenStack a force to reckon with in the cloud ecosystem.

As OpenStack adoption increases across verticals, the SDN controller which manages all of the virtual networking is taking more of a center stage. OpenDaylight, with its flexible and extensible architecture, support for multivendor networking devices, network programmability enabling control of the underlay and overlay, and tight integration with OpenStack is becoming the de facto choice for Neutron backend in NFV deployments.

Over the last 6 weeks, several colleagues and I were involved in a massive effort, spanning multiple teams and time zones, to test and improve the scale and performance of OpenDaylight. The scope of this work is in line with the objectives of the S3P WorkGroup and we are quite happy with the progress made thus far. This blog post goes over all the hardening that went into OpenDaylight in the scale and performance realms for the Nitrogen Release.

Our lab inventory consisted of 13 Dell R630 Nodes, with Intel Haswell processors (28 cores and 56 threads), 128G of memory and  Intel x710 quad port NIC. We were using custom-built Carbon SR-2 RPMs (custom built to test some patches before they were merged). We deployed in both a clustered and standalone configuration. Two different configurations were used:

  1. 3 OpenStack controllers, 3 ODLs clustered, 1 OpenStack undercloud and the rest of the nodes as compute nodes
  2. 1 OpenStack controller, 1 ODL, 1 OpenStack undercloud and the rest of the nodes as compute nodes

Browbeat was used to orchestrate the tests with Rally, monitor the environment using Collectd/Graphite/Grafana stack and store test results in Elasticsearch. Browbeat takes a simple YAML-based configuration file of the control plane tests you want to run and orchestrates them on the OpenStack cloud. Some of the Rally scenarios that were run included creating networks, subnets, neutron ports, routers, security group rules, and booting VMs on subnets, with each done 500 “times” at varying concurrencies of  8, 16 and 32.  While the “times” denotes the total number of resources you want to create of each type, the concurrency denotes the number of resources you want to create concurrently. So, one could effectively vary the load on OpenDaylight using these two knobs.

One of the biggest issues on which we focused during this round of testing was out-of-memory errors and subsequent death of the OpenDaylight process when creating several Neutron resources at scale. Using our automation tools (Ansible/ Collectd/Graphite and Grafana), we were able to actively monitor the heap memory usage and use Eclipse MAT to analyze the hprof files dumped on OOM. Ansible was used to install collectd (light-weight daemon to monitor system resource usage) on the nodes, Graphite was the data store for this time series data and Grafana was used to visualize the data as graphs. This is a great example of using several open source technologies to make another open source technology better. The memory leaks we were observing were traced back to unclosed transactions in the openflowplugin. Our test team took this information back to the ODL upstream teams, who promptly addressed the issue with fixes included in the Nitrogen release. It was inspirational to see that level of collaboration in the community.

We also looked into the networking-odl v2 driver which is the glue that connects OpenStack Neutron to Opendaylight. A journal table is used by the driver to keep a queue of  the operations that occurred on neutron and need to be mirrored to the OpenDaylight controller. We identified several optimization possibilities in the way journaling was done. We also profiled the Galera database cluster used to house this table. With a few optimizations, we were able to achieve a 20x reduction in the CPU consumption of the mysqld process.

Clustering was another area of focus. We identified an issue where OpenStack VMs wouldn’t boot when clustered OpenDaylight configuration was being used. To give a bit of context, OpenStack considers a VM active only when the actual plumbing of the OVS interfaces and flows happens on the hypervisor, so OpenStack’s compute service Nova waits on the Neutron port to be set to active for a VM to be considered “active”. When using OpenDaylight, a websocket is used to communicate the port status information from the OpenDaylight controller to networking-odl. This in turns sets the port status in the Neutron database. In a clustered OpenDaylight setup there are 3 ODLs and 3 instances of networking-odl, one on each OpenStack controller. Haproxy assigns a Virtual IP (VIP) to the ODL cluster; this VIP could be on any one of the OpenDaylight nodes (not necessarily ODL cluster leader). However, Neutron events that trigger flow creation and activation of the operational port occur only on the leader, so when the VIP is not on the leader the websocket notifications aren’t established against the leader, causing a failure in communication of port status information between OpenDaylight and Neutron via networking-odl. The issue was fixed by having each networking-odl instance on the OpenStack controller establish a websocket with the local OpenDaylight member. This was a major win in hardening the integration points between OpenDaylight and OpenStack. A good amount of failover testing was also done to test the clustering feature of ODL.

This testing also led to the overall improvement of OpenStack as we were able to tune certain kernel parameters to help scale OpenStack to hundreds of networks and instances.  Overall, we are extremely confident that all the work that went into improving the scalability, stability and performance of OpenDaylight, means an OpenDaylight release that is better than ever before. In my colleague Daniel Farrell’s words,  “It seems to me that this was one of the more important stability improvements in ODL’s history. The combination of expert performance testers, dedicated hardware, close support from experts with direct access the testing environment and tight connections to the relevant upstream projects had never happened so well.”

I have to say that I have had a very positive experience working with the upstream OpenDaylight community. It is really heartening to see the developers, packaging team and performance engineers come together and do more than what an individual can ever do. As a performance engineer, I was thrilled to see the OpenDaylight community treat performance and scale as first class citizens and focus on them pre-release vs post-release.


Introducing The Linux Foundation’s Open Source Networking Days

By Blog

This post, by Arpit Joshipura, first appeared on

One of my primary goals at The Linux Foundation is to foster innovation across the entire open source networking ecosystem. This involves coordinating across multiple open source projects and initiatives and identifying key areas for collaboration to create an open source networking stack.

We are working across the entire ecosystem with industry-leading partners — from developers to service providers to vendors — to unify various open source components and create solutions that will accelerate network transformation. As part of this journey, I am pleased to introduce Open Source Networking Days (OSN Days), a series of free events that are hosted and organized by local user groups and The Linux Foundation members, with support from our projects, including DPDK,, ONAP, OpenDaylight, OPNFV, PNDA, and others.

OSN Days are a fantastic opportunity for network developers and users to learn how ONAP, OPNFV, OpenDaylight  and other open source initiatives are changing NFV/SDN orchestration and networking solutions. Stops on the tour include: ParisMilan, Stockholm, London, Tel Aviv, and Japan. Register today for an upcoming OSN Day in your region.

The day-long events will start with a plenary session where attendees will hear from site hosts and The Linux Foundation speakers on the state of the industry and the collaboration and touch points between projects that make up the open source networking stack. Presenters will also explore how business opportunities like 5G and IoT are enabled by network transformation.  In the afternoon, events may feature technical sessions, tutorials, demonstrations, and workshops that empower attendees to participate, contribute, and deepen their knowledge of open source networking.

Our first OSN Day kicks off October 9 in Paris, followed by stops in Milan (October 12), Stockholm (October 13), London (October 16), Tel Aviv (October 19), and Japan (October 19). Thanks to our incredible site hosts Amdocs, ATOS, Cloudify, Ericsson, Huawei, NEC, Orange, RedHat, SUSE and Vodafone, along with our high-caliber roster of speakers, for helping to make these OSN Days a reality!

More details about the events, including site-specific agendas, registration info, and details on hotel and travel, can be found here: If you have any questions, or would like to host an event yourself in the future, please email

 –Arpit Joshipura

Enterprise Networking Planet: OpenDaylight Carbon SDN Platform Expands Performance and Scalability

By In The News

The OpenDaylight Project released its newest milestone today with the official general availability of the new Carbon platform.

Carbon is the sixth major release from OpenDaylight and follows the Boron release that debuted in September 2016. OpenDaylight first started in April 2013 as a Linux Foundation Collaborative Project, with the goal of building an open SDN platform.

Phil Robb, Interim Executive Director for OpenDaylight told EnterpriseNetworkingPlanet that from the beginning, OpenDaylight was designed to be a general-purpose development platform for building programmable networks He noted that early releases of OpenDaylight put the core platform in place.

Virtualization Review: New OpenDaylight SDN Controller Boosts IoT Support

By In The News

It also includes an integrated framework for network functions virtualization (NFV) management.

The OpenDaylight Project has produced the sixth version of its open source software-defined networking (SDN) controller, describing the “Carbon” release as more mature, with more support for Internet of Things (IoT) deployments, among other enhancements…

In the new Carbon release, the project focuses on better IoT, Metro Ethernet and cable operator support, along with improved integration with the OpenStack project, which provides a cloud OS. Other areas of focus included integrated NFV management and S3P, standing for security, scale, stability and performance.

Fierce Telecom: OpenDaylight Foundation shows off Carbon for Metro Ethernet and the IoT

By In The News

While the Open Network Automation Platform (ONAP) is adding members and is working on a wide range of technologies to support software defined networking (SDN), the OpenDaylight Foundation,yet another project overseen by the Linux Foundation, has released the sixth iteration of its open source SDN controller.

The revision, named Carbon, improves support for Metro Ethernet and for cable networks, as well as internet of things (IoT) deployments, OpenDaylight said. The previous revision, Boron, was released last September.

Telecom Asia: OpenDaylight announces Carbon SDN platform

By In The News

The OpenDaylight Project, which delivers an open source platform for programmable, software-defined networks, has announced the sixth major release of its open source SDN controller. The latest release, named Carbon, features enhancements designed to better support for metro Ethernet and cable operators, as well as Internet of Things (IoT) deployments, OpenDaylight said. The previous revision, Boron, was debuted in September 2016.


By Uncategorized

Platform For Network-Driven Businesses

Boron (B) is the fifth release of OpenDaylight (ODL), the leading open source platform for programmable, software-defined networks. OpenDaylight has become the de facto platform for service providers and enterprises making the transition to SDN. Learn more about how OpenDaylight is being used in different kinds of organizations here.

With the Boron release, OpenDaylight marks a new milestone in technology and community maturity. Boron is the result of significant collaboration between users, network equipment vendors, and the growing ecosystem of systems integrators and application developers building OpenDaylight-based solutions for unique use cases and user requirements. Building on their deployment experiences and experimentation, several leading user organizations have invested their own resources into the OpenDaylight developer community, originating more than half of the new projects proposed for Boron.

Boron brings a strong practical focus on two leading types of deployments, with enhancements to cloud and NFV support as well as large-scale network engineering. New operational tooling, enhanced performance and improved documentation round out the improved deployment experience. Boron also provides new tooling and documentation to support application developers, as well as greater integration with larger industry frameworks from OPNFV and OpenStack to CORD and Atrium Enterprise.

New Features in OpenDaylight Boron

Common SDN toolchains

Boron includes improvements to several projects related to cloud and NFV, including OpenFlow, OVSDB and OVS/ The Genius project is an application composition pipeline, useful for service function chaining. OpenStack-related capabilities have been re-architected within a unified development framework for better scalability and performance, including clustering and persistence. The NetVirt project improves coordination between OpenStack Neutron and the OpenDaylight controller, as well as enhanced support for IPv6, Security Groups (via OpenFlow configuration), VLANs, and other important capabilities.

Boron increases support for new methods of approaching classic traffic engineering challenges faced by telco networks. OpenDaylight already provides an exceptionally broad range of southbound protocols, and now continues to build out and standardize how protocols such as OpenFlow, BGP, PCEP, MPLS and NETCONF are used, and how to model them effectively.

Operational tooling

With active deployments supporting revenue-generating services, operational tooling is becoming critical for network managers. The Cardinal project monitors the health of the controller itself, delivered as a service to existing, deployed networki monitoring and analytics tools. The Centinel analytics engine now enables new levels of end-to-end data collection and machine learning to support performance monitoring and bandwidth managment across WAN links. EMAN is an energy management plug-in based on the IETF EMAN spec. The OCP plug-in enables operators to jointly manage radio and fronthaul network provisioning.

Application developer tooling

YangIDE provides new editing support for anyone who needs to work with Yang models — either create new models, make changes to existing models, or to integrate models into new services and custom appliances. NetIDE faciliates interoperability between open source SDN controllers: apps written to Ryu/Floodlight/Pyretic controllers can now be run on OpenDaylight-managed infrastructure. A new approach to clustering in OpenFlow provides high availability to non-distributed, or “singleton” applications without introducing overhead and latency from maintaining and managing multiple instances at once; this allows developers to write to OpenDaylight without needing to know the details of the platform HA implementation.

Architecture Diagram

View the complete ODL Feature List and descriptions.

Support OpenDaylight

By Uncategorized

The OpenDaylight Foundation promotes and advances the global development, distribution and adoption of the ODL platform. The work we do is sustained by our members, who are committed to collaborating within an open governance framework to solve our industry’s shared challenges.

Learn more about:

  • How the Project is governed, including the Technical Steering Committee and Board of Directors
  • Our advisors and ambassadors
  • Becoming a member, to support and advance OpenDaylight and the open SDN ecosystem