By Abhijit Kumbhare, OpenDaylight TSC Chair 

The OpenDaylight community is pleased to announce the availability of the 12th release of the most widely deployed open source SDN controller platform, OpenDaylight Magnesium!  In addition to ongoing deployments, maturity and stability, the release brings two new projects, new features that help with service provider use cases, deeper cross-project collaboration with ONAP, and improvements in S3P (stability, security, scalability and performance). 

OpenDaylight celebrates its seventh year as a project (!!!) and introduces Magnesium as the world enters an unprecedented time, unlike anything this generation has ever seen. I am continually impressed and humbled by the strength and cohesiveness of the ODL community, over the past several years now, especially, in this uncertain time. Thank you to all involved! Read on for release highlights. 

Magnesium Highlights

• New projects: The Magnesium release includes two new projects—DetNet and Plastic. The new projects deal with deterministic networking for performance sensitive traffic and model-to-model translations, respectively. 
• New Features for Service Provider Use Cases: The community contributed several new features to the TransportPCE and BGPCEP projects, making OpenDaylight even better suited to service provider use cases. TransportPCE took a step forward in the control of open optical infrastructure domains while the BGPCEP project added support for missing features from RFC 5440 and a BGP-LS topology provider for segment routing. In addition to these two major areas of enhancements, the Genius project, that provides generic network interfaces, utilities, and services, added support for Bidirectional Forwarding Detection (BFD) for OpenFlow tunnels. This makes it easier to detect if the tunnel is bidirectional or not.
• ONAP Integration: the OpenDaylight community provided an optimized distribution of the Magnesium release to the ONAP project so that OpenDaylight can be consumed efficiently by a number of ONAP controllers. Distribution is being integrated successfully by the ONAP Common Controller Software Development Kit (CCSDK) project and will be available in the upcoming ONAP Frankfurt release.
• Improved Functionality, Scalability, and Stability: The Magnesium release continued to make progress around S3P. For example, the OpenFlow plugin and the Netvirt project improved cluster stability, scale, and performance. The projects also include updated documentation, upgrade process description, and numerous bug fixes. The Daexim (data export/import) project has improved scalability and supports processing of very large data sets. Projects such as Genius, OVSDB, NETCONF, and AAA include a variety of improvements in the areas of scalability, performance, and bug fixes. Moreover, the Magnesium release includes Java Developer Kit (JDK) 11 that brings with it a number of security features such as Transport Layer Security (TLS) 1.3, newer ciphers, key agreement improvements, and root certificate additions. In total, Magnesium includes 70+ improvements and bug fixes!

In addition to new features, projects, integrations, and updates, ODL is seeing ongoing industry momentum. Member company Lumina Networks contributed to the Plastic and BGPCEP projects and also released updates to the  Lumina SDN Controller 11.2.0 to align to theOpenDaylight Sodium release.An early version of the Magnesium release TransportPCE project was used in a live demo at the Optical Fiber Communications Conference (OFC) 2020 in San Diego, CA, to show interoperability of OpenROADM equipment from six suppliers and to control a low latency optical layer build from ROADMs and OTN flexponders. Orange played a central role in moving the project forward. And PANTHEON.tech continued their contributions to the core OpenDaylight platform as well as individual plugins; specific contributions covered JDK 11, YANG Parser, In-memory data tree, OVSDB, and distributed datastore efforts.

Visit the Magnesium Release Homepage

Download Magnesium

Comments from key contributors:

PANTHEON.Tech: “PANTHEON.tech’s focus remained on the core platform, while we also dabbled into individual plugins to deliver significant performance, scalability and correctness improvements. We have successfully transitioned the OpenDaylight codebase to require Java 11 and improved the YANG Parser, In-memory Data Tree, OVSDB Project & Distributed Datastore.”

Orange: “TransportPCE magnesium release is a major step forward in the control of open optical infrastructure domains. It completes previous releases focusing on an OpenROADM based WDM layer, proposing an experimental support for OTN, that will be hardened over the magnesium release train. The Northbound interface is upgraded to OpenROADM Service 5.1. Current set of APIs have been enriched to expose an abstracted WDM/OTN topology in accordance with T-API version 2.1.2 to higher layer controllers/orchestrator and to provide an interconnection with the GNPy tool for path calculation/validation including nonlinear impairments. An experimental inventory feature is also introduced for OpenROADM 1.2.1 devices, planning an extended support to 2.2.1 in the next magnesium releases.”

Lumina Networks: “The Magnesium release is a great example of the community’s commitment to enabling open control of multi-vendor networks. With this release, we were happy to represent the interests of several network operators to continue to progress to the functionality and scalability of heterogeneous environments. The Plastic project in particular, is a data model translator which breaks the dependencies on vendor libraries and plugins. We’re thrilled to work with the community on projects like these which ease the extensibility of open source network control.”

This post originally appeared on the Inocybe Technologies blog. 

With development efforts focused on code cleanup, bug fixing, and security, OpenDaylight Oxygen showcases platform maturity with a keen focus on quality.  Although the community has rallied around stabilizing and improving existing functionality to reassure operational reliability, there were also some great features contributed.

Under the hood, all ODL projects upgraded odlparent and yangtools versions.  This was a cross project effort involving every project in the Oxygen distribution.  Changes adapted from the odlparent upgrade involve enforcing better code quality across the project through enhanced checkstyle and findbugs rules, as well as dependency upgrades.  One noteworthy upgrade, the move from karaf 4.0.9 to 4.1.3, enables runtime resolution of feature versions based on feature ranges. This unlocks developer agility, catalyzing the adaptation of future upgrades.  Yangtools contributions centralized around bug fixes and better code organization. Another noteworthy change is that the in-memory data tree now enforces mandatory leaf node presence for operational store by default.  This behaviour change may cause a few surprises during application upgrade, and can be disabled as a stop-gap measure, although it is not recommended.

The rest of the kernel contributors also focused mostly on bug-fixing during the Oxygen release, although there were a few other noteworthy changes.  The MD-SAL project team continued development on the second version of the binding specification, which will be utilized in future releases of ODL. The current plan is to stick with the first version of the binding specification for Fluorine though, and the community is working to close several gaps and deficiencies in what is currently available.  The NETCONF development team contributed functionality to utilize key based authentication with southbound NETCONF devices, which hitherto was limited to basic authentication. The AAA team debuted a new MD-SAL based authentication Realm which enables replication of ODL account information across a cluster, although it is not enabled by default.

A lot of work was done to complete the transition of all ODL projects off the configuration subsystem.  The configuration subsystem was replaced with blueprint many releases ago, but has been maintained as a deprecated system to ensure proper transition for non-ODL projects.  There are plans to remove the configuration subsystem in Fluorine, since the code is costly to maintain further into the future. However, there may be some compatibility layer maintained in Fluorine to bridge the gap before complete removal.  Additional work was done to transition projects off the DataChangeListener (DCL) in favor of DataTreeChangeListener (DTCL), which showcases much better performance. The controller development team plans to remove DCL APIs in Fluorine, so existing applications must migrate to using DTCL.  The Removal of core APIs is tricky, since the ODL development community does not have much insight into how they are consumed outside of the open-source codebase. However, the general strategy is to deprecate, maintain for several releases with clear messaging of upcoming demise, followed by actual removal.  Since DCL has been deprecated since Beryllium, it is time to finally get rid of it altogether. It also allows for continued transition off controller MD-SAL APIs, which have unfortunately been carried as baggage for a long time.

The protocol and application stacks received some upgrades.  BGP added support for EVPN VPWS Flexible Cross-Connect Service based on draft-sajassi-bess-evpn-vpws-fxc.  The netvirt & genius projects added support for IPv6 enabled L3VPNs utilizing BGPVPNs as well as support for MPLS/GRE tunnel creation for L3VPNS in OVS 2.8+.  Although less visible to end-users, these projects made great improvements in transaction chaining and threading under the hood. OVSDB and Openflowplugin projects focused efforts on closing several clustering bugs as well as creating more user-friendly libraries for utilizing functionality.  The Service Function Chaining project added support for statistics for Rendered Service Paths and bump-in-the-wire Service Functions, thus improving the usability of SFC from an operational standpoint. The JSON-RPC project was officially added to the release, and contains bindings to interact with ODL utilizing ZMQ.  This accelerates integration of non-Java based projects with the controller.

The pattern in the ODL development community is trending towards improving what exists rather than reinventing the wheel.  Each subsequent release comes with a facelift for existing projects, which should come as no surprise as the project continues to mature.

For additional information on the OpenDaylight Oxygen release, please visit The Linux Foundation’s Oxygen release blog: https://goo.gl/fhnVfh

Written by Ryan Goulding, Principal Software Engineer at Inocybe Technologies.

From Sai Sindhur Malleni

One of my responsibilities as a performance engineer working on OpenStack is to make sure that OpenStack scales, no matter what the use case is or the backend technologies being used. The flexibility to use multiple open source and proprietary backends to support the various services in OpenStack such as Neutron, Cinder and Glance is what makes OpenStack a force to reckon with in the cloud ecosystem.

As OpenStack adoption increases across verticals, the SDN controller which manages all of the virtual networking is taking more of a center stage. OpenDaylight, with its flexible and extensible architecture, support for multivendor networking devices, network programmability enabling control of the underlay and overlay, and tight integration with OpenStack is becoming the de facto choice for Neutron backend in NFV deployments.

Over the last 6 weeks, several colleagues and I were involved in a massive effort, spanning multiple teams and time zones, to test and improve the scale and performance of OpenDaylight. The scope of this work is in line with the objectives of the S3P WorkGroup and we are quite happy with the progress made thus far. This blog post goes over all the hardening that went into OpenDaylight in the scale and performance realms for the Nitrogen Release.

Our lab inventory consisted of 13 Dell R630 Nodes, with Intel Haswell processors (28 cores and 56 threads), 128G of memory and  Intel x710 quad port NIC. We were using custom-built Carbon SR-2 RPMs (custom built to test some patches before they were merged). We deployed in both a clustered and standalone configuration. Two different configurations were used:

1. 3 OpenStack controllers, 3 ODLs clustered, 1 OpenStack undercloud and the rest of the nodes as compute nodes
2. 1 OpenStack controller, 1 ODL, 1 OpenStack undercloud and the rest of the nodes as compute nodes

Browbeat was used to orchestrate the tests with Rally, monitor the environment using Collectd/Graphite/Grafana stack and store test results in Elasticsearch. Browbeat takes a simple YAML-based configuration file of the control plane tests you want to run and orchestrates them on the OpenStack cloud. Some of the Rally scenarios that were run included creating networks, subnets, neutron ports, routers, security group rules, and booting VMs on subnets, with each done 500 “times” at varying concurrencies of  8, 16 and 32.  While the “times” denotes the total number of resources you want to create of each type, the concurrency denotes the number of resources you want to create concurrently. So, one could effectively vary the load on OpenDaylight using these two knobs.

One of the biggest issues on which we focused during this round of testing was out-of-memory errors and subsequent death of the OpenDaylight process when creating several Neutron resources at scale. Using our automation tools (Ansible/ Collectd/Graphite and Grafana), we were able to actively monitor the heap memory usage and use Eclipse MAT to analyze the hprof files dumped on OOM. Ansible was used to install collectd (light-weight daemon to monitor system resource usage) on the nodes, Graphite was the data store for this time series data and Grafana was used to visualize the data as graphs. This is a great example of using several open source technologies to make another open source technology better. The memory leaks we were observing were traced back to unclosed transactions in the openflowplugin. Our test team took this information back to the ODL upstream teams, who promptly addressed the issue with fixes included in the Nitrogen release. It was inspirational to see that level of collaboration in the community.

We also looked into the networking-odl v2 driver which is the glue that connects OpenStack Neutron to Opendaylight. A journal table is used by the driver to keep a queue of  the operations that occurred on neutron and need to be mirrored to the OpenDaylight controller. We identified several optimization possibilities in the way journaling was done. We also profiled the Galera database cluster used to house this table. With a few optimizations, we were able to achieve a 20x reduction in the CPU consumption of the mysqld process.

Clustering was another area of focus. We identified an issue where OpenStack VMs wouldn’t boot when clustered OpenDaylight configuration was being used. To give a bit of context, OpenStack considers a VM active only when the actual plumbing of the OVS interfaces and flows happens on the hypervisor, so OpenStack’s compute service Nova waits on the Neutron port to be set to active for a VM to be considered “active”. When using OpenDaylight, a websocket is used to communicate the port status information from the OpenDaylight controller to networking-odl. This in turns sets the port status in the Neutron database. In a clustered OpenDaylight setup there are 3 ODLs and 3 instances of networking-odl, one on each OpenStack controller. Haproxy assigns a Virtual IP (VIP) to the ODL cluster; this VIP could be on any one of the OpenDaylight nodes (not necessarily ODL cluster leader). However, Neutron events that trigger flow creation and activation of the operational port occur only on the leader, so when the VIP is not on the leader the websocket notifications aren’t established against the leader, causing a failure in communication of port status information between OpenDaylight and Neutron via networking-odl. The issue was fixed by having each networking-odl instance on the OpenStack controller establish a websocket with the local OpenDaylight member. This was a major win in hardening the integration points between OpenDaylight and OpenStack. A good amount of failover testing was also done to test the clustering feature of ODL.

This testing also led to the overall improvement of OpenStack as we were able to tune certain kernel parameters to help scale OpenStack to hundreds of networks and instances.  Overall, we are extremely confident that all the work that went into improving the scalability, stability and performance of OpenDaylight, means an OpenDaylight release that is better than ever before. In my colleague Daniel Farrell’s words,  “It seems to me that this was one of the more important stability improvements in ODL’s history. The combination of expert performance testers, dedicated hardware, close support from experts with direct access the testing environment and tight connections to the relevant upstream projects had never happened so well.”

I have to say that I have had a very positive experience working with the upstream OpenDaylight community. It is really heartening to see the developers, packaging team and performance engineers come together and do more than what an individual can ever do. As a performance engineer, I was thrilled to see the OpenDaylight community treat performance and scale as first class citizens and focus on them pre-release vs post-release.

This blog was originally published on Inocybe Technologies’ blog here.

With the release of Nitrogen, OpenDaylight continues to solidify itself as the de facto open networking controller standard.  Although the release contains widespread enhancements and improvements,  the main focus of the OpenDaylight Nitrogen release is the major version upgrade of the underlying Apache Karaf container from 3.0.8 to 4.0.9.  Among other functions, Apache Karaf is used to coordinate ODL microservices and lifecycle, provide a common logging infrastructure, enable remote management through JMX and a unix-like shell, facilitate dynamic configuration and hot deployment, and provide a common set of base resources across the product.  As such, Karaf has several responsibilities and has served as a crucial piece of infrastructure to the overall OpenDaylight architecture since its inclusion in the Helium release. It’s kind of a big deal ;)!

Since ODL Karaf dependency management is centralized in the odlparent project, an upgrade may initially seem trivial—  just a one-liner version bump, right?  Unfortunately, that is far from the case.  As previously stated, each release of Karaf provides a set of common dependencies that should remain constant across applications in the container.  This is done in order to centralize on a base set of hypothetically stable, interoperable and secure dependencies based on what is available from upstream projects at the time.  Thus, even a minor version bump of the Karaf container version in odlparent can result in the need for a number of changes cascading across consuming projects in ODL— especially if Karaf’s upstream dependencies contain API changes.

Major versions of Apache Karaf, which often contain new features, dependencies, frameworks, and API changes, are arduous and expensive to consume.  Moving to a new Karaf version becomes similar to transferring an existing building onto a new foundation;  there will be some jagged pieces along the bottom that will need work to fit properly.  With that said, it is critically important for ODL to keep current with Karaf releases in order to consume security fixes and avoid upstream dependency end-of-life scenarios.  Centralization of common ODL dependencies in odlparent eases some of the heartburn associated with cross-project upgrades.  Inclusion of the functionality to utilize version ranges in ODL, which continues to be developed, will ease upgrade woes in future releases.  Additionally, a significant amount of testing infrastructure was added in the Nitrogen release timeframe to enhance runtime dependency resolution checks and identify problems earlier and more predictably.

Another noteworthy change which occurred in the Nitrogen timeframe is the decision of odlparent contributors to disaggregate the subproject release from the traditional simultaneous release plan.  This means that odlparent now releases independently of the rest of ODL projects, and consuming projects depend on a released version rather than a snapshot.  Snapshot artifacts are prone to change from day to day, which can cause nuisance to consumers.  For example, imagine that a snippet of code compiles on Tuesday, and without any changes, fails to compile Wednesday morning.  This problem is avoided by depending on released artifacts, as released artifacts should never change without a proper version bump.  Consuming projects can treat the released odlparent artifacts as true upstream dependencies, and upgrades of those dependencies are now consumed with greater precision and control.  Breaking odlparent out of the simultaneous release additionally allows the odlparent development team to continue to perform necessary upgrades of widely used core dependencies without breaking downstream projects, enabling a disaggregated and saner development workflow.  Although odlparent is the only project to break off from the simultaneous release during the Nitrogen timeframe, contributors of other projects have expressed interest in following a similar plan in the near future.  This is exciting for developers and users alike, as it assists in breaking down the traditionally monolithic release process of OpenDaylight into more manageable chunks.

Maturity of the platform and its community is recognized by the decision to invest in an infrastructure focused release.  Focus is garnered around not only doing things, but doing them right.  Upgrade of the Karaf version in ODL Nitrogen is a double-edged sword;  the foundation of ODL is greatly improved but at the expense of requiring the sole focus of an entire, albeit shortened, release cycle to adapt to changes.  The improvements weren’t free, but were necessary in order to continue to deliver a stable and secure platform.  Fortunately for ODL application developers and consumers, several articles and code examples are available which provide assistance in transitioning to the updated ODL Nitrogen release.  Additionally, the support lifecycle of the previous release, ODL Carbon, has been extended beyond the traditional lifespan in order to help ease transition woes.

If you’d like to learn more about the OpenDaylight Nitrogen Release, the ODL Foundation’s Nitrogen blog is available here: https://www.opendaylight.org/blog/2017/09/26/opendaylight-introduces-nitrogen. Dive in and download OpenDaylight’s Nitrogen release today from: https://www.opendaylight.org/downloads.

Written by Ryan Goulding, Principal Software Engineer at Inocybe Technologies.

This post was originally published on Inocybe Technologies blog, reposted with their permission.

End users can rest assured that the early days of OpenDaylight releases that were jam-packed with tens-to-hundreds of new under-supported features are far gone. The latest release, Carbon, showcases the maturity and production-grade quality that Platform users have come to expect. Carbon provides significant improvements to security, stability and network programmability.

A driving goal for the OpenDaylight Carbon release is to improve the stability and reliability of ODL services.  Namely, several projects are converted to use Aries Blueprint for service activation over the bespoke configuration subsystem, an effort which was started in Boron and is improved in Carbon.  Blueprint is better documented and easier to debug, resulting in a more effective and satisfying application development experience.  Since Blueprint supports parallel service activation, there is less latency between starting the controller and utilizing the provided services.  Upgradability is improved through the Blueprint adoption, since efforts are made to separate application configuration from code wiring.  This is useful since most operators upgrading OpenDaylight wish to maintain configuration between releases, but pick up internal wiring changes.

Initial groundwork to add Apache Karaf 4.X features for each project was performed in hopes of transitioning to the newer container in the Nitrogen release.  Additionally, enhanced testing was added to ensure that features import all of the appropriate runtime bundles, improving stability of ODL features.  This groundwork should greatly help the community developers to perform the very non-trivial Karaf upgrade during the Nitrogen release cycle.

The RFC 6020 implementation of the YANG 1.0 Data Modeling Language is superseded by an implementation of RFC 7950, the YANG 1.1 Data Modeling Language.  For application developers this means that they’re now able to use YANG 1.1 constructs in their YANG models. On a similar note, interoperability with southbound NETCONF devices utilizing RFC 7950 is made possible in the Carbon release.

The clustered NETCONF implementation is greatly stabilized through re-architecture around the cluster singleton service, as well as greatly increased test coverage.  End users can expect a consistent clustered NETCONF experience to that of the Boron release, but have more peace of mind surrounding the stability of NETCONF in a distributed controller deployment.

A forward looking version of the MD-SAL Binding Specification version 2 is included in the Carbon release, though there are not yet any consuming applications.  The new version of the binding specification solves several deficiencies discovered in the original binding specification.  This implementation is Twirl-based, which has a similar function to the xtend implementation in the V1 spec, but generates the code in Scala instead of Java.  Don’t worry about running out to learn Scala; the generated Scala code is injected into the Java Runtime Environment, and is accessible to traditional Java clients.

Carbon contains an implementation of the recently (and finally) standardized RFC 8040, RESTCONF.  Hitherto, OpenDaylight users are probably most familiar interacting with the RESTCONF Draft 02 API.  The DRAFT 02 API still exists for compatibility purposes, since many pieces of software still rely on that API contract.  The new RFC 8040 RESTCONF API implementation is made available through a separate endpoint.  Users are encouraged to start exploring and using the standard version of the API, since it is still unclear how long the community should support the DRAFT 02 version.

Additionally, security of RESTCONF is improved through the addition of a model-based authorization schema in the AAA project.  Operators can now dynamically restrict sets of URL endpoints to specific classes of users at runtime.  This enhanced authorization mechanism is available for both RESTCONF versions.  AAA contributors have also added support for model-based certificate management.  Although the certificate management functionality is currently only integrated with OVSDB in the Carbon release, there are plans to provide hooks for use with other southbound protocols in the future.

An initial implementation of IETF Call Home based Draft 08 is added to the NETCONF project offering.  The implementation is currently not cluster aware, but offers the base functionality for Call Home functionality.  Overall, this improves the integration points for ODL, and enhances an operator’s ability to automate orchestration of ODL as part of a greater architecture.

Carbon debuts a new project called Daexim, a utility which allows the import and export of data from the MD-SAL datastore in JSON format.  Daexim is limited in the sense that it cannot tolerate YANG data model changes between releases.  However, developers can write external logic to manipulate data between import and export, providing for easier upgradability between releases of ODL.

Additionally, Carbon includes the first incarnation of jsonrpc, a project aimed towards enhancing external communication and federation with the controller.  For now, jsonrpc exposes a shim for ZMQ, a well tested, commodity message bus implementation.  Instead of utilizing RESTCONF, NETCONF or some other northbound interface, application developers can hook into the bus to manipulate data.  In essence, this unlocks the capability to write controller applications using non-JRE languages that support ZMQ integration.  This is compelling from the standpoint that it unlocks the ability for an entirely new set of developers to become involved with the project.

Overall, Carbon provides greater stability, security and enhanced network programability.  Groundwork is put in place to perform the Karaf upgrade in the Nitrogen release, and service activation is greatly stabilized and better tested to ensure a more consistent and friendly operational experience.  New functionality is added to help communicate with the controller, export data, and orchestrate ODL as part of a greater solution.  Dive in and download OpenDaylight’s Carbon release today from: https://www.opendaylight.org/downloads.

Written by Ryan Goulding, Senior Software Engineer at Inocybe Technologies.

Carbon has been released! OpenDaylight’s sixth platform release, Carbon, adds new enhancements to better support Metro Ethernet and cable operators as well as Internet of Things (IoT) deployments. As OpenDaylight continues to mature, a growing number of member use cases are emerging from companies including CenturyLink, China Mobile, Inocybe Technologies and Tencent.

OpenDaylight’s latest release further advances the platform’s scalability and robustness, with new capabilities supporting multi-site deployments for geographic reach, application performance and fault tolerance. Southbound protocols OpenFlow and Netconf gained in scalability and features, as did various administrative utilities.

Carbon also streamlines service function chaining by providing an integrated framework for NFV management. Much of this integration work and new capabilities available in Carbon were showcased as part of a proposed “Nirvana Stack,” presented in Boston last month.

As these toolchains are further integrated into higher level open source projects like ONAP, OpenStack and OPNFV, they are increasingly enabling innovators to productively explore new use cases such as IoT.

To learn more about Carbon or download the release, visit here.

We would like to thank our supportive community of developers and members who helped make the release possible!