Part 3 in a 3 part series

A starting point for a better software development life cycle is to first identify and eliminate those activities that add little or no value for either customers or the company, or does so at a slow pace. Once these are removed, it will be easier to focus on improving the software development process and product development.

To do so, you first need a strategy that’s aligned with technology. Think about how you can deliver more customer value and reduce cost? How can you drive your company to improve communication flow, drive more incremental value, and deliver higher quality software.  Simply put, how do you achieve faster delivery, lower costs with more flexibility?

Here is where DevOps enters the game and how it can be a game changer. DevOps is not a “thing”. It is not a magic bullet. DevOps is a set of organizational philosophies that define how product development is done. DevOps depends on selecting appropriate development tools that enhance the development process and enhance communication and collaboration between all of the groups involved in product development: Engineering, integration, testing, the customer, etc. DevOps helps to deliver features faster, stabilizes operating environments, improves communication and collaboration, thereby providing more time for innovation. Once DevOps processes are established within an IT organization, organizations can experience greater customer satisfaction, cost reduction, and a more reliable product.

The DevOps implementations vary from company to company, but there are key elements and practices that all organization should always use as a common basis.  For example, the assembling of major DevOps elements such as Continuous Integration, Delivery and Deployment, source code management, configuration management and provisioning, creation and configuration of virtual development environments, etc., have been supported by best practices and tools in the market. Open source tools like Vagrant, Ansible, Puppet, Jenkins, Docker and others can be used to create a consistent DevOps infrastructure.

Recent Puppet surveys about DevOps and the effect that DevOps has on businesses, says that high-performing IT organizations deploy products 200X more frequently than low performers. These organizations have 24X faster recovery times and 3X lower change failure rates. Lean management and continuous delivery practices create conditions for delivering value faster and sustainably – spending 22 percent less time on unplanned work and rework. For network security issues alone, these DevOps IT teams spend 50 percent less time remediating security issues.

Northforge Innovations works with companies to implement DevOps into their software development process.  Our expertise in DevOps and network infrastructure is key in improving the culture and environments needed to make DevOps effective.  We have years of experience and expertise in DevOps including Configuration Management, Continuous Integration (CI) and Continuous Delivery/Deployment (CD), Build and Test Automation, Virtualization and Containers, Repository Management, and Cloud.

By Fabio V.

Part 2 in a 3 part series

How many times have you seen situations when internal communications fail and everyone suspects that it’s directly impacting the quality and reliability of the software modules being developed?  In fact, you may not even be building the right product or services.

  • Without a well-maintained flow of communication, information is not properly propagated or tracked. This can result in incomplete design, missing required features, and need for significant changes during the development process.
  • When applications are built in different environments by different teams (development, integration, test, etc.) environmental differences and version mismatches can result in different application behavior. This makes problems difficult or impossible to reproduce, substantially increases overhead for the involved teams, and can cause major delays in schedule.
  • Installation instructions are used for deployment in different environments. Those instructions can often change and may not be 100% correct, resulting in errors when the application isn’t installed properly or acts differently.
  • Manually moving and installing the application costs time. Instructions can be forgotten while moving or installing from environment to environment.
  • Repetitive and manual tests slow the release progress if there is a battery of tests to run through every time there is a new package.


Catching errors late in the process causes everybody to rush to make deadlines. The deadlines are often missed anyway and haste late in the process frequently results in reduced quality—so the product is both late and buggy (and probably over budget).

Implementing a DevOps approach can prevent these problems—because DevOps is all about business agility and continuous delivery.

In the last blog post in this three-part series, we’ll look at how we could solve it and how to achieve results for companies and their customers.

By Fabio V.

Part 1 in a 3 part series

Customers are constantly changing their requirements and seeking new and more powerful solutions. Service providers are being pushed to look at their own operations and improve their development cycle to meet these changing requirements.

There are several development lifecycle models such as waterfall, iterative, agile, etc., and the appropriateness of each depends both on the company’s culture and the project. But nowadays, the success of a project depends not only on the choice of a development model and its suggested practices, activities, and deliveries, but also on how the organization executes on the model. The challenges associated with customer needs and a competitive business market argue for a more communicative, flexible and automated operation.

DevOps can be the answer. DevOps is a set of IT practices that stress collaboration and communication between software developers and IT professionals, while at the same time automating the process of software delivery. For better collaboration and communication, DevOps aims at establishing a culture and environment for faster development and testing, and enables more frequent and more reliable software releases. Companies that incorporate DevOps practices into their operations get more done through continuous software delivery, less complexity, and faster resolution of problems.

Consider an example of a fictitious solution provider that does not implement DevOps with two development teams working on two separated modules of a common networking/security application. The organization has a typical organization structure with architects, developers, testers, managers, etc., working in a custom software development lifecycle process supported by common tools and manual processes. There is an integration team responsible for integrating the modules developed by the two teams and a test team responsible for installing a package candidate in different environments in order to execute tests.  We can imagine the problems that can arise in this environment.

Each developer has his or her own task estimates to finish development and push the code to a source repository. Sometimes, internal communications does not flow and specific dependencies are not raised or notified. Once the code is pushed, the integration team merges the modules and builds the application in a local machine. Afterwards, a release package and its installation instructions are passed on to the test team who will install it in different environments to manually execute their test plan to verify functionality, security, and performance. Problems will be documented and prioritized.

This is the first in three blog posts on DevOps.  The next two posts cover the possible problems and consequences, and how we can achieve better results for companies.

By Fabio V.


Intel’s Data Plane Development Kit (DPDK) has emerged as the key enabler for building the high-performance data planes needed by network functions virtualization applications. DPDK is a set of software libraries and drivers that can be integrated with virtual network functions (VNFs) or into virtual switches (or both) for fast packet performance in a server powered by an Intel® architecture processor. DPDK, initially developed by Intel, is now a Linux Foundation open source project.

Communications service providers are turning to NFV to help improve the agility and lower the costs of their network services. DPDK is one of the critical underlying technologies that improves data plane throughput so that VNFs deliver the deterministic response like the legacy, appliance-based applications they are replacing. With a long list of DPDK project implementations, combined with years of network software technology expertise, Northforge works with companies seeking a performance edge for their NFV applications.

Download Intel Network Builder’s solution brief on Northforge’s work with DPDK applications here.

In a building, each floor depends on the strength of the floors below it. Ensuring that the fifth floor is reinforced provides very little comfort if there is a structural problem on the third floor. In order to ensure that the building keeps standing, you need to reinforce every floor starting from the bottom.

And so it is with protocol stacks. Providing security at the transport layer (e.g., TLS) has questionable value if the packet exchange is compromised at the network or data link layer. Yet, we rarely worry about protecting these layers.

This new technical brief from Northforge describes some of the common attacks that can occur at the data link layer and how MAC-layer Security or MACsec (IEEE std 802.1AE™) can be used to provide hop-by-hop or end-to-end authentication and encryption to protect the lowest floors of your protocol building.

For a technical brief on MACsec, download here.

High performance I/O from a NIC

XEN and DPDK can be complementary. Clearly an EAL can be used within a VM running on top of the XEN hypervisor to provide high performance I/O from a NIC. However XEN and DPDK can also be alternatives for implementing a solution to the same problem. Consider a system that needs to perform the following sequence of functions

Using DPDK in a single VM, this could be decomposed as follows:



The solution can be implemented natively on top of XEN in multiple VMs like this:



The world of using virtual machines to implement virtualized network functions is still new. Each application is different, either administratively, functionally, or in performance requirements and therefore each will need an implementation that addresses its specific requirements. DPDK and XEN are two of the tools that can assist with these design problems.

If you need help designing and implementing your VNFs call Northforge Innovations. This is what we do.

By Larry S.







Implementing DPDK and Xen

With DPDK, packet processing is performed at the application layer in the virtual machine. Receive processing is based on polling the receive interface (using the EAL) rather than on interrupts. Interrupts require a fair amount of overhead in the “normal” case, but when interrupts must be propagated from the host operating system to the hypervisor to the guest operating system to the application, they end up being very expensive.

With DPDK, threads communicate through the use of shared memory queues. DPDK provides mechanisms for lockless (i.e. no blocking or synchronization needed) ring buffers that support single and multiple writers as well as single and multiple readers. These are called “rte-rings”.

There’s a better approach for increased performance and efficiency

For Rx (Receive Processing), the NIC (Network Interface) performs a Direct Memory transfer (DMA) to a buffer ring. If the NIC supports Receive Side Scaling (RSS), it can queue packets to different threads on different cores based on packet filters setup on the NIC. This increases the packet processing performance by spreading it over multiple cores.

The Proc function (Processing) can be scaled by decomposing it into a sequence of steps that can be organized as a pipeline. For example, if the processing is organized as three sequential steps, the three threads can be assigned to different cores and once the pipeline is full the system is, in effect, working on three packets simultaneously. There are a couple of different models for this depending on whether RSS is being used (which is shown in the top example in the following diagram.

Implementing DPDK and Xen fig 2

The Tx (Transit Processing) is initiated by putting the packet on a shared queue. The Tx process can then transfer the packet to the NIC for transmission.

DPDK is intended for a solution where all of the threads are running on the same Virtual Machine.

Packet Processing with XEN

​Not all packet processing solutions are designed to run on a single Virtual Machine. There are administrative reasons for splitting the system across multiple virtual machines. For example, if the packet stream represents multiple customers, then it might be desirable to split the processing across multiple VMs to provide separation and protection between customers as well as facilitating billing. There are also functional reasons for splitting the system across multiple virtual machines. For example, if the server is providing both a client focused capability (such as DHCP) and also a network service such as an IP Router, then running these on different VMs makes sense.

DPDK can provide performance benefits within a single VM, but splitting the processing across VMs is a bit more problematic since each VM appears as an independent and self-contained machine, each with its own memory. Providing communications and data movement between these VMs could be done using a networking function (i.e. transmitting and receiving packets through a virtual switch), but there are performance problems with this. Using shared memory between processes is much faster.

This is where XEN can play a part. XEN provides inter-VM shared memory using a page grant mechanism. The XEN hypervisor runs in Domain 0 (what we normally think of as kernel mode). It has access to the hardware page tables and memory management functions. Most of the work is done in the User Domain, however. So, a process in Dom 0 (e.g., a NIC driver) or Dom U in a VM (e.g., an application) can share a page but making a request to XEN. XEN enters the page into a grant table and returns a handle for the page. The handle can then be provided to another process running on another VM to grant access to the page. This mechanism is called “xenstore”.

XEN provides support for non-blocking ring buffers in xenstore similar to DPDK rte-rings. These are called xenstore-rings.

Implementing DPDK and Xen fig 3

In the final section we compare and summarize the two solutions. Comparing XEN and DPDK Solutions – A summary


By Larry S.

In general, packet processing applications follow a standard regimen:

  •     Receive a packet (Rx)
  •     Process a packet (Proc)
  •     Transmit a packet (Tx)

The Rx part is, more or less, the same regardless of the type of packet processing. The Proc part is the heart of the application.

Use multiple cores in your general packet processing model to enhance performance fig b

This could be an Ethernet Switching application, an IP Routing application, Deep Packet Inspection (DPI), or a protocol process such as DHCP.

In gateway applications (routing and switching) the Tx is usually a different interface than the Rx interface whereas with a protocol process the Tx is usually a response to the sender and therefore to the same interface as the Rx.

​The performance requirements vary from application to application, but a 1Gbps Ethernet can transport approximately 1.5 million packets per second, so the requirements could be very steep.

One approach to achieving high performance is to employ multiple cores. For Rx, this can be done by using a hardware capability called Receive Side Scaling (RSS). With RSS the Network Interface (NIC) can queue packets to different threads on different cores based on fields in the packet. For Proc, this can be done by breaking the processing into multiple tasks and creating a pipeline. (sidebar on pipeline processing). Tx usually requires very little processing so there is little motivation for partitioning it.

Use multiple cores in your general packet processing model to enhance performance fig b

By effectively using multiple cores, the packet processing performance of the system is greatly enhanced because the system is processing multiple packets simultaneously.

Based on the general packet processing model described here, we will next show how this model can be implemented using DPDK and XEN: Packet Processing with DPDK and XEN


By Andrei C.

No host OS + paravirtualization support = performance improvement

XEN is a hypervisor. A hypervisor is a supervisory program (think, operating system) that provides support for virtual machines. Parallels (Parallels), VMWare (Dell), and Virtual Box (Oracle) are all hypervisors. They provide an environment that hosts a number of processes (virtual machines) where each virtual machine believes it is running on the underlying hardware. Each virtual machine contains a guest operating system (e.g., Windows, macOS, Linux) and one or more processes/applications running within the guest operating system. Each of these hypervisors sits on top of a host operating system (e.g., Windows, macOS, Linux).

It is common to run a hypervisor like Virtual Box on a Mac and load one or more Windows virtual machines in order to run applications that only run on Windows.

XEN is a bit different in several ways from the three hypervisors listed above. First, it is a “bare metal” system. It runs as the lowest level, right on top of the hardware — there is no host operating system. As you’d expect, this improves performance and efficiency.

Second, XEN supports paravirtualization. With paravirtualization, XEN provides APIs for many system functions and the guest operating system in the virtual machines can be rebuilt to access system resources and the I/O subsystem via this API.

There are several Linux distributions that have been recompiled to use the XEN interface. This can improve performance and also allow stronger support for virtual machines on CPUs that don’t have good VM support (some x86s, older ARMs, etc.) XEN also provides support for accessing the underlying hardware like the other hypervisors (this is called Hardware Virtual Machine, HVM). The final difference is that XEN is open-source.


One of the major problems with implementing network processing applications in virtual machines is implementing high performance I/O. Packet processing applications often need to process tens of thousands or even millions of packets per second. This is difficult at the application layer, in general, but even more difficult when the system I/O calls are made to a guest operating system which has to access the hard through the hypervisor and the host operating system.

The Data Plane Development Kit (DPDK) is a solution to this problem. DPDK is a framework that provides for creating software libraries tailored for specific hardware architectures (e.g., X86) and specific operating systems (e.g., Linux). These libraries (called Environment Abstraction Layer or EAL) provide high performance, generic (i.e., hardware and OS independent) access to hardware and operating resources including the I/O subsystem.

Using the DPDK EAL allows the development of high performance user-mode packet processing applications which can also be tuned to exploit multi-core CPUs.

​Next, we provide an overview of a general packet processing model: General Packet Processing Model


By Larry S.


First in a five-part series

It is increasingly common to implement packet processing functions in virtual machines. This is what Network Functions Virtualization (NFV) is all about.

The most common implementation model for network functions has been to replicate the functions in the devices that are distributed around the network. This is the easiest way to do it, but taking a step back it becomes clear that this is not the most efficient way to do it both from a management point of view as well as from an overall efficiency point of view.

Consider DHCP for example. A network could have 50 edge routers running DHCP, but there is no reason why DHCP has to run in each edge router. The traffic load associated with DHCP is small so it makes sense to run each of the 50 DHCP instances as a Virtual Machine (VM) or possibly as a thread of a DHCP VM on a centralized server. There is the efficiency benefit in statistically multiplexing the computational load (all 50 instances are never running at the same time) and several management benefits such as only having to update a single system to fix bugs and add features and only having to configuration a single local system. Additionally, modern cloud computing technology provides the ability to migrate VMs for redundancy and to cloud burst for unexpected load peaks.

NFV requires high-performance software-based implementations of these packet processing functions in a virtual machine environment. These implementations must be able to read packets from the network, process them, and send packets out to the network. There are a variety of tools and techniques for implementing packet processing in Virtual Machines. In this blog series we will discuss two common, but very different approaches. One is the XEN Hypervisor and the other is the Data Plane Development Kit (DPDK). Depending on the functionality required, these two solutions can be used independently or concurrently. Over a sequence of blog posts we will describe these two approaches and how they can be used to implement high-performance packet-processing software.

​In the next blog post we provide a brief overview of the XEN hypervisor and DPDK.

-Larry S.