Feature Articles: Technological Development for Network Virtualization

Fundamental Research Activities on Network Virtualization

Kazuaki Obana, Kazuhisa Yamada, Yoshihiro Nakajima,
Hitoshi Masutani, Katsuhiro Shimano,
and Masaki Fukui


NTT Network Innovation Laboratories has been conducting a great deal of research in the areas of network virtualization and software-defined networking (SDN). In this article, we describe Lagopus, a high-performance OpenFlow 1.3 capable SDN software switch from the O3 Project. We also briefly explain our research activities involving network virtualization.

Keywords: NFV, SDN, OpenFlow


1. Introduction

One requirement in building future networks or next-generation networks is to construct a virtualized network in which its QoS (quality of service), performance, and functions are adaptable to each user’s needs. Thus, technology for expanding the applicable area of fine flow control via software-defined networking (SDN) and technology for adapting the deployment, correction, and removal of high-performance and highly functional software by means such as network functions virtualisation (NFV)*1, are becoming increasingly important. In this article, we introduce NTT Network Innovation Laboratories’ research activities on SDN and network virtualization.

*1 The British spelling used by the European Telecommunications Standards Institute

2. O3 Project: automating the configuration of network nodes

We aim to establish network virtualization technology that enables many network operators and service providers who share network resources to design and build networks and to manage network operations for their own purposes. This will enable network operators to reduce the time needed to design, build, and modify their networks to only about 10% of the time previously required over a wide area network. As a result, service providers will be able to dramatically reduce the time required to initiate and withdraw services. Moreover, user satisfaction is expected to increase as users gain faster access to their desired services.

This project is known as the O3 (O Three) Project*2 (Fig. 1) and is based on research on network virtualization technology consigned via the Ministry of Internal Affairs and Communications. It is being jointly promoted by five companies.

Fig. 1. Overview of O3 project.

This project aims to verify and commercialize research and development (R&D) results through prototypes and verification testing of network virtualization technology. At the same time, the project also aims to share and standardize research results globally, making some of the results open to the public and providing them to domestic and overseas telecommunications carriers and service providers and vendors. An outline of the target R&D areas in this project is as follows:

(1) Technology for developing network management and control platform software

This technology is used to develop SDN platform software that enables the construction of networks while meeting service requirements and enabling fast provision of services. Specifically, it integrates a wide area network by ensuring common handling of information to control diverse networks such as optical, wireless, and packet communication networks (network configuration information, communication status information, etc.), and enables development of platforms for flexibly and promptly performing control functions.

(2) Technology for developing software for network design, construction, and operation management

This technology is used to develop software for network design, construction, and operation management that runs on the platform mentioned in item (1). Specifically, it is used to develop design software (to verify designed networks) that is necessary in order to apply SDN to a wide area network, construction software (to interconnect with existing networks and transition from existing networks to SDN), and operation management software (to identify faults between layers and respond to them faster and to control service quality).

(3) Technology for developing virtualization-compatible network devices

This is used to develop SDN network devices that can be controlled by technologies outlined in (1) and (2). Specifically, it is used to develop the interfaces and driver functions that enable use of low-cost, high-performance resources through all layers, optimization of service construction costs, and control of existing optical, wireless, and packet transport network devices, as well as software communication devices that can freely change configurations and functions.

NTT Network Innovation Laboratories is developing software communication devices that are compatible with SDN ((3) in the list above) and that consist of software programs on general-purpose server hardware.

*2 O3 stands for the overall concepts of this project: open, organic, and optima. In June 2013, five companies (NEC Corporation, Nippon Telegraph and Telephone Corporation, NTT Communications Corporation, Fujitsu Limited, and Hitachi, Ltd.) began working jointly on R&D to conduct verification tests. The five companies aim to verify and commercialize these technologies and will promote the sharing and standardization of the research results globally.

3. SDN software switch: Lagopus

The use of software-oriented network nodes on general-purpose servers is important in order to simplify transfer networks and reduce their cost. These nodes are now more common than hardware-oriented network nodes for this purpose. NTT Network Innovation Laboratories successfully launched a prototype high-performance SDN software switch called Lagopus as part of the O3 project. Although SDN has already been introduced in corporate networks such as those at datacenters, this prototype SDN software switch achieves high performance even when large scale flow entries are set, and it functions as a technical element enabling application of SDN to wide area networks such as those of telecommunications carriers and Internet providers [1]. The target performance and flexibility of this prototype switch are shown in Fig. 2. In preparation for using this prototype switch for NFV in the future, the switch was given a modular architecture with remarkable expandability that makes it easy to expand and upgrade.

Fig. 2. Flexibility and performance target of Lagopus switch.

This prototype is compliant with long-term-support OpenFlow 1.3, the SDN-enabling specification, and also achieves good results in OpenFlow conformance tests. Furthermore, the development and implementation of the Flexible parallel Flow processing Framework (fff: Fig. 3) makes high-performance packet transfer possible with the prototype.

Fig. 3. SDN software switch with fff.

In general, a kernel space program is superior to a user space one in terms of processing speed. There are a number of problems with kernel space programs, however, including the need for the program to be maintained for the latest kernels. This prototype is a user space program and is implemented in parallel, with multiple threads enabling sufficient performance to be achieved.

Problems with SDN software switches are solved as follows:

(1) Faster forwarding with parallelization (multiple threads enabled)

Flows are identified, and packets in a flow are handled in a single set of pipelines to avoid order reversal of packets. The sets of pipelines are parallelized and multi-threaded to achieve greater speed [1].

(2) Faster searching of large scale flow entries with an ‘fff’ look-up algorithm

A novel algorithm has been developed in order to speed up the process of looking up large scale flow entries under complicated search conditions, such as the don’t care tolerance usually used in SDN (OpenFlow) in each set of pipelines. The prototype achieves high performance in searching flow tables, and the performance does not decrease much even if large scale flow entries are set into the flow tables. In addition, an fff look-up algorithm is implemented with as high a cache hit ratio as possible in order to reduce the number of times the memory needs to be accessed [1].

When 100K entries, which suffice as the first target, are added to flow tables, and all packet headers are rewritten, this prototype achieves 10-Gbit/s wire rate transferring of large packets. This makes it one of the highest performance SDN software switches ever.

We developed an agent function, which interprets the OpenFlow control protocol from the SDN controller and controls our software switch, in collaboration with the NTT Software Innovation Center, which has developed Ryu, as mentioned in one of the Feature Articles in this issue [2]. In the O3 project, our prototype is connected via Ryu with SDN control platform software.

The prototype will be extended to enable it to handle larger flow entries. R&D to achieve higher performance in applying software nodes to wide area networks will be promoted, as well as R&D for handling new protocols and management functions with high reliability in order to make software nodes more reliable and better operable.

4. Research to provide virtual networks to service providers

A virtual technology that can provide functions of future networks and network resources on a logical network is necessary for service providers. Network virtualization technology, which is one of the technologies applied to achieve SDN and NFV, can divide a physical network into multi-layered subnetworks. In each subnetwork, a different transport control protocol can be used, and specific applications or services can also be assigned. This is a major advantage that enables new services to be built quickly and flexibly on a carrier network.

We have taken part in developing the Network Platform for Flexibly-Programmable Advanced Service Composition. This platform includes routers, switches, servers, and network processors. It was developed during the period from fiscal year (FY) 2008 to FY2010 in a joint research project launched by NICT (National Institute of Information and Communication Technology), the University of Tokyo, NTT, NEC, Hitachi, and Fujitsu Laboratories in order to study network technology after the NGN (next-generation network).

Since 2011, a new research project to promote research on a next-generation virtual network platform has been executed under the Commissioned Research of NICT. The same organizations are participating in this project, and KDDI Laboratories and other organizations have also joined it. In this project, we have developed a virtual network platform consisting of a node system called VNode, which enables connections with nodes in other networks, as well as a network management system for the entire virtual network platform. The basic components of the VNode system are shown in Fig. 4.

Fig. 4. Basic components of VNode system.

Our VNode prototype system is shown in Fig. 5. We conducted experimental trials on a testbed (Fig. 6) that is based on overlay networks on JGN-X (Japan Gigabit Network-eXtreme), which is operated by NICT. As part of efforts to realize a global virtual network, we conducted experimental trials on interconnecting our platform with foreign virtual network platforms. Some challenges arose in the trials between the virtual network testbed on JGN-X and a testbed on GENI (the Global Environment for Network Innovation), which is promoted by the National Science Foundation in the USA. Their architecture and implementation are so different that it was difficult to interconnect them. Furthermore, ordinary virtual network technology cannot guarantee QoS (the number of transmitted packets and latency). Thus, it is difficult to provide high-quality service. We installed our newly developed VNode at the University of Utah, which is a research partner of NICT, and we connected VNode to the ProtoGENI testbed. We then carried out the world’s first successful trial to control multi-domain networks between Japan and the USA.

Fig. 5. Prototype VNode system.

Fig. 6. Testbed on JGN-X.

5. Future work

We will continue to promote the implementation of many kinds of applications on our testbed and to conduct experimental trials. We will also work to create the environment that will encourage many users to install our VNode system.


[1] NTT press release,
[2] R. Kubo, T. Fujita, Y. Agawa, and H. Suzuki, “Ryu SDN Framework—Open-source SDN Platform Software,” NTT Technical Review, No. 12, Vol. 8, 2014.
Kazuaki Obana
Senior Research Engineer, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.E. and M.E in electrical engineering from Waseda University, Tokyo, in 1995 and 1997, respectively. After joining NTT Optical Network Systems Laboratories in 1997, he was engaged in R&D of priority queuing by time and place information, a reverse proxy with a shaping function, and a fast transmitting system with multi-lanes. From 2008 to 2012, he worked on R&D of customer premises equipment in Promotion Project 1, NTT Cyber Solutions Laboratories. Since moving to NTT Network Innovation Laboratories in 2012, he has been engaged in R&D of programmable nodes.
Kazuhisa Yamada
Senior Research Engineer, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.E. in electronics engineering from Gunma University in 1987, after which he joined NTT Electrical Communication Laboratories, where he engaged in research on a programmable transport system. Since 1998, he has been developing carrier network operation systems, and since 2003, he has also been developing a digital television relay network operation system. His current research interests are network virtualization and network management technologies for future networks.
He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE).
Hitoshi Masutani
Senior Research Engineer, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.E. in communication engineering and the M.E. in electrical, electronic and information engineering from Osaka University in 1999 and 2001, respectively. After joining NTT Network Innovation Laboratories in 2001, he studied multicast networking and SIP-based home networking. In 2005, He moved to the Visual Communication Division of NTTBizlink, where he was responsible for developing and introducing visual communication services, including an IP-based high-quality large scale video conferencing system and a real-time content delivery system on IPv6 multicast. He also worked on developing their service order management system and network management system for video conference services.
Yoshihiro Nakajima
Research Engineer, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.S. in information science, the M.E. in engineering, and the Ph.D. in computer science from the University of Tsukuba, Ibaraki, in 2003, 2005, and 2008, respectively. He was a research fellow of the Japan Society for the Promotion of Science from 2005 to 2008. He joined NTT Network Innovation Laboratories in 2008 and studied high-performance stream processing systems and multilayer dynamic network provisioning/control systems. He is currently studying SDN and network virtualization technologies. He is a member of the Association for Computing Machinery, the Information Processing Society of Japan, and IEEE.
Katsuhiro Shimano
Senior Research Engineer, Supervisor, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.S. in physics from Waseda University, Tokyo, in 1991 and the M.S. in physics from the University of Tokyo in 1993. Since joining NTT in 1993, he has studied optical networks and network management and related areas such as optical network management systems, GMPLS, and traffic engineering. He also spent time at the headquarters working on NGN network architecture from the first phase of construction. Recently, he has been leading the research on SDN and network virtualization at NTT Network Innovation Laboratories.
Masaki Fukui
Director, Media Innovation Laboratory, NTT Network Innovation Laboratories.
He received the B.E., M.E., and Ph.D. in applied physics from Osaka University, in 1987, 1989, and 1995, respectively. In 1989, he joined NTT Transmission Systems Laboratories. He was engaged in researching optical information processing (optical computing) and WDM transmission systems. From 2001 to 2011, he worked for NTT EAST R&D Center, where he worked on the development of network systems for IP services and optical transport network systems. In 2011, he moved to NTT’s laboratories again. He is currently engaged in research on the architecture of future networks and network virtualization (SDN).