To view PDF files

You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Feature Articles: R&D Efforts in Cloud Computing Platform Technologies through Open Innovation

OpenStack Activities

Hirofumi Ichihara, Shintaro Mizuno, Kenichi Sato, Tomoko Inoue, Takashi Natsume, Masahito Muroi, and Masahisa Kawashima


In the three years since the founding of OpenStack, numerous companies have joined the community, an ecosystem has been established, and the commercial use of OpenStack has grown. This article describes recent developments in OpenStack, public-cloud and private-cloud efforts at NTT laboratories, research and development of hybrid clouds at NTT Innovation Institute, Inc. (NTT I3), and NTT activities in the OpenStack community.

Keywords: OpenStack, IaaS, OSS


1. Introduction

OpenStack [1] is an open source software (OSS), Infrastructure as a Service (IaaS) platform project that provides users with infrastructure such as computers and networks as virtual resources. OpenStack functions have increased with every release cycle, as in the Icehouse release of April 2014 (Fig. 1). OpenStack Summit Atlanta 2014, which was held in the month following that release, highlighted the maturity of the OpenStack ecosystem*1.

Fig. 1. OpenStack release schedule.

At this summit, the launch of OpenStack Marketplace was announced as a site for gathering information on products and services using OpenStack. Through Marketplace, 14 companies provide OpenStack distribution and 17 companies provide public-cloud and private-cloud services using OpenStack. In addition to users and cloud service providers, various companies such as vendors, consultants, and integrators participate in the OpenStack community and form an ecosystem. The ecosystem can support multiple use cases at NTT business companies, and it also prevents vendor lock-in. This ecosystem enables NTT services to effectively use OpenStack and facilitates the research and development of cloud services at NTT laboratories through collaboration with domestic and international partners.

*1 Ecosystem: A mechanism for developing collaborative relationships among companies, developers, and users and forming a single environment through mutual cooperation

2. OpenStack efforts at NTT laboratories

NTT laboratories have been investigating a method for combining OpenStack and a virtual network controller in order to deploy complex networks on virtual environments since 2012 [2]. NTT laboratories have also been testing and developing a public cloud using OpenStack. This testing revealed that the Folsom release targeted for 2012 was deficient in some functions needed for commercial use of OpenStack, so we added a number of functions to OpenStack. We describe here some representative functions developed by NTT laboratories.

2.1 Multi-plugin support for virtual network function

A multi-plugin function (Metaplugin) was added to enable multiple plugins to be simultaneously pluggable with Neutron, the OpenStack network controller. Although Neutron allows a plugin to be selected in accordance with the subordinate network, it has not allowed the simultaneous use of multiple plugins. To support use cases that involve the control of a complex network that might have multiple types of subordinate vendor switches, the Metaplugin function was added to enable multiple plugins to be simultaneously pluggable, as shown in Fig. 2.

Fig. 2. Example of using Metaplugin.

2.2 Transaction function

Transaction processing*2 appeared to be deficient across the entire OpenStack Folsom release, so NTT laboratories investigated transaction processing in OpenStack. It was found, for example, that a process in which an error occurred during OpenStack operations would often leave behind erroneous settings and unnecessary resources that had been created during the execution process. NTT laboratories added transaction processing across all of OpenStack in order to appropriately delete resources and correct settings after the occurrence of a processing anomaly.

2.3 Resource management function

A variety of issues were revealed in relation to OpenStack resources when using OpenStack in commercial services. For example, while a commercial service would normally initiate billing at the time of resource creation by a user, there was no billing system in OpenStack. We also found that OpenStack did not have a sufficient mechanism for checking whether an operation to OpenStack was valid. NTT laboratories created a resource management function that manages resources from outside OpenStack and supports operations initiated by users (Fig. 3).

Fig. 3. Example of using the resource management function.

These functions added by NTT laboratories were provided to the OpenStack community. The Metaplugin function was incorporated in Neutron code. NTT laboratories are also contributing as a community member to the development of the Taskflow library to achieve transaction processing. We have been discussing within the community how to deal with problems solved by the resource management function.

*2 Transaction processing: Information processing that guarantees the flow of a sequence of indivisible operations

3. NTT laboratories and NTT I3 efforts going forward

NTT laboratories will continue efforts in developing cloud services using OpenStack based on requirements and use cases of NTT business companies, while NTT Innovation Institute, Inc. (NTT I3) will survey the North American market and research and develop cloud services as next-generation cloud solutions applicable to the needs of that market. The following describes these activities in more detail.

3.1 Public-cloud efforts

NTT laboratories have adopted the use of unmodified OpenStack community code as a major research policy of public clouds since 2014. The reason for this is that NTT laboratories have added more than 100 of their own software patches to the community code in their development work, and NTT-modified code has been vastly different from the community version. This difference has made it extremely difficult to keep up with updates to the community version of the code. To provide public-cloud services using OpenStack code without internally incorporating our own patches, we will provide our patches to the community. If there are some patches that have not been incorporated into the community code, we will implement alternative functions in an external system managed outside of OpenStack (Fig. 4). This approach will make it easier to keep up with updates within the OpenStack community.

Fig. 4. Configuration using OpenStack community version.

3.2 Private-cloud efforts

NTT laboratories have been researching and developing public clouds and have been participating in the OpenStack community since the beginning. We regard research on public clouds to be an activity that is important for expanding the provision of the NTT Group’s cloud services as a counter to Amazon Web Services (AWS)*3. However, we are conscious that the demand for private clouds is increasing rapidly. In researching public clouds, NTT laboratories have mainly been targeting clouds of medium scale or larger that are configured using at least several dozen appliances of various types. We plan to research private clouds that can be completely configured using OSS. We expect that the private clouds will be able to extend their own configurations to the needed scale from an initial small-scale configuration (Fig. 5).

Fig. 5. Proposal for private-cloud configuration.

3.3 Enhancing a company’s entire IT infrastructure

NTT I3 is engaged in a platform project called Elastic Service Infrastructure (ESI) to enhance the operation of a company’s information technology (IT) infrastructure, including datacenters and enterprise WANs (wide area networks) [3]. The NTT Group’s main business line for companies consists of virtual private network (VPN) products, hosting, managed IT, and network security, and ESI promotes the strategic evolution of these products and services into the cloud era.

We developed ESI because of the gap between the operational efficiency of the cloud and that of the current corporate IT infrastructure. The advances in cloud computing have substantially raised the efficiency of operating clusters of computers, but this has affected only a portion of the corporate IT infrastructure. A datacenter installed with computers includes network devices such as firewalls and load balancers, and the various bases of corporations connect to the datacenter over an enterprise WAN (VPN). However, the operation of these network devices and the enterprise WAN has been complicated and dependent on manual labor, thereby taking a lot of time and money.

The ESI makes the operation of datacenters and enterprise WANs efficient through cloud-based technologies such as virtualization and software operation. For example, the operation of network devices such as firewalls and load balancers can be made more efficient with a software controller that consolidates the management of those devices as virtual appliances. This approach is called network function virtualization (NFV)*4 for enterprises.

An important factor in making this approach successful is the construction of an ecosystem consisting of many virtual appliance products and infrastructure-operation products for the virtualization era. Therefore, OpenStack API (application programming interface)*5 was adopted in the ESI project.

*3 AWS: The general name for cloud services provided by Amazon
*4 NFV: Technology for achieving the functions of network devices using software
*5 API: Interface for using an application from the outside

4. Community activities

Since 2010, directly after the founding of OpenStack, 21 developers across the entire NTT Group have been contributing to the OpenStack community. These contributions have included reviews of patches proposed to the community, the provision of bug patches to the community, and proposals for new functions.

The contributions that NTT laboratories have made to repair bugs and propose new functions are many and varied (Fig. 6). Many of these contributions have involved functions that were deemed necessary for the provision of commercial services by the NTT Group. However, such use cases associated with a single corporate group and approval of related proposals requires long discussions and much time to gain acceptance from the community. An important point is that many other companies have also proposed many patches to satisfy the use cases of their own services. Under these conditions, the shortest way for a company’s own patches to be reviewed and approved by developers is to make many contributions to the community. It is important to gain the confidence of others and form new professional relationships by contributing high-quality reviews of patches proposed within the community and by providing patches that have value for the community. NTT laboratories are making more contributions and developing stronger relationships with the aim of improving our standing within the OpenStack community.

Fig. 6. Contributions to OpenStack community.

5. Concluding remarks

The expansion of the OpenStack ecosystem has enabled cloud providers to provide cloud services with a variety of formats. As part of this ecosystem, NTT laboratories and NTT I3 are continuing to develop services using OpenStack tailored to use cases and are endeavoring to make substantial contributions to the OpenStack community.


[1] Website of OpenStack,
[2] S. Mizuno, H. Sakai, D. Yokozeki, K. Iida, and T. Koyama, “IaaS Platform Using OpenStack and OpenFlow Overlay Technology,” NTT Technical Review, Vol. 10, No. 12, December 2012.
[3] M. Kawashima, “Network Infrastructure Elasticity,” NTT I3 blog, August 2014.
Hirofumi Ichihara
Research Engineer, Cloud System SE Project, NTT Software Innovation Center.
He received the M.E. in information networking from Osaka University in 2012. He has been working on a cloud system using OpenStack at NTT since 2012. His current research interests include software-defined networking (SDN) and NFV.
Shintaro Mizuno
Senior Research Engineer, Cloud System SE Project, NTT Software Innovation Center.
He received the B.E. and M.E. in mechanical and environmental informatics from Tokyo Institute of Technology in 1995 and 1997, respectively. Since joining NTT in 1997, he has been working on authentication systems, secure communication systems, and information security. His current research interests include cloud management systems and SDN.
Kenichi Sato
Senior Research Engineer, Supervisor, Development Project Leader, Cloud System SE Project, NTT Software Innovation Center.
He received the B.E. and M.E. in precision mechanical engineering from the University of Tokyo in 1991 and 1993, respectively. He joined NTT Network Information Systems Laboratories in 1993. He studied intelligent agent communication, electronic payment systems, and transaction processing. From 2002 to 2005, he developed and operated a web-based ASP (application service provider) service at NTT Resonant Inc. He is currently studying cloud computing platforms based on OpenStack.
Tomoko Inoue
Research Engineer, Cloud System SE Project, NTT Software Innovation Center.
She received the B.A. in literature from Ritsumeikan University, Kyoto, in 2003 and the M.A. in informatics from Kyoto University in 2005. She joined NTT WEST in 2005 and moved to NTT Information Sharing Platform Laboratories in 2011. Since 2011, she has been interested in enterprise cloud network systems and is studying the architecture and construction of virtual networks in a cloud environment.
Takashi Natsume
Engineer, Cloud System SE Project, NTT Software Innovation Center.
He received the B.E. and M.E. in information and communication engineering from the University of Tokyo in 1997 and 1999, respectively. He joined NTT in 2013. He is engaged in system design of public-cloud systems based on OpenStack and functional verification of OpenStack.
Masahito Muroi
Cloud Architect, Cloud System SE Project, NTT Software Innovation Center.
He received the B.E. and M.E. in computer science from the University of Electro-Communications, Tokyo, in 2009 and 2011, respectively. He has been working as a cloud architect since 2011. He started developing a public cloud using OpenStack in 2012. He has recently been focusing on a carrier-grade cloud using the Juno release.
Masahisa Kawashima
VP, Product Management, Cloud Infrastructure, NTT Innovation Institute, Inc.
He received the Ph.D. in electrical engineering from Waseda University, Tokyo, in 1994 and the M.Sc. in technology management from MIT’s Sloan School of Management, USA, in 2002.
Since joining NTT in 1994, he has been engaged in R&D and business development of the NTT Group. He has accomplished several initiatives to update NTT’s business operations using new technology trends and played a leading role in organizing the NTT Open Source Software Center in 2006 and NTT WEST’s strategic Wi-Fi service renewal in 2011. He is currently working on redefining telecommunications infrastructure business by taking advantage of the potential of software and virtualization technologies. He oversees business deployment of cloud orchestration and SDN/NFV technologies at NTT I3.