Feature Articles: R&D Efforts in Cloud Computing Platform Technologies through Open Innovation
In the three years since the founding of OpenStack, numerous companies have joined the community, an ecosystem has been established, and the commercial use of OpenStack has grown. This article describes recent developments in OpenStack, public-cloud and private-cloud efforts at NTT laboratories, research and development of hybrid clouds at NTT Innovation Institute, Inc. (NTT I3), and NTT activities in the OpenStack community.
Keywords: OpenStack, IaaS, OSS
OpenStack  is an open source software (OSS), Infrastructure as a Service (IaaS) platform project that provides users with infrastructure such as computers and networks as virtual resources. OpenStack functions have increased with every release cycle, as in the Icehouse release of April 2014 (Fig. 1). OpenStack Summit Atlanta 2014, which was held in the month following that release, highlighted the maturity of the OpenStack ecosystem*1.
At this summit, the launch of OpenStack Marketplace was announced as a site for gathering information on products and services using OpenStack. Through Marketplace, 14 companies provide OpenStack distribution and 17 companies provide public-cloud and private-cloud services using OpenStack. In addition to users and cloud service providers, various companies such as vendors, consultants, and integrators participate in the OpenStack community and form an ecosystem. The ecosystem can support multiple use cases at NTT business companies, and it also prevents vendor lock-in. This ecosystem enables NTT services to effectively use OpenStack and facilitates the research and development of cloud services at NTT laboratories through collaboration with domestic and international partners.
2. OpenStack efforts at NTT laboratories
NTT laboratories have been investigating a method for combining OpenStack and a virtual network controller in order to deploy complex networks on virtual environments since 2012 . NTT laboratories have also been testing and developing a public cloud using OpenStack. This testing revealed that the Folsom release targeted for 2012 was deficient in some functions needed for commercial use of OpenStack, so we added a number of functions to OpenStack. We describe here some representative functions developed by NTT laboratories.
2.1 Multi-plugin support for virtual network function
A multi-plugin function (Metaplugin) was added to enable multiple plugins to be simultaneously pluggable with Neutron, the OpenStack network controller. Although Neutron allows a plugin to be selected in accordance with the subordinate network, it has not allowed the simultaneous use of multiple plugins. To support use cases that involve the control of a complex network that might have multiple types of subordinate vendor switches, the Metaplugin function was added to enable multiple plugins to be simultaneously pluggable, as shown in Fig. 2.
2.2 Transaction function
Transaction processing*2 appeared to be deficient across the entire OpenStack Folsom release, so NTT laboratories investigated transaction processing in OpenStack. It was found, for example, that a process in which an error occurred during OpenStack operations would often leave behind erroneous settings and unnecessary resources that had been created during the execution process. NTT laboratories added transaction processing across all of OpenStack in order to appropriately delete resources and correct settings after the occurrence of a processing anomaly.
2.3 Resource management function
A variety of issues were revealed in relation to OpenStack resources when using OpenStack in commercial services. For example, while a commercial service would normally initiate billing at the time of resource creation by a user, there was no billing system in OpenStack. We also found that OpenStack did not have a sufficient mechanism for checking whether an operation to OpenStack was valid. NTT laboratories created a resource management function that manages resources from outside OpenStack and supports operations initiated by users (Fig. 3).
These functions added by NTT laboratories were provided to the OpenStack community. The Metaplugin function was incorporated in Neutron code. NTT laboratories are also contributing as a community member to the development of the Taskflow library to achieve transaction processing. We have been discussing within the community how to deal with problems solved by the resource management function.
3. NTT laboratories and NTT I3 efforts going forward
NTT laboratories will continue efforts in developing cloud services using OpenStack based on requirements and use cases of NTT business companies, while NTT Innovation Institute, Inc. (NTT I3) will survey the North American market and research and develop cloud services as next-generation cloud solutions applicable to the needs of that market. The following describes these activities in more detail.
3.1 Public-cloud efforts
NTT laboratories have adopted the use of unmodified OpenStack community code as a major research policy of public clouds since 2014. The reason for this is that NTT laboratories have added more than 100 of their own software patches to the community code in their development work, and NTT-modified code has been vastly different from the community version. This difference has made it extremely difficult to keep up with updates to the community version of the code. To provide public-cloud services using OpenStack code without internally incorporating our own patches, we will provide our patches to the community. If there are some patches that have not been incorporated into the community code, we will implement alternative functions in an external system managed outside of OpenStack (Fig. 4). This approach will make it easier to keep up with updates within the OpenStack community.
3.2 Private-cloud efforts
NTT laboratories have been researching and developing public clouds and have been participating in the OpenStack community since the beginning. We regard research on public clouds to be an activity that is important for expanding the provision of the NTT Group’s cloud services as a counter to Amazon Web Services (AWS)*3. However, we are conscious that the demand for private clouds is increasing rapidly. In researching public clouds, NTT laboratories have mainly been targeting clouds of medium scale or larger that are configured using at least several dozen appliances of various types. We plan to research private clouds that can be completely configured using OSS. We expect that the private clouds will be able to extend their own configurations to the needed scale from an initial small-scale configuration (Fig. 5).
3.3 Enhancing a company’s entire IT infrastructure
NTT I3 is engaged in a platform project called Elastic Service Infrastructure (ESI) to enhance the operation of a company’s information technology (IT) infrastructure, including datacenters and enterprise WANs (wide area networks) . The NTT Group’s main business line for companies consists of virtual private network (VPN) products, hosting, managed IT, and network security, and ESI promotes the strategic evolution of these products and services into the cloud era.
We developed ESI because of the gap between the operational efficiency of the cloud and that of the current corporate IT infrastructure. The advances in cloud computing have substantially raised the efficiency of operating clusters of computers, but this has affected only a portion of the corporate IT infrastructure. A datacenter installed with computers includes network devices such as firewalls and load balancers, and the various bases of corporations connect to the datacenter over an enterprise WAN (VPN). However, the operation of these network devices and the enterprise WAN has been complicated and dependent on manual labor, thereby taking a lot of time and money.
The ESI makes the operation of datacenters and enterprise WANs efficient through cloud-based technologies such as virtualization and software operation. For example, the operation of network devices such as firewalls and load balancers can be made more efficient with a software controller that consolidates the management of those devices as virtual appliances. This approach is called network function virtualization (NFV)*4 for enterprises.
An important factor in making this approach successful is the construction of an ecosystem consisting of many virtual appliance products and infrastructure-operation products for the virtualization era. Therefore, OpenStack API (application programming interface)*5 was adopted in the ESI project.
4. Community activities
Since 2010, directly after the founding of OpenStack, 21 developers across the entire NTT Group have been contributing to the OpenStack community. These contributions have included reviews of patches proposed to the community, the provision of bug patches to the community, and proposals for new functions.
The contributions that NTT laboratories have made to repair bugs and propose new functions are many and varied (Fig. 6). Many of these contributions have involved functions that were deemed necessary for the provision of commercial services by the NTT Group. However, such use cases associated with a single corporate group and approval of related proposals requires long discussions and much time to gain acceptance from the community. An important point is that many other companies have also proposed many patches to satisfy the use cases of their own services. Under these conditions, the shortest way for a company’s own patches to be reviewed and approved by developers is to make many contributions to the community. It is important to gain the confidence of others and form new professional relationships by contributing high-quality reviews of patches proposed within the community and by providing patches that have value for the community. NTT laboratories are making more contributions and developing stronger relationships with the aim of improving our standing within the OpenStack community.
5. Concluding remarks
The expansion of the OpenStack ecosystem has enabled cloud providers to provide cloud services with a variety of formats. As part of this ecosystem, NTT laboratories and NTT I3 are continuing to develop services using OpenStack tailored to use cases and are endeavoring to make substantial contributions to the OpenStack community.