Battle of Equals between NTT Laboratories and NTT Business Companies via OpenStack—
A New Challenge toward the Next-generation Cloud Infrastructure

Transition to open source software (OSS) for economical and rapid development

Hayashi: As a cloud evangelist this past year, I have taken on the work of steering the cloud toward business with an objective point of view. Although any company is apt to promote its own products, I have tried to offer advice from a neutral standpoint, but at the same, I believe I can say without bias that the cloud services offered by NTT Communications (NTT Com) are superior. One reason for this is the use of an OpenStack system, which NTT Com and NTT laboratories have been collaborating on. Today, I would like to ask both of you about NTT’s involvement with OpenStack. To begin with, I would like to ask Mr. Sakai about OpenStack activities at NTT Software Innovation Center (SIC).

Sakai: Despite being part of NTT laboratories, SIC is performing research and development with a focus on practical use. At SIC, about 40 individuals are presently involved in OpenStack development in cooperation with NTT business companies such as NTT Communications and NTT DOCOMO with the aim of releasing a commercial system. OpenStack is a system that implements resources such as servers, networks, and storage on the cloud using virtualization technology (Fig. 1). Until recently, the approach taken by NTT laboratories was to execute projects on their own and develop services and products superior to those of other companies as proprietary technologies. OpenStack, however, represents a complete reversal to that approach as it aims to create a better system through the joint efforts of many people around the world.

Fig. 1. OpenStack components in next-generation cloud.

Hayashi: SIC’s participation in OpenStack began quite early. Can you provide some details on this involvement?

Sakai: There had been an increasing number of cases in which general enterprises throughout the world were adopting Amazon Web Services^*1 to run their internal business applications. Although NTT laboratories are generally quick to pick up on new technologies, there have been not a few times in which global competitors have invested a large amount of resources and taken the lead while we were still in the development stage. In technology fields in which the commoditization of services is accelerating, as in cloud-based IaaS, it is believed that the development of services by a large community of software developers in an open-source manner can result in good products inexpensively and quickly compared with the development of services as proprietary technologies by a single company. It is for this reason that we chose to make a transition to commercial development using OSS five years ago.

We have been participating in the OpenStack community nearly from the beginning of its development in 2010. At that time, a new version of the software was released every three months (every six months at present), and beginning with the initial release named OpenStack Austin, every subsequent release has been given a name in alphabetical order (Table 1). SIC has been involved with releases since the second release named OpenStack Bexar, so we have been participating in the OpenStack community practically from the very start. An OpenStack version named OpenStack Mitaka is scheduled for release in the spring of 2016. Here, the name “Mitaka” refers to the train station nearest to NTT Musashino R&D Center in which SIC is located, so the selection of this name for this release is truly gratifying for us.

Table 1. List of OpenStack release names.

Hayashi: Mr. Okuhira, please tell us about the work of Cloud Services at NTT COM and your duties as Director.

Okuhira: I began my cloud-related work in 2011, the year that the Cloud Services department was established. Prior to that, my work focused on the development of large-scale financial systems. In that work, I frequently interfaced with researchers in NTT laboratories in such fields as e-money, smart cards, and encryption, and I received much help from these individuals in various ways from the time that I was a young employee. The system that Cloud Services initially worked on was Cloudⁿ, which was launched in the United States in March 2012. Cloudⁿ is a public cloud featuring rich application programming interfaces (APIs) and self-management functions. Although NTT Com has traditionally focused on the global datacenter business, cloud-related orders are on the increase, and while revenues are still low, the cloud business is seen as one pillar of future growth.

Hayashi: Can you tell us how OpenStack came to be adopted in Cloudⁿ?

Okuhira: OpenStack was one of several options as the initial platform for Cloudⁿ, but since we sought stability in the early stages of Cloudⁿ as a commercial system, we decided to go with Apache-based CloudStack originally developed by Cloud.com but now provided as OSS (Fig. 2). OpenStack later came to be used by SIC and elsewhere within the NTT Group as its stability improved, and it was decided in 2013 to adopt it in the Cloudⁿ platform. Actually, it was decided to use OpenStack for a new service three months before the launch of that service. A number of problems arose before and after this service launch, but SIC staff responded to them rapidly, which enabled early implementation. At present, we are providing a stable service without any major problems.

Fig. 2. Different configurations of Cloudⁿ Compute.

Cloud Services is also developing a next-generation cloud infrastructure (Fig. 3) that will adopt OpenStack Juno as one of its platforms. In the year leading up to the Juno release, I met with Mr. Sakai every week and had many discussions with him on how to combine Juno with software-defined networking (SDN). We also incorporated the opinions of the business companies in a process that promoted community activities.

Fig. 3. Enterprise Cloud: features of next-generation cloud infrastructure.

A relationship on equal terms between NTT laboratories and business companies bears fruit

Hayashi: It appears that the collaboration between SIC and NTT Com is producing results.

Sakai: In the development of a commercial system, it has often been the case that NTT laboratories would listen to the needs of a business company and then go on to improve product quality to a level that could withstand the demands of commercial operation. However, if such a relationship were to continue, it would not be unusual for the relationship between the business company and NTT laboratories to unconsciously fall into a consignment type of relationship between an order placer and an order receiver. However, that kind of relationship has not occurred between Mr. Okuhira and me. I have been able to voice the opinions of NTT laboratories without restraint, and we have been able to talk until reaching an agreement on equal terms. Forming such a relationship has been difficult in the past, but it evolved quickly in this project.

Okuhira: A multi-tenant scheme and scalability are essential to the provision of cloud services, and for a service provider, high availability (HA) is an important requirement. These functions, however, are not incorporated in OpenStack. I have had many discussions with staff at SIC as to how we can develop these functions in a sound way that befits NTT services.

Sakai: OpenStack is an accumulation of various types of OSS, so its development and troubleshooting in the event of a problem requires the use of all knowledge pertaining to such a variety of OSS. Indeed, when a problem occurs just before service launch, an immediate investigation is necessary to determine what exactly is happening. However, since OpenStack operates on the basis of many distributed modules, logs for analysis purposes will be output in a separate and scattered manner at such a time. Clarifying the cause of the problem under such conditions would be time consuming, and for this reason, we have attached common identifications (IDs) to logs across modules. The specifications for this common ID and the HA function mentioned earlier by Mr. Okuhira are currently being proposed by SIC to the OpenStack community, which has praised these proposals.

Hayashi: The NTT Group received the Superuser Award at the recent OpenStack Summit Tokyo. Can you explain the reasons for receiving this award?

Sakai: More than 5000 people representing 56 countries attended the OpenStack Summit Tokyo held in Shinagawa from October 27–30, 2015. The OpenStack Summit welcomes OpenStack developers from around the world twice a year, and this meeting was held in Japan for the first time. The Superuser Award goes to one company for its achievements in implementing OpenStack in a way that creates value and for its contributions to the OpenStack community. The NTT Group was one of four finalists and was the first company from Asia to be chosen for this award (Photo 1). Through this award, NTT Com was of course commended for its application of OpenStack to Cloudⁿ, while NTT Resonant was commended for deploying OpenStack in the infrastructure of the “goo” web portal. It was also noted that NTT DOCOMO and NTT DATA use OpenStack, the former in its mail services and the latter in SI (system integration) matters. In short, the NTT Group received the Superuser Award because of the way in which its business companies were using OpenStack in actual business applications and for its contributions to the OpenStack community.

Photo 1. Receiving the Superuser Award and view of trophy.

Okuhira: I believe that this award is the result of five years of contributions led by Mr. Sakai. The services provided by the business companies have become more stable as a result. We have also been recognized for our decision to continue using OpenStack in the next-generation cloud infrastructure. When the press release announcing this award [1] was issued, we received many inquiries from overseas affiliated companies, and I sensed their great expectations for us. I also think that this news served to publicize our cloud services among our customers and to spread the word that the NTT Group is deeply involved in OpenStack activities.

Hayashi: Which OpenStack components are the focus of attention at SIC?

Sakai: One is the Neutron module for network virtualization. In the beginning, the implementation of network-related functions was lacking compared to functions for servers and storage. For example, a single virtual machine was unable to support multiple network interfaces, so we focused our energy on upgrading such functions that were insufficient for enterprise use and proposed those changes to the OpenStack community. By the way, NTT Com adopted network virtualization by SDN in commercial services early on by global standards, and NTT laboratories made contributions here as well, such as by modifying plug-ins in OpenStack to incorporate a variety of SDN products. More recently, Neutron adopted the “Ryu” Border Gateway Protocol function developed and proposed by SIC as an open source SDN infrastructure. In this way, I am proud of the contributions that we at SIC are continuing to make to the OpenStack community in this field. At the same time, we are actively contributing to the Swift module that provides object storage services. In OpenStack, engineers actively involved in development work are chosen to be “core developers” in specific modules via a recommendation system. SIC has produced several core developers in the Neutron and Swift modules. Recent efforts in this regard include our proposal for a function that spans multiple OpenStack modules, by which I mean the application of common IDs to logs as I described earlier, and studies on new technologies such as NFV (network functions virtualization) and containers. A project for providing an HA function that we have named “Masakari” after my nickname also attracted interest at the recent OpenStack Summit.

Hayashi: There is also the idea of performing independent development outside of contributions to the OpenStack community. Where do you draw the line between those two pursuits?

Sakai: This is a difficult problem. Certainly, in the past, there was a strategy of holding on to independently developed functions to differentiate our services, but recently, we are apt, in principle, to submit our developments to the community. This is because failing to submit such a function early on may result in the appearance of that function in the community through other code. That function may then become mainstream, which would make us mere imitators. Of course, for functions supporting NTT carrier-grade operations or NTT proprietary advanced security functions, for example, we have decided that it is best that we keep these closed and continue to develop them independently. In such cases, however, we would try as much as possible to leave the main body of OpenStack untouched by implementing these original functions in plug-in form as external functions that would interface with OpenStack. In a May 2015 press release, it was announced that NTT laboratories had incorporated an NTT original high-speed secret sharing engine called “Super High-speed Secret Sharing” in OpenStack Swift in plug-in form thereby enhancing storage confidentiality [2]. This is a good example of such independent development.

Personnel training toward global rollout of the next-generation cloud infrastructure

Hayashi: What kinds of contribution activities take place within the OpenStack community?

Sakai: OpenStack is truly an open community. In the OSS community, although completed source code is open, development tends to be closed to specific companies. In contrast, the development process in OpenStack is very democratic; someone who would like to add something to OpenStack will make a proposal, which can be incorporated in OpenStack only if everyone else approves. This process begins by submitting one’s desired function in the form of specifications, writing the code for that function, repeatedly making corrections based on reviews received from core developers, and finally obtaining consensus. It is only in this way that a proposal can come to be incorporated within the body of OpenStack. What source code will come to be incorporated in releases that are issued once every six months depends on the judgment of people in charge of each module, but some problems can arise in this process. For example, an abundance of proposed functions may result in some being overlooked, or the addition of new functions A and B may produce a conflict. Such problems have to be solved. A person who can consistently solve software bugs is very useful and highly appreciated in the OpenStack community, and his or her opinions may have great influence. In such ways, SIC will continue to be active in the OpenStack community in cooperation with the entire NTT Group.

Okuhira: At NTT Com too, we are adding members involved in OpenStack and seek to educate and train a variety of personnel in this field. In the end, however, there is a limit to the scale of human resources that we can contribute, but we seek to be active in the OpenStack community in cooperation with the entire NTT Group centered around SIC.

Hayashi: In the next-generation cloud infrastructure, what kind of new services using OpenStack will be offered?

Okuhira: In the next-generation cloud infrastructure, OpenStack will be used in virtual servers. We also plan to develop OpenStack into a form in which it can be used in financial systems and private networks while we fortify network communications between datacenters. Up until recently, OpenStack had been used only in services within Japan, but the Enterprise Cloud using the next-generation cloud infrastructure is being rolled out globally. In the past, the sales system was based on paper-based applications, but from here on, it will be a paperless system and all services will be usable through APIs. In addition, use of the Keystone module will make it unnecessary to independently install user authentication/authorization functions for each component.

Hayashi: I see that there are plans to develop services using OpenStack even for industries that demand high reliability such as the financial sector. What is the outlook for research and development that includes OpenStack?

Sakai: We are looking to create added value by combining a customer’s business data with information from the Internet of Things (IoT). One example is predictive maintenance, which is now a popular topic. As a typical scenario in this field, we can envision how the analysis of IoT data can indicate that a particular piece of equipment is about to fail. Then, in addition to advising the customer of this situation, we could also provide information on where an order for new equipment at minimal cost could be placed, or in the case of repairs, when the engineer in charge could visit the customer’s premises to replace components, etc. Such processing will require more than just collecting IoT data in a simple and safe manner. It will involve a number of issues that have to be resolved, such as processing collected data and matching it up with cloud data as well as ensuring the secure flow of data between enterprises. I would like to see SIC make an all-out effort to make NTT’s network and cloud a main player in achieving such a world.

Hayashi: Please tell us about future developments centered on the next-generation cloud infrastructure.

Okuhira: Our plan is to extend OpenStack to include a menu oriented to enterprise applications such as SAP^*2 and Oracle^*3. Furthermore, in addition to providing APIs and automation and self-management functions, we are also thinking about a service implementation system that enables a customer with insufficient facilities to add new facilities in a short period of time. Moreover, in addition to developing services, we are also thinking about ways of optimizing maintenance operations after service launch, an optimal way of writing API reference codes, and a mechanism that would enable customers to link up with each other so that necessary services could be provided when needed. It is not our aim to just keep up with the times but to be ahead of the times.

Experiencing the joy of people around the world using the results of our research

Hayashi: What would you say to young researchers?

Sakai: I believe that the joy of putting something that one has developed out into the world and seeing customers use it for the first time is a sensation that no one who has never felt it can understand. To achieve widespread use of a development as opposed to simply introducing it into business, many difficult and undesirable things need to overcome, but for people who would like to experience such emotion and joy together, I heartily recommend that they enter the field of OSS. The code that one writes becomes a foundation to be used by other people in the world, and there is much joy in seeing that happen. To begin with, please use yourself what you have developed!

Okuhira: I would like to ask young researchers in NTT laboratories to become involved early with NTT business companies. In particular, I think that accumulating a variety of experiences is good for researchers in OSS development. If you visit a business company, you will have to handle complaints from customers, so that will also help to improve your communication skills. Having the feeling that your group’s technology in itself is not suitable for business is also a valuable experience.

Hayashi: Finally, can you tell us how you spend your free time?

Sakai: Well, there is a racetrack nearby, so I often go there with my children. Watching horses in a green, natural environment different from my everyday environment is emotionally soothing. A horse’s eyes are truly beautiful. I also have fun playing mahjong with my family at home.

Okuhira: I enjoy working up a sweat playing softball with others in my neighborhood, and I also participate in PTA (parents and teachers association) activities. When at work, it is easy for me to motivate people as needed because of my position, but with people in my neighborhood, I have a horizontal relationship, so if I cannot skillfully convey what I would like others to do, they will not do it. I therefore think about the best way of communicating with people to make things move along. I believe that it’s important to have points of contact with all kinds of people and to have conversations with people outside the company. We should not become obsessed with our work; private activities can also be useful in various aspects of our lives.

Hayashi: Thank you very much!

References

[1]	NTT Com press release issued in October 2015. http://www.ntt.com/aboutus_e/news/data/20151027.html
[2]	NTT press release issued in May 2015. http://www.ntt.co.jp/news2015/1505e/150518a.html