Open Source Software Initiatives Supporting NTT Group Software Development and Operations

1. OSS Technology Initiative: PostgreSQL

The NTT Open Source Software Center (NTT OSS Center) is collaborating with NTT Group companies and the open source software (OSS) community, conducting technical development, and accumulating know-how regarding various OSS, from operating systems (OS) to middleware, so that software development, maintenance, and operation of information technology (IT) systems centered on OSS can be performed stably and efficiently in the businesses being operated by the Group companies. Database management systems (DBMSs) hold a very important position in information and communication systems, and the NTT OSS Center has put great effort into expanding application of PostgreSQL^* since its inception. The objectives have included enhancing performance and functionality as database sizes have increased, and facilitating migration from commercial DBMSs. There have also been initiatives to develop large-scale databases using external table functionality, to conduct research and development (R&D) on distributed databases, and to improve compatibility so that Structured Query Language (SQL) dialects defined by commercial DBMSs can run unmodified on PostgreSQL (Fig. 1).

Fig. 1. Contributions to PostgreSQL development.

The latest version, PostgreSQL 10, introduces declarative table partitioning, which was developed under the leadership of members of the NTT OSS Center. Partitioning is a function whereby performance or operability is improved by dividing and storing data in multiple tables, and it is particularly useful when handling very large datasets. In previous versions, PostgreSQL did not provide a dedicated partitioning function, and pseudo-partitioning had to be implemented by combining table inheritance functions, triggers, and CHECK constraints. With the introduction of a dedicated partitioning function, partitioning can be applied by simply declaring a partitioning structure in SQL when creating the table (CREATE TABLE statement). Data insertion is faster that it was with the earlier pseudo-partitioning practice.

To further expand PostgreSQL into enterprise systems, the NTT OSS Center is working to implement sharding, which will enable table partitions to be distributed over multiple servers to distribute load and improve performance. The new partitioning and foreign data wrapper (FDW) function are necessary as basic functions of sharding. The NTT OSS Center has also led the enhancements of FDW, which enables tables on a remote server to be accessed as though they were local tables. Some functionality is still not adequate for implementing sharding, for example, support for distributed transactions, but the NTT OSS Center will continue development in collaboration with the community and will work to expand PostgreSQL into a database that can be scaled out to larger sizes.

Note that in addition to partitioning, PostgreSQL 10 incorporates various other new functionalities and improvements, including a new logical replication function that enables replication by table, enhancements to parallel queries so they can be used in more cases, and a new quorum commit functionality developed under the leadership of those at the NTT OSS Center.

2. Technically verified OSS suite: OSSVERT

Since its inception, the NTT OSS Center has been operating its OSS Suites Verified Technically (OSSVERT) program. In OSSVERT, the NTT OSS Center verifies the operation of systems composed of OSS that is recommended and supported by the Center and provides technical materials summarizing the results. Technical materials provided include model overviews, installation procedures, environment definitions, and verification reports. Use of the recommended middleware configurations and various settings verified and provided by OSSVERT facilitates rapid and efficient system design and construction. The main OSS in OSSVERT configurations is maintained and supported by the NTT OSS Center, so safe and secure operational support is available.

In the past, OSSVERT initiatives have dealt with verifying the construction of three-tier web systems on physical servers, but with recent trends to expand the use of OSS, the NTT OSS Center has begun using OSSVERT to verify other types of OSS that are expected to be used more widely in the future. Specific initiatives include verifying the OpenStack (RDO Liberty) run-time environment, cloud environments utilizing Amazon Web Services (AWS), and three-tier web systems built on Docker.

Beyond simply verifying operation, the NTT OSS Center has also been using OSSVERT to test new combinations of middleware in order to evaluate their potential for service provision and to discover any constraints or issues as early as possible. Software is checked for any problems particularly from the perspective of availability and performance under high loads, so that any necessary cautions or considerations when designing and building systems can be clearly identified, and any defects are discovered, investigated, and analyzed. This enables solutions or workarounds to be found quickly.

A strength of OSSVERT is that it provides information that can be used immediately based on testing. A certain amount of information can be obtained from technology books and the Internet, but it is very difficult to find information on how to appropriately operate various enterprise functionalities using the latest configurations. OSSVERT provides reference information suitable for enterprise use based on practical testing.

3. Use of OSS for enterprise application development: Macchinetta framework

It is becoming very common to use platform software called an application framework (AP framework) when developing applications for various types of business operations (business APs). An AP framework is software that mediates between the business AP and the software operating environment composed of hardware, OS, and middleware. An AP framework can increase productivity by providing various common functions, and it has other benefits such as promoting more uniform quality since the framework has conventions that reduce differences among programmers developing the application, and more efficient maintenance and management, since common knowledge based on the framework makes systems easier to understand.

The NTT Software Innovation Center has developed the Macchinetta framework as a new AP framework that can be used throughout the NTT Group [1]. This will improve efficiency for personnel training, software development, and maintenance, and it also has the benefit of eliminating duplicate investment since it is used throughout the Group. The basic policies in developing this framework were to use OSS of a global standard, to ensure it was applicable on a wide range of IT systems, and not to be constrained by proprietary technology.

The Macchinetta framework is composed of a recommended OSS stack model, which is a stack of OSS products needed for common functionality, and technical documentation to facilitate development using the AP framework (Fig. 2). The recommended OSS stack model has been tested in multiple combinations of OSS products so that they can be stably used in such combinations. In addition to components for developing web systems with popular architectures, it also includes batch processing architectures necessary for business APs, and client-side components for improving operability. It has been adopted as the standard AP framework for NTT Group companies doing systems development, such as NTT DATA and NTT Comware. In principle, it will be used for all new projects and has a large and growing record of practical applications.

Fig. 2. Overview of Macchinetta framework.

To provide even higher quality support and increase the reliability of the AP framework, we continue to participate actively in improving OSS and are conducting R&D in cooperation with the OSS community. To implement major functionalities such as web application server-side processing and batch processing in the Macchinetta framework, we are using an OSS set centered around the Spring Framework [2], which is developed by the Spring community. We are providing bug patches to improve the quality of this OSS and proposing new functionality based on needs within the NTT Group. In particular, we are contributing by working on vulnerabilities that could lead to security risks. These are investigated and analyzed in collaboration with NTT Group companies, and bug-fix proposals are created in communication with principal developers in the community.

An active OSS community is also essential to ensure that the OSS we are using has many users and continues to be developed and maintained for a long time. We are actively contributing to invigorating the community such as by making presentations explaining the latest technologies at events sponsored by the community in Japan and around the world, and by promoting technical and event information on the web.

4. Future plans

The NTT OSS Center and NTT Software Innovation Center endeavor to increase the usability of OSS by testing coordinated operation of OSSVERT and the Macchinetta framework as described above (Fig. 3). We are also providing the necessary technologies and tools for software development and operation, supplying know-how, and supporting problem solving so that system updates for IT system security and modernization can be done and the OSS can be maintained and operated stably and efficiently over long periods of time. Through these endeavors, we are promoting the use of IT in NTT Group companies and helping to reduce their capital expenditure and operating expenses.

Fig. 3. Software development and operation with OSSVERT and the Macchinetta framework.

References

[1]	G. Suzuki, A. Kanamaru, T. Iwatsuka, J. Katada, S. Okada, S. Mochida, K. Natsukawa, K. Motohashi, T. Hishiki, T. Kaneko, K. Tanabe, H. Izumoto, M. Sakai, K. Yamashita, and Y. Iwaki, “Improving the Efficiency of Application Development Based on the Macchinetta Framework,” NTT Technical Review, Vol. 15, No. 2, 2017. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201702fa2.html
[2]	NTT DATA, “Complete Introduction to Spring: Java Application Development Using the Spring Framework,” Shoeisha, 2016 (in Japanese).

Trademark notes

All brand names, product names, and company names that appear in this article are trademarks or registered trademarks of their respective owners.