Improvements to NICE-ELWISE Test Efficiency
We have developed an automatic testing tool that can increase the efficiency of testing work for NICE-ELWISE, which is an information distribution platform based on IC (integrated circuit) cards composed of two completely different types of software: a web-server-based operations and management system called NICE and embedded software for ELWISE cards. We investigated the features of both types of software and determined which tests are suitable for automation. This article describes our investigation and presents evaluation results.
1. Effectiveness of automating testing
Automated software testing can generally reduce the amount of testing work, and it can also help maintain a fixed level of software quality. However, in some cases, the cost of creating and maintaining the automatic test code can be high, so the desired effect is not achieved, even if all test items are automated. Consequently, it is important to take a strategic approach to test automation , .
2. NICE-ELWISE architecture
NTT Laboratories has developed an information distribution platform based on IC (integrated circuit) cards called NICE (network-based IC card environment)  and an extremely multipurpose IC card conforming to ISO 14443 Type B standards, called ELWISE , and has been promoting business based on these products . The architecture of NICE-ELWISE is shown in Fig. 1. The NICE-ELWISE software can be broadly divided into the NICE web-server-based operations and management system and the IC card embedded software, including the JICSAP2 infrastructure, the card manager, and various libraries for functions such as encryption (JICSAP: Japan IC Card System Application Council).
3. NICE-ELWISE testing
3.1 Testing overview
At NTT Laboratories, whenever a product is passed from the developers to the shipping stage for version upgrades or bug fixes, or when there is a change to its external environment such as a security patch to the operating system or middleware used by the product, the product’s individual functions as well as its overall quality are tested before it is delivered to businesses to ensure that no problems will result. Test items include operational tests done by the shipping personnel and items that provide an overall assessment, including items from test documents created by the developers. The latter include tests of functions that are used infrequently, such as when the batteries expire or there is a memory fault in the card.
3.2 Test features
The features of the NICE-ELWISE tests are listed in Table 1. Integration testing for web software is in the form of user operations from the application screen and entails frequent patches of the operating system and middleware as well as many structural variations. For embedded software, input/output interface tests are used and updates to IC chips or card firmware (APE: application execution environment) are infrequent, but tests for compatibility with reader/writer equipment are necessary.
3.3 Testing issues
Since there are many products and versions, the cost of compatibility testing is very high and requires knowledge of the implementation and overall know-how about each product. The fact that it is difficult for someone unfamiliar with a product to perform the tests has been an issue in efforts to complete the testing work. For these reasons, we studied ways in which overall product quality testing can be done in a short amount of time to a consistent level by persons that have not had the opportunity to accumulate detailed implementation knowledge.
4. Testing suited to automation
In general, tests that are suited to automation are ones for which little work to update the test automation code is required. Examples of suitable tests include tests of functions for which changes to the application programming interface are infrequent, tests where the code is used many times, such as load testing, or simple test scenarios where the code can be created easily. In contrast, tests that are not well suited to automation include tests for functions that are changed frequently, such as the graphical user interface (GUI), regression testing that requires changes or additions to the test code, testing of configuration variations such as environment settings, and tests where the results from various input values must be checked.
5. Application of automation
Because research and development of the basic functions of NICE-ELWISE was complete, no further significant expansion in functionality was expected, business deployment was proceeding, and usage patterns had become well-known, we decided that it had become easier to automate many tests that had previously been done manually by shipping personnel. For example, GUI testing is generally not considered suitable for automation because of the frequent changes, but no further significant changes were expected to the GUI for NICE. Furthermore, while configuration variation tests in multiple environments are not usually suitable for automation, the conditions for embedded systems are fixed; in particular, those for NICE are fixed. Finally, periodic checks related to security patches are also required, so test code for this purpose can be used many times over. For these reasons, we decided that the software was suitable for test automation.
6. Clarification of test viewpoints
To ensure uniformity in the level of test work, we first established clear test viewpoints to be used by the shipping personnel.
(1) To check the functionality at the individual product level, we took the user viewpoint and performed system operations correctly under normal conditions and in normal environments.
(2) To check the overall product quality, it is necessary to perform checks at all levels, from the upper application level to the lower libraries, ensuring that they are operating together correctly.
7. Test organization
By using the test viewpoints instead of testing the products individually, we studied ways of testing that encompass test items from the upper application levels all the way down to lower-level libraries. For example, we selected upper application-level commands that could operate all normally used functions by studying all of the interfaces between each product and between the products and hardware in detail. In this way, the tests to be performed by the shipping personnel could be designed to cover about 70% of the NICE test items, 85% of the JICSAP2 infrastructure items, and 90% of the library items relative to the entire test item set. As shown in Fig. 2, the test organization comprises two configurations: configuration 1 tests operation of functions linking internal IC card software and configuration 2 tests functions linking the server, terminal, and IC card.
8. Evaluation of test optimizations
We studied automation suitability and established clear test viewpoints as described above and then developed test automation tools.
(1) Efficiency of configuration 1
We evaluated the efficiency of configuration 1 by measuring the time taken to perform the tests manually and by using the test tools. Automation reduced the testing time by 98%, and testing for each version could be accomplished in approximately 30 minutes.
(2) Efficiency of configuration 2
For configuration 2, we not only examined the increase in efficiency after test tool introduction, but also considered that a person with limited NICE testing experience could now perform the tests to the same level as a NICE developer. The testing time required by an inexperienced NICE tester is shown in Fig. 3. For test procedures using the test tool and requiring fewer than five operations, a significant reduction in test times was seen after the first attempt. Since tool operation is very simple, no further effect was observed from the second attempt onwards, regardless of the tester’s experience level.
For test items requiring five or more procedural steps, such as those requiring preparation before running the test tool, additional time was required for inexperienced testers on the initial attempt, but from the second attempt, once they had completed the entire test once and understood the procedure, a significant reduction in required time was observed because test settings and conditions were fixed.
By performing the tests many times and mastering the procedures, inexperienced testers could perform them to the same level as a developer (shown for reference), so a time saving of about 71% can be expected.
The results shown that tests can now be performed to the same level as an experienced tester, even when test personnel change, once the new personnel have mastered the test operations.
9. Future developments
By studying the suitability of automatic testing taking into consideration the particular characteristics of the software, we can learn how various approaches—such as building product linkage test environments that clarify a product’s interfaces down to the hardware level and maintaining the quality and level of testing by establishing environmental conditions from the user viewpoint—can be applied to general software development. We are advancing the application of these methods as one way to increase software development productivity in the future.