Move&Flick: Text Input Application for Visually Impaired People—From Research and Development to Distribution of the Application

1. Introduction

Many visually impaired people are interested in smartphones with touch screens since more smartphone applications are becoming available that are designed for such users, and some users are highly motivated to use devices that have become popular with many people. However, smartphone applications that demand a lot of text input via touch screens pose a challenge for the visually impaired. It is difficult to input text using the standard touch operation providing voice feedback that is similar to the key operation used with the numeric keyboards for feature phones, as the touch screens have no surface indentations.

Voice input is standard in smartphones and can be very helpful for people with visual impairments. However, this input is difficult to use in public spaces such as trains or department stores, as visually impaired users are reluctant to provide revealing details that people nearby, who are strangers, can hear.

While fluency in the standard operation of smartphones can be acquired with long-term training, most visually impaired people continue to use feature phones due to their ease of use, or they use a feature phone for text input and a smartphone for other applications. There has long been a demand for a new text input interface/application for smartphones that is easy to learn and to use by visually impaired users.

2. Easy-to-acquire gestures on touch screens for visually impaired people

NTT Service Evolution Laboratories has been researching and developing a new text input method for smartphone screens for visually impaired users. To ensure practicality, we focused on single finger gestures on the screen and attempted to minimize the importance of touch positions on the screen. Of course, multi-touch operations do exist and are already used in various applications. However, there is a lack of commonality, meaning that the same gestures can yield different commands depending on the application, and this tends to confuse the visually impaired. This is a unique problem for multi-touch gestures.

We asked the visually impaired staff of NTT CLARUTY to try various single finger gestures for screen input and identified which gestures were easy for the visually impaired to acquire. It is clearly difficult for visually impaired users to learn complicated gestures such as ideograms, as the lack of visual feedback means that they cannot understand what errors were made and how to correct them, and voice feedback makes it difficult for users to understand what errors were made and how to correct them. Even with an encircling gesture, which most people assume to be fairly simple, the finger of the visually impaired person tended to hit the side of the screen. Our investigation showed that they were able to acquire a gesture consisting of two continuous finger strokes with no intermediate finger lift; each stroke was made in one of eight directions, as shown in Fig. 1. The first stroke selects a consonant (or a command group), and the second stroke selects a vowel (or a command of the selected group). A Japanese character is input as combinations of a consonant and a vowel. The name Move&Flick replicates the basic gestures.

Fig. 1. Basic gestures of Move&Flick.

3. Techniques to facilitate learning the basic gestures of Move&Flick

Move&Flick is designed to help visually impaired users acquire the gestures with high accuracy and to increase the recognition accuracy of the basic gestures. Move&Flick has an algorithm to detect finger movement direction using radial areas with dead zones, as shown in Fig. 2. The finger movement direction is selected when a finger moves from the circle area into any of the eight radial areas. The selected area is provided as feedback to the user by voice. No movement direction is output while the finger is in the circle area or dead zones. This design helps the visually impaired to understand the ideal finger movement directions and to easily acquire the basic gestures. In addition, the dead zones suppress detection errors even if the operation is rough and quick, as may be expected with more experienced users.

Fig. 2. Algorithm to detect movement direction based on radial areas with dead zones.

Move&Flick also has an algorithm to accurately detect the change point, which is when the first stroke changes to the second stroke. This is necessary because the distance of finger movements differs with each input of each person, and the accuracy with which the change point is detected affects the accuracy with which the second stroke is detected. The details of this algorithm are shown in Fig. 3. The change point is Point Y, and the point at which the first stroke direction is detected is Point B, which does not coincide with Point Y. The algorithm confirms whether the detected first stroke direction and the direction of the finger motion is the same every time the finger moves a prescribed distance, which is the radius of C2 shown in Figs. 3(ii) and 3(iv), and sets Point Y when the first stroke direction and the direction of finger motion are different. The algorithm reduces the need to be careful about moving the finger the correct distance.

Fig. 3. Algorithm to detect the change point from first movement direction to second movement direction.

NTT Service Evolution Laboratories developed these algorithms as part of a project organized by the Ministry of Internal Affairs and Communications in the Japanese Government.

4. Design of Move&Flick

Japanese text input applications for the visually impaired include not only characters but also options such as delete, kana-kanji transformation, and options such as a voice reader. Various studies have been done on text input, but few text input applications have been released that implement the research achievements. One reason is that the application designs failed to implement all required commands.

Japanese characters outnumber alphanumeric characters, and there are several operations such as kana-kanji transformation and kana-katakana transformation that alphabet text input applications do not have. Therefore, NTT Service Evolution Laboratories in conjunction with NTT CLARUTY worked on enhancing the basic gestures of Move&Flick to simplify the input of all commands and effectively map gestures to commands.

To increase the level of modality, we focused on motions before the basic gestures. We implemented a means of changing the mode of input by entering a single tap immediately before entering the basic gestures. Most of the visually impaired users were able to learn this single tap operation in a short time. The operation in Move&Flick is used to change the consonant.

We also examined how the smartphones were held. The visually impaired tend to hold the smartphone with the non-dominant hand and operate the device with the dominant hand. Visually impaired users can discern the device width since smartphones have touch screens that are basically as wide as the device and the non-dominant hand has physical contact with both sides of the device. Therefore, they can accurately touch the left and right areas, each of which has the width of a single finger (Fig. 4). To input a character, the basic gestures are performed after touching the center area. To input an option, the basic gestures are performed after touching the left or right area.

Fig. 4. Three screen areas are easily selected.

The mapping of gestures to commands must be easy to remember if the visually impaired are to learn how to operate Move&Flick. NTT Service Evolution Laboratories and NTT CLARUTY carefully refined the assignment of consonants and vowels and the classification of option commands. Since some visually impaired people are already familiar with smartphones such as iPhones^*, the mapping was made as universal as possible to avoid confusion. NTT Service Evolution Laboratories had the visually impaired staff of NTT CLARUTY check each mapping change. Thus, Move&Flick was designed using a human-centered approach.

5. Experiment to evaluate the Move&Flick design

NTT Service Evolution Laboratories conducted an experiment to evaluate whether subjects were able to input kanji using Move&Flick after one hour of instruction and training. All subjects were visually impaired, and half of them had prior experience in operating smartphones. The results showed that all subjects were able to input kanji. Those experienced in the use of smartphones indicated that they wanted to use Move&Flick. Of the subjects without any experience in using smartphones, 60% also stated their desire to use Move&Flick. These results confirmed the feasibility of Move&Flick as an application for visually impaired users.

6. Preparation for application distribution

NTT DOCOMO developed Move&Flick for distribution based on the technical specifications provided by NTT Service Evolution Laboratories. Because visually impaired users tend to prefer the iPhone, the application program was written to ensure compliance with App Store Review Guidelines. In addition, NTT DOCOMO, together with NTT CLARUTY and a software vendor, improved the application based on the opinions of visually impaired people obtained in a survey. Some key steps in the development are given below.

7. Introduction of Move&Flick for the visually impaired

NTT DOCOMO advertised Move&Flick on its website, and NTT CLARUTY advertised Move&Flick in mailing lists and in communities for the visually impaired. NTT DOCOMO also held a trial session at the docomo Lounge, its main showroom in the Marunouchi area of Tokyo, to let a lot of visually impaired people know about Move&Flick, which is a new concept of text input appropriate for people with all levels of visual impairment. Visually impaired people and media representatives attending the session were able to try out Move&Flick. An NTT CLARUTY staff member served as a guide and introduced Move&Flick to the participants as a useful application enabling the visually impaired to input text on a touch screen.

NTT DOCOMO, together with NTT CLARUTY, has published audio manuals for Move&Flick based on the Digital Audio-based Information System format so that visually impaired users can learn how to use Move&Flick by themselves. In addition, the company provides a mock-up of Move&Flick (Fig. 5) to help users understand the operation of Move&Flick at docomo Hearty Plaza Marunouchi, a space dedicated to helping all users, including the visually impaired, how to use mobile devices. These mock-ups of Move&Flick have also been used at various exhibitions for NTT DOCOMO staff to show them how to use Move&Flick.

Fig. 5. Application mock-up.

8. Future prospects

NTT DOCOMO is proposing Move&Flick as a new text input application for the visually impaired. Various improvements have already been made to Move&Flick based on feedback and comments received from visually impaired users. While there are several existing smartphone applications that attempt to support the visually impaired, their acceptance is weak due to handicaps in their use. NTT DOCOMO will continue to strive to support the visually impaired with the goal of enabling them to fully enjoy their daily lives.

	Ryosuke Aoki Researcher, Networked Robot and Gadget Project, NTT Service Evolution Laboratories. He received a B.E. in engineering and an M.S. and Ph.D. in information sciences from Tohoku University, Miyagi, in 2005, 2007, and 2014. Since joining NTT in 2007, he has been conducting research in the areas of human computer interaction, multimodal interaction, motor learning, and service design. He has been a Visiting Assistant Professor at Tokyo Denki University since April 2016.
	Ryo Hashimoto Researcher, Universal UX Design Project, NTT Service Evolution Laboratories. He received a B.E. in engineering and an M.E. in informatics from Kyoto University in 2008 and 2010. Since joining NTT in 2010, he has been studying human computer interaction, universal design, and instructional design. He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE).
	Misa Hirao Researcher, Universal UX Design Project, NTT Service Evolution Laboratories. She received a B.A. in policy management in 2008 and a Master of Media and Governance in 2010 from Keio University, Kanagawa. She joined NTT Cyber Solution Laboratories in 2010, where she has been studying human-computer interaction, especially universal design and user-centered design.
	Masahiro Watanabe Senior Research Engineer, Universal UX Design Project, NTT Service Evolution Laboratories. He received a B.E. and M.E. in mechanical engineering and a Dr.E. in electronics, information, and communication engineering from Waseda University, Tokyo, in 1991, 1993, and 2003. He joined NTT Basic Research Laboratories in 1993 and studied the human auditory system. Since moving to NTT Cyber Solutions Laboratories in 1999, he has been studying human-computer interaction, especially universal design and user-centered design. He is a member of IEICE, the Human Interface Society, and the Acoustical Society of Japan.
	Tomohiro Yamada Executive Research Engineer, Supervisor, Networked Robot and Gadget Project, NTT Service Evolution Laboratories. He received an M.S. in electronics engineering from Niigata University in 1992 and an MBA from the University of Birmingham, UK, in 2007. He joined NTT in 1992, where he has been researching content distribution systems and technologies to support home appliances. He is the W3C (World Wide Web Consortium) Advisory Committee Representative of NTT and also a member of the Association for Computing Machinery, the Institute of Electrical and Electronics Engineers, and the Information Processing Society of Japan.
	Kazuma Ito Assistant Manager, Accessibility Promotion Office, Sales Department, NTT CLARUTY CORPORATION. He graduated from the Department of Management and Information Science from Niigata University of Management in 2003. He completed a course at the National Vocational Rehabilitation Center for Persons with Disabilities in 2005, after which he joined NTT CLARUTY. He has been working on improving web accessibility for persons with disabilities and developing barrier-free products and services.
	Akihito Tanaka Associate Manager, Accessibility Promotion Office, Sales Department, NTT CLARUTY CORPORATION. He graduated from the Faculty of Informatics, Department of Computer Science, Shizuoka University, in 2001. He worked at a company where he developed information systems from 2001 to 2003. He completed a course at the National Vocational Rehabilitation Center for Persons with Disabilities in 2004 and joined NTT CLARUTY the same year. He has been engaged in improving web accessibility for persons with disabilities and supporting the development of barrier-free products and services.
	Tomoaki Kodaka Manager, Accessibility Promotion Office, Sales Department, NTT CLARUTY CORPORATION. He graduated from the Aoyama Gakuin University Faculty of Law in 1987. From 1987 through 2002, he worked for Japan Finance Corporation. Since joining NTT CLARUTY in 2004, He has been engaged in managing efforts to evaluate and verify the accessibility of products and services to achieve universal design.
	Akihiro Ohtsu Senior Vice President, Executive Manager, Sales Department, NTT CLARUTY CORPORATION. He joined NTT in 1981. He was on loan to NTT CLARUTY from NTT EAST in 2012. He has been engaged in driving employment for persons with disabilities, increasing market sales, and negotiating with other companies including NTT Group companies.
	Akihiko Sasaki NTT DOCOMO, INC. He joined NTT DOCOMO in 1998. He has been involved in developing smartphone applications and user interfaces designed for visually impaired users, specifically the Move&Flick application.
	Yukio Takayashiki FUJITSU LIMITED. He was temporarily transferred to NTT DOCOMO in 2012, where he worked on the development of Move&Flick and mail applications such as docomo mail. He is now engaged in developing software for the Internet of Things at Fujitsu.
	Mitsutoshi Ehara NTT DOCOMO, INC. He joined NTT DOCOMO in 1998. He has been developing character input operations in the Move&Flick smartphone application for visually impaired users.