Development of a Mobile Personal Agent that Expands Human Potential

1. Introduction

In recent years, there has been growing interest in personal mobility vehicles, which are vehicles for short-distance travel for small numbers of people. These include two-wheeled self-balancing electric vehicles, micro-electric vehicles, and electric wheelchairs. It is forecast that the value of the market for these vehicles will expand to 120 billion yen by 2020 [1]. Against this background, NTT DOCOMO proposed the provision of an Environment for Comfortable Transportation as part of the initiative Business Creation in View of 2020 in its New Initiatives Toward Delivery of Medium-term Targets announced in April 2015 [2]. Specifically, NTT DOCOMO established DOCOMO BIKESHARE with joint investments by NTT Urban Development, NTT DATA, and NTT FACILITIES [3]. As the name indicates, DOCOMO BIKESHARE is a bicycle sharing business. In addition, NTT DOCOMO concluded a business alliance with WHILL KK. [4] to undertake a mobility sharing business using a personal mobility vehicle called the WHILL Model A [5]. The NTT Group is expected to further accelerate entry into this market.

2. Use of ICT in a personal mobility sharing service

We believe that two types of ICT (information and communication technology) have applications in a personal mobility sharing service. One is automation of the operational guidance provided for users. Since personal mobility vehicles are unfamiliar to first-time users, it is necessary to teach users how to perform such functions as powering-on the vehicle. If such operational guidance can be provided without human intervention, the workload on staff who handle the renting out of personal mobility vehicles can be reduced. The second application is automated assessment of the skill levels of personal mobility users. The number of traffic accidents involving electric wheelchairs, which have already been widely adopted, is increasing. Since no driving or user license is required to operate an electric wheelchair, there are many users who have not had sufficient practice. To prevent accidents, it is necessary to assess the operational skill level of users. However, it is difficult to determine whether or not a particular user is sufficiently skilled. Provision of the results of automatic assessment as feedback will encourage users to pay more attention to operations and help them to improve their skills.

3. NTT activities

NTT is developing a cloud-based interaction control technology called R-env^TM [6] (Fig. 1) that makes it easy to develop applications that focus on interactions with humans. The objective of this technology is to achieve a cloud-based multi-device interaction service that expands the potential of people by understanding their statuses and conditions through verbal and non-verbal communication between them and various robots, sensors, and other devices that surround them, and by using this understanding to encourage new actions and to enhance their awareness of their surroundings.

Fig. 1. Development of a device-associated service using R-env^TM.

R-env provides a cloud environment that allows device-associated services (i.e., services that combine robots, sensors, gadgets, and applications) to be developed, debugged, and executed using a single browser. Anyone can easily develop device-associated services by creating a state transition diagram that combines the actions of devices connected to R-env and conditions for transitions to the next actions using the graphical user interface of a browser. We have developed a prototype personal agent that can be used when a personal mobility vehicle is connected to a device such as a smartphone. The agent understands the user’s operational status and operational skill level from information gleaned from the user’s interactions with the personal mobility vehicle such as his/her dialog with and operations of the vehicle, and uses this understanding to provide feedback such as prompting operations through a dialog or otherwise enhancing his/her awareness about operations. The personal agent applied to personal mobility will provide operational guidance and rate the driver’s level of operational skill without any need for human intervention.

4. Joint experiment by three companies

To evaluate the effectiveness of this mobility-assisting personal agent, NTT Service Evolution Laboratories, NTT DOCOMO, and WHILL initiated a joint experiment on October 27, 2015 [7]. Specifically, NTT DOCOMO’s mobility sharing system [8] was installed in each WHILL Model A personal mobility vehicle, and the prototype personal agent was installed in the WHILL Model A and in a smartphone (Fig. 2). In the agent, the analysis engine for determining the driver’s level of operational skill, and the dialog control and speech synthesis technologies, all developed by NTT, work together. This experiment is aimed at assessing the effectiveness of operational training given to a first-time driver through a dialog with the agent.

Fig. 2. Overview of associated device and analysis engine.

The operational training developed by NTT has three features. First, the linkage between a WHILL Model A and a smartphone turns the WHILL Model A into a robot. As a mobility vehicle, the WHILL Model A is characterized by its excellent design, operability, and running ability. Incorporation of a smartphone into a WHILL Model A makes it possible to implement various personal agent functions such as dialogs with the user, displaying of images, determination of the user’s operational status from the analysis of data from sensors installed in the WHILL Model A, and provision of feedback of analysis results to the user.

Second, it is possible to provide operational training that is linked to data from sensors in the WHILL Model A. It is important that the operational guidance items provided to the user are given based on a particular user’s operational status. The mobility-assisting personal agent ascertains the user’s operational status through data obtained from various sensors and provides operational training appropriate for the situation. This enhances the effectiveness of the training.

Third, the level of operational skill of the user can be determined. At present, there is no benchmark for operational skills of personal mobility users, as is the case with the operation of electric wheelchairs, even though the latter are becoming increasingly popular. The Rehabilitation Center for Persons with Disability and the Association for Technical Aids provide a practice checklist [9]. However, the checklist only identifies items of required practice such as how to power-on an electric wheelchair and other basic operations, and how to move forward or turn right or left. It does not provide any way of determining if the user is able to satisfy each item in the checklist.

Noting that smartphones have acceleration and angular velocity sensors, we developed our own operational skill benchmark [10] based on the data from such sensors mounted on the WHILL Model A, and implemented it as part of the mobility-assisting personal agent function. The operational training includes training in such basic operations as powering a vehicle on/off and adjusting the speed, basic running operations such as moving forward and turning right or left, and finally, steering a figure eight course, which is used to provide feedback on his/her operational skill level to the user (Fig. 3).

Fig. 3. Overview of operational training.

5. Feasibility test on Soleil Hill

To evaluate the effectiveness of the mobility-assisting personal agent with the above features in providing operational training, we conducted a feasibility test of a WHILL Model A sharing service with visitors to Nagai Uminote Soleil Hill [11], Yokosuka City, from November 10 through December 9, 2015. Soleil Hill is a large, barrier-free park with a variety of amenities, including a restaurant, spa, and playground equipment, and is thronged with visitors on weekends. This site was selected because it was expected that there would be many visitors who could be targets for the sharing service, namely, those who can reach the park entrance by car but cannot walk around in the park, or those who like to try out new vehicles. Two WHILL Model A units were stationed at the entrance to Soleil Hill. Anyone wanting to apply to ride one of the vehicles was requested to learn how to operate the vehicle without assistance before venturing out into the park. The feasibility test examined whether or not the number of staff members needed to explain operation of the WHILL Model A could be reduced and if it was possible to accurately determine an individual applicant’s operational skill level. People ranging in age from their 20s to their 90s rode the WHILL Model A units. It was confirmed that the operational training given by the personal agent was able to eliminate user concerns about operating a WHILL Model A. There was no significant gap between the level of operational skill determined by the personal agent and the user’s self-assessment thereof [12]. Thus, we were able to confirm that the operational training given by the agent was effective.

6. Future prospects

The joint experiment confirmed that the operational training given by the mobility-assisting personal agent was effective. We are preparing to study an operational skill-rating method that does not simply assume a flat road but takes other road conditions such as brick paving and slopes into consideration, because the current benchmark for the level of operational skill does not take road conditions into consideration. We also aim to enable various indoor and outdoor devices to work together in order to provide a variety of empowering services that adapt to each user’s conditions. These include a navigation service that enables personal mobility users to avoid barriers and a running assistance service that assists personal mobility users based on the user’s condition and the environmental conditions. Furthermore, we plan to increase the number of partner enterprises that use R-env and the number of devices that can be connected to it so that personal agents can operate on various devices, including communication robots and home appliances. Our goal is to accelerate the realization of a world in which such personal agents make use of information about not only users’ mobility but also their current situations, in order to understand their conditions, interact with them, and enable them to recognize what actions are required (Fig. 4).

Fig. 4. Concepts of services made possible by R-env.

References

	Tatsuya Ishihara Research Engineer, Networked Robot & Gadget Project, NTT Service Evolution Laboratories. He received an M.E. in information science and technology from the University of Tokyo in 2006. He joined NTT Cyber Solutions Laboratories in 2006. From 2011 to 2014, he was with NTT EAST. He joined NTT Service Evolution Laboratories in 2014. His research interests include robotics and human-machine interfaces. He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE) and the Robotics Society of Japan (RSJ).
	Masanao Nakano Researcher, Networked Robot & Gadget Project, NTT Service Evolution Laboratories. He received an M.E. in information science from Kobe University, Hyogo, in 2014. He joined NTT Service Evolution Laboratories in 2014. His research interests include data analysis and human-robot interaction.
	Takayoshi Mochizuki Senior Research Engineer, Networked Robot & Gadget Project, NTT Service Evolution Laboratories. He received an M.A. in human sciences from Waseda University, Tokyo, in 2001. He joined NTT Service Evolution Laboratories in 2014. His research interests include human-robot interaction and human-machine interfaces.
	Tomohiro Yamada Executive Research Engineer, Supervisor, Networked Robot and Gadget Project, NTT Service Evolution Laboratories. He received an M.S. in electronics engineering from Niigata University in 1992 and an MBA from the University of Birmingham, UK, in 2007. He joined NTT in 1992 and has been conducting research on content distribution systems and technologies to support human-robot communication. He is the W3C (World Wide Web Consortium) Advisory Committee Representative of NTT and also a member of the Association for Computing Machinery, the Institute of Electrical and Electronics Engineers, and the Information Processing Society of Japan.
	Shin Mitsuhashi Assistant Manager, Smart-life Solutions Department, NTT DOCOMO, INC. He joined NTT Communications in 2004 and moved to NTT DOCOMO in 2012. He is currently involved in mobility share business endeavors.
	Muneaki Fukuoka Chief Technology Officer, WHILL KK. He worked for Olympus Corporation as a medical device engineer from 2008 to 2012 and joined WHILL as one of three founders in 2012.