Evaluation of Adaptability to Unfamiliar Environments Using Virtual Reality

1. Significance of evaluating ability to adapt to the environment while moving

Accidents related to mobility for the elderly, such as falling down and automobile accidents, are societal issues. Falling while walking accounts for a significant proportion of deaths among the elderly. Elderly persons also account for 18.1% of traffic accidents, and the proportion of accidents caused by elderly drivers tends to increase yearly [1].

NTT Service Evolution Laboratories is conducting research on estimating users’ mobility function from inertial data and surface electromyogram data [2, 3]. Mobility behaviors, such as walking or driving a car, are considered to be performed based on a general motor control model. This involves a sequence of processes: that of perceiving the state of one’s body and the external environment through visual, auditory, and somatic senses; cognitive processing of the perceived information in the brain; sending instructions to the musculoskeletal system; and the mobility processes in which the musculoskeletal system receives the instructions from the brain and muscles contract or relax accordingly. To understand the causes of elderly people falling down while walking or being involved in traffic accidents and prevent such incidents, it is necessary to examine all of these faculties, not just mobility. Cattell subdivided intelligence into two aspects: crystallized intelligence and fluid intelligence [4]. Fluid intelligence is the ability to learn new things or solve problems to adapt to new environments, and is thought to decline with age. Conversely, crystallized intelligence includes the abilities an individual acquires through experience, training, and practice throughout their life and tends to be stable despite the passing of years. When walking or driving a car, one often leaves familiar roads and drives on new and unfamiliar routes or encounters unexpected situations such as pedestrians who suddenly appear. When appropriate mobility behaviors are required in an environment that is changing from minute to minute, fluid intelligence, or the cognitive ability to adapt immediately to unfamiliar or unexpected aspects in the environment, is important. Therefore, evaluating and improving the user’s ability to adapt to unfamiliar or unexpected environments while moving are important in preventing accidents related to mobility.

2. Virtual-reality system for evaluating ability to adapt to the environment while moving

To measure users’ ability to adapt to their environment requires observation of their behavior in unfamiliar and unexpected environments, evaluation and analysis of the environment, and their bodily movement in response. Various manufacturers have recently begun selling consumer-oriented virtual reality (VR) systems that anyone can use. VR is able to simulate and reproduce various environments and provide information on those environments and the user’s movements in those environments. The concept for this research was to use VR to create environments that are unfamiliar to the user then measure their ability to adapt to those environments by observing their movements and performance in the environments.

To evaluate the ability to adapt to the environment while moving requires analysis of bodily information, including how behaviors differ when the ability to adapt to the environment is strong or weak. The level of dexterity (skill) with the upper limbs (including upper arms, forearms, and hands) is understood to be related to cognitive functions [5]. Considering the ability to adapt to the environment while moving as one of these cognitive functions, our concept was to evaluate this ability by focusing on dexterity in the upper limbs. There are many methods for evaluating dexterity in the upper limbs, but three basic upper-limb mobility functions have been identified related to dexterity, which are spacing, timing, and grading [6]. Spacing is the ability to move the hand in the correct direction, timing is the ability to move the hand by correctly adjusting the timing, and grading is the ability to correctly adjust the effort applied while moving. NTT Service Evolution Laboratories is conducting joint research with Prof. Kensaku Mori and Assoc. Prof. Masahiro Oda of the School of Information Science, Nagoya University, and Prof. Hitoshi Hirata and Designated Assistant Prof. Shintaro Oyama of the Department of Hand Surgery in the School of Medicine, Nagoya University, for implementing a system using VR to evaluate the ability to adapt to the environment while moving that is based on the dexterity of the upper limbs.

For upper limb dexterity, we focused specifically on the task of catching a falling ball. To simulate an environment in which the users were not accustomed to, we conceived three scenarios that would produce differences from behavior in the real world. The first was to change the characteristics of the object (ball). In particular, we examined changes in the acceleration of gravity and the size of the object and created an implementation in which these parameters are variable. The second was to change the characteristics of the environment space. We examined spaces with obstacles that cause the object to bounce and in which the acceleration of the object (ball) changes three dimensionally. We implemented these obstacles and the space so that the range, rebound coefficients, and acceleration settings could be set arbitrarily. The third scenario was to change the motion characteristics of the user. In this case, we set the parameters, i.e., coefficients used to project real-world upper-limb movements onto movements of the avatar in VR space; either exaggerating or attenuating the movements.

Figure 1 shows images of using a prototype of our VR system. In this task of catching a falling ball from the scenario of changing the characteristics of the environment space, a prescribed range (shown in red) is an obstacle, which bounds the environment in which the ball is falling, simulating changes in the characteristics of the environment. Through multiple trials, the users were able to adapt to this unfamiliar environment and consistently catch the ball. The system measures the user’s movement data (from the controller) and their performance while performing the task in the specified environment. A sequence of user operations for one task (Fig. 1(a)) and the user’s VR perspective (Fig. 1(b)) are shown. As shown in Fig. 1(a), the user in the real world wears a head-mounted display (HMD) and uses controllers, which are part of the VR system. The video shown on the display in Fig. 1(a) is the VR data that the user receives through the HMD. Video selected from what the user sees in the HMD is shown in Fig. 1(b). The left is a scene with the red ball falling from above, the center shows the ball bouncing off of the obstacle (the red region), and the right shows the ball having been caught by the user.

Fig. 1. Images of a prototype of our VR system.

Figure 2 shows graphs of (a) the cumulative number of successes and (b) reproducibility of the motion when a single user performed multiple trials of the task. Reproducibility of motion is computed as the correlation coefficient for the motion data between trials. It is clear that both the cumulative successes and movement reproducibility increased with the number of trials. By quantifying changes in performance such as success rates and motion reproducibility as trials accumulate, we expect to be able to evaluate the user’s ability to adapt to the environment while moving.

Fig. 2. An analysis example from a user.

3. Future development

This article introduced a VR system for evaluating the ability to adapt to an environment while moving. It enables movement data and performance measurements to be collected during the process of adapting to changes in the environment. We intend to continue studying methods for evaluating the cognitive function of the ability to adapt to the environment while moving, on the basis of data from participants with various attributes, using measured motion and performance data. We also intend to continue updating the tasks and configurations in our prototype VR system.

We plan to investigate the significance of this evaluation of adaptability in the real world, beyond falling or traffic accidents, to issues with a wide range of cognitive functions, such as forgetfulness. We will also investigate how this system can be used to improve adaptability to the environment while moving.

During the current novel coronavirus pandemic, we frequently encounter situations in which we cannot go out for a walk or other exercises. The need for technologies for training our bodies while at home is higher than ever before. Many devices are being brought to market that are designed to improve mobility-related functions, such as muscle mass, and we expect that training cognitive functions using devices/systems, such as that introduced in this article, will also become more common. We will contribute to extending the healthy lives of everyone by creating technologies to evaluate and improve cognitive function in various ways and implement them in peoples’ homes as evaluation and training systems.

References

[1]	Tokyo Metropolitan Police Department, Preventing Traffic Accidents Involving the Elderly (in Japanese), https://www.keishicho.metro.tokyo.jp/kotsu/jikoboshi/koreisha/koreijiko.html
[2]	T. Isezaki, N. Fujii, S. Ishii, T Watanabe, and O. Mizuno, “Physical Function Estimation Based on Inertial Data of the Motion of Stepping Over an Obstacle,” 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2016), Orlando, FL, USA, Aug. 2016.
[3]	T. Isezaki, H. Kadone, A. Niijima, R. Aoki, T. Watanabe, T. Kimura, and K. Suzuki, “Sock-type Wearable Sensor for Estimating Lower Leg Muscle Activity Using Distal EMG Signals,” Sensors, Vol. 19, No. 8, 1954, 2019.
[4]	R. B. Cattell, “Theory of Fluid and Crystallized Intelligence: A Critical Experiment,” J. Educ. Psychol., Vol. 54, No. 1, pp. 1–22, 1963.
[5]	J-Y. Yoon, T. Okura, K. Tsunoda, T. Tsuji, Y. Kohda, Y. Mitsuishi, C. Hasegawa, and H. Kim, “Relationship between Cognitive Function and Physical Performance in Older Adults,” Japanese J. Phys. Fit. Sport. Med., Vol. 59, No. 3, pp. 313–322, 2010.
[6]	T. Fukui, K. Inoue, and K. Tunehisa, “Development of an Apparatus for Hand Dexterity Evaluation: Study of a Clinical Adaptation,” Kawasaki Medical Welfare Journal, Vol. 17, No. 2, pp. 389–394, 2008 (in Japanese).