Ambient Assistance Technology Supporting Human Behavior by Understanding the Relationship between People and Environments

1. Introduction

Our “Point of Atmosphere” world, where natural interactions that harmonize people and environments are possible, is composed of information and communication technologies (ICTs) that help people choose the best behavior. These ICTs can determine the situations people are in, objects surrounding people, and their relationships with others as well as their behaviors, intent, and feelings in a wide range of daily activities. This suggests that ICT will go beyond online services and evolve to provide value by directly influencing various human behaviors in the real world (Fig. 1).

Fig. 1. Evolution of ICT.

Based on the trends in this ICT evolution, we are investigating ambient assistance technology, which models real-world environments (spaces, objects, and other people) that surround people engaged in various activities. The constructed models are combined with the models of perceptional, psychological, physiological, and physical states of people. Then, the choice of the best action is naturally supported through the devices that are human-environment interfaces (Fig. 2).

Fig. 2. A world enabled by ambient assistance technology.

2. A new world born of ambient assistance technology

What new conveniences will come about in our lives through the development and use of ambient assistance technology? In a city, for example, the possibility of colliding with a dangerously driven car when that car is in the pedestrian’s blind spot can be determined using models of the person and those around him or her as well as environmental models for roads and traffic flows to instantly derive a simulation to enable the person to take advanced precautions, such as evacuating to a nearby building, through digital signage around the person or wearable devices. For those who want to lose weight, a map app will guide them on a route with a slight detour to appropriately increase their activities without affecting the arrival time at their destinations, and an artificial intelligence agent could present easy-to-accept and optimized exercises to a person with the right timing to spark motivation and help exercising become a habit. In the office, these technologies enable environmental measurements to, e.g., increase productivity and encourage creative collaboration by adjusting temperature for comfort, adjusting lighting (brightness, color) to encourage concentration or relaxation, and creating sound fields to raise confidentiality or conversely to encourage information exchange, based on the modes of behavior in which people work (at their desks, in meetings, brainstorming, presentations, etc.) and surrounding situations (room layouts, desk alignments, gatherings of people, office equipment arrangements, etc.). This will naturally guide people towards the best actions to enable them to casually notice things that they would not normally notice, enable them to change their behaviors to achieve their goals, naturally change the way they work with influence from the environment, and increase their productivity and promote creative collaboration.

3. Towards ambient assistance technology

To make the aforementioned world a reality, we are developing ambient assistance technology to support optimized human behaviors. To do this, the following technical elements are required (Fig. 3). The first element is ambient sensing technology. Human behavior is based on various abilities, such as physical and psychological, that mutually influence each other. Additionally, environments in which people are partaking in various activities and their relationships with others are also considered to affect the demonstration of their abilities. Ambient sensing technology enables the generation of models for people and their surrounding environments by continually collecting and digitizing status information of the activity areas centered on active people without interfering with their original activities. The second element is real-world modeling technology. Ambient sensing technology collects information on the states of active people and their surrounding environments to reveal the mechanisms (models) of the changing states of people’s abilities or physical spaces. Real-world modeling technology integrates such digital models of people and the environment to enable reproduction (simulation) of real-world situations in the past, present, and future. The third element is ambient feedback technology. Real-world modeling technology can simulate human behavior and psychological abilities in certain environments and situations. Ambient feedback technology supports user behaviors by extracting optimal behaviors in a situation such as specific training methods and influencing the user through wearable devices and devices embedded in the surrounding environment.

Fig. 3. Elements of ambient assistance technology.

4. Ambient sensing technology

Ambient sensing technology focuses on human behaviors and the states of their surrounding environments in the real world and digitizes them immediately. Specifically, while defining targets of human and environmental sensing, we are studying the following technologies: personal space life log collection technology that continuously collects and digitizes state information of the activity area (personal space) in which the person is without causing them stress; personal space life modeling (twin modeling) technology to generate personal modeling (behavior, body movement, perception, emotions, physiology and ecology, etc.) and surrounding environment modeling (physical space, environmental state, surrounding object recognition, communications, etc.). For example, for human sensing and modeling, we have been working on state estimation (central fatigue estimation [1, 2] and state transition estimation of muscle activity patterns [3]) based on the collection and analysis of biometric data including heart rate and myoelectric data collected from wearable devices worn on the body in the form of shirts, socks, hats, etc. For sensing and modeling the environment, we have been working on technologies that estimate road conditions a user is currently walking on (inclinations, steps, stairs, etc.) obtained from body movements through the smartphone the user is carrying [4] and technology that maps sensing information from many people to physical space and detects changes in environmental conditions over a wide area [5]. Starting with devices to simultaneously record personal and environmental information acquired to date, we plan to study new sensing and modeling approaches combining environments and people that take into account the abilities of a person appearing in an environment, changes in the states of the environments appearing with a person’s behavior, etc.; thus, encouraging further research and development of ambient sensing technology.

5. Real-world modeling technology

Real-world modeling technology accumulates personal and environmental twin models obtained from ambient sensing technology and lays over high-precision spatial information and environmental monitoring, which is a bird’s-eye-view real-world model contrasting personal space, and incorporates physical laws, traffic rules, etc to achieve an autonomous real-world model that is constantly updated. In the real-world model, it can be expected that higher-order situations (contexts) will become clear from the reproduced relationships between people and between people and the environment. For example, a “cognitive map” that shows how a person perceives his or her place in geospatial space is dependent on his/her spatial cognitive ability, sense of direction, and spatial knowledge, which vary from person to person. It is also said that a place can be felt to be relatively closer than the actual geographical distance depending on the familiarity with it and attractiveness [6, 7]. Thus, reproducing real-world maps that look different and are distorted for each person from their spatial cognitive models and high-precision spatial information will reveal what a person who gets lost easily is actually looking at and in what kind of situations and what intentions caused them to lose their way, which can then be used to naturally help the person avoid such situations. Also, it could be possible to reproduce human memory models for changes in time by reproducing how information is transferred between people and environments in the real world by simulating past-present-future time lines from accumulated real-world models. With real-world modeling technology, therefore, we plan to investigate the digitizing of things that cannot be directly measured with individual sensors and models.

6. Ambient feedback technology

Ambient feedback technology supports people to engage in optimal behaviors calculated using real-world modeling technology. This technology will require methods to simulate human behavior and select optimal behaviors based on those simulation results. Therefore, we are considering using real-world models not only of oneself but also of other people. We aim to calculate optimal behaviors by comparing other people’s behaviors calculated from the models of others. This other person behavior is simulated by applying the same situation and purpose of behavior as oneself to the model of the other person. For example, by comparing the behaviors of sports professionals with one’s own behavior it might be possible to calculate more specific training methods. People will require support in performing behaviors after calculating optimal behaviors. Smartphone event notifications are often used for this purpose. In the future, we will consider more natural and casual methods so that anyone can benefit regardless of ICT literacy. For example, if the digital signage now widely in use advances further, anything in a city could become a signage device. If so, it might be possible to provide timely messages that are easy for people to perceive not only through the visual and auditory senses but also through any or all five senses. In addition, if wearable devices evolve, it might be possible for people to engage in optimal behaviors by just wearing such a device.

We conducted research on comparing human behavior by extracting the characteristics of professional and amateur movements [3]. In terms of wearable devices, we conducted research on inducing changes in muscle activity and walking without any user intention by applying contact stimulation to the sole of the foot from the insoles of shoes that change in hardness and shape [8]. To encourage people to engage in optimal behaviors, we have been researching message-generation methods that are easy for people to accept, based on behavioral change models [9]. We will keep expanding these techniques to enable the implementation of ambient feedback technology.

7. Conclusion

Although efforts to develop ambient assistance technology have only just begun, we are aiming for a world in which anybody can enjoy its benefits regardless of their ICT literacy. We will work quickly and steadily towards achieving the challenging goals of deeply understanding people and environments in the real world and modeling them to extract optimized behaviors of people from simulations and support people in engaging in those behaviors naturally.

References

[1]	T. Kondo, Y. Yamato, M. Nakayama, A. Chiba, K. Sakaguchi, T. Nishiguchi, T. Masuda, and T. Yoshida, “Natural Sensing with ‘hitoe’ Functional Material and Initiatives towards Its Applications,” NTT Technical Review, Vol. 15, No. 9, 2017. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201709fa3.html
[2]	K. Eguchi, R. Aoki, S. Shimauchi, A. Chiba, and N. Asanoma, “Biosignal Processing Methods Targeting Healthcare Support Services,” NTT Technical Review, Vol. 16, No. 8, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201808fa5.html
[3]	T. Isezaki, R. Aoki, T. Indo, S. Deshpande, and L. Tamir, “Transition Characteristics Analysis of Muscle Activity Patterns for Firefighters,” Proc. of the 41st Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 23–27, Berlin, Germany, July 2019.
[4]	Y. Kurauchi, N. Abe, H. Konishi, and H. Sehimo, “Barrier Detection Using Sensor Data from Multiple Modes of Transportation with Data Augmentation,” Proc. of the IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC 2019), pp. 15–19, Milwaukee, USA, July 2019.
[5]	Press release issued by NTT on Nov. 22, 2018 (in Japanese). https://www.ntt.co.jp/news2018/1811/181122b.html
[6]	Y. Wakabayashi, “The Evolution of Map: Geospatial Information and Future of Humans,” Sogensha Inc., 2018 (in Japanese).
[7]	Y. Wakabayashi, “Spatial Analysis of Cognitive Maps,” Chijin Shobo, 1999 (in Japanese).
[8]	M. Sasagawa, A. Niijima, K. Eguchi, R. Aoki, T. Isezaki, T. Kimura, and T. Watanabe, “Lower Limb Muscle Activity Control by using Jamming Footwear,” EMBC 2019, pp. 3302–3305, Berlin, Germany, July 2019.
[9]	T. Sato, R. Aoki, M. Koyasu, I. Shinozaki, N. Oshima, N. Mukawa, T. Watanabe, and T. Indo, “Koudouhennyou no Tameno Ninchiteki Fukyouwa ni okeru Message Teijihouhou (A Message Presentation Method in ‘Cognitive Dissonance’ for Behavior Change),” DICOMO2019, pp. 670–675, Fukushima, Japan, 2019 (in Japanese).