Feature Articles: R&D on Devices Using Life-assist Technologies
Development of Applications for a Wearable Electrode Embedded in Inner Shirt
NTT has developed a conductive fabric called hitoe that enables continuous measurement of the biological signals of the person wearing it. Heartbeat variations and electrocardiogram signals detected through hitoe are transmitted wirelessly by a compact dedicated device to a smartphone or tablet, where they can be readily checked using an application. Such technology is expected to lead to the creation of new services in fields such as sports training, health enhancement, security and safety, medical care support, and entertainment. In this article, we introduce some examples of approaches to application development.
Keywords: wearable, electrocardiogram, smartphone
NTT has collaborated with the textile manufacturer Toray Industries, Inc., to develop a conductive fabric called hitoe that enables continuous measurement of the biological signals of the person wearing it . Hitoe is a woven fabric that was created by coating fiber surfaces with an electrically conductive polymer material (PEDOT-PSS: poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate)) . Integrating this fabric with an inner shirt makes it possible to easily record heartbeat variations and electrocardiogram signals over the long term in a variety of daily life scenarios (Fig. 1). These physiological signals are measured and then transmitted wirelessly by a compact dedicated device to a smartphone or tablet, where they can be readily checked by using the appropriate application.
Heartbeat variations are naturally of interest in the field of healthcare, but they can also be used as health indicators in sports or dietary activities . It has previously been reported that hitoe can stably measure vital data during sports activities, for example, in sports with intense movements such as badminton . In addition, variations in heartbeat are thought to be related to the sympathetic nervous system and are known to act as indicators of emotional stress . By focusing on the development of both physical and emotional aspects, we can expect to see new services created in the fields of sports training, health enhancement, sleep management, medical care support, security/safety, and entertainment. To cover such a variety of services, it is essential to implement highly applicable systems in which smartphones and tablets are linked to a cloud service. In this article, we introduce three initiatives from the entertainment field and the security/safety field as applications of hitoe.
2. Applications in the entertainment field
Hitoe was exhibited at the Niconico Super Conference 3, an event attracting 100,000 fans of the Niconico website, on April 26–27, 2014, at Makuhari Messe, Chiba, Japan . We asked various guests to wear hitoe shirts during the conference, and we held five events including talk shows and games. We describe two events here: the First hitoe Cup Heartbeat Shogi Tournament and the Analog Game Workshop at Heartbeat University.
2.1 First hitoe Cup Heartbeat Shogi Tournament
Heartbeat Shogi is a name we coined for this event. It is a shogi (Japanese chess) TV program in which the heart rates of the players wearing hitoe shirts were measured and displayed to visitors on monitors and to an online audience over the Internet. We were looking for a new form of entertainment in which viewers could not only enjoy the battles taking place on the game boards and the body language of the players but could also understand the physiological changes the players were experiencing that we usually have no means of knowing about from their outside appearance. We set up a tournament and invited three professional players to take part in it (Fig. 2). The players wore hitoe inner-shirts during the matches.
The heart rates of the players during the final game are shown in Fig. 2(b). The heart rate is plotted along the vertical axis and the shogi move number along the horizontal axis. The heart rate of Shingo Ito, 5-dan (on the right in Fig. 2(a)), was within the 70- to 80-bpm level throughout the entire game. The heart rate of Naohiro Ishida, 4-dan (on the left in Fig. 2(a)), on the other hand, tended to rise from the middle game and even reached the 160-bpm level in the endgame. Such information was delivered to the audience in real time by integrating the wearable technology with an Internet image relay.
If we focus on more subtle heartbeat variations, we see that the heart rate of Ishida peaked at 7-2 rook in the middle game (Fig. 2(b)). This occurred at a game position where he took Ito’s bishop. Ishida’s heart rate continued to rise even in the endgame until Ito resigned. We think that this gives a glimpse into the correspondence between heartbeat variations and game position and can become a method of conveying a further sense of excitement to the audience.
Naohiro Ishida won the tournament and was given the title of First hitoe Heartbeat Shogi Meijin by Hiroo Unoura, President & CEO (Chief Executive Officer) of NTT. A better understanding of the relationship between heartbeat variations and emotional state was gained from the post-game discussion. Ishida, 4-dan, thought back and said that “The endgame was so tense, I could feel my own heartbeat. Even though I was playing aggressively and was in the lead, one false move and I would have lost the game.” We can infer from this that the anticipation of making the most of his chances in the game caused him inner tension. Ito, 5-dan, said that “By the endgame, I thought I had already lost, so I was completely relaxed.” This perspective might be one reason why his heartbeat variations were stable through the game. With shogi, decisive moments and relative superiority according to the game position can be seen on the board. However, the players’ own evaluations of the board positions and their level of tension can usually only be determined by their expressions or actions. We expect that the display of heart rates as an element that mirrors changes in the tension and emotional state of the players during games such as shogi will add new interest for viewers.
2.2 Analog game workshop at Heartbeat University
Heartbeat University is an educational program led by instructors with interests in particular areas. The first instructor we invited to be part of this program was Jun Kusaba, the originator of Game Market, Japan’s largest analog game experience and sales event and also the executive of the Gaming History Society. He gave a lecture and led workshops in an analog game. During the workshops, participants who were acting as students played an analog game under the coaching of Mr. Kusaba. The students wore hitoe shirts, and their heart rates were measured (Fig. 3(a)). We used the card game called Divinare in the workshop sessions (Fig. 3(b)). There are cards in four colors (red, blue, yellow, and green), and the game involves correctly predicting how many cards of each color were dealt. In this game, the luck and hunches of the players feature highly, in contrast to shogi, and even a beginner has a chance of winning. It was therefore expected that the heartbeat variations would be different from those of the shogi players. Each student was given three points if the predicted number of cards matched the correct number of cards, one point if the number differed by one, and minus one point otherwise.
They played four rounds and competed for the total number of points. An example of a student’s heartbeat variations is shown in Fig. 3(c). A rise in heart rate can be seen near the end of each round. This was assumed to be related to the way that the difference in his predicted number of cards and the correct number of cards was revealed at the end of each round; the previously unknown correct number of cards is gradually revealed as the game progresses. One student replied in a questionnaire given after the game was finished, that after the second round, “I was in fine form so I thought I might win if I carried on” and after the fourth round, “I thought that I would definitely hold on to my lead and receive a prize.” This shows that it is possible for such anticipation and agitation to be reflected in the heart rate.
However, variations in heart rate such as those of this student were not the only interesting example. There was another student whose heart rate remained about 30 bpm higher than usual from beginning to end, irrespective of developments during the game. To integrate biosignal measurements effectively into entertainment, it is important that repeatability between large numbers of people is confirmed and also that variations in the biosignals are presented in correspondence with the event in a way that the audience can understand. It is essential to explore the concept of creating events based on an understanding of the mechanisms of the body, then to search for methods of applying biological information from various viewpoints.
3. Application to security/safety
It is now possible to buy and sell an enormous variety of goods over the Internet, and the demand for distribution and delivery of goods is therefore very high. This has increased the importance of understanding the physical condition and stresses of vehicle drivers. Focusing on heartbeat variations and perspiration as stress markers during driving is thought to be effective for this . We had people wear hitoe shirts and then measured their heartbeat variations while they participated in driving tests under real-life conditions. An example of an electrocardiogram waveform taken while a subject was driving is shown in Fig. 4(a). The subject’s R waves, which are features of the electrocardiogram waveform, can clearly be seen. Photos taken when the driver was traveling along a road through a pass in a mountain region and along a suburban road with good visibility are shown in Figs. 4(b) and 4(c). The road through the mountain pass was a narrow one-lane road with a sequence of curves such as that shown in the photo. In contrast, the road with good visibility was a two-lane road that was fairly straight.
We can visualize heartbeat variations from an electrocardiogram waveform by representing them in Poincaré plots, as shown in Figs. 4(d) and 4(e). A Poincaré plot is a graph in which the n-th RR interval (the interval between two consecutive R waves; see Fig. 4(a)) from the start of measurement is plotted on the horizontal axis, and the next (n+1)-th RR interval is plotted on the vertical axis. When the driver is in a stressful situation, the fluctuations in heartbeat variations are small, and the plotted points clump together . Conversely, when the driver is relaxed, the plots scatter. A comparison of the plots of Figs. 4(d) and 4(e) shows that the plot for the mountain pass road is clumped closer together than that for the good-visibility road, suggesting the driver was under more stress on the mountain road. A questionnaire was later given to the driver, who said that on the mountain pass road, “I was driving cautiously since I didn’t know when an oncoming car would appear from a blind corner,” which verified that the driver had a certain sense of tension. This was a fundamental evaluation, but in the future, we plan to estimate driver fatigue, impaired physical condition, or drowsiness, which will contribute to improved security and safety while driving and also effective employee management support.
4. Future developments
In this article, we introduced three applications of hitoe. The ways in which these applications are implemented and combined with other technology will differ with the intended use, but it is important that each application has a high level of usability. We should provide a system that is easy to use, even for people who are not experienced with smartphones, and a full range of functions corresponding to individual needs. Apart from the methods described here, we are also promoting links with the GPS (global positioning system) functions of smartphones, various sensors, SNSs (social networking services), and digital games with business partners.