You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Feature Articles: NTT’s Medical and Health Vision toward Creation of Bio-digital Twin

Vol. 19, No. 7, pp. 40–45, July 2021. https://doi.org/10.53829/ntr202107fa5

New Technology for Measurement and Analysis of Biological Sounds and Electrocardiographic Signals—Toward Early Detection of Heart Disease and Rehabilitation by Using Personal Heart Modeling

Masahiro Nakano, Ryohei Shibue, Kunio Kashino,
Shingo Tsukada, and Takayuki Ogasawara

Abstract

Biological sounds and electrocardiographic signals are important for understanding the physical and psychological state of a person. In this article, signal-processing and machine-learning technologies involving measuring biological sounds by using a wearable acoustic-sensor array, called a telestethoscope, as well as new measurement and analysis technology, i.e., tensor electrocardiogram using hitoe^TM, are introduced. The possibility of early detection of diseases by using such personal heart modeling and its application to post-onset rehabilitation are also discussed.

Keywords: telemedicine, tensor electrocardiogram, wearable devices

1. Estimating the state of mind and body through biological sounds

The biological information obtained from measuring the living body reflects the function, morphology, and dynamic characteristics of the living body. We are researching and developing the concept of bio-digital twins to improve people’s well-being by, for example, early detection of diseases. With a bio-digital twin, various types of biological information as signals are captured, and signal-processing and machine-learning technologies are used to model the physical and mental states of each person on the basis of the observed signals. Initiatives that focus on the sounds generated by living organisms are first introduced.

1.1 Telestethoscope

Doctors and nurses use auscultation to determine the presence or absence of abnormalities and the degree of urgency. Skilled medical practitioners can visualize in their minds the generation and transmission of biological sounds such as breathing, movement of heart valves, and blood flow. Inspired by such auscultation by medical professionals, we are conducting research and development on a wearable device called the telestethoscope (see Fig. 1(a)).

The telestethoscope is used for acquiring biological sounds (acoustic signals) from multi-channel microphones and electrocardiograms (ECGs). The acquired acoustic signals and ECG can be sent to a remote terminal via a network. Sounds from various points on the wearer’s body can be heard by touching the screen of that terminal. An overview of the telestethoscope is shown in Fig. 1.

The telestethoscope will allow medical personnel to listen to acoustic signals of patients remotely without having to come into contact with patients; therefore, it will be useful for examining patients with infectious diseases and assessing the urgency of patient care from a distance (e.g., when the patient is at home or in a rural area) (Fig. 1(b)). Since it is possible to record and share biological sounds, it is also possible to repeatedly auscultate offline or chronologically study the status by comparing current and past sounds. In addition to the above use by medical professionals, we are also researching its use by the general public for self-healthcare, as described below.

1.2 Adding descriptive text to heart sounds by using the telestethoscope

It is not easy for non-specialists to understand the information about living organisms by simply listening to biological sounds. However, signal processing and machine-learning technology makes it possible to convert biological sounds into a form that is easy for many people to understand. Therefore, we are investigating converting (translating) biological sounds directly into simple wording (Fig. 1(c)).

The most-straightforward means of explaining biological sounds would be to classify them as “normal” or “abnormal.” However, if a person’s sounds are classified as “abnormal,” a more-detailed explanation may be needed. For example, it would be useful to know what abnormalities are present and whether the person in question should go to the hospital immediately. To address this issue, we previously proposed a method called conditional sequence-to-sequence caption generation, which allows the user to specify a numerical value called specificity that represents the level of detail of the description to be generated [1]. For example, if the heart sounds and a large number as a desired level of detail are input into the system, a detailed explanation can be output.

1.3 Estimating health status via video images from heart sounds acquired from the telestethoscope

Another means of converting biological sounds into an easy-to-understand form is to generate moving images that represent biological functions in real time (Fig. 1(d)). We have attempted to reconstruct the movement of the heart from the sound of the heart. A normal heart moves in a cyclic fashion in which the transitions occur in the following order: atrial systole, isovolumic systole, ejection phase, isovolumic diastole, and filling phase. We therefore aim to estimate the transitional state of the heart from measuring heart sounds then estimate and reconstruct the movement of the heart in three dimensions (3D). Since the heart pumps blood around the body, a specific physical relationship between its pressure and volume is maintained through the state transitions. Focusing on this physical relationship, we devised a method of constructing a 3D moving image. This method uses the physical relationship to improve the accuracy of cardiac motion estimation by restraining it in accordance with the state transition, which is estimated from the heart sounds [2].

Accurately estimating and predicting the physical and psychological state of an individual from easily observable biometric information, such as sound, will necessitate many other innovations. We will continue to research and develop signal-processing and machine-learning technologies for measuring biological functions and dynamics from multiple perspectives, such as sound information and electrocardiographic information, as described below.

Fig. 1. System overview of telestethoscope. (a) Telestethoscope wearable device, (b) remote auscultation, (c) automatic sound-description generation, and (d) visualizing estimated 3D cardiac motion in accordance with the estimated heart states.

2. Wearable 3D ECG device using new lead system and ECG analysis method called tensor ECG guided by the new system

To determine the physiological function of the heart, ECGs are widely used in, for example, diagnosis at medical institutions, vital-sign monitoring, medical examinations, and automated external defibrillators. With advances in information and communications technology, such as the Internet and smartphones, as well as in information-processing technologies, such as machine learning, ECGs are being applied in new fields such as artificial-intelligence-guided healthcare. Due to the increase in cardiac diseases in the super-aging society, a society whose population aged 65 or older is over 21%, the need for home care and telemedicine using electrocardiography is rapidly increasing. Taking advantage of our experience in developing a wearable ECG device using hitoe^TM, we are developing a wearable device for continuous measurement of ECGs and an ECG analysis method by combining medical knowledge on clinical ECGs with recent information-processing technology [3].

2.1 3D ECG induction suitable for long-term recording in wearable device

ECGs are analyzed on the basis of their potential difference (scalar quantity) and shape (pattern), which are obtained from multiple bioelectrodes placed at specified points (standard 12 leads) on the surfaces of the extremities and chest. As a method of recording electrocardiographic potential more three-dimensionally, a vector ECG (which uses a three-axis Cartesian-coordinate system) involves using electrodes on the chest, head, and lower extremities and a resistance-correction circuit. Since both standard-12-lead and vector ECGs involve using electrodes on the extremities, they are easily affected by body movements, so they are basically recorded while the patient is in a resting state. However, electrodes attached to the chest are less affected by body motion and can produce relatively large cardiac potentials, so they are used for Holter ECG monitors (portable devices for cardiac monitoring), exercise-load, and sports ECGs.

For long-term stable recording of a 3D ECG, we devised a wearable devise for measuring ECG with its reference point set in the apex region of the heart (i.e., on the left anterior wall, where the heart is closest to the thorax) and with counter poles in three linearly independent directions (Fig. 2(a)) [4]. By bipolar induction^*1 generated by placing two horizontal leads on opposite sides of the thorax and vertical leads on the upper right precordium in line with the electromotive-force base axis^*2, 3D electrocardiographic potential can be stably recorded (Fig. 2(b)). The electrodes and wires are integrated into an elastic belt that can be easily attached by simply tightening the shoulder and waist parts of the belt. We have also developed a non-invasive cardiac polygraph with hemodynamics for simultaneously measuring cardiac output and deep vascular pulse waves (Fig. 2(c)).

2.2 New method for analyzing ECGs: tensor ECG

Abnormalities in ECGs are shown as not only a standardized abnormal waveform (pattern) but also as a slight distortion of the shape or change in potential. A method of quantitatively evaluating such atypical ECG abnormalities has not been developed. We are currently designing individual criteria for specific diseases and special patterns to identify them. Regarding the collective action potential of the myocardium, the electromotive force of the cardiac potential has three plateau phases with continuous depolarization, whereas the cardiac potential at the body surface does not usually show such a plateau. We use the molecular-biology and physiological constraints of the membrane potential of each period (from phases 0 to 4) of the action potential to estimate the collective action potential of the myocardium from the electrocardiographic potential on the body surface (i.e., an inverse problem) (Fig. 2(d)). From the conversion of electrocardiographic potential to action potential, atypical distortions are magnified and made clearly visible (Fig. 2(e)). We are examining whether the parameters (tensor[s]) obtained during the conversion to action potential of the myocardium can be used as indicators for quantifying and uniformly evaluating complex abnormal patterns and minute distortions in ECGs. We anticipate that tensor ECG for multi-mode data measurement and analysis will be useful in diagnosing arrhythmias associated with heart failure, ischemic heart disease, and sudden cardiac death.

Fig. 2. Overview of tensor ECG: a new induction and analysis method.

*1	Bipolar induction: An ECG records the flow of electricity in the heart, and induction is the process of creating a flow of electricity from a negative electrode to a positive electrode by attaching electrodes across the heart. Bipolar induction is a method of acquiring an ECG by using two electrodes.
*2	Base axis of electromotive force: Since the heart is a 3D structure, its electrical charge continuously changes three-dimensionally. An ECG can be regarded as a recording of the electromotive force generated throughout the heart, and the direction of the electromotive-force vector is called the base axis.

3. Application to rehabilitation using wearable devices

Wearable devices are useful not only for early detection and accurate diagnosis of diseases but also after the onset of disease. One example of a wearable device is one for improving rehabilitation quality we have been developing and medically validating with Fujita Health University and Toray Industries, Inc. since 2017 [5]. Taking stroke rehabilitation as an example, the more opportunities for rehabilitation (exercise training) after the occurrence of a stroke, the better the outcome can be expected [6]. However, rehabilitation with therapists is limited, so stroke patients are encouraged to be as active as possible in their daily lives by getting out of bed or doing rehabilitation exercises on their own. However, it is difficult for a patient with a one-side-paralyzed (hemiparetic) body to exert vigorous effort on their own, so they need support to achieve a better recovery. Against this background, we developed a wearable-device that enables precise monitoring of activities of a stroke patient on a 24-hour basis (Fig. 3). For example, a hemiparetic patient wears our device, which he/she can put on and take off by him/herself, and data are automatically collected on a server through a relay such as a smartphone or gateway. The device includes an algorithm for processing data for (i) advanced downsampling that avoids overloading the network of hospital facilities while preserving the characteristics of the patient’s activity and (ii) imputation processing assuming partial data loss. Medical validation of the index obtained by these processes is also underway.

Fig. 3. Overview of the rehabilitation support system.

This device is used on the basis of three types of outputs. The first is a report for the patient and their family. The recovery process can be clearly conveyed through numerical records of the gradual increase in the time spent in active movement. The illustrations change as the patient progresses, so the patient and his/her therapist can discuss the progress made after each session, which will lead the patient to the next exercise. The second output is the detailed measurement results used in medical-team meetings. Medical-care providers working with a large number of patients cannot keep track of specific individuals. However, on viewing the 24-hour record output from the device, the entire team can share changes in a certain patient’s daily activities. The third output is listed information based on measurement data.

Regarding medical research, facts are revealed by verifying statistical certainty derived from a large number of measurement results. However, the amount of activity data over a 24-hour period is large; therefore, we designed a system for automated preprocessing so that it can be easily used in medical research. The system also supports the 6-minute walk test, which is widely used in rehabilitation studies. This allows the results of 24-hour activity monitoring to be easily compared with previous studies. We expect this will lead to new insights in rehabilitation research. The system has been commercialized and is being used for medical research. We believe that it will further contribute to better patient recovery through combining it with advanced technologies such as machine learning.

References

[1]	S. Ikawa and K. Kashino, “Neural Audio Captioning Based on Conditional Sequence-to-sequence Model,” Proc. of the Detection and Classification of Acoustic Scenes and Events 2019 (DCASE 2019) Workshop, pp. 99–103, New York, USA, Oct. 2019.
[2]	M. Nakano, R. Shibue, K. Kashino, S. Tsukada, and H. Tomoike, “Gaussian Process with Physical Laws for 3D Cardiac Modeling,” Proc. of the 28th European Signal Processing Conference (EUSIPCO 2020), pp. 1452–1456, Amsterdam, Nederland, Jan. 2020.
[3]	Y. Tsukada, M. Tokita, H. Murata, Y. Hirasawa, K. Yodogawa, Y. Iwasaki, K. Asai, W. Shimizu, N. Kasai, H. Nakashima, and S. Tsukada, “Validation of Wearable Textile Electrodes for ECG Monitoring,” Heart Vessels, Vol. 34, No. 7, pp. 1203–1211, 2019.
[4]	S. Tsukada, “Wearable Textile Electrodes for Long-term Vector ECG Monitoring ‘Tensor Cardiography,’” 14th International Symposium on Medical ICT (ISMICT 2020), Nara, Japan, May 2020.
[5]	T. Ogasawara, K. Matsunaga, H. Ito, and M. Mukaino, “Application for Rehabilitation Medicine Using Wearable Textile ‘hitoe’,” NTT Technical Review, Vol. 16, No. 9, pp. 6–12, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201809fa2.html
[6]	M. Yagi, H. Yasunaga, H. Matsui, K. Morita, K. Fushimi, M. Fujimoto, T. Koyama, and J. Fujitani, “Impact of Rehabilitation on Outcomes in Patients with Ischemic Stroke: A Nationwide Retrospective Cohort Study in Japan,” Stroke, Vol. 48, No. 3, pp. 740–746, 2017.

	Masahiro Nakano Researcher, NTT Communication Science Laboratories. He received a B.E. and M.E. in information science and technology from the University of Tokyo in 2009 and 2011. He joined NTT Communication Science Laboratories in 2011, where he is currently working on Bayesian machine learning and its application to biomedical information processing. He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE).
	Ryohei Shibue Researcher, NTT Communication Science Laboratories. He received a B.E. and M.E. in information science and technology from the University of Tokyo in 2015 and 2017. He joined NTT Communication Science Laboratories in 2017 and studied statistical modeling for eye movements. He is currently studying cardiovascular signal processing. He is a member of the Japan Statistical Society and the Japan Neuroscience Society.
	Kunio Kashino Senior Distinguished Researcher at NTT Communication Science Laboratories, and Visiting Professor at the National Institute of Informatics (NII). He received a Ph.D. from the University of Tokyo in 1995. His research interest includes audio and video analysis, synthesis, search, and recognition algorithms and their implementation. He is a fellow of IEICE and a member of the Institute of Electrical and Electronics Engineers (IEEE), the Association for Computing Machinery, Information Processing Society of Japan, Japanese Society for Artificial Intelligence, and the Acoustic Society of Japan.
	Shingo Tsukada NTT Fellow, NTT Basic Research Laboratories. Shingo Tsukada received an M.D. from Toyama University School of Medicine and a medical license in 1990 and Ph.D. in medicine from the University of Tsukuba in 2003. He was a visiting researcher at the University of California at San Diego from 2003 to 2005. He joined NTT Basic Research Laboratories in 2010 as a research specialist. He has been studying the mechanism and activity control of signal transduction of brain cells and cardio vascular regulation. His current interests include the detection of biomedical signals and functional modification using novel wearable-type and implant-type bioelectrodes based on the composites of conductive polymers with various fibers and textiles. He is an inventor of the textile bioelectrode hitoe^TM. He is a member of the Physiological Society of Japan, the Japan Society of Applied Physics, the Japanese Circulation Society, and the Japanese Orthopedic Association.
	Takayuki Ogasawara Senior Research Engineer, Molecular and Bio Science Research Group, NTT Basic Research Laboratories and Bio-Medical Informatics Research Center. He received a B.E. and M.E. in applied physics from Hokkaido University in 2006 and 2009. He also completed the Developmental Brain Science Master’s Program at the Research and Education Center for Brain Science, Hokkaido University, in 2009. He joined NTT in 2009. His current interests include human sensing and medical Internet of Things (IoT) technologies. He received the 2018 Best Magazine Paper Award of the IEICE Communications Society. He is a member of IEEE and IEICE.

↑ TOP