Personal Heart Modeling Using Mobile Sensing

1. Personalized biomedical modeling

Even in the fields of medicine and health, the traditional limitations of physical location, time, and distance are giving way to continuous advances in sensing, communications, artificial intelligence (AI), and other information-processing technologies. An increasing number of people use wearable devices, such as smartwatches, for their personal health management. Further utilization of biometric information in daily life will enable continuous healthcare, useful not only when ill but also before and after treatment, and personalized healthcare, not just patterned treatments for diseases. Research and development and practical applications in this field is very active worldwide.

As part of NTT’s Medical and Health Vision, a simulator called the Bio-Digital Twin is being developed to support risk control and improved wellness [1] by predicting future physical and mental conditions of a person. The Bio-Digital Twin is a computational model that captures a wide variety of biomedical information, from molecular and cellular scales to organs and their interconnections, even the environment in which a person lives, expressing those connections as a network. One of the components of the Bio-Digital Twin is personal heart modeling. Personal heart modeling means measuring the operation and condition of an individual’s heart then mathematically and logically representing the person’s heart in a computer in such a way that allows simulation under various assumptions.

Our research team aims to obtain more detailed biometric information than with conventional wearable devices such as smartwatches, while still enabling measurement in daily life without any undue burden [2].

2. New sensing

One of our prototype biometric sensing instruments is a wearable electrocardiogram (ECG) device. Electrocardiography, as commonly used in medical institutions, uses 10 electrodes placed on the arms, legs, and chest to obtain 12 potential differences. Smartwatches that obtain a simple ECG measuring the potential difference just between a person’s two hands are becoming popular. In contrast, our prototype is centered on the apical region of the chest where the heart is closest to the rib cage as a reference point with opposite electrode poles in three nearly orthogonal directions to capture the electrical activity of the heart in three dimensions (Fig. 1(a)). In our prototype, the electrodes and wiring are integrated with a stretchable shoulder belt, making it easy to fit to any sized adult by adjusting the shoulder and waist straps [3].

Another prototype biometric sensing instrument is what we call the “AI Telestethoscope,” which is capable of measuring multi-channel acoustic signals simultaneously with an ECG. We prototyped both a wearable-type (Fig. 1(b)) and handheld-type (Fig. 1(c)) AI Telestethoscope. They both enable sensed data to be transmitted in real time to a remote location where a doctor or nurse can listen to the sound coming from specific locations on the body surface by interactively selecting them on a terminal screen. Unlike conventional wireless electronic stethoscopes that use a single microphone, our AI Telestethoscopes use multiple microphones to simultaneously capture sound signals at multiple locations to capture the sounds of the heart’s activity in three dimensions.

Fig. 1. Wearable ECG device (a) and AI Telestethoscopes (wearable type (b), handheld type (c)). Prototypes for research purpose.

3. Information processing of signals to explore inside a living body

Information processing is key to estimating the internal state and functioning of a living body from biomedical signals measured on the surface of the body. The novel sensing instruments introduced in the previous section are designed to capture precise biometric information for personal heart modeling.

The heart works in accordance with the cyclical activity of heart muscle cells called cardiomyocytes. There are two types of cardiomyocyte activity: electrical and mechanical. An ECG generally shows the combined effects of the actions of many cardiomyocyte cells in the heart that can be observed as electrical potential differences on the body surface. On the other hand, heart sounds are mainly caused by the mechanical activity of the heart, especially the opening and closing of the four valves inside the heart. Abnormalities in the heart can also generate distinctive sounds, such as the vibrations caused by turbulent blood flow which can be heard as a so-called “heart murmur.”

Both the bio-electrical and bio-acoustic signals that can be observed on the surface are the result of a mixture of many factors. It is not easy to estimate the cause backwards from the effect. That is, to understand the condition of the heart inside the body by tracing its physical mechanism in reverse from surface observations.

We are approaching this seemingly ill-posed problem by using three-dimensional (3D) electrical and acoustical information to derive 3D location clues, and by introducing various information-processing techniques such as the use of physical constraints and machine learning [4].

When testing for heart abnormalities, it is not uncommon for a typical ECG to show nothing more than irregularity in a few waveforms. To solve this problem, we attempt to visualize, from the ECG waveforms, the electrical action potential of localized groups of cardiomyocytes by estimating statistical parameters related to changes in cardiomyocyte populations (tensor ECG) [5]. Visualization of action potential information at various locations in the heart is expected to help with early detection of arrhythmias associated with heart failure, ischemic heart disease, and sudden cardiac death because abnormalities can be more clearly expressed than with standard ECGs.

We also expect to be able to estimate what sounds are coming from which locations in the heart by capturing heart sounds in three dimensions. The location of the sound, its tone, and its acoustic characteristics are known to be important clues for determining the presence and severity of heart disease. Historically, sounds have been captured on the surface of the body using a manual stethoscope or phonocardiograph, and there has been no better way to listen to particular sounds within the body in a non-invasive manner.

To this end, we developed a novel oscillator decomposition technique for heart sounds that can estimate internal sound sources from sounds observed on the body surface, called PCUSD (Physically Constrained Unsupervised Signal Decomposition) [6]. PCUSD focuses on the periodicity of heart movement and the mechanism of heart-sound generation. Normally, the heart moves periodically, going through four stages (cyclical phases) in sequence: S1, systolic, S2, and diastolic, where S1 corresponds to the first sound of a heartbeat and S2 corresponds to the second sound. Different heart valves open and close during each stage, and their vibrations generate heart sounds. We can assume there are multiple vibrational components to the sounds made by each valve and that their amplitudes change in accordance with the condition of the heart during the sequential phases of the cardiac cycle. This led us to construct a probabilistic generation model of the mechanisms of heart sounds based on physical modeling of the heart valves.

Figure 2 shows an example of applying PCUSD to a single channel of heart sound. We examined the estimation accuracy of the S1 and S2 intervals. The figure shows a case of mitral regurgitation with associated heart murmur. The presence of a heart murmur (in this case, between S1 and S2) generally makes it even more difficult to estimate the heart’s condition. The right panel shows the results from a conventional method [7] that does not use a generation model. It shows that it is sometimes prone to estimation errors. In contrast, the left panel shows our PCUSD’s results, indicating improved estimation accuracy for the same S1 and S2 intervals. Table 1 shows this improvement numerically.

Fig. 2. Estimation of S1 and S2 segments.

Table 1. S1 and S2 segment estimation accuracy of PCUSD and baseline method [7]. F1 score is an accuracy measure, the larger the better.

It is straightforward to apply PCUSD to multi-channel heart sounds. Figure 3 shows four acoustic channels (Fig. 3(a)) observed on the body surface of a patient with aortic stenosis and eight internal sound source waveforms (Fig. 3(b)) estimated with PCUSD from the four acoustic channels. Because PCUSD uses a physical model of valve vibration, all (or almost all) these components are assumed to originate from the four valves of the heart. Each of these components can be heard as sound. AI analysis of each waveform is expected to make it possible to reliably diagnose the condition of a malfunctioning heart valve and its progressive deterioration from heart sounds—acoustic biomedical information—captured on the body surface.

Fig. 3. Sounds captured on the body surface (a) and estimated in-body vibration waveforms (b).

4. Toward utilization

The new sensing and estimation technologies described above can potentially provide a more detailed understanding of heart activity than standard ECG and electronic stethoscope technologies currently available. Some heart abnormalities are more likely to cause electrocardiographic abnormalities and be detected using an ECG. Others are more likely to cause heart sound abnormalities and be detected through auscultation. There is yet another set of abnormalities detectable by comparison of the heart’s output of electrical and acoustic signals.

We aim to actualize our vision of personal heart modeling by integrating our new technologies to estimate the condition of the heart based on real-time biomedical data capture and information processing with other relevant patient information such as their electronic health records, family medical history, recent blood tests, pre-existing medical imaging, and current public health status. This approach can potentially help people adopt a healthier lifestyle more appropriate for their personal risks of heart disease and lead to earlier detection of any heart abnormalities if they occur.

When discussing the usefulness of new technologies for medical and healthcare applications, solid research and validation is of paramount importance. Therefore, we are collaborating with multiple medical institutions, research institutions, and specialized hospitals to assess the potential of our technologies to meaningfully improve early detection of certain types of heart diseases, provide useful support for people suffering from heart failure, serve as a tool for patients and doctors during cardiac rehabilitation. Together with our collaborators, we will continue to actively verify the feasibility of this ongoing research.

References

[1]	Press release issued by NTT, “Medical and Health Vision: Realization of a Bio Digital Twin—NTT Corporation Working on the ‘Future of Medicine’ to Predict Diseases, and Mental and Physical Conditions, Enabling People to Have Health and Hope for the Future,” Nov. 17, 2020. https://group.ntt/en/newsrelease/2020/11/17/201117c.html
[2]	M. Nakano, R. Shibue, K. Kashino, S. Tsukada, and T. Ogasawara, “New Technology for Measurement and Analysis of Biological Sounds and Electrocardiographic Signals—Toward Early Detection of Heart Disease and Rehabilitation by Using Personal Heart Modeling,” NTT Technical Review, Vol. 19, No. 7, pp. 40–45, 2021. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr202107fa5.html
[3]	Y. Tsukada, M. Tokita, H. Murata, Y. Hirasawa, K. Yodogawa, Y. Iwasaki, K. Asai, W. Shimizu, N. Kasai, H. Nakashima, and S. Tsukada, “Validation of Wearable Textile Electrodes for ECG Monitoring,” Heart Vessels, Vol. 34, No. 7, pp. 1203–1211, 2019.
[4]	M. Nakano, R. Shibue, K. Kashino, S. Tsukada, and H. Tomoike, “Gaussian Process with Physical Laws for 3D Cardiac Modeling,” Proc. of the 28th European Signal Processing Conference (EUSIPCO 2020), pp. 1452–1456, Amsterdam, Netherlands, Jan. 2020.
[5]	S. Tsukada, “Wearable Textile Electrodes for Long-term Vector ECG Monitoring ‘Tensor Cardiography,’” Proc. of the 14th International Symposium on Medical ICT (ISMICT 2020), Nara, Japan, May 2020.
[6]	R. Shibue, M. Nakano, T. Iwata, K. Kashino, and H. Tomoike, “Unsupervised Heart Sound Decomposition and State Estimation with Generative Oscillation Models,” Proc. of the 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC 2021), pp. 5481–5487, Virtual, Nov. 2021.
[7]	C. D. Papadaniil and L. J. Hadjileontiadis, “Efficient Heart Sound Segmentation and Extraction Using Ensemble Empirical Mode Decomposition and Kurtosis Features,” IEEE J. Biomed. Health Inform., Vol. 18, No. 4, pp. 1138–1152, 2014.