Heart Rate Measurement with Video Camera Based on Visible Light Communication

1. Introduction

Recent advances in light emitting diodes (LEDs) have opened up numerous possibilities for new visible light communication (VLC) technologies and systems [1]. VLC systems typically use LEDs and photodiodes. However, much progress has also been made recently in image sensor technology. Consequently, optical camera communication (OCC), a VLC system that uses LEDs and image sensors, has been attracting a lot of attention. A typical OCC system is composed of a single camera with multiple sensors distributed in sight of the camera [2]. OCC is advantageous in that it can obtain data from multiple points together with the sensor positions.

From another point of view, as the population of elderly citizens increases, healthcare monitoring is becoming an important and promising application. For example, suppose a situation where a member of a gym is wearing a heart rate (HR) sensor that transmits his or her heart beat (HB) in real time (Fig. 1(a)). The gym staff could provide immediate aid to the member if they observed danger signs in the member’s HR [3]. This technology will also be beneficial for small children and infants. One potential application is monitoring the HRs of sleeping infants in child care centers (Fig. 1(b)). Monitoring their HRs would enable a caregiver to quickly deal with abrupt HR changes such as those associated with sudden infant death syndrome [4].

Fig. 1. (a) In this scenario, a member of a gym is wearing a heart rate (HR) sensor. The gym staff can advise the member by monitoring the HR or provide immediate aid if danger signs are observed. (b) In this scenario, the OCC system is used to support child care center caregivers in monitoring babies who are napping. Information can be easily superimposed on the captured image by using OCC.

However, with the conventional radio frequency wireless network, this task is not as easy as one may think. The network may not work well if there are a large number of devices [5] and positional information is difficult to obtain.

Therefore, we have developed an OCC system [6] and an event timing encoding (ET encoding) technique [7] that can precisely transfer the intervals of HB events with a conventional camera. The high spatial directionality of visible light enables multiple data channels to be transmitted simultaneously through different optical paths along with the positional information of sensors. In addition, OCC can potentially be combined with natural image analysis in the future.

2. Method to transfer precise HB timing

As explained in the introduction, the system is composed of a sensor unit and camera (image sensor). The capturing speed of a commonly used camera is usually limited to 30 or 60 frames per second, which is not high enough to observe the event timing of the HB. Furthermore, the system only detects the ‘on/off’ state of the LED for simplicity and robust processing. If someone simply flashes the LED at a sensor unit when an HB event occurs, the captured images on the camera side give us only 33 ms of time resolution. However, the requirement is less than 4-ms resolution even for simple HR analysis.

ET encoding is designed to overcome this problem. The idea is that it first signalizes the occurrence of the HB event and then refines the relative timing with consecutive LED patterns. In practice, the unit simulates the frame rate of the camera and switches the LED ‘on’ and ‘off’ using this clock (LED clock). When the unit detects an HB event, it marks the timing location of the event within this LED clock frame. The unit first sends an ‘on’ signal after the detection to inform the decoder of the LED clock frame. When the camera detects this first ‘on’ signal, we can determine on the camera side that the event occurred somewhere in the previous frame shot. The unit continuously sends details about the location of the event in the LED clock frame. From these data, we can refine the timing of the event. In other words, ET encoding tells the LED clock frame that the event is occurring and sends the natural binary expression of the event timing in the frame. The flow chart for sending the event timing by flashing an LED is shown in Fig. 2. The encoding image is shown in Fig. 3.

Fig. 2. Flow chart for sending the event timing by flashing an LED.

Fig. 3. Example of LED flashing patterns. The LED is switched on immediately as the first bit of the packet after the event detection. The precise event timing is marked on the LED clock frame (see magnified view). The LED pattern from the second bit is determined according to this marked location.

The time resolution of the HB interval in ET encoding is calculated as

where T_min is the shortest term of the HB interval and F_z is the capturing speed of the camera (which is the same as the flashing frequency of the LED). This ET encoding is more efficient than directly transferring the HB time interval under the condition of

where T_max is the longest term of the HB interval.

3. Demonstration and comparison

The ET encoding was implemented in our HR monitoring system. The unit was custom-built and can be attached to “hitoe” sensing fabric developed by Toray Industries, Inc. and NTT. It samples an electrocardiogram signal with 500 Hz and 10 bits of resolution. A microcomputer installed in the unit searches for the HB timing and utilizes 5 bits (five on/off flashes of an LED) to send one HB interval.

A comparison between the HR observed from direct monitoring and the HR decoded from the LED flashing patterns captured by the camera is shown in Fig. 4. HB intervals were calculated for HRs obtained by using four different methods: an HR extracted from the original waveform; an HR transferred by the interval encoding method; an HR reconstructed using an event signal (the same as using only the first signal in ET encoding); and an HR decoded from ET encoding. When 5 bits of data were transferred from the unit, ET encoding gave an almost identical HR.

Fig. 4. (a) Waveform of electrocardiogram. (b) Results for HRs, extracted from original waveform; HRs reconstructed using event signal (same as using only first signal in ET encoding); and HRs decoded from ET encoding at 5 bit/packet. From the data, we can trace the activity of participants who stood up from a seated posture. (c) Error between HRs observed from direct monitoring and the HR decoded from different methods. An almost identical HR was obtained with ET encoding.

4. Future development

This article explained efficient encoding methods that can be used in OCC with a low-frame-rate image sensor. We introduced ET encoding, which sends the rough timing of the HB in the first signal and refines its location. Theoretically, ET encoding can send the HR with better resolution than directly sending the interval data in many cases.

We implemented ET encoding in a custom-made electrocardiogram sensor. The results showed that ET encoding transferred HRs with higher resolution when the same bit/beat was used. Concerns that the visible light might be irritating to the human eye can be allayed by using a near-infrared LED.

This system will be implemented in the future to obtain HR data and will be particularly useful for evaluating health risks. Since the system utilizes a common camera, it can be combined with natural image analysis. In addition, data transferrable by ET encoding are not limited to HR. Other data such as spikes in neuronal activity can also be transferred by ET encoding. Further research will open up new possibilities of OCC.

References

[1]	M. Khalighi and M. Uysal, “Survey on Free Space Optical Communication: A Communication Theory Perspective,” IEEE Commun. Surv. Tutor., Vol. 16, No. 4, pp. 2231–2258, 2014.
[2]	M. Kurihara, Y. Honji, J. Fujimori, Y. Oikawa, and Y. Yamasaki, “Examination of Sound Field Visualization-method Using Small Visualization Devices Built with MEMS-Mic and LED,” Proc. of the 2nd Technical Committee Meeting on Acoustic Imaging, Acoustical Society of Japan, 2009 (in Japanese).
[3]	X. Jouven, J. P. Empana, P. J. Schwartz, M. Desnos, D. Courbon, and P. Ducimetière, “Heart-rate Profile during Exercise as a Predictor of Sudden Death,” N. Eng. J. Med., Vol. 352, No. 19, pp. 1951–1958, 2005.
[4]	S. M. Pincus, T. R. Cummins, and G. G. Haddad, “Heart Rate Control in Normal and Aborted-SIDS Infants,” American J. Physiol., Vol. 264, No. 3, R638-R646, 1993.
[5]	L. Chevalier, S. Sahuguede, and A. Julien-Vergonjanne, “Optical Wireless Links as an Alternative to Radio-frequency for Medical Body Area Networks,” IEEE J. Sel. Area. Commun., Vol. 33, No. 9, pp. 2002–2010, 2015.
[6]	G. Pablo Nava, H. D. Nguyen, Y. Kamamoto, T. G. Sato, Y. Shiraki, N. Harada, and T. Moriya, “A High Speed Camera-based Approach to Massive Sound Sensing with Optical Wireless Acoustic Sensors,” IEEE Trans. Computational Imaging, Vol. 1, No. 2, pp. 126–139, 2015.
[7]	T. G. Sato, Y. Shiraki, and T. Moriya, “Heart Rate Measurement Based on Event Timing Coding Observed by Video Camera,” IEICE Trans. Commun., Vol. E100.B, No. 6, pp. 926–931, 2017.