The neural network allows to determine heart rate and blood oxygen saturation without using medical equipment or a fitness tracker. The technology simplifies the process of medical diagnostics and makes it completely contactless.
Modern fitness trackers allow you to measure a large number of physiological parameters using photoplethysmography. This method works as follows. The beam of light passing through the wrist is partially reflected from the blood vessels under the skin. By the amount of reflected light per unit of time, you can determine the heart rate, respiratory rate, blood oxygen saturation, and other parameters. Photoplethysmography is an inexpensive alternative to the traditional approach of plethysmography, which requires sophisticated equipment and skilled medical professionals. Recent studies have shown that the plethysmographic signal is strong enough to be picked up by observing changes in skin color in a series of images taken with a conventional camera, including a smartphone camera. Thus, there is a possibility of completely contactless medical diagnostics.
A neural network extracts from a video recording of a face the change in skin color over time at individual pixels. Each pixel is an independent sensor that registers three time dependencies corresponding to the R -, G -, and B-components of the signal. The image inevitably has noise associated with the video camera, light fluctuations, and the movement of the patient’s face relative to the camera. To suppress these noises, spatial averaging over several adjacent pixels is used (see Figure 1).
Then a convolutional neural network (for example, ResNet) based on 3N signals, where N is the number of pixels, forms a single time dependence, equivalent to the light reflection coefficient, recorded by a fitness tracker or a medical sensor. To do this, in particular, the neural network splits the signal from each pixel into two components: a constant in time and a variable. For the calculation of physiological parameters, the variable part of the signal is of interest, since the constant part is related to the illumination of the face. The heart rate, respiratory rate, and SpO2 level (blood oxygen saturation) are extracted from the variable signal using standard methods. The scheme of the algorithm is shown in Fig. 2.
Thus, computer vision methods provide an opportunity to develop new, completely contactless diagnostic methods, bringing the possibilities of remote medical consultations to a new level.