Application of Frequency Features of Optical Flow for Event Detection in Video-EEG Monitoring Data

. The work is devoted to the study of the frequency features of the optical flow obtained from the video record of long-term video-electroencephalographic (video-EEG) monitoring data of patients with epilepsy. It is necessary to obtain features to recognize epileptic seizures and differentiate them from non-epileptic events. We propose to analyze the periodograms of the smoothed optical flow computed from the fragments of the patient ’ s video recordings. We use Welch's method to obtain periodograms. The values of the power spectral density of the optical flow at the selected frequencies are used as features. Using the clustering algorithm, seven groups of events are identified in video recordings and combined into three generalized classes. We train SVM classifier and conduct recognition of events in a test sample of 103 video fragments in four patients. The experiment indicates the accuracy of event classification equal to 90.3%. © 2021 Journal of Biomedical Photonics & Engineering.


Introduction
In clinical practice, video-electroencephalographic monitoring, a method of long-term synchronous recording of an electroencephalogram (EEG) and video image, has become widely used. Simultaneous video recording of the patient's clinical state and the bioelectrical activity of the brain makes it possible to reliably diagnose epileptic seizures and differentiate them from non-epileptic events [1,2]. For recording and visual analysis of video-EEG data, physicians use specialized software (for example, Galileo NT Line package), which has a set of functions for signal processing and analysis, as well as statistical data analysis. If diagnostically important fragments of the EEG are found, the physician needs to revise the area of interest in the video recording for visual assessment and differentiation of an epileptic and artifact event. Visual analysis of video data is extremely laborious [21]; therefore, it becomes necessary to develop methods for automatic registration of epileptic seizures from video sequences obtained during video-EEG monitoring.
Some works on the analysis of video recordings for the detection and recognition of epileptic events are known. The work [3] presents methods for measuring the motion strength and motor activity of newborns using video recording. The quantitative characteristics obtained in the form of signals are used to differentiate myoclonic and clonic seizures and to distinguish seizures from the normal behavior of the newborn. The motion strength is defined as the area of the moving parts of the infant's body. To outline such fragments, the wavelet transform of frames, median filtering, and segmentation operations using an adaptive version of the k-means algorithm are used. The change in time of the coordinates of characteristic points, selected automatically on the child's limbs and tracked using the KLT (Kanade-Lucas-Tomasi) algorithm, generates signals that characterize motor activity. Modifications of methods for measuring motion strength and motor activity are presented in the works [4,5].
In the works [6,7], the authors solved the problem of real-time detection of clonic seizures in newborns by the sequence of images obtained from video cameras. The value of the filtered average optical flow calculated as the sum of the binarized pixel-by-pixel difference in luminance of adjacent frames of the video sequence is analyzed. In Ref. [6], a feature of a seizure is the periodicity of the optical flow, which is detected using a hybrid autocorrelation-YIN estimation technique. In Ref. [7], the Maximum Likelihood criterion is used to determine the periodicity of the optical flow.
The paper [8] presents an algorithm for recognizing convulsive seizures in real-time from a sequence of frames recorded by a video camera. To analyze video images and detect events, the method described in Ref. [9] is used. The method consists in computing the components of the optical flow associated with group transformations of objects in frames (translation, rotation, dilatation, shear) and band-pass temporal filtering to identify an occurrence of clonic movements.
In the works [10,11], the authors analyze video sequences recorded by multiple cameras to detect nocturnal motor seizures and central apneas occurring in the aftermath of epileptic seizures. The characteristics of the sigmoid patterns of the time-frequency spectrum (modulation maximum amplitude and total spectral power modulation at the time of the event) in the range of 0.1-1 Hz of translation, dilatation, and shear rates are used as features of a diagnostic event. These velocities are calculated by the method described in Ref. [9] from the optical flow generated by the patient's movements.
Previously, the authors of studies [12,13] proposed to detect events in a video recording by the magnitude of the optical flow, which characterizes the degree of mobility of the frame area in which the patient is located. The algorithm was designed to detect both convulsive and non-convulsive seizures. The tests showed that the detected events quite accurately coincided with the events detected during the analysis of the EEG wavelet spectrograms. In Ref. [14], an algorithm for the synchronous analysis of the EEG signal and video recording for the detection and differentiation of diagnostic and artifact events was proposed. The algorithm combines a threshold detector of brain activity using the ridges of wavelet spectrograms [15] and a threshold detector of events in terms of optical flow [12,13]. The results of the analysis of clinical data recorded on EBNeuro equipment and Galileo NT Line package software showed the fundamental possibility of reliable distinguishing artifact events from epileptic seizures [14]. However, for more reliable recognition of epileptic seizures, additional features are needed.
The study of publications has shown that frequency analysis of signals obtained from an optical flow is widely used to extract features of epileptic seizures. This work aims to study the frequency features of the optical flow of video recording for the recognition of diagnostic and artifact events during the synchronous analysis of video-EEG data. Unlike the well-known works, the proposed technique is not restricted to a specific seizure type, and this study will consider a wider range of events recorded on video. We propose to analyze the periodograms of the smoothed optical flow, computed from the fragments of the video recording of the patients. Since this work does not consider the multidirectional components of the optical flow, the classical method of spectral analysis of signals will be used to obtain periodograms. As features, we will use the values of the power spectral density (PSD) at the selected frequencies.

Materials and Methods
As mentioned above, this work aims to obtain and study the features for recognizing epileptic seizures in the data of long-term video-EEG monitoring and their differentiation with non-epileptic events. In this study, we used video recordings from long-term video EEG monitoring data of seven adult patients with an active level of wakefulness. The recordings were obtained with an HD camera using the Galileo NT Line package. The camera is fixed to the ceiling of the hospital room. The duration of the recordings is from 5 to 24 h or more. Each video consists of files in AVI format containing a fragment of three minutes in length.
We will recognize events based on the analysis of the patient's movement recorded by a video camera. To measure the intensity of the patient's movement, we use the value of the smoothed total optical flow calculated in the region of interest in each frame of the video sequence. The optical flow is computed by the Lucas-Kanade method. To smooth the activity measure, we use a discrete version of the Kalman-Bucy filtering algorithm.
We form the feature descriptions of events recorded on video based on the frequency analysis of the smoothed measure of patient's activity. To compute periodograms, we use Welch's method with a Hamming window of three sizes. We propose to use the values of the power spectral density of a measure of the patient's mobility at 14 selected frequencies as the features of events. To analyze the obtained feature descriptions of events and study the possibility of using pattern recognition algorithms for detecting events, we use a clustering algorithm with automatic choosing the optimal number of clusters. We use the Support Vector Machine (SVM) classifier to recognize events in video recordings. The classifier was trained on 78 three-minute video fragments of three adult patients with an active level of wakefulness. For testing, we used 103 video fragments of the other four adult patients.
In the following sections of the article, we describe in detail the application of the listed above methods and data in our research. common approach is based on the analysis of optical flow. In works [13,14] we proposed to detect diagnostic events using the measure, characterizing the degree of activity of the region of interest. The region of interest is the part of the frame where the patient is located. A measure of the activity of the region of interest is the total value of the optical flow calculated for each frame of the video sequence:  (1), we applied the Lucas-Kanade algorithm [16]. Two examples of seizure and food intake events from video-EEG data of two patients and computed optical flow vector fields are shown in Fig. 1  As far as the noise component is present in model (1), we use the smoothed value of the activity measure ! . (#) to detect events. The smoothed ! . (#) value is obtained using a discrete version of the Kalman-Bucy filtering algorithm [17]. We apply the Kalman-Bucy algorithm because it provides the optimal estimate in the sense of minimum error variance. The graphs of the values of measure of activity !(#) and smoothed measure of activity ! . (#) obtained from a real video recording are shown in Fig. 2. In this video from 126 to 180 sec, an epileptic seizure is captured. The decision to fix a diagnostic event is made according to the threshold rule. To increase the reliability of event detection, it is necessary to take into account not only the amplitude but also the frequency characteristics of the measure of activity. In the next subsection, we will present a technique for extracting the frequency features of the smoothed measure of patient's activity ! . (#).

Extracting frequency features of events
To construct periodograms, we use the Welch method with a Hamming window with 50% overlap [18].
where + (+) (<) is the discrete-time transformation of the segment @; B is the energy of the window C.
Using formulas (2) and (3), we obtained periodograms of different frequency resolutions at three window sizes: 10, 200, and 600 samples. Fig. 3 shows the examples of periodograms computed at window size equal to 200 from video records of different events in one of the patients. We studied 78 fragments of video recordings of three patients that fixed the following events: epileptic seizure, intense movement in the frame, sleep, rest, smooth movement, food intake. Fig. 3 shows that different events take different values of the power spectral density at a particular frequency value. To study the structure of the data that contain the obtained spectral characteristics of the optical flow, we use the technique of cluster analysis. The application of the clustering algorithm to distinguish groups of events by levels of power spectral density of periodograms obtained from video recordings of different events is described in the next subsection.

Partitioning frequency feature space
To study the possibility of using classifiers for detecting events, we applied a clustering algorithm. The algorithm is based on the search for locally optimal data partitions with automatic selection of the optimal number of clusters [19]. For optimization, the criterion for the minimum of the sum of intraclass variances is used: where F 4 9 is the center of group G 4 , to which the object F 8 is assigned; E(F 8 , F 4 9 ) is the distance from object F 8 ∈ G 4 to the center of group F 4 9 ; ! 4 5 is the intraclass variance. Here, F 8 ∈ I < , F 4 9 ∈ I < , where N is the dimension of the feature space. The optimal number of clusters J in the range K − 1 ≤ J ≤ L + 1 is determined from the condition for the maximum of the functional  In the study, we used a set of feature descriptions of 78 fragments of video recordings of events taken from the data of 24-h video-EEG monitoring of three adult patients with an active level of wakefulness. The feature space is formed from the PSD values of the periodograms of the optical flow obtained with three window sizes (see Section 3). The PSD values were taken at 14 frequencies selected in the range from 0.5 to 8.8 Hz. Thus, the dimension of the composited feature vectors is equal to 42. The optimal number of clusters was chosen in the range from 2 to 8.
The results given in Table 1 show that the clusters of feature descriptions of several types of events are wellseparated in the feature space, and the centers of several clusters are located relatively close. For example, the centers of the clusters "Seizure" and "Sleep", "Smooth movement" and "Movement" are significantly distant, while the centers of the clusters "Smooth movement" and "Rest after seizure", "Eating" and "Rest after seizure", as well as "Seizure" and "Movement" are located relatively close, since the nature of the patient's movements in these groups of events may be similar. Therefore, we combined closely spaced clusters and formed the following three generalized classes. The first class, "Seizure/Movement", includes epileptic seizures and episodes of intense movement in the region of interest (clusters of events "Seizure" and "Movement"). The second class, "Sleep", includes the patient's sleep and state of rest (clusters of events "Sleep" and "Sleep + movement"). The third class, "Smooth movement", includes events associated with the movements of low intensity. This class combines episodes of eating, smooth posture changes, and working with a smartphone or laptop ("Eating", "Smooth movement", and "Rest after seizure" event clusters). Thus, the frequency features extracted from the video recordings of long-term monitoring of patients made it possible to distinguish three types of patterns that characterize generalized classes of events.
In the next section, to confirm the possibility of detecting events using frequency features, we will train the SVM classifier and recognize events from the test set of fragments from long-term video recordings of patients with epilepsy.

Results
We used the SVM algorithm [20] to classify events. A set of 78 feature descriptions of video recordings of events, described in Section 4, was used to train the classifier. Events are grouped into three generalized classes. In the SVM algorithm, we used a potential function in the form of a dot product. The leave-one-out cross-validation procedure was used for training. The results of training and quality measure values are given in Table 2.
Feature descriptions of 103 three-min fragments of video recordings of long-term monitoring in four patients formed the test sample. The analyzed video fragments captured events given in Table 1 and their combinations, including seven seizures and five events of food intake. It should be noted that one event could be recorded in several consecutive three-minute video fragments. Table 3 presents the results of testing and values of recognition quality measures.

Discussion
The relatively low percentage of correctly recognized events from the "Seizure/Movement" class can be explained in the following way. The camera recorded most of the epileptic seizures from the test sample on several consecutive 3-min videos. Some records contain only the initial or final seizure stages. The duration of these stages is significantly shorter than three minutes. The size of the window used in calculating the estimates of periodograms by formulas (2), (3) is much less than the number of frames in a three-minute video  Detection of nocturnal motor seizures in children 94% recording, and the spectra of weighted segments (3) can vary significantly. In this case, the frequency pattern of the three-minute fragment, characterized by the estimate (2), may be deformed and assigned to the non-proper class. It should be noted that in the considered test set of video recordings, all epileptic seizures captured in several consecutive three-minute fragments are detected correctly in at least one of the fragments. In general, the percentage of correctly classified events is equal to 90.3% of the total. Table 4 shows the characteristics of event recognition obtained by analyzing video sequences of video-EEG monitoring data by different methods presented in the works that we considered in Section 1. From the data in Tables 3 and 4 and taking into account the above remarks, it follows that the results of event detection (precision, recall, and specificity) obtained in this work correspond to the performance of the algorithms proposed in Refs. [6][7][8][9][10][11].
It is possible to increase the accuracy of event classification by reducing the duration of the analyzed video fragments and augmenting the training sample of events. Differentiation of seizures and artifacts of chewing and movement will be carried out in the synchronous analysis of video and EEG recordings.

Conclusions
We studied the possibility of using the frequency characteristics of the video recordings for analyzing long-term video-electroencephalographic monitoring data. Periodograms of the smoothed optical flow, calculated from the fragments of video recordings, which captured various events, were computed. To obtain periodograms, we used Welch's method with the Hamming window of three sizes. The values of the spectral power density of the optical flow at fourteen selected frequencies are used as features of events. We applied the clustering algorithm with automatic selection of the optimal number of clusters to obtain optimal data partition. The preliminary results of the frequency analysis of the optical flow, computed from the video-EEG monitoring data of patients, were obtained. Seven groups of events were identified in the feature space using the clustering algorithm. These groups were combined into three generalized classes. To confirm the possibility of detecting events by frequency J of Biomedical Photonics & Eng 7 (3) 30 Sep 2021 © J-BPE 030301-7 characteristics of video recordings, we trained an SVM classifier and conducted recognition of events in a test sample of 103 video fragments in four patients. The percentage of correctly classified events is equal to 90.3% of the total. Further research will be aimed at improving the accuracy of detecting events in video recordings of patients by frequency features and developing an algorithm for synchronous analysis of video-EEG monitoring data.

Disclosures
All authors declare that there is no conflict of interests in this paper.