Emotion Recognition from Long-Term Physiological Signals using Wearables

Abstract

Emotion recognition is a research field of high interest due to its clinical and technological value in the advancing world. There remains a shortage of studies in emotion recognition from physiological signal datasets collected from real-life settings using wearable sensors such as smart wristbands, and less studies are performed on the deep learning data modelling techniques explored on these datasets. This research added insights about two different approaches to study emotion recognition on such dataset. The first part of this thesis works on counting models that were trained and fitted on the emotional valence and arousal count labels. The count models are based on regression analysis models such as Poisson, Negative Binomial and Random Forest. Implementing these regression methods showed count prediction rates of 55-70%. This suggests that physiological signals are sensitive to counts of emotion valence and arousal labels, resulting in patterns that can be detected and used to predict these counts. Longer recordings of emotion labels and larger counts would help improve the detection of such patterns using Regression modelling techniques. In the second part of this thesis, deep learning approaches were applied on the valence and arousal labeled Heart Rate (HR) data individually to classify the images into two classes each (binary classification): High (H) and Low (L) labels. A configurable Convolutional Neural Network (CNN) framework was proposed and used for classification, and five types of image encodings were performed on the labelled HR and Galvanic Skin Response (GSR) time signals after preprocessing. Metrics evaluating the performance of the classification of the CNN's were computed and compared to see the effect of varying the image-encoding methods. Combination of the output of CNNs in terms of class probabilities was performed to improve binary classification of emotional valence from image transforms of daily HR signals by training a Support Vector Machine (SVM) classifier on more than one imaging technique. Significant improvement of more than 20 % (up to 97.3% accuracy) was seen with this configuration on the HR signals as opposed to classification based on individual image-encoded input. GSR valence signals showed weak performance in comparison to HR valence signals, attributed to noisier signals that were not well-represented in the image transforms which resulted in reduced GSR samples used to train the CNN models. Arousal labels were also not detected in comparison to valence labels, suggesting mislabeled signals or slower activations of the effects of arousal on the physiological signals. This means that the proposed configurable CNN-SVM scheme is optimal for detecting emotional valence, which is of a high physiological and psychological importance compared to arousal. The proposed HR-based scheme allows for an automated and continuous monitoring of emotional valence as an important risk factor for heart health, depression and anxiety using only heart rate as a stand-alone and reliable predictor from wearable sensors.

Date of Award	Dec 2021
Original language	American English