Conversational Emotion Climate Recognition Using Higher-Order Spectra, Emotion Dynamics and Deep Learning

Abstract

This research embarked on a journey to explore the potential of bicoherence—a technique traditionally utilized for physiological signals like EEG and ECG—in the realm of emotion recognition from conversational audio signals. At the heart of the study lay pivotal questions: Can bicoherence unravel unique emotional nuances in audio signals that conventional spectral features might miss? And, when combined with Convolutional Neural Networks (CNNs), how does this feature set fare against traditional emotion recognition methodologies?

Our findings indicate significant promise. Bicoherence-derived features, when paired with CNNs, achieved a notable AUC of 0.8512 for arousal prediction on the IEMOCAP dataset. While some datasets like SEWA Hungarian posed generalization challenges, others like the SEWA Serbian revealed exceptional potential, boasting an AUC as high as 0.9990, Notably, for valence, models trained on SEWA datasets exhibited commendable AUCs, reaching up to 0.9789.

But the research wasn’t without challenges. There were generalization issues when models trained on one dataset were tested on another, emphasizing the importance of diverse and comprehensive data. Furthermore, the study opened doors to deeper inquiries: Which specific CNN architectures can best leverage bicoherence-based features? How do these features stack up on various datasets, and are there specific contexts where they shine?

In the pursuit of a holistic and nuanced emotion recognition system, this study underscores the potential of bicoherence, while also highlighting areas ripe for future exploration. As the field marches ahead, the insights gleaned from this research pave the way for innovations that can revolutionize mental health diagnostics, interventions, and beyond.

Date of Award	18 Dec 2023
Original language	American English
Supervisor	Hadjileontiadis (Supervisor)