Computer Vision

The majority of studies developed to recognize human emotions are limited to a single modality namely facial expressions or speech. This book introduced a multimodal approach to improve the accuracy of the emotion recognition by combining audio and visual data. Furthermore a CNN model has been proposed to automatically extract facial features that uniquely differentiate facial expressions and this method has been applied to recognize the cognitive states of learners in E-learning environments and the learners'' facial expressions are mapped to cognitive states such as boredom confusion engagement and frustration. The objectives are as follows: - Multimodal feature extraction and fusion from face image and speech: Geometric-based SURF features from face image are considered as are spectral and prosodic features from speech.- To combine the scores obtained from individual models the proposed linear weighted fusion approach was used.- To recognize learners'' cognitive states in e-learning environments a Hybrid CNN model has been proposed.