Facial Expression-Based Customer Sentiment Analysis for Service Quality Improvement using Deep Learning Techniques

Abstract

In today’s customer-centric economy, understanding and responding to customer sentiment is vital for service excellence. Traditional feedback mechanisms, such as surveys and reviews, are often limited by response bias and delayed insights. Hence, this research presents a deep learning approach for improving service delivery through customer sentiment and facial expression analysis. The study leverages specifically MobileNetV2 and InceptionV3, to classify facial expressions into three sentiment classes: Satisfied, not satisfied and Neutral. A local dataset comprising of 900 annotated facial images of locally sourced dataset was curated to address ethnic bias in existing datasets to ensure local relevance while Facial Expression recognition 2013 dataset comprises 48x48 pixel grayscale photos of faces.The methodology involved rigorous data preprocessing, including grayscale normalization, face alignment, and augmentation. MobileNetV2 and InceptionV3 models were trained and evaluated using stratified 80:20 train-test split with categorical cross-entropy as the loss function. Performance was assessed using metrics such as accuracy, precision, recall, F1-score, and confusion matrix. Inception reported an accuracy of 94.01% precision of 0.89 recall of 0.98 and an f1 score of 0.94. MobileNet on the other hand reported an accuracy of 90.12%, precision of 0.89, recall of 0.91 and f1-score of 0.91. This shows that inception model outperformed mobileNet in terms of accuracy, precision, recall and f1-score. The results demonstrate the feasibility of using facial expression recognition for sentiment tracking in service environments such as banks, schools, and retail outlets.