Multimodal-Based Stream Integrated Neural Networks for Pain Assessment
- 1 December 2021
- journal article
- research article
- Published by Institute of Electronics, Information and Communications Engineers (IEICE) in IEICE Transactions on Information and Systems
- Vol. E104.D (12), 2184-2194
- https://doi.org/10.1587/transinf.2021edp7065
Abstract
Pain is an essential physiological phenomenon of human beings. Accurate assessment of pain is important to develop proper treatment. Although self-report method is the gold standard in pain assessment, it is not applicable to individuals with communicative impairment. Non-verbal pain indicators such as pain related facial expressions and changes in physiological parameters could provide valuable insights for pain assessment. In this paper, we propose a multimodal-based Stream Integrated Neural Network with Different Frame Rates (SINN) that combines facial expression and biomedical signals for automatic pain assessment. The main contributions of this research are threefold. (1) There are four-stream inputs of the SINN for facial expression feature extraction. The variant facial features are integrated with biomedical features, and the joint features are utilized for pain assessment. (2) The dynamic facial features are learned in both implicit and explicit manners to better represent the facial changes that occur during pain experience. (3) Multiple modalities are utilized to identify various pain states, including facial expression and biomedical signals. The experiments are conducted on publicly available pain datasets, and the performance is compared with several deep learning models. The experimental results illustrate the superiority of the proposed model, and it achieves the highest accuracy of 68.2%, which is up to 5% higher than the basic deep learning models on pain assessment with binary classification.Keywords
This publication has 44 references indexed in Scilit:
- Pain communication through body posture: The development and validation of a stimulus setPain, 2014
- Continuous Pain Intensity Estimation from Facial ExpressionsLecture Notes in Computer Science, 2012
- A local approach based on a Local Binary Patterns variant texture descriptor for classifying pain statesExpert Systems with Applications, 2010
- The painful face – Pain expression recognition using active appearance modelsImage and Vision Computing, 2009
- Classification accuracy comparison: Hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiorityRemote Sensing of Environment, 2009
- The structure, reliability and validity of pain expression: Evidence from patients with shoulder painPain, 2008
- Research on Recognition for Facial Expression of Pain in NeonatesActa Optica Sinica, 2008
- Segregation of Form, Color, Movement, and Depth: Anatomy, Physiology, and PerceptionScience, 1988
- Acute pain response in infants: a multidimensional descriptionPain, 1986
- Generalized procrustes analysisPsychometrika, 1975