Repeatability of Commonly Used Speech and Language Features for Clinical Applications
Open Access
- 2 December 2020
- journal article
- Published by S. Karger AG in Digital Biomarkers
- Vol. 4 (3), 109-122
- https://doi.org/10.1159/000511671
Abstract
Introduction: Changes in speech have the potential to provide important information on the diagnosis and progression of various neurological diseases. Many researchers have relied on open-source speech features to develop algorithms for measuring speech changes in clinical populations as they are convenient and easy to use. However, the repeatability of open-source features in the context of neurological diseases has not been studied. Methods: We used a longitudinal sample of healthy controls, individuals with amyotrophic lateral sclerosis, and individuals with suspected frontotemporal dementia, and we evaluated the repeatability of acoustic and language features separately on these 3 data sets. Results: Repeatability was evaluated using intraclass correlation (ICC) and the within-subjects coefficient of variation (WSCV). In 3 sets of tasks, the median ICC were between 0.02 and 0.55, and the median WSCV were between 29 and 79%. Conclusion: Our results demonstrate that the repeatability of speech features extracted using open-source tool kits is low. Researchers should exercise caution when developing digital health models with open-source speech features. We provide a detailed summary of feature-by-feature repeatability results (ICC, WSCV, SE of measurement, limits of agreement for WSCV, and minimal detectable change) in the online supplementary material so that researchers may incorporate repeatability information into the models they develop.Keywords
This publication has 24 references indexed in Scilit:
- Real-time robust recognition of speakers' emotions and characteristics on mobile platformsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective ComputingIEEE Transactions on Affective Computing, 2015
- Tutorial on use of intraclass correlation coefficients for assessing intertest reliability and its application in functional near-infrared spectroscopy–based brain imagingJournal of Biomedical Optics, 2015
- Fitting Linear Mixed-Effects Models Usinglme4Journal of Statistical Software, 2015
- Fully automated assessment of the severity of Parkinson's disease from speechComputer Speech & Language, 2013
- Acoustic characteristics of vowel sounds in patients with Parkinson diseaseNeuroRehabilitation, 2013
- The Relationship of Lexical Richness to the Quality of ESL Learners’ Oral NarrativesThe Modern Language Journal, 2012
- Reliability and Validity in NeuropsychologyPublished by Springer Science and Business Media LLC ,2010
- Effects of Bilateral Subthalamic Nucleus Stimulation and Medication on Parkinsonian Speech ImpairmentJournal of Voice, 2007
- Assessing intrarater, interrater and test–retest reliability of continuous measurementsStatistics in Medicine, 2002