Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA)
Open Access
- 16 May 2022
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Psychology
- Vol. 13, 894478
- https://doi.org/10.3389/fpsyg.2022.894478
Abstract
Language sample analysis (LSA) is an important practice for providing a culturally sensitive and accurate assessment of a child's language abilities. A child's usage of literate language devices in narrative samples has been shown to be a critical target for evaluation. While automated scoring systems have begun to appear in the field, no such system exists for conducting progress-monitoring on literate language usage within narratives. The current study aimed to develop a hard-coded scoring system called the Literate Language Use in Narrative Assessment (LLUNA), to automatically evaluate six aspects of literate language in non-coded narrative transcripts. LLUNA was designed to individually score six literate language elements (e.g., coordinating and subordinating conjunctions, meta-linguistic and meta-cognitive verbs, adverbs, and elaborated noun phrases). The interrater reliability of LLUNA with an expert scorer, as well as its' reliability compared to certified undergraduate scorers was calculated using a quadratic weighted kappa (Kqw). Results indicated that LLUNA met strong levels of interrater reliability with an expert scorer on all six elements. LLUNA also surpassed the reliability levels of certified, but non-expert scorers on four of the six elements and came close to matching reliability levels on the remaining two. LLUNA shows promise as means for automating the scoring of literate language in LSA and narrative samples for the purpose of assessment and progress-monitoring.This publication has 26 references indexed in Scilit:
- Automatic generation of the index of productive syntax for child language transcriptsBehavior Research Methods, 2013
- Properties of the Narrative Scoring Scheme Using Narrative Retells in Young School-Age ChildrenAmerican Journal of Speech-Language Pathology, 2010
- Comparison of classification accuracy using Cohen’s Weighted KappaExpert Systems with Applications, 2008
- Narrative in adolescent specific language impairment (SLI): a comparison with peers across two different narrative genresInternational Journal of Language & Communication Disorders, 2007
- The Index of Narrative Microstructure: A Clinical Tool for Analyzing School-Age Children’s Narrative PerformancesAmerican Journal of Speech-Language Pathology, 2006
- Spoken language samples of New Zealand children in conversation and narrationAdvances in Speech Language Pathology, 2004
- Literate Language Features in Spoken Narratives of Children With Typical Language and Children With Language ImpairmentsLanguage, Speech, and Hearing Services in Schools, 2001
- Index of Productive SyntaxApplied Psycholinguistics, 1990
- SPECIFIC LANGUAGE IMPAIRMENT AS A MATURATIONAL LAG: EVIDENCE FROM LONGITUDINAL DATA ON LANGUAGE AND MOTOR DEVELOPMENTDevelopmental Medicine and Child Neurology, 1987
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968