Dialogue systems for language learning: a meta-analysis

Bibauw, Serge [UCL] Van den Noortgate, Wim [KULeuven] François, Thomas [UCL] Desmet, Piet [KULeuven]

The present study offers a meta-analysis of effectiveness studies on dialogue-based CALL, i.e., systems allowing a learner to practice a foreign language (L2) by interacting with a conversational agent. Through a systematic inclusion and exclusion process, screening 419 publications, we identified 17 relevant meta-analysable studies. We made use of Morris and DeShon’s (2002) formulas to compute comparable effect sizes across designs, including k = 100 individual effect sizes, which were analysed through a multilevel random-effects model. Results confirm that dialogue-based CALL practice has a significant medium effect size on L2 proficiency development (d = 0.59). We performed extensive moderator analyses to explore the relative effectiveness on several learning outcomes of different types and features of dialogue-based CALL (type of interaction, modality, agent embodiment...). In particular, our study confirms the effectiveness of form-focused and goal-oriented systems, system-guided interactions, corrective feedback provision, and gamification features. Effects for lower proficiency learners, and on vocabulary, morphosyntax, holistic proficiency and accuracy are clearly established. Finally, we discuss further evolutions to be expected in this promising field of dialogue-based CALL and we offer a first insight into opportunities and limitations for language learning.