The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline

Shardlow, Matthew [Manchester Metropolitan University] Alva-Manchego, Fernando [Cardiff University] Batista-Navarro, Riza [University of Manchester] Bott, Stefan [Universitat Pompeu Fabra] Calderon Ramirez, Saul [Tecnológico de Costa Rica] Cardon, Rémi [UCL] François, Thomas [UCL] Hayakawa, Akio [Universitat Pompeu Fabra] Horbach, Andrea [University of Hildesheim] Hülsing, Anna [University of Hildesheim] Ide, Yusuke [Nara Institute of Science and Technology] Marvin Imperial, Joseph [National University Philippines] Nohejl, Adam [Nara Institute of Science and Technology] North, Kai [George Mason University] Occhipinti, Laura [University of Bologna] Peréz Rojas, Nelson [Tecnológico de Costa Rica] Raihan, Nishat [George Mason University] Ranasinghe, Tharindu [Aston University] Solis Salazar, Martin [Tecnológico de Costa Rica] Štajner, Sanja [Karlsruhe] Zampieri, Marcos [George Mason University] Saggion, Horacio [Universitat Pompeu Fabra] et al.[show all]

We report the findings of the 2024 Multilingual Lexical Simplification Pipeline shared task. We released a new dataset comprising 5,927 instances of lexical complexity prediction and lexical simplification on common contexts across 10 languages, split into trial (300) and test (5,627). 10 teams participated across 2 tracks and 10 languages with 233 runs evaluated across all systems. Five teams participated in all languages for the lexical complexity prediction task and 4 teams participated in all languages for the lexical simplification task. Teams employed a range of strategies, making use of open and closed source large language models for lexical simplification, as well as feature-based approaches for lexical complexity prediction. The highest scoring team on the combined multilingual data was able to obtain a Pearson’s correlation of 0.6241 and an ACC@1@Top1 of 0.3772, both demonstrating that there is still room for improvement on two difficult sub-tasks of the lexical simplification pipeline.