Mapping Subjectivity in Press Discourse: Human vs. Transformer Comprehension of Belgian French Press Articles

Escouflaire, Louis [UCL] Descampe, Antonin [UCL] Fairon, Cédrick [UCL]

In this paper, we compared the linguistic comprehension of subjectivity in French press articles by a fine-tuned transformers model and by human annotators. We found that, even though large language models and annotators rate the same articles as objective or subjective, they appear to be influenced by different tokens in the text to reach the same conclusions. To visualize the differences in explanations made by humans and by the fine-tuned model, we produced “human attention maps” which display the tokens that human readers consider most as linguistic “indicators of subjectivity”. Our dataset consists of 150 excerpts from articles randomly selected from the RTBF Corpus, which contains 750,000 articles published on the website by the Belgian French public service media. Half of the 150 articles were classified by the media as “op-eds” (opinion articles), the other half was tagged as “news” (information articles). Opinion articles are expected to be more subjective than information articles, and thus to contain more linguistic indicators of subjectivity. We fine-tuned CamemBERT, a French-trained version of the transformers model RoBERTa, using a corpus of 5,000 opinion and 5,000 information articles extracted from the RTBF Corpus. The fine-tuned model was then used to classify the 150 articles, and we used a deep learning explanation method called layer-wise relevance propagation to identify which tokens were attributed the most attention in each text by the model. In parallel, over the course of 4 weeks, 36 students in journalism read and annotated 50 articles each. Annotators evaluated the overall subjectivity of each article and highlighted the tokens that they considered as indicators of subjectivity. Each article was reviewed by at least 10 annotators. The results of both experiments were then compared quantitatively and qualitatively using attention maps at token-level, and confronted with theoretical foundations of linguistic subjectivity in press discourse.