Analysis of a news generating tool in different contexts

During last years, fake news generators have done big progress and are now able to product really convincing results. In this work, we evaluate the quality of the news outputted by a state-of-the-art-news generator depending of the input data it is given. The goal is to check if the news are of worst quality when the generator works on topics far from the ones it was trained on, and if this change in quality is detected by both human beings and machine learning tools. To achieve this goal, we start by building our database. The original news come from crawling and the generated ones are obtained by giving some part of the original ones to our news generator. We then set up an experiment plan to collect human evaluations and we build two classifiers to do our automated evaluation. We observe that the news generated on new topics are of worst quality than the other ones, and that it is at least detected by machine learning tools. For the human evaluation, the results tend to be similar but they are less significant and ask for further investigations. We conclude that it shows that a state-of-the-art news generator, Grover, is not really durable in time. We finish by insisting on all the questions it raises, like if it is the case for other generators, when and how much to re-train the generator or what to do to mitigate the problem.