Lie-o-matic: using natural language processing to detect contradictory statements - Beatriz Souto de Sá Baldaia (2020)
The Information age brought the digitization of data, and, consequently, a faster and wider flow and production of information. People, such as journalists, struggle to cope with the increasing data disclosure and to monitor and verify the spread information, that might be corrupted (containing lies, inconsistencies, contradictions, etc.). Considering this current problem and the constant evolution in Natural Language Processing (NLP) techniques and machine learning, we are interested in taking advantage of those recent developments to tackle the specific NLP task of detecting contradictions in text. This dissertation investigates the effect of various datasets, from different domains and tasks (like contradictions in a different context or arguments of support and attack), on the learning performance of a supervised learning classification model for detecting contradictions. Hence, we address the problem as a binary classification task, fine-tuning a sentence-pair classification task, built on top of a pre-trained BERT model, to later run prediction of if two texts are contradictory or not. Literature on contradiction detection has focused almost on separating antonyms and contrasting words. To the best of our knowledge, no systematic investigation has considered transfer learning for the task of contradiction detection. To illustrate this idea, contradictions in a political domain were used as a case study. Since we are testing transfer learning, we conducted experiments using as source task domain data collected from four publicly available corpora: MultiNLI, US2016, Argumentative Microtext, and Argument Annotated Essays. Then, for target domain, we built two datasets containing pairs of contradictions from two different sources, an online article exposing Donald Trump contradictory claims, and MultiNLI corpus (but only instances of government genre). To evaluate the conducted experiments, we measure classification performances mainly through ROC and Precision-Recall curves analysis. The findings from the research answer our research question that, indeed, other datasets can be used to boost an inference model learning performance on a target task, although the results are not too significant to strongly assure the consistency and reliability of our findings. The findings offer insights into what kind of relationship between documents one should focus on when resorting to transfer learning for detection of contradictions. We conclude that the domain's task, context and language patterns (linguistic markers characteristic of a person speech), have a bigger impact and, thus, can be helpful if different data contains similarities in these three factors. Nonetheless, we faced some limitations in our research, such as the lack of robustness in the testing dataset built from Donald Trump contradictions, because of missing professional annotators for that task, and the already great classification results when only using the target domain for training and testing, leading to a small margin for improvements.