Séminaire de Recherche en Linguistique

In this talk, I will first discuss our departmental submission to the “Build it, Break it” shared task on sentiment analysis, focusing on the limitations of NLU systems highlighted by the results on the task. I will further discuss the annotations that we currently have available for such “common sense” data. Current practices of operationalizing annotations for natural language understanding datasets often assume one single label per item. I will show that such practices obscure systematic variability in the data, which should be analyzed and captured. Current work in part-of-speech and word sense tagging as well as relation extraction embraces uncertainty in the annotations when learning predictive models from those annotations. Similarly, I propose that NLU moves towards systems that predict the distributions of labels instead of a single label, to fully capture human interpretations of language.