Big claims, low outcomes: fact checking ChatGPT’s efficacy in handling linguistic creativity and ambiguity

Abstract

Ambiguity has always been a pain in the neck of Natural Language Processing (NLP). Despite enormous AI tools for human language processing, it remains a key concern for Language Technology Researchers to develop a linguistically intelligent tool that could effectively understand linguistic ambiguity and creativity possessed by human language. In this regard, the newly designed AI tool ChatGPT has dramatically attracted human attention due to its remarkable ability to answer human questions from a wide range of domains, which needs a reality check. This article scrutinises ChatGPT’s ability to answer and interpret neologisms, codemixing, and linguistically ambiguous sentences. For this, we have tested lexically, syntactically, and semantically ambiguous expressions, codemixed words, as well as a few language game instances. The findings show that ChatGPT still fails to understand linguistically complex sentences, specifically those common in everyday discourse or not part of any standard textbook. More specifically, semantically ambiguous sentences and language games remain an uphill task for ChatGPT to understand. This has implications for further improving the output of ChatGPT.