Our journal article “Screening Articles for Systematic Reviews with ChatGPT” co-authored with Eugene Syriani and Gauransh Kumar has been in the three nominees for the 2024 Best Paper Award in the Journal of Computer Languages (COLA), and eventually finished as the runner-up. Even the nomination is a massive honor and we congratulate the eventual winners.
We believe the nomination was due to the particularly thorough and detailed evaluation of the LLM at hand. Should you be working on LLM-related topics, this article might be super useful for you. Feel free to reach out to any of the authors for discussions.
About the article. We wrote this article to shed light on how reliable ChatGPT and similar LLMs can be in automating the screening process in a systematic review. Systematic reviews can be sources of immensely valuable knowledge but producing them is a cumbersome endeavor. It is more tempting to rely on LLMs, but what are the real benefits, limits, and threats to validity by doing so? We elaborate on such questions in this one.