The controversy surrounding Generative Artificial Intelligence (AI) for text arises from its remarkable capacity to generate entire paragraphs that closely resemble human-written content, and answer diverse questions spanning various disciplines, including technical and complex high-order thinking queries. Additionally, it can compose and comment on code in multiple programming languages, translate and improve text in various languages, and perform numerous other tasks. However, if misused, these AI assets can quickly lead to issues such as plagiarism and integrity breaches, primarily affecting the education and research sectors.
ChatGPT, an example of one of the most advanced chatbots to date, has garnered significant attention from various communities, including AI experts and those whose fields could be impacted by it, such as journalists, educators, and researchers. This attention has resulted in a multitude of blogs and posts published in digital press, magazines, and social networks. Nevertheless, there is still a scarcity in terms of research papers due to ChatGPT relative novelty. Currently, most existing papers are preprints, with peer reviewed journal or conference papers being limited. These publications primarily explore ChatGPT capabilities, limitations, and its applications across different domains like medicine, nurse education, journalism, media education, learning, as well as teaching and assessment mainly in higher education. Some efforts have also focused on leveraging ChatGPT as a research tool, ranging from academic paper writing to idea generation and literature review, with instances where ChatGPT has even been credited as a co-author in certain papers.
Still in ChatGPT leverage in research, how would be the future of publishing if peer review itself would be made using such AI text generation models? Supporting the advancement of knowledge, publishing research plays a pivotal role. However, a crucial and intricate prerequisite is necessary: the peer review process. During this stage, research papers submitted to journals undergo evaluation by experts in the respective field. Their responsibility is to scrutinize the manuscripts for originality, validity, and significance, ensuring that only high-quality submissions are accepted and published. Peer reviewers hold the key to distinguish between groundbreaking discoveries and potentially misleading or erroneous findings. In this sense, AI based review assistant tools, such as Raxter.io, are rapidly evolving to support reviewers in furnishing feedback to authors and offering recommendations on enhancing the manuscript. It is perfect as long as these tools are used in tandem with reviewers expertise to assist them in their hard and time-consuming task. In contrast, it would be problematic if reviewers only rely on these tools to provide their feedback and notifications.
Journal editors and academic publishers need to be vigilant about the possibility of reviewers using ChatGPT or another to generate their reviews. It becomes crucial for them to take proactive steps to safeguard the integrity, validity, and fairness of the review process; as it is possible to pilot these tools to review a research paper. To ensure ethical use and maintain appropriate oversight, measures should be adopted to place the review process at the forefront of the rapidly advancing AI text generation models, currently in the spotlight. By doing so, it is possible to establish a responsible and trustworthy environment for peer review, where AI is leveraged thoughtfully to advance the quality of scholarly publications and not the contrary. Presented below are some recommendation proposals to address or at least mitigate potential misuse of generative AI for research papers peer review:
- Ability of text generative models in conducting papers reviewing should be assessed to highlight their strengths and weaknesses. For this, comprehensive case studies are necessary to confront their generated reviews to expert ones for different papers of several fields. Editors can then gain valuable insights into the limitations associated with their use in the review process and the distinguishable style of their responses.
- Besides the common use of anti-plagiarism solutions, it has become imperative to adopt AI tools to detect AI generated text and differentiate it from human-written one. Recent endeavors have indeed been initiated to combat the misuse of ChatGPT, particularly in academic dishonesty. While these detectors may not be entirely infallible and can be influenced by truncated or modified AI generated text, they still offer editors a valuable means to scan review content and detect any portions produced using AI text generators.
- It is extremely important to rely on the expertise in the field when assigning reviewers to a submitted paper. While this criterion is undeniably significant, it alone may not be adequate. A wiser approach would be to prioritize reviewers who possess both expertise and essential abilities to conduct meticulous and comprehensive evaluations, providing consistent and well-structured reviews. Here, services that enable academics to monitor, authenticate, and showcase their peer review contributions for conferences and journals are useful. Leveraging such services, e.g., Publons, can prove highly beneficial in identifying top reviewers with publisher-verified peer review contributions, and thus facilitating the selection of the most suitable candidates.
- It is judicious to go beyond conventional methods and make earnest efforts to revolutionize the reviewing process. Customized review inquiries that delve into the nuances of the paper subject matter should be incorporated, rather than relying on predetermined checklists and simplistic rating scales with binary or choice-based questions. By doing so, responses cannot be generated by AI based tools. Even if a reviewer only relies on an AI chatbot, it becomes evident to the editor, as the responses would lack accuracy in relation to the specific review queries.
In all this, ethics stand as the primary means to address these concerns. Publishers and relevant actors might have to implement a focused policy, enhancing reviewers understanding of the implications associated with the Human-Machine leverage of generative AI guided peer review.