New Study Finds ChatGPT Ineffective for Software Engineering Prompts
In the realm of AI chatbots, ChatGPT has gained popularity for its ability to provide conversational answers to a wide range of questions. However, a recent study has revealed that when it comes to software engineering prompts, ChatGPT may not be the most reliable resource.
Traditionally, programmers have relied on platforms like Stack Overflow for advice on their projects. Stack Overflow operates on a question-and-answer model, similar to ChatGPT. The key difference is that with Stack Overflow, users have to wait for someone to respond to their question, whereas ChatGPT provides instant responses. This convenience has led many software engineers and programmers to turn to ChatGPT for their queries.
To determine the effectiveness of ChatGPT in answering software engineering prompts, researchers at Purdue University conducted a study. They provided ChatGPT with 517 Stack Overflow questions and assessed the accuracy and quality of its answers.
The results were concerning. Out of the 512 questions, ChatGPT provided incorrect answers for 259 (52%) of them, while only 248 (48%) were correct. Furthermore, a staggering 77% of the answers were unnecessarily verbose.
Despite the high rate of incorrect answers, the study did find that ChatGPT’s responses were comprehensive and addressed all aspects of the question 65% of the time. To gain further insights into the quality of ChatGPT’s responses, the researchers sought the opinions of 12 participants with varying levels of programming expertise.
Interestingly, the participants preferred Stack Overflow’s responses over ChatGPT’s in various categories. However, they struggled to identify incorrect answers generated by ChatGPT, overlooking them 39.34% of the time. According to the study, this was due to the well-articulated nature of ChatGPT’s responses, which made it easy to overlook any incorrect information.
The generation of plausible-sounding yet incorrect answers is a significant issue with chatbots in general, as it can contribute to the spread of misinformation. Coupled with the low accuracy scores, these findings raise concerns about relying on ChatGPT for software engineering prompts.
In conclusion, while ChatGPT may be a handy resource for general information needs, it falls short when it comes to software engineering prompts. Programmers and software engineers should exercise caution and consider alternative platforms like Stack Overflow for reliable and accurate advice in their projects.