ChatGPT solved a coding test of exactly 728 questions: Here are the results!

MetaversePlanet July 11, 2024

0 1 minute read

ChatGPT, one of the world’s most advanced artificial intelligence models, solved a 728-question coding test. Here’s what it can do…

The research, published in the June issue of IEEE Transactions on Software Engineering, tested the ChatGPT 3.5 model on 728 coding challenges from the LeetCode platform. The test was conducted in five programming languages: C++, Java, JavaScript, and Python. The renowned AI model successfully passed the test.

ChatGPT has exceeded the results of the coding test conducted in the past months

One of the most prominent features of AI technology is its ability to write computer code. A new study evaluating ChatGPT’s performance in this area shows that the ChatGPT 3.5 model, developed by OpenAI, received at least a passing grade.

Among 728 coding problems, ChatGPT showed very successful results on pre-2021 problems. It was able to solve easy-level problems with 89 percent accuracy, medium difficulty problems with 71 percent accuracy, and difficult problems with 40 percent accuracy.

However, ChatGPT’s performance dropped significantly on problems added after 2021: it achieved 52 percent success on easy problems, 40 percent on medium difficulty, and only 0.66 percent on difficult problems.

Yutian Tang, a researcher from the University of Glasgow, explained the reason for this decline:

“On post-2021 algorithm problems, ChatGPT’s ability to produce functionally correct code is affected. It sometimes struggles to grasp the meaning of questions even on easy-level problems.”

The research also showed that ChatGPT was better at correcting human errors than its own and could produce code that required 50 percent less runtime and memory usage compared to humans.

So, what do you think about ChatGPT’s performance? Don’t forget to share your opinions with us in the comments section.