AI Coding Research Wins Penn State Award
Source: psu.edu
The International Conference on Mining Software Repositories in 2025 gave awards for outstanding research. Nathalia Nascimento and Everton Guimarães, assistant software engineering professors at Penn State Great Valley, were awarded the Distinguished Paper Award. Their research focused on how well four large language models solve data science coding problems. Two student researchers assisted them.
Large language models (LLMs) are AI algorithms that can generate code to help data scientists analyze and visualize data. A team of Penn State Great Valley professors and students studied how reliable these models are for coding tasks. Their work earned them the Distinguished Paper Award from the Association for Computing Machinery’s Special Interest Group on Software Engineering (ACM SIGSOFT) at an international conference in April.
Nathalia Nascimento and Everton Guimarães conducted a study with research assistants Sai Sanjna Chintakunta and Santhosh Anitha Boominathan. The study evaluated the performance of four AI assistants: Microsoft Copilot, ChatGPT, Claude, and Perplexity Labs. The team examined how well these models solved data science coding challenges, including analytical, algorithmic, and visualization problems. The researchers explored if task type or difficulty affected the code output quality.
The research team created a dataset to assess LLM performance in data science coding tasks. The students extended this work in a journal, including another LLM in their analysis.
The researchers stated that all models performed above a 50% success rate. ChatGPT and Claude had success rates above 60%, but none reached 70%. The researchers noted that the mid-range success rate shows the models' strengths and limitations.
The team wrote that their study offers a framework for evaluating LLMs in data science. It helps practitioners select models for specific tasks and sets standards for AI assessments. The researchers submitted their paper to the International Conference on Mining Software Repositories (MSR), a venue for software analytics research, which was co-located with the International Conference on Software Engineering.
The Great Valley research team’s paper, “How Effective are LLMs for Data Science Coding? A Controlled Experiment,” was accepted for the technical track of the MSR conference. After their presentation, ACM SIGSOFT gave the team the Distinguished Paper Award.
Nascimento said they were honored to receive the recognition. Student researcher Chintakunta said that the professors' support made a difference, and she is proud of their accomplishments.