News
AI Coding Research Wins Penn State Award
Source: psu.edu
Published on May 28, 2025
Core topic: AI Coding Research
Keywords: AI coding research, Penn State award, large language models, data science, coding challenges, ACM SIGSOFT, AI algorithms, AI assistants, code generation, Distinguished Paper Award, AI performance, research study, AI reliability, AI coding tasks
Main keywords: AI coding research, Penn State award, large language models, data science, coding challenges, ACM SIGSOFT, AI algorithms, AI assistants, code generation, Distinguished Paper Award, AI performance, research study, AI reliability, AI coding tasks
Supporting n-grams: Penn State Great Valley, Distinguished Paper Award, large language models, data science coding, AI algorithms, code output quality, ACM SIGSOFT, coding challenges, LLM performance, AI assistants
Penn State Researchers Win Award for AI Coding Research
Researchers at Penn State Great Valley, Nathalia Nascimento and Everton Guimarães, have received the Distinguished Paper Award for their groundbreaking research on AI coding. The study, presented at the International Conference on Mining Software Repositories, evaluated how well large language models (LLMs) perform in solving data science coding problems. This recognition highlights the significance of their work in advancing AI algorithms for practical applications in data science.
The research focused on four prominent large language models: Microsoft Copilot, ChatGPT, Claude, and Perplexity Labs. These AI assistants were tested on their ability to generate code for various data science tasks, including analytical, algorithmic, and visualization challenges. The team meticulously analyzed the code output quality, exploring how task type and difficulty influenced the models' performance.
Evaluating AI Performance in Data Science
Nascimento and Guimarães, along with student researchers Sai Sanjna Chintakunta and Santhosh Anitha Boominathan, created a comprehensive dataset to assess the performance of LLMs in data science coding tasks. The study revealed that all models achieved a success rate above 50%, with ChatGPT and Claude exceeding 60%. However, none of the models reached a 70% success rate, indicating both the strengths and limitations of current AI technologies.
"This research provides a framework for evaluating large language models in data science," Nascimento said. "It helps practitioners select the right models for specific tasks and sets a standard for AI assessments." The study not only contributes to the field of AI research but also offers practical insights for data scientists and developers who rely on AI tools in their work.
Recognition and Future Implications
The research team's paper, titled "How Effective are LLMs for Data Science Coding? A Controlled Experiment," was accepted for the technical track of the MSR conference. Following their presentation, the Association for Computing Machinery’s Special Interest Group on Software Engineering (ACM SIGSOFT) awarded them the Distinguished Paper Award. This prestigious recognition underscores the impact of their work in the AI and software engineering communities.
"We are honored to receive this award," Nascimento said. "It validates our efforts to push the boundaries of AI coding research and inspire further innovation in the field." Student researcher Chintakunta added, "The support from our professors has been invaluable, and we are proud of what we have accomplished together."