Structural and functional analysis of artificial intelligence models for code generation

Abstract

Lucrarea de față își propune să examineze performanța unor modele generative populare, precum ChatGPT, Claude, Gemini și Grok Web, în crearea unui cod funcțional, eficient și uș de menținut.

Nowadays, generative Artificial Intelligence (AI) tools are gaining more and more attention and slowly becoming a standard in software development; therefore, there is an increasing need to determine how effectively these tools perform beyond simply delivering code that runs. The present thesis, titled ”Structural and Functional Analysis of Generative AIs for Code Generation”, aims to examine the performance of some popular generative AI models such as ChatGPT, Claude, Gemini and Grok Web, in creating code that is functionally valid, efficient, and maintainable. To accomplish this, each model was evaluated on six real world coding problems sourced from LeetCode, covering a diverse set of algorithmic challenges such as dynamic programming, graph traversal, and array manipulation. A consistent prompting strategy was applied to collect Python code samples from each model, which were then evaluated using popular software engineering metrics [11]; such as the number of lines of code, maintainability index, Cyclomatic complexity and Halstead complexity [16]. With the collected results, a detailed statistical analysis was performed, using different statistical methods such as ANOVA, Post hoc analysis and Nonparametric statistics, in order to determine which models consistently performed best. The results show that the type of problems has the most impact on code complexity and length; however, when considering code maintainability, the specific AI model plays a significant role. This research goes beyond simple pass or fail benchmarks and provides a broader and detailed understanding of how generative AI tools behave in practical programming tasks. The paper work also covers the fundamental models of Gen-AI, such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and transformers. These findings can be utilized by developers, teachers, and tool creators to select the most suitable AI assistant for their specific needs and to gain a clearer understanding of the models’ strengths as well as their current limitations.