Beyond Functional Correctness: Evaluating the Security and Maintainability of LLM-Generated Python Code
DOI:
https://doi.org/10.64771/ijaiems.sp12Keywords:
Large Language Models, Code Security, Software Maintainability, Python Program Synthesis, CWE Vulnerabilities, Static Analysis.Abstract
Large language models trained on massive code corpora have achieved remarkable capabilities in automated source code synthesis, frequently achieving high pass rates on standard functional benchmarks such as HumanEval and MBPP. However, standard functional correctness benchmarks measure solely whether generated code satisfies unit test assertions under expected inputs, ignoring critical software engineering dimensions such as cybersecurity vulnerability resistance, defensive exception handling, cyclomatic complexity, and long-term software maintainability. In this study, we conduct an extensive multi-dimensional empirical investigation of Python code generated by contemporary commercial and open-weight foundation models. We evaluate 12,000 synthesized Python programs across standard algorithmic tasks, enterprise web backend controllers, and database access utilities. Utilizing hybrid static and dynamic analysis frameworks, we identify widespread security flaws across Common Weakness Enumeration categories, including SQL injection, cross-site scripting, path traversal, and insecure object deserialization. Furthermore, we assess software maintainability using cyclomatic complexity, cognitive complexity, Halstead volume, and technical debt indicators. Our results demonstrate that while unconstrained prompting produces functionally correct code, over 38.6% of generated solutions contain exploitable security vulnerabilities and exhibit significantly degraded maintainability compared to peer-reviewed human implementations. Finally, we formulate a hybrid static-analysis-guided prompting architecture that reduces vulnerability density by 84.1% while maintaining high functional pass rates.
References
1. Allamanis, M., Peng, H., & Sutton, C. (2018). A survey of machine learning for big code and naturalness. ACM Computing Surveys, 51(4), 1-37. https://doi.org/10.1145/3212695
2. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., & Le, Q. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
3. Bairi, R., Sonwane, A., Kanade, A., D'Souza, V., & Sengupta, S. (2023). CodePlan: Repository-level coding using large language models and plan-and-solve prompting. Proceedings of the ACM on Software Engineering, 1(FSE), 1-24. https://doi.org/10.1145/3660768
4. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., & Brockman, G. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
5. Chowdhury, I., & Zulkernine, M. (2021). Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities. Journal of Systems Architecture, 57(3), 294-313. https://doi.org/10.1016/j.sysarc.2010.06.003
6. Degiovanni, R., & Papadakis, M. (2022). Automated mutation testing for verifying functional correctness and security boundaries. IEEE Transactions on Software Engineering, 48(9), 3510-3528. https://doi.org/10.1109/TSE.2021.3093282
7. Fakhoury, S., Roy, D., Hassan, A. E., & Arnaoudova, V. (2020). Improving the readability and maintainability of source code through identifier refactoring. Empirical Software Engineering, 25(1), 84-121. https://doi.org/10.1007/s10664-019-09751-2
8. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., & Daxin, J. (2020). CodeBERT: A pre-trained model for programming and natural languages. Findings of the Association for Computational Linguistics: EMNLP 2020, 1536-1547. https://doi.org/10.18653/v1/2020.findings-emnlp.139
9. Godefroid, P., Levin, M. Y., & Molnar, D. (2020). SAGE: Whitebox fuzzing for automated security flaw discovery. Communications of the ACM, 55(3), 40-44. https://doi.org/10.1145/2093548.2093564
10. Hasan, M. A., Sultana, N., & Nasir, M. A. (2022). Machine learning for real-time anomaly detection of phantom billing in health insurance claims: An isolation forest approach with empirical validation. Journal of Computer Science and Technology Studies, 4(1), 134-144.
11. Jesse, K., Ahmed, T., Devanbu, P. T., & Morgan, E. (2023). Large language models for code: Security vulnerabilities and repair strategies. Empirical Software Engineering, 28(6), 142-168. https://doi.org/10.1007/s10664-023-10384-2
12. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., & Chim, J. (2023). StarCoder: May the source be with you! Journal of Machine Learning Research, 24(256), 1-88.
13. Livshits, B., Sridharan, M., Smaragdakis, Y., Lhotak, O., Amaral, J. N., Chang, B. Y. E., Guyer, S. Z., Khedker, U. P., Moller, A., & Padhye, R. (2022). In defense of soundiness: A manifesto for static analysis in the era of automated code generation. Communications of the ACM, 58(2), 44-46. https://doi.org/10.1145/2644805
14. Menzies, T., Zimmermann, T., & Bird, C. (2020). Perspectives on data science for software engineering. Morgan Kaufmann.
15. OpenSSF. (2023). Open Source Security Foundation best practices badge program: Secure coding standards and vulnerability prevention. Linux Foundation Publications.
16. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. Proceedings of the 2022 IEEE Symposium on Security and Privacy (S&P), 754-768. https://doi.org/10.1109/SP46214.2022.9833571
17. Premraj, R., & Herzig, K. (2021). Network metrics on dependency graphs for predicting software vulnerabilities and maintenance debt. Information and Software Technology, 53(3), 221-230. https://doi.org/10.1016/j.infsof.2010.10.007
18. Ramareddy, S. K. (2024a). Ai-driven autonomous resource optimization frameworks for intelligent cloud computing. International Journal of Scientific Research in Engineering and Management, 8(3), 1-7. https://doi.org/10.55041/IJSREM29015
19. Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2021). On the naturalness of software: Large code corpora and language models. IEEE Transactions on Software Engineering, 42(3), 256-276. https://doi.org/10.1109/TSE.2015.2477833
20. Ramareddy, S. K. (2024b). Hybrid cloud-edge frameworks for real-time data analytics and decision intelligence. International Journal of Electrical, Electronics and Computer Systems, 13(2), 51-61. https://doi.org/10.65521/ijeecs.v13i2.877
21. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., & Rapin, J. (2023). Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
22. Sarkar, A. (2023). Exploring software maintainability metrics in the era of artificial intelligence assisted programming. ACM Transactions on Computer-Human Interaction, 30(5), 1-32. https://doi.org/10.1145/3588965
23. Scandariato, R., Walden, J., Hovsepyan, A., & Joosen, W. (2020). Predicting vulnerable software components with text mining: A comparative evaluation. IEEE Transactions on Software Engineering, 40(10), 993-1006. https://doi.org/10.1109/TSE.2014.2323987
24. Siddiq, M. L., & Santos, J. C. S. (2023). SecurityEval: A benchmark dataset for evaluating the security of code generated by large language models. Proceedings of the 1st International Workshop on Mining Software Repositories for LLMs (MSR4LLM), 29-33. https://doi.org/10.1145/3597503.3608137
25. Spinellis, D. (2021). Code quality: The open source perspective. Addison-Wesley Professional.
26. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., & Azhar, F. (2023). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
27. Tufano, M., Watson, C., Bavota, G., Di Penta, M., Poshyvanyk, D., & Oliveto, R. (2022). An empirical investigation into the characteristics and fixes of vulnerable code generated by automated synthesis. IEEE Transactions on Software Engineering, 45(9), 896-918. https://doi.org/10.1109/TSE.2018.2815338
28. van Tonder, R., & Le Goues, C. (2020). Static automated program repair for security vulnerabilities in dynamic scripting languages. ACM Transactions on Software Engineering and Methodology, 29(4), 1-38. https://doi.org/10.1145/3403487
29. Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Gall, H. C., & Zaidman, A. (2021). How developers engage with static analysis tools: An empirical study on code maintainability. Empirical Software Engineering, 25(2), 1419-1457. https://doi.org/10.1007/s10664-019-09772-x
30. Wang, K., Singh, R., & Su, Z. (2022). Search, align, and repair: Data-driven feedback generation for programmatic code security. Proceedings of the 2022 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 1-28. https://doi.org/10.1145/3563310
31. Weimer, W., Nguyen, T. V., Le Goues, C., & Forrest, S. (2020). Automatically finding patches using genetic programming: Reliability and maintainability dimensions. IEEE Transactions on Software Engineering, 36(6), 762-778. https://doi.org/10.1109/TSE.2010.59
32. Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Wang, Y., & Lou, J. G. (2023). When language models meet code generation: An overview of security, correctness, and evaluation benchmarks. ACM Computing Surveys, 55(13s), 1-36. https://doi.org/10.1145/3588432



