Topics, Trends, and Sentiments in Software Testing: An Analysis of Developers’ Engagement on Stack Overflow
DOI:
https://doi.org/10.24203/dfjd8332Keywords:
Software testing, Stack Overflow, Topic Modeling, Sentiment Analysis, Developer Engagement, Machine Learning Testing, ChatGPTAbstract
This study investigated software testing discussions on Stack Overflow from 2020 to 2024 to uncover key trends, topics, and developer sentiments. 14 key topics, including unit testing, machine learning testing, mobile testing (especially Flutter), and Docker testing were identified. The study revealed a decline in developer engagement, as the number of posts answered and with accepted answers decreased, particularly after 2022. Sentiment analysis showed a predominance of negative sentiments across most topics, especially in mobile and machine learning testing. While some topics like machine learning testing initially saw positive sentiment, this shifted toward frustration as the years progressed. These findings suggest that the rise of AI-based tools, such as ChatGPT, has affected the way developers engage with traditional forums like Stack Overflow. The decline in engagement and the prevalence of negative sentiments highlight the challenges developers face in software testing. This research points to the need for further investigation into how AI tools influence developer behavior and their reliance on peer support platforms. Additionally, it suggests exploring how sentiment analysis can be integrated into software testing tools to better address developer frustrations and improve support for testing emerging technologies. The study provides insights that could guide the development of more effective tools and frameworks to enhance the software testing process.
References
REFERENCES
[1] A. Salahirad, G. Gay, and E. Mohammadi, "Mapping the structure and evolution of software testing research over the past three decades," Journal of Systems and Software, vol. 195, p. 111518, 2023/01/01/ 2023, doi: 10.1016/j.jss.2022.111518.
[2] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, "Software Testing With Large Language Models: Survey, Landscape, and Vision," IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911-936, 2024, doi: 10.1109/TSE.2024.3368208.
[3] D. Chevers, A Software Development Approach for Driving Competitiveness in Small Firms. Auerbach Publications, 2023.
[4] I. Santos, E. F. Coutinho, and S. R. S. Souza, "Software testing ecosystems insights and research opportunities," presented at the Proceedings of the XXXIV Brazilian Symposium on Software Engineering, Natal, Brazil, 2020. [Online]. Available: https://doi.org/10.1145/3422392.3422458.
[5] F. S. Ahmed, A. Majeed, T. A. Khan, and S. N. Bhatti, "Value-based cost-cognizant test case prioritization for regression testing," PLOS ONE, vol. 17, no. 5, p. e0264972, 2022, doi: 10.1371/journal.pone.0264972.
[6] R. Blanco, M. Trinidad, M. J. Suárez-Cabal, A. Calderón, M. Ruiz, and J. Tuya, "Can gamification help in software testing education? Findings from an empirical study," Journal of Systems and Software, vol. 200, p. 111647, 2023/06/01/ 2023, doi: 10.1016/j.jss.2023.111647.
[7] Y. Wu, J. Su, D. D. Moran, and C. D. Near, "Automated software testing starting from static analysis: current state of the art," arXiv preprint arXiv:2301.06215, 2023, doi: 10.48550/arXiv.2301.06215.
[8] S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, "Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions," presented at the Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 2024. [Online]. Available: https://doi.org/10.1145/3613904.3642596.
[9] A. M. Alghamdi, W. Aljedaani, H. Jalali, S. Ludi, and M. M. Eler, "Understanding developer challenges and trends in web accessibility: a stack overflow analysis," Universal Access in the Information Society, 2024/11/19 2024, doi: 10.1007/s10209-024-01174-3.
[10] T. Varun Kumar, "A Comprehensive Empirical Study Determining Practitioners' Views on Docker Development Difficulties: Stack Overflow Analysis," International Journal of Innovative Research in Computer and Communication Engineering, vol. 12, no. 1, pp. 157-164, 2024.
[11] J. He et al., "Representation Learning for Stack Overflow Posts: How Far Are We?," ACM Trans. Softw. Eng. Methodol., vol. 33, no. 3, p. Article 69, 2024, doi: 10.1145/3635711.
[12] S. Izzat and N. N. Saleem, "Software testing techniques and tools: A review," Journal of Education and Science, vol. 32, no. 2, pp. 30.0-44.0, 2023, doi: 10.33899/edusj.2023.137480.1305.
[13] H. M. Ali, M. Y. Hamza, and T. A. Rashid, "A comprehensive study on automated testing with the software lifecycle," arXiv preprint arXiv:2405.01608, 2024, doi: 10.48550/arXiv.2405.01608.
[14] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, "An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation," IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85-105, 2024, doi: 10.1109/TSE.2023.3334955.
[15] M. A. Zmuda, "A Software Framework Based on JUnit for Automated Software Testing in Computer Science Courses," in 2024 IEEE Frontiers in Education Conference (FIE), 13-16 Oct. 2024 2024, pp. 1-8, doi: 10.1109/FIE61694.2024.10892903.
[16] V. Yadav, R. K. Botchway, R. Senkerik, and Z. K. Oplatkova, "Robotic automation of software testing from a machine learning viewpoint," in Mendel, 2021, vol. 27, no. 2, pp. 68-73, doi: 10.13164/mendel.2021.k.0e3.
[17] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, "Software testing with large language models: Survey, landscape, and vision," IEEE Transactions on Software Engineering, 2024, doi: 10.1109/TSE.2024.3368208.
[18] D. Elsner, D. Bertagnolli, A. Pretschner, and R. Klaus, "Challenges in regression test selection for end-to-end testing of microservice-based software systems," presented at the Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, Pittsburgh, Pennsylvania, 2022. [Online]. Available: https://doi.org/10.1145/3524481.3527217.
[19] R. Santos, I. Santos, C. Magalhaes, and R. d. S. Santos, "Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing," in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), 27-31 May 2024 2024, pp. 353-360, doi: 10.1109/ICST60714.2024.00039.
[20] M. Boukhlif, N. Kharmoum, and M. Hanine, "LLMs for Intelligent Software Testing: A Comparative Study," presented at the Proceedings of the 7th International Conference on Networking, Intelligent Systems and Security, Meknes, AA, Morocco, 2024. [Online]. Available: https://doi.org/10.1145/3659677.3659749.
[21] N. Weeraddana, M. Alfadel, and S. McIntosh, "Characterizing Timeout Builds in Continuous Integration," IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1450-1463, 2024, doi: 10.1109/TSE.2024.3387840.
[22] A. W. Wambua, "What Do Flutter Developers Ask About? An Empirical Study on Stack Overflow Posts," Journal of Software Engineering Research and Development, vol. 12, no. 1, pp. 7: 1-7: 13, 2024, doi: 10.5753/jserd.2024.3620.
[23] A. Alanazi and R. Alfayez, "What is discussed about Flutter on Stack Overflow (SO) question-and-answer (Q&A) website: An empirical study," Journal of Systems and Software, vol. 215, p. 112089, 2024, doi: 10.1016/j.jss.2024.112089.
[24] C. C. Silva, M. Galster, and F. Gilson, "Applying short text topic models to instant messaging communication of software developers," Journal of Systems and Software, vol. 216, p. 112111, 2024/10/01/ 2024, doi: https://doi.org/10.1016/j.jss.2024.112111.
[25] A. Abdelrazek, Y. Eid, E. Gawish, W. Medhat, and A. Hassan, "Topic modeling algorithms and applications: A survey," Information Systems, vol. 112, p. 102131, 2023/02/01/ 2023, doi: https://doi.org/10.1016/j.is.2022.102131.
[26] N. Prakash, H. Wang, N. K. Hoang, M. S. Hee, and R. K.-W. Lee, "PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models," presented at the Proceedings of the 31st ACM International Conference on Multimedia, Ottawa ON, Canada, 2023. [Online]. Available: https://doi.org/10.1145/3581783.3613836.
[27] J. Zimmermann, L. E. Champagne, J. M. Dickens, and B. T. Hazen, "Approaches to improve preprocessing for Latent Dirichlet Allocation topic modeling," Decision Support Systems, vol. 185, p. 114310, 2024/10/01/ 2024, doi: https://doi.org/10.1016/j.dss.2024.114310.
[28] B.-X. Du and G.-Y. Liu, "Topic analysis in lda based on keywords selection," Journal of Computers, vol. 32, no. 4, pp. 1-12, 2021, doi: 10.53106/199115992021083204001.
[29] C. Treude and M. Wagner, "Predicting good configurations for GitHub and stack overflow topic models," presented at the Proceedings of the 16th International Conference on Mining Software Repositories, Montreal, Quebec, Canada, 2019. [Online]. Available: https://doi.org/10.1109/MSR.2019.00022.
[30] A. Pereira, K. Gama, C. Zimmerle, and F. Castor, "Reactive Programming with Swift Combine: An Analysis of Problems Faced by Developers on Stack Overflow," in Proceedings of the XXXVII Brazilian Symposium on Software Engineering, 2023, pp. 109-115.
[31] M. U. Haque, L. H. Iwaya, and M. A. Babar, "Challenges in docker development: A large-scale study using stack overflow," in Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM), 2020, pp. 1-11, doi: 10.1145/3382494.3410693.
[32] M. Wankhade, A. C. S. Rao, and C. Kulkarni, "A survey on sentiment analysis methods, applications, and challenges," Artificial Intelligence Review, vol. 55, no. 7, pp. 5731-5780, 2022, doi: 10.1007/s10462-022-10144-1.
[33] K. L. Tan, C. P. Lee, and K. M. Lim, "A Survey of Sentiment Analysis: Approaches, Datasets, and Future Research," Applied Sciences, vol. 13, no. 7, p. 4550, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/7/4550.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Anthony Wambua
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The articles published in International Journal of Computer and Information Technology (IJCIT) is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
