User Rating Prediction Method Based on Fine-tuning of Large Language Models
DOI:
https://doi.org/10.24203/tka4w802Keywords:
large language model, fine-tuning, attention mechanism, interpretable predictionAbstract
Online reviews in social networks reflect users' preferences for specific attributes of products. Accurate predictions of user ratings based on these reviews can help businesses better understand genuine user feedback. The purpose of this study is to fine-tune large language models using online reviews and corresponding user rating data, generating a large model for predicting user ratings based on reviews. An attention mechanism is introduced to calculate attention weights for fine-grained review texts, reflecting the contribution of different text features to user rating prediction. By visualizing these weights, the process of calculating the predicted rating values can be explained. Experimental results show that the proposed interpretable user rating prediction method can effectively visualize the attention weights of important text features in the decision-making process of the large rating prediction model. Compared to the baseline model, the mean absolute error is reduced by 1.96, and the root mean square error is reduced by 1.73.
References
[1] Bahtar A Z, Muda M. The impact of User–Generated Content (UGC) on product reviews towards online purchasing–A conceptual framework[J]. Procedia Economics and Finance, 2016, 37: 337-342.
[2] Cai Y, Ke W, Cui E, et al. A deep recommendation model of cross-grained sentiments of user reviews and ratings[J]. Information Processing & Management, 2022, 59(2): 102842.
[3] Lei X, Qian X, Zhao G. Rating prediction based on social sentiment from textual reviews[J]. IEEE transactions on multimedia, 2016, 18(9): 1910-1921.
[4] Cheng Z, Ding Y, Zhu L, et al. Aspect-aware latent factor model: Rating prediction with ratings and reviews[C]// Proceedings of the 2018 World Wide Web Conference on World Wide Web, Lyon, Apr 23-27, 2018. France: ACM, 2018: 639-648.
[5] Sadiq S, Umer M, Ullah S, et al. Discrepancy detection between actual user reviews and numeric ratings of Google App store using deep learning[J]. Expert Systems with Applications, 2021, 181: 115111.
[6] Devlin J , Chang M W , Lee K , et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Under-standing[C] //Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa-tional Linguistics: Human Language Technologies, Min-neapolis, Jun 2-7, 2019. USA: Association for Computa-tional Linguistics, 2009: 4171-4186.
[7] Ekgren A, Gyllensten A C, Gogoulou E, et al. Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish[C]//Proceedings of the 13th Language Resources and Evaluation Confer-ence, Marseille, Jun 20-25, 2022. France: European Language Resources Association, 2022: 3509-3518.
[8] Rives A, Meier J, Sercu T, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences[J]. Proceedings of the National Academy of Sciences of the United States of America, 2021, 118(15):e2016239118.
[9] Liu Y, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[J]. arXiv preprint arXiv.1907.11692, 2019.
[10] Yang Z , Dai Z , Yang Y ,et al.XLNet: Generalized Autoregressive Pretraining for Language Understanding[C] // Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Dec 8-14, 2019. Canada: ACM, 2019: 5753–5763.
[11] Khan Z Y, Niu Z, Sandiwarno S, et al. Deep learning techniques for rating prediction: a survey of the state-of-the-art[J]. Artificial Intelligence Review, 2021, 54: 95-135.
[12] Zhao G, Qian X, Xie X. User-service rating prediction by exploring social users' rating behaviors[J]. IEEE transactions on multimedia, 2016, 18(3): 496-506.
[13] Lai C H, Hsu C Y. Rating prediction based on combination of review mining and user preference analysis[J]. Information Systems, 2021, 99: 101742.
[14] Ma X, Lei X, Zhao G, et al. Rating prediction by exploring user’s preference and sentiment[J]. Multimedia Tools and Applications, 2018, 77: 6425-6444.
[15] Shi W, Wang L, Qin J. Extracting user influence from ratings and trust for rating prediction in recommendations[J]. Scientific reports, 2020, 10(1): 13592.
[16] Purkaystha B, Datta T, Islam M S. Rating prediction for recommendation: Constructing user profiles and item characteristics using backpropagation[J]. Applied Soft Computing, 2019, 75: 310-322.
[17] Ding N, Qin Y, Yang G, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models[J]. Nature Machine Intelligence, 2023, 5(3): 220-235.
[18] Tinn R, Cheng H, Gu Y, et al. Fine-tuning large neural language models for biomedical natural language processing[J]. Patterns, 2023, 4(4): 100729.
[19] Hilmkil A, Callh S, Barbieri M, et al. Scaling federated learning for fine-tuning of large language models[C]// Proceedings of International Conference on Applications of Natural Language to Information Systems, Saarbrucken, Jun 23-25, 2021. Germany: Springer, 2021: 15-23.
[20] Bakker M, Chadwick M, Sheahan H, et al. Fine-tuning language models to find agreement among humans with diverse preferences[J]. Advances in Neural Information Processing Systems, 2022, 35: 38176-38189.
[21] Luo L, Ning J, Zhao Y, et al. Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks[J]. Journal of the American Medical Informatics Association, 2024: ocae037.
[22] Niu Z, Zhong G, Yu H. A review on the attention mechanism of deep learning[J]. Neurocomputing, 2021, 452: 48-62.
[23] Tokuoka Y, Yamada T G, Mashiko D, et al. An explainable deep learning-based algorithm with an attention mechanism for predicting the live birth potential of mouse embryos[J]. Artificial Intelligence in Medicine, 2022, 134: 102432.
[24] Tal O, Liu Y, Huang J, et al. Neural attention frameworks for explainable recommendation[J]. IEEE Transactions on Knowledge and Data Engineering, 2019, 33(5): 2137-2150.
[25] Gu R, Wang G, Song T, et al. CA-Net: Comprehensive attention convolutional neural networks for explainable medical image segmentation[J]. IEEE transactions on medical imaging, 2020, 40(2): 699-711.
[26] Andresini G, Appice A, Caforio F P, et al. ROULETTE: A neural attention multi-output model for explainable network intrusion detection[J]. Expert Systems with Applications, 2022, 201: 117144.
[27] Liu X, Pan Z, Yang H, et al. An Adaptive Moment estimation method for online AUC maximization[J]. PloS one, 2019, 14(4): e0215426.
[28] Chen H, Zheng L, Al Kontar R, et al. Stochastic gradient descent in correlated settings: A study on gaussian processes[J]. Advances in neural information processing systems, 2020, 33: 2722-2733.
[29] McAuley J J, Leskovec J. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews[C]//Proceedings of the 22nd international conference on World Wide Web, Rio de Janeiro, May 13-17, 2013. Brazil: ACM, 2013: 897-908.
[30] Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of machine learning research, 2020, 21(140): 1-67.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Qi Zhang, Hao Zhong
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The articles published in International Journal of Computer and Information Technology (IJCIT) is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
