Is ChatGPT a Good Recommender? A Preliminary Study

Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.

[1]  Haofen Wang,et al.  Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System , 2023, ArXiv.

[2]  Fangzhao Wu,et al.  Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems , 2023, ArXiv.

[3]  Dan Su,et al.  A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity , 2023, IJCNLP.

[4]  Tianming Liu,et al.  ChatAug: Leveraging ChatGPT for Text Data Augmentation , 2023, ArXiv.

[5]  Zhaopeng Tu,et al.  Is ChatGPT A Good Translator? A Preliminary Study , 2023, ArXiv.

[6]  Jie Yang,et al.  GreenPLM: Cross-lingual pre-trained language models conversion with (almost) no cost , 2022, ArXiv.

[7]  Jingqi Gao,et al.  Equivariant Contrastive Learning for Sequential Recommendation , 2022, ArXiv.

[8]  Jie Yang,et al.  Revisiting Bundle Recommendation: Datasets, Tasks, Challenges and Opportunities for Intent-aware Product Bundling , 2022, SIGIR.

[9]  Jingren Zhou,et al.  M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems , 2022, ArXiv.

[10]  Jagendra Singh,et al.  A Novel Deep Neural-based Music Recommendation Method considering User and Song Data , 2022, 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI).

[11]  Peilin Zhou,et al.  Decoupled Side Information Fusion for Sequential Recommendation , 2022, SIGIR.

[12]  Yingqiang Ge,et al.  Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) , 2022, RecSys.

[13]  Alexander M. Rush,et al.  Multitask Prompted Training Enables Zero-Shot Task Generalization , 2021, ICLR.

[14]  Chuhan Wu,et al.  FeedRec: News Feed Recommendation with Various User Feedbacks , 2021, WWW.

[15]  Jeremy Blackburn,et al.  "It Is Just a Flu": Assessing the Effect of Watch History on YouTube's Pseudoscientific Video Recommendations , 2020, ICWSM.

[16]  Raciel Yera,et al.  Content-based group recommender systems: A general taxonomy and further improvements , 2021, Expert Syst. Appl..

[17]  Jieming Zhu,et al.  SimpleX: A Simple and Strong Baseline for Collaborative Filtering , 2021, CIKM.

[18]  Guoguang Liu,et al.  An ecommerce recommendation algorithm based on link prediction , 2021 .

[19]  Yongfeng Zhang,et al.  Personalized Transformer for Explainable Recommendation , 2021, ACL.

[20]  Federica Cena,et al.  Logical foundations of knowledge-based recommender systems: A unifying spectrum of alternatives , 2021, Inf. Sci..

[21]  James Y. Zou,et al.  Language Models as Recommender Systems: Evaluations and Limitations , 2021 .

[22]  Xianyi Zeng,et al.  An interactive knowledge-based recommender system for fashion product design in the big data environment , 2020, Inf. Sci..

[23]  Chitra Dadkhah,et al.  A survey of attack detection approaches in collaborative filtering recommender systems , 2020, Artificial Intelligence Review.

[24]  Ji-Rong Wen,et al.  S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization , 2020, CIKM.

[25]  Xing Xie,et al.  MIND: A Large-scale Dataset for News Recommendation , 2020, ACL.

[26]  M. de Rijke,et al.  Challenges and research opportunities in eCommerce search and recommendations , 2020, SIGIR Forum.

[27]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[28]  Jesús Bobadilla,et al.  Deep Learning Architecture for Collaborative Filtering Recommender Systems , 2020, Applied Sciences.

[29]  Achraf Gazdar,et al.  A new similarity measure for collaborative filtering based recommender systems , 2020, Knowl. Based Syst..

[30]  Dominik Kowald,et al.  The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study , 2019, ECIR.

[31]  Jianfeng Gao,et al.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation , 2019, ACL.

[32]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[33]  Nima Jafari Navimipour,et al.  A Systematic Study on the Recommender Systems in the E-Commerce , 2020, IEEE Access.

[34]  Fernando Ortega,et al.  Classification-based Deep Neural Network Architecture for Collaborative Filtering Recommender Systems , 2020, Int. J. Interact. Multim. Artif. Intell..

[35]  Archana Bhise,et al.  Smart Billing Using Content-Based Recommender Systems Based on Fingerprint , 2020 .

[36]  Xiangnan He,et al.  MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video , 2019, ACM Multimedia.

[37]  Li Wei,et al.  Recommending what video to watch next: a multitask ranking system , 2019, RecSys.

[38]  Deqing Wang,et al.  Feature-level Deeper Self-Attention Network for Sequential Recommendation , 2019, IJCAI.

[39]  Xing Xie,et al.  NPA: Neural News Recommendation with Personalized Attention , 2019, KDD.

[40]  Chen Ma,et al.  Hierarchical Gating Networks for Sequential Recommendation , 2019, KDD.

[41]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[42]  Peng Jiang,et al.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer , 2019, CIKM.

[43]  Yang Liu,et al.  Fine-tune BERT for Extractive Summarization , 2019, ArXiv.

[44]  Orhan Firat,et al.  Massively Multilingual Neural Machine Translation , 2019, NAACL.

[45]  Kim-Kwang Raymond Choo,et al.  Privacy-aware smart city: A case study in collaborative filtering recommender systems , 2018, J. Parallel Distributed Comput..

[46]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[47]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[48]  Julian J. McAuley,et al.  Self-Attentive Sequential Recommendation , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[49]  Ankur Bapna,et al.  The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation , 2018, ACL.

[50]  Ke Wang,et al.  Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding , 2018, WSDM.

[51]  Maksims Volkovs,et al.  Content-based Neighbor Models for Cold Start in Recommender Systems , 2017, RecSys 2017.

[52]  Piji Li,et al.  Neural Rating Regression with Abstractive Tips Generation for Recommendation , 2017, SIGIR.

[53]  Mirella Lapata,et al.  Learning to Generate Product Reviews from Attributes , 2017, EACL.

[54]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[55]  Jianfeng Gao,et al.  Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access , 2016, ACL.

[56]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[57]  Pasquale Lops,et al.  Learning Word Embeddings from Wikipedia for Content-Based Recommender Systems , 2016, ECIR.

[58]  Alexandros Karatzoglou,et al.  Session-based Recommendations with Recurrent Neural Networks , 2015, ICLR.

[59]  Zachary Chase Lipton.  A Critical Review of Recurrent Neural Networks for Sequence Learning , 2015, ArXiv.

[60]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[61]  Lars Schmidt-Thieme,et al.  BPR: Bayesian Personalized Ranking from Implicit Feedback , 2009, UAI.

[62]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.