TencentRec
- 27 May 2015
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 227-238
- https://doi.org/10.1145/2723372.2742785
Abstract
With the arrival of the big data era, opportunities as well as challenges arise in both industry and academia. As an important service in most web applications, accurate real-time recommendation in the context of big data is of high demand. Traditional recommender systems that analyze data and update models at regular time intervals cannot satisfy the requirements of modern web applications, calling for real-time recommender systems. In this paper, we tackle the ``big\", ``real-time\" and ``accurate\" challenges in real-time recommendation, and propose a general real-time stream recommender system built on Storm named TencentRec from three aspects, i.e., ``system\", ``algorithm\", and ``data\". We analyze the large amount of data streams from a wide range of applications leveraging the considerable computation ability of Storm, together with a data access component and a data storage component developed by us. To deal with various application specific demands, we have implemented several classic practical recommendation algorithms in TencentRec, including the item-based collaborative filtering, the content based, and the demographic based algorithms. Specially, we present a practical scalable item-based CF algorithm in detail, with the super characteristics such as robust to the implicit feedback problem, incremental update and real-time pruning. With the enhancement of real-time data collection and processing, we can capture the recommendation changes in real-time. We deploy the TencentRec in a series of production applications, and observe the superiority of TencentRec in providing accurate real-time recommendations for 10 billion user requests everyday.Keywords
Funding Information
- 973 Program (2014CB340405)
- National Natural Science Foundation of China (61272155,61272340)
- Beijing Natural Science Foundation (4152023)
This publication has 31 references indexed in Scilit:
- Big data: the driver for innovation in databasesNational Science Review, 2014
- TeRecProceedings of the VLDB Endowment, 2013
- Real-time top-n recommendation in social streamsPublished by Association for Computing Machinery (ACM) ,2012
- StreamRecPublished by Association for Computing Machinery (ACM) ,2011
- Fast online learning through offline initialization for time-sensitive recommendationPublished by Association for Computing Machinery (ACM) ,2010
- Collaborative filtering for orkut communitiesPublished by Association for Computing Machinery (ACM) ,2009
- Google news personalizationPublished by Association for Computing Machinery (ACM) ,2007
- Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensionsIEEE Transactions on Knowledge and Data Engineering, 2005
- Item-based top-Nrecommendation algorithmsACM Transactions on Information Systems, 2004
- Mining high-speed data streamsPublished by Association for Computing Machinery (ACM) ,2000