The unified logging infrastructure for data analytics at Twitter
- 1 August 2012
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 5 (12), 1771-1780
- https://doi.org/10.14778/2367502.2367516
Abstract
In recent years, there has been a substantial amount of work on large-scale data analytics using Hadoop-based platforms running on large clusters of commodity machines. A less explored topic is how those data, dominated by application logs, are collected and structured to begin with. In this paper, we present Twitter's production logging infrastructure and its evolution from application-specific logging to a unified "client events" log format, where messages are captured in common, well-formatted, flexible Thrift messages. Since most analytics tasks consider the user session as the basic unit of analysis, we pre-materialize "session sequences", which are compact summaries that can answer a large class of common queries quickly. The development of this infrastructure has streamlined log collection and data analysis, thereby improving our ability to rapidly experiment and iterate on various aspects of the service.This publication has 11 references indexed in Scilit:
- Trojan data layoutsPublished by Association for Computing Machinery (ACM) ,2011
- Hadoop++Proceedings of the VLDB Endowment, 2010
- Data warehousing and analytics infrastructure at facebookPublished by Association for Computing Machinery (ACM) ,2010
- HadoopDBProceedings of the VLDB Endowment, 2009
- Building a high-level dataflow system on top of Map-ReduceProceedings of the VLDB Endowment, 2009
- Modeling actions of PubMed users with n-gram language modelsInformation Retrieval Journal, 2008
- Practical guide to controlled experiments on the webPublished by Association for Computing Machinery (ACM) ,2007
- Modeling Online Browsing and Path Analysis Using Clickstream DataMarketing Science, 2004
- Funnel report mining for the MSN networkPublished by Association for Computing Machinery (ACM) ,2001
- Basic local alignment search toolJournal of Molecular Biology, 1990