Event detection in finance using hierarchical clustering algorithms on news and tweets
Open Access
- 10 May 2021
- journal article
- research article
- Published by PeerJ in PeerJ Computer Science
- Vol. 7, e438
- https://doi.org/10.7717/peerj-cs.438
Abstract
In the current age of overwhelming information and massive production of textual data on the Web, Event Detection has become an increasingly important task in various application domains. Several research branches have been developed to tackle the problem from different perspectives, including Natural Language Processing and Big Data analysis, with the goal of providing valuable resources to support decision-making in a wide variety of fields. In this paper, we propose a real-time domain-specific clustering-based event-detection approach that integrates textual information coming, on one hand, from traditional newswires and, on the other hand, from microblogging platforms. The goal of the implemented pipeline is twofold: (i) providing insights to the user about the relevant events that are reported in the press on a daily basis; (ii) alerting the user about potentially important and impactful events, referred to as hot events, for some specific tasks or domains of interest. The algorithm identifies clusters of related news stories published by globally renowned press sources, which guarantee authoritative, noise-free information about current affairs; subsequently, the content extracted from microblogs is associated to the clusters in order to gain an assessment of the relevance of the event in the public opinion. To identify the events of a day d we create the lexicon by looking at news articles and stock data of previous days up to d−1 Although the approach can be extended to a variety of domains (e.g. politics, economy, sports), we hereby present a specific implementation in the financial sector. We validated our solution through a qualitative and quantitative evaluation, performed on the Dow Jones’ Data, News and Analytics dataset, on a stream of messages extracted from the microblogging platform Stocktwits, and on the Standard & Poor’s 500 index time-series. The experiments demonstrate the effectiveness of our proposal in extracting meaningful information from real-world events and in spotting hot events in the financial sphere. An added value of the evaluation is given by the visual inspection of a selected number of significant real-world events, starting from the Brexit Referendum and reaching until the recent outbreak of the Covid-19 pandemic in early 2020.Keywords
Funding Information
- The Centre for Advanced Studies at the Joint Research Centre of the European Commission
This publication has 35 references indexed in Scilit:
- A Survey of Techniques for Event Detection in TwitterComputational Intelligence, 2013
- Semantics-based information extraction for detecting economic eventsMultimedia Tools and Applications, 2012
- Understanding bag-of-words model: a statistical frameworkInternational Journal of Machine Learning and Cybernetics, 2010
- Textual analysis of stock market prediction using breaking financial newsACM Transactions on Information Systems, 2009
- Sure Independence Screening for Ultrahigh Dimensional Feature SpaceJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Hierarchical Clustering Algorithms for Document DatasetsData Mining and Knowledge Discovery, 2005
- The Adaptive Markets HypothesisThe Journal of Portfolio Management, 2004
- Using contextual analysis for news event detectionInternational Journal of Intelligent Systems, 2001
- A Survey of Recent Advances in Hierarchical Clustering AlgorithmsThe Computer Journal, 1983
- A vector space model for automatic indexingCommunications of the ACM, 1975