Online supervised spam filter evaluation
- 1 July 2007
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 25 (3)
- https://doi.org/10.1145/1247715.1247717
Abstract
Eleven variants of six widely used open-source spam filters are tested on a chronological sequence of 49086 e-mail messages received by an individual from August 2003 through March 2004. Our approach differs from those previously reported in that the test set is large, comprises uncensored raw messages, and is presented to each filter sequentially with incremental feedback. Misclassification rates and Receiver Operating Characteristic Curve measurements are reported, with statistical confidence intervals. Quantitative results indicate that content-based filters can eliminate 98% of spam while incurring 0.1% legitimate email loss. Qualitative results indicate that the risk of loss depends on the nature of the message, and that messages likely to be lost may be those that are less critical. More generally, our methodology has been encapsulated in a free software toolkit, which may used to conduct similar experiments.Keywords
This publication has 9 references indexed in Scilit:
- Models and Statistical Inference: The Controversy between Fisher and Neyman–PearsonThe British Journal for the Philosophy of Science, 2006
- An evaluation of statistical spam filtering techniquesACM Transactions on Asian Language Information Processing, 2004
- Receiver Operating Characteristic (ROC) Curve: Practical Review for RadiologistsKorean Journal of Radiology, 2004
- "In vivo" spam filteringACM SIGKDD Explorations Newsletter, 2003
- Evaluating cost-sensitive Unsolicited Bulk Email categorizationPublished by Association for Computing Machinery (ACM) ,2002
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- Support vector machines for spam categorizationIEEE Transactions on Neural Networks, 1999
- Evaluating and optimizing autonomous text classification systemsPublished by Association for Computing Machinery (ACM) ,1995
- Theory of Statistical EstimationMathematical Proceedings of the Cambridge Philosophical Society, 1925