Logistic Regression Reloaded: Online Algorithms

[SIGIR 2011 Tutorial Abstract]

David D. Lewis
David D. Lewis Consulting Chicago, IL

sigir11tutorial@DavidDLewis.com ABSTRACT
The goals of this tutorial are to: 1) give attendees a sense of whether logistic regression is appropriate for a given task, and whether a batch or online approach is most appropriate, 2) give them an understanding of online learning in general and of online learning of logistic regression models in particular, 3) expose attendees to particular online learning algorithms for logistic regression, presented both mathematically and through code examples taken from open source machine learning packages, and 4) provide tips on setting learning rates and other parameters of online learning algorithms. They thus are natural in situations where training data becomes available incrementally, but a trained model is needed at all points in time. In articial intelligence parlance, they are anytime algorithms. Second, online machine algorithms have a small memory footprint, since typically only one or a few examples are in memory at a time. They thus play the traditional role of external memory algorithms. In more modern terminology, the combination of anytime behavior and external memory behavior puts these algorithms in the camp of streaming data algorithms. These characteristics have become more important as logistic regression is applied to very large data sets in computational advertising, recommendation systems, electronic discovery of legal documents, and other applications. Online learning algorithms have seen increased interest for other reasons. First, they are often simple to implement: many of the algorithms take the form of weight updates that can be presented on a single PowerPoint slide. Second, despite the fact that online algorithms inevitably produce only an approximation to the result of batch mode training, they can nonetheless produce better classiers than batch mode algorithms when training sets are very large and time is limited. Third, online algorithms provide straightforward ways to deal with some forms of time-varying data and concept drift. Finally, rich theoretical connections between online learning and other areas of theoretical computer science and optimization have led to increased understanding of the conceptual foundations of online learning.

Categories and Subject Descriptors

F.1.2 [Computation By Abstract Devices]: Modes of ComputationOnline computation; I.2.6 [Articial Intelligence]: LearningConnectionism and neural nets; H.2.8 [Database Management]: Database ApplicationsData mining; G.3 [Probability and Statistics]: Robust regession

General Terms
Algorithms, Theory, Experimentation

Keywords
supervised learning, regularization, external memory algorithms, text classication

2. 1. TUTORIAL DESCRIPTION
This tutorial will begin with a brief introduction to logistic regression, a exible, eective approach to supervised learning of classiers. Practical issues with applying logistic regression to high dimensional data in general, and text in particular, will be emphasized. The bulk of the tutorial will then focus on so-called online algorithms for tting logistic regression models, presenting a historical and practical view, many concrete algorithms, and demonstrations with open source software. Why online logistic regression? Online machine learning algorithms have two major characteristics. First, they process training data sequentially, so that a trained model becomes available before all training data has been processed.
Copyright is held by the author/owner(s). SIGIR11, July 2428, 2011, Beijing, China. ACM 978-1-4503-0757-4/11/07.

RELEVANCE TO IR

Logistic regression is a highly eective technique for supervised learning and has become one of the mainstays of machine learning. Ecient, easily usable software has become widely available. Within IR, logistic regression has become widely used in text categorization and text mining, as well as in areas which overlap with natural language processing, such as question answering and summarization. It also has an interesting history of creative uses in text retrieval, particularly in fusion of multiple systems. More generally, logistic regression is a good rst supervised learning approach to learn about, since the semantics of the model are clear, and issues of optimization, generalization, and regularization are more clearly distinguished than for some learning approaches. In the past three to ve years, online logistic regression has, somewhat surprisingly, become the dominant approach in many application areas for logistic regression.