A Software System for Topic Extraction and Document Classification
Davide Magatti and Fabio Stella
Department of Informatics, Systems and Communications
Universit degli Studi di Milano-Bicocca
Milan, Italy
Email: {magatti, stella} @disco.unimib.it
Abstract
A software system for topic extraction and automatic
document classification is presented. Given a set of doc-
uments, the system automatically extracts the mentioned
topics and assists the user to select their optimal number.
The user-validated topics are exploited to build a model for
multi-label document classification. While topic extraction
is performed by using an optimized implementation of the
Latent Dirichlet Allocation model, multi-label document
classification is performed by using a specialized version of
the Multi-Net Naive Bayes model. The performance of the
proposed conceptual model is investigated by using 10,056
documents retrieved from the WEB through a set of queries
formed by exploiting the Italian Google Directory. This
dataset is used for topic extraction while an independent
dataset, consisting of 1,012 elements labeled by humans, is
used to evaluate the performance of the Multi-Net Naive
Bayes model. The results are satisfactory, with precision
being consistently better than recall for the labels associated
with the four most frequent topics.
1. Introduction
The continuously increasing amount of text available on
the WEB, news wires, forums and chat lines, business com-
pany intranets, personal computers, e-mails and elsewhere
is overwhelming [1].
Information, is switching from useful to troublesome.
Indeed, it is becoming more and more evident that while
the amount of data is rapidly increasing, our capability to
process information remains constant. This trend strongly
limits the extraction of valuable knowledge from text and
thus drastically reduces the competitive advantage we can
gain. Search engines have exacerbated such a problem by
dramatically increasing the amount of text available in a
matter of a few key strokes.
While Wigner [2] examines the reasons why a great
body of physics can be neatly explained with elementary
mathematical formulas, the same does not apply to sciences
that involve human beings. Indeed, in this case it is becom-
ing increasingly evident how complex theories will never
Marco Faini
DocFlow Italia S.p.A.
Centro Direzionale Milanofiori, Strada 4 Palazzo Q8
20089 Rozzano, Italy
Email: marco.faini@docflow.it
share the elegance of physics. Therefore, when dealing with
sciences that involve human beings
... we should stop acting as if our goal is
to author extremely elegant theories, and instead
embrace complexity and make use of the best ally
we have: the unreasonable effectiveness of data.
as discussed by Alon Halevy, Peter Norvig and Fernando
Pereira in their recent paper titled “The Unreasonable Ef-
fectiveness of Data” [3].
The authors conclude their paper by stating
Choose a representation that can use unsuper-
vised learning on unlabeled data, which is so much
more plentiful than labeled data.
and by giving the following suggestion
Represent all the data with a nonparametric
model rather than trying to summarize it with a
parametric model, because with very large data
sources, the data holds a lot of detail.
In this paper the authors share this point of view and
follow the suggestion to design and implement a software
prototype for topic extraction and multi-label document clas-
sification. The software prototype implements a conceptual
model which processes a corpus of unstructured textual data
to discover which topics are mentioned, and then exploits the
extracted topics to learn a supervised classification model for
multi-label document classification.
The rest of the paper is organized as follows; Section 2
introduces Text Mining and gives basic elements concerning
the Latent Dirichlet Allocation model and the Multi-Net
Naive Bayes model. Section 3 describes the main compo-
nents of the software prototype along with their functional-
ities. Section 4 is devoted to numerical experiments and to
multi-label document classification performance evaluation.
Finally, Section 5 presents conclusions and discusses further
developments.
2. Text Mining
Text Mining (TM) [1], [4] is an emerging research area
which aims to solve the problem of information overload,
i.e. to automatically extract knowledge from semi and un-
structured text. Therefore, TM can be interpreted as the
natural answer to the unreasonable effectiveness of data
which aims to efficiently memorize the huge amount of data
available through the WEB. The main techniques used by
TM are; data mining, machine learning, natural language
processing, information retrieval and knowledge manage-
ment. Typical tasks of TM are; text categorization, document
clustering and organization, and information extraction.
2.1. Probabilistic Topic Extraction
Probabilistic Topic Extraction (PTE) is a particular form
of document clustering and organization used to analyze the
content of documents and the meaning of words with the aim
to discover the topics mentioned in a document collection.
A variety of models have been proposed, described and
analyzed in the specialized literature [5], [6], [7], [8]. These
models differ among them in terms of the assumptions they
make concerning the data generating process. However, they
all share the same rationale, i.e. a document is a mixture of
topics.
To describe how the PTE model works, let P (z) be
the probability distribution over topics z, P (w|z) be the
probability distribution over words w given topic z. The
topic-word distribution P (w|z) specifies the weight to the-
matically related words. A document is assumed to be
formed as follows: the ith word wi is generated by first
extracting a sample from the topic distribution P (z), then
by choosing a word from P (w|z). We let P (zi = j) to be
the probability that the j th topic was sampled for the ith
word token while P (wi |zi = j) is let to be the probability
of word wi under topic j. Therefore, the PTE induces the
following distribution over words within a document:
X
T categorization and represents a building block of several
P (wi ) = P (wi |zi = j) P (zi = j) (1)
j=1
where T is the number of topics.
Hofmann [6], [9] proposed the probabilistic Latent Se-
mantic Indexing (pLSI) method which makes no assump-
tions about how the mixture weights in (1), i.e. P (zi = j),
are generated. Blei et al. [7] improved the generalizabil-
ity of this model to new documents. They introduced a
Dirichlet prior, with hyperparameter α, on P (zi = j), thus
originating the Latent Dirichlet Allocation (LDA) model. In
2004, Griffiths and Steyvers [10] introduced an extension of
the original LDA model which associate a Dirichlet prior,
with hyperparameter β, also to P (wi |zi = j). The authors
suggested the hyperparameter to be interpreted as the prior
observation count on the number of times words are sampled
from a topic before any word from the corpus is observed.
This choice can smooth the word distribution in every topic
with the amount of smoothing determined by β. The authors
showed that good choices for the hyperparameters α and
β will depend on number of topics T and vocabulary size
W , and that accordingly to the results of their empirical
investigation α = 50T and β = 0.01 to work well with many
different document collections.
Topic extraction, i.e. estimation of the topic-word dis-
tributions and topic distributions for each document, can
be implemented through different algorithms. Hofmann [6]
used a direct estimation approach based on the Expectation-
Maximization (EM) algorithm. However, such an approach
suffers from problems involving local maxima of the likeli-
hood function. A better alternative has been proposed by Blei
et al. [7] which directly estimate the posterior distribution
over z given the observed words w. However, many text
collections contain millions of word tokens and thus the
estimation of the posterior over z requires the adoption of
efficient procedures. Gibbs sampling, a form of Markov
Chain Monte Carlo (MCMC), is easy to implement and
provides a relatively efficient method of extracting a set of
topics from a large corpus.
It is worthwhile to mention that topic extraction can be
performed when the number of topics T is given. However,
in many cases this quantity is unknown and we have to
resort to empirical procedures and/or to statistical measures
such as perplexity to choose the optimal number of topics
to be retained after the unsupervised learning task has been
performed.
2.2. Text Categorization
Document classification, the task of classifying natural
language documents into a predefined set of semantic cate-
gories, has become one of the key methods for organizing
online information. It is commonly referred to as text
applications such as: web pages categorization, newswire
filtering and automatic routing of incoming messages at call
centers, ...
Text categorization is distinguished into binary, multi-
class and multi-label settings. In the binary setting there
are exactly two classes, i.e. relevant and non relevant, spam
and non spam or sport and non sport. Some classification
tasks require more than two classes, i.e. an e-mail routing
agent at a call center need to forward an incoming message
to the right operator depending on the specific nature of
the message contents. Such cases belong to the multi-class
setting where documents can be labeled with exactly one out
of K classes. Finally, in the multi-label setting there is no
one-to-one correspondence between class and document. In
such a setting, each document can belong to many, exactly are judged to be too frequent/rare within a document
one or no category at all. and/or across the document corpus. This software com-
Several supervised learning models have been described ponent exploits a general purpose Italian vocabulary to
in the specialized literature to cope with text categorization. obtain the word-document matrix following to the bag-
However, the most studied models are Support Vector Ma- of-words model. Furthermore, the following document
chines and Naive Bayes. representations are allowed; binary, i.e. a word token
The Support Vector Machines (SVMs) approach has been belongs or not to a given document, term frequency,
proposed by Joachims [11] and has been extensively investi- i.e. how may times a word token is mentioned in a
gated by many researchers [12], [13], [14], [15] to mention given document, and term frequency inverse document
a few. The SVM model has been further extended to cope frequency first introduced in [28]. The following docu-
with cases when little training data is available. For example, ment formats are valid inputs for the software system;
a news-filtering service, which requires thousands of labeled pdf, word and txt. The user is allowed to interact with
data, it is very unlikely to please even the most patient user. the Text Pre-Processing software component through
To cope with such a case the Transductive Support Vector the GUI depicted in Figure 1.
Machines (TSVMs) is presented and discussed in [16] and
• Topic Extractor. This software component offers topic
further developed in [17], [18], [19].
extraction and topic selection functionalities. The topic
Several works have extensively studied the Naive Bayes
extraction functionality is implemented through a cus-
model for text categorization [20], [21]. However, these pure
tomized version of the Latent Dirichlet Allocation
Naive Bayes classifier models considered a document as a
(LDA) model [10]. LDA learning, i.e. topic extraction,
binary feature vector, and so they cannot utilize the term
is obtained by using the Gibbs sampling algorithm
frequencies in a document, resulting in poor performances.
which has been implemented in the C++ programming
The multinomial Naive Bayes text classifier has been shown
language on a single processor machine. The topic
to be an effective alternative to the basic Naive Bayes model
selection functionality assists the user in choosing the
by a number of researchers [22], [23], [24].
optimal number of topics to be retained. Topic selec-
However, the same researchers have also given disap-
tion is implemented through a hierarchical clustering
pointing results compared to many other statistical learning
procedure based on the symmetrized Kullback Liebler
methods, such as nearest neighbor classifiers [25], support
distance between topic distributions [8]. Each retained
vector machines [11] and boosting [26]. Recently Sang-
topic z = j is summarized through the estimate of
Bum Kim et al. [27] revisited the naive Bayes frame-
its prior probability P (z = j), a sorted list of its
work and proposed a Poisson Naive Bayes model for text
most frequent words w, together with the estimate
classification with a statistical feature weighting method.
of their conditional probabilities of occurrence given
Feature weighting has many advantages when compared to
the topic, i.e. the value of the conditional probability
previous feature selection approaches, especially when the
P (w|z = j).
new training examples are continuously provided. In this
paper a Multi-Net Poisson Naive Bayes model has been
used to implement the multi-label document classification
functionality.

3. The Software System

The software system described in this paper consists of

three main components; namely Text Pre-processor, Topic
Extractor and Multi-label Classifier. These components,
which are also available as stand-alone applications have
been integrated to deploy a software system working on
Windows XP and Vista operating systems. A brief descrip-
tion of the aims and functionalities offered by the three
software components follows.

• Text Pre-processor. This software component im-

plements functionalities devoted to document pre-
processing and document corpus representation. It of-
fers stopwords removal, different word stemming op-
tions and several filters to exclude those words which Figure 1. Text Pre-Processor GUI.
 • Multi-label Classifier. Implements a supervised multi-
label classification model. This model exploits the
output from the Topic Extractor software component,
i.e. conditional probability distribution of words given
the topic. It uses a customized version of the Multi-
Net Poisson Naive Bayes (MNPNB) model [27]. The
MNPNB model allows to select the following bag-
of-words representations; binary, term frequency and
term frequency inverse document frequency. Each new
document, represented according to the bag-of-words
model, is automatically labeled depending on the user
specified value for the posterior threshold. If the pos-
terior probability of a given topic is greater than the
user specified posterior threshold, then the document
receives the label associated with the considered topic.
This software component has been implemented in C#
and is available through dedicated GUIs (Figure 2 and Figure 3. MNPNB classifier: results GUI.
Figure 3) as well as a WEB service.
S ALUTE , S CIENZA , S OCIETA , S PORT, T EMPO L IBERO.
4. The Italian Google Directory Each first level topic is associated with a second and third
The performance of the software system is investigated level sub-topic structure summarized in a words list.
using a document corpus collected by exploiting the topics
structure offered by the Italian Google Directory (gDir)1 . 4.1. Document Corpus
This topic structure (Figure 4) relies on the Open Directory
Project (DMOZ) which manages the largest human-edited The document corpus has been collected by submitting a
directory available on the web; each branch of the directory set of 273 queries to the Google search engine. Each query
can be extended only by editors having a deep knowledge contains a pair of words, randomly selected from the union
about the specific topic to be edited. Furthermore, the of the words lists associated to second and third level sub-
editors’ community guarantees fairness and correctness of topic structures.
the gDir topics structure.
The topics associated with the first level of gDir are
the following; ACQUISTI , A FFARI , A RTE , C ASA , C OM - Some examples of the submitted random queries, are as
PUTER , C ONSULTAZIONE , G IOCHI , N OTIZIE , R EGIONALE , follows; "veicoli realtá virtuale", "body art
1. http://www.google.com/Top/World/Italiano/.

Figure 2. MNPNB classifier: labeling GUI. Figure 4. The Italian Google Directory.
altre culture" and "multimedia politica". • P (S ALUTE) = 0.0787,
The PDF filter, offered by the Google search engine, has • P (C OMPUTER) = 0.0696,
been used to ensure that only pdf format files are retrieved. • P (T EMPO L IBERO ) = 0.0884
The random query process retrieved 14,037 documents, each • P (A MMINISTRAZIONE) = 0.1021.
associated with one or more gDir first level topics. Furthermore, each topic is summarized in a words list,
whose 20 most frequent words are reported in Table 1 and
4.2. Text Pre-processing Table 2. It is worthwhile to mention that for each pair
of words wi and topics j, an estimate of the conditional
The document corpus, consisting of the 14,037 retrieved probability of the word wi given the topic j, i.e. P (wi |j)
documents, has been submitted to the Text Pre-processor (Eq. 1), is provided.
software component. Therefore, PDF files have been trans-
formed to plain text, submitted to stopwords removal and 4.4. Multi-Label Classification
to word stemming. Furthermore, size-based file selection
has been applied to include only those PDF files with size The performance of the software system, as a whole, has
between 2 and 400 KB. been estimated by submitting a new document corpus to
The obtained document corpus consists of 10,056 doc- the Multi-label Classifier. This document corpus has been
uments (D = 10, 056). The global vocabulary, which has
been formed by including only those words occurring at
Table 1. S ALUTE and C OMPUTER.
least in 10 and in no more than 450 documents, consists of
48,750 word tokens (W = 48, 750). The document corpus S ALUTE 0.0787 C OMPUTER 0.0696
is represented in a word-document matrix W D consisting of cellule 0.0032 blog 0.0063
emissioni 0.0029 google 0.0035
W × D (term frequency) elements (48, 750 × 10, 056). The nutrizione 0.0028 linux 0.0034
word-document matrix is inputed to the Topic Extractor molecolare 0.0026 copyright 0.0033
software component as described in the next subsection. proteine 0.0022 wireless 0.0030
dieta 0.0022 source 0.0029
climatici 0.0021 access 0.0028
4.3. Topic Extraction foreste 0.0021 client 0.0027
cancro 0.0021 multimedia 0.0027
aids 0.0020 hacker 0.0026
The Topic Extractor software component has been in- disturbi 0.0020 password 0.0026
voked with the following learning parameters; 12 topics infermiere 0.0019 giornalismo 0.0025
cibi 0.0019 browser 0.0023
(T = 12), alpha prior equal to 1.67 (α = 1.67), which tumori 0.0019 provider 0.0022
implements the α = 20 T rule cited in [7], beta prior equal veterinaria 0.0018 telecom 0.0022
to 0.01 (β = 0.04), which implements the β = 200 W rule
obesitá 0.0018 brand 0.0022
clinico 0.0018 book 0.0021
cited in [7]. The Gibbs sampling procedure has been run serra 0.0017 chat 0.0021
100 times with different initial conditions and different virus 0.0017 wiki 0.0021
initialization seeds. Each run consisted of 500 sampling infezioni 0.0017 piattaforme 0.0021
iterations. The topics extracted through the last 99 runs of
the Gibbs sampling learning procedure have been re-ordered Table 2. T EMPO L IBERO and A MMINISTRAZIONE.
to correspond as best as possible with the topics obtained T EMPO L IBERO 0.0884 A MMINISTRAZIONE 0.1021
through the first run. Correspondence was measured by sconto 0.0077 locazione 0.0021
means of the simmetrized Kullback Liebler distance. aeroporto 0.0038 federale 0.0021
salone 0.0035 direttivo 0.0021
The 12 topics extracted from the Topic Extractor soft- spiaggia 0.0028 finanze 0.0020
ware component have been summarized through their cor- lago 0.0028 versamento 0.0020
responding 500 most frequent words. Among the extracted colazione 0.0026 lire 0.0019
albergo 0.0025 commi 0.0019
topics, the four most interesting ones have been manually vacanza 0.0025 prescrizioni 0.0018
labeled as follows; piscina 0.0024 vietato 0.0018
vini 0.0023 contrattuale 0.0018
• S ALUTE (medicine and health), bagni 0.0023 richiedente 0.0018
• C OMPUTER (information and communication technolo- voli 0.0021 utilizzatore 0.0017
gies), pensione 0.0021 agevolazioni 0.0017
biglietto 0.0020 contabile 0.0017
• T EMPO L IBERO (travels and holidays), notti 0.0020 appalto 0.0017
• A MMINISTRAZIONE (bureaucracy, public services). escursioni 0.0020 affidamento 0.0017
agevolazioni 0.0020 redditi 0.0017
The structure of the four topics, is described in Table 1 archeologico 0.0019 sanzione 0.0017
and Table 2. Each topic is associated with an estimate of its piatti 0.0019 somme 0.0016
prior probability; bicicletta 0.0019 indennitá 0.0016
collected by using the same random querying procedure
described in subsection 4.1. Its documents have been man-
ually labeled, according to the 12 first level gDir topics,
independently by three humans.
The labeled document corpus consists of 1,012 docu-
ments. It is worthwhile to mention that each document can
be associated with one or more labels, i.e. a document can
mention one or more topics. In detail, 478 documents are
singly labeled, 457 are doubly labeled, while only 77 are
associated with three labels.
The Multi-label Classifier, queried by using the binary
document representation and by setting a posterior threshold
equal to 0.5, achieves an accuracy equal to 73%, which
can be considered satisfactory. The estimates of precision
and recall for the four selected topics; namely C OMPUTER,
S ALUTE, A FFARI and T EMPO L IBERO, are reported in Table
3.
The best result is achieved for the topic A MMINIS -
TRAZIONE , where the precision equals 92%, i.e. if the
Multi-label Classifier labels a document with the label A M -
MINISTRAZIONE , then the labeling is wrong with respect to
the manual labeling with probability 0.08. Furthermore, the
recall equals 79% which means that the documents manually
labeled with A MMINISTRAZIONE are correctly labeled by
the Multi-label Classifier with probability 0.79. The topic
C OMPUTER achieves a precision value equal to 85% which
is slightly lower than that achieved for A MMINISTRAZIONE.
However, the achieved recall value drops from 79% of
A MMINISTRAZIONE to 59%. The topic T EMPO L IBERO
achieves a precision value equal to 78%, i.e. slightly lower
than those achieved for C OMPUTER, while the achieved
recall value is equal to 44%. Thus, the achieved recall value
is significantly lower than that achieved for C OMPUTER.
Finally, the topic S ALUTE achieves performances compa-
rable to those of T EMPO L IBERO. Indeed, the precision
equals 76%, while the recall equals 41%. Around 57% of
documents manually labelled with T EMPO L IBERO and/or
with S ALUTE, are not identified as such by the Multi-label
Classifier.
It is worthwhile to notice that for each label the achieved
recall value is consistently lower than the precision value.
A possible explanation for this behavior is as follows; the
manual labeling procedure is both complex and ambiguous;
it could label documents by using a broader meaning for
each topic. Therefore, it is expected somewhat that auto-
matic document classification could not achieve excellent
performance with respect to both precision and recall. It is
expected that the ‘purer’ a topic is, the better the perfor-
mance achieved for the automatic documents classification
task will be.
However, it is important to keep in mind the difficulty
of the considered labeling task, together with the fact that
human labeling of documents can result in ambiguous and
contradictory label assignment.
Table 3. Precision/Recall for S ALUTE (S AL .),
C OMPUTER (C OM .), T EMPO L IBERO (T E L.) and
A MMINISTRAZIONE (A MM .).
S AL . C OM . T E L. A MM .
Precision 76 85 78 92
Recall 41 59 44 79
5. Conclusions and Future Work
The overwhelming amount of textual data calls for ef-
ficient and effective methods to automatically summarize
information and to extract valuable knowledge. Text Mining
offers a rich set of computational models and algorithms
to automatically extract valuable knowledge from huge
amounts of semi and un-structured data.
In this paper a software system for topic extraction and
document classification has been described. The software
system assists the user in correctly discovering which main
topics are mentioned in a document collection. The dis-
covered topic structure, after being user-validated, is used
to implement an automatic document classifier. This model
suggests labels to be used for each new document submitted
by the user. It is important to mention that each document is
not restricted to receive a single label but can be labeled with
more topics. Furthermore, each topic, for a given document,
is associated with a probability value that informs the user
about the fitting of the topic to the considered document.
This feature offers an important opportunity to the user who
can sort his/her document collection in descending order of
probability for each topic.
However, it must be clearly stated that many improve-
ments can be achieved by taking into account specific
user requirements. This aspect is under investigation, and
particular attention is being dedicated to non-parametric
models for the discovery of hierarchical topic structures.
Finally, the interplay between topics taxonomies and topic
extraction algorithms offers an interesting research direction
to explore.
References
[1] R. Feldman and J. Sanger, The Text Mining Handbook. New
York: Cambridge University Press, 2007.
[2] E. Wigner, “The unreasonable effectiveness of mathematics
in the natural sciences,” Communications in Pure and Applied
Mathematics, vol. 13, no. 1, pp. 1–14, February 1960.
[3] A. Halevy, P. Norvig, and F. Pereira, “The unreasonable
effectiveness of data,” IEEE Intelligent Systems, vol. 24, no. 2,
pp. 8–12, 2009.
[4] M. W. Berry and M. Castellanos, Survey of Text Mining II:
clustering, classification and retrieval. London: Springer,
2008.
 [5] T. L. Griffiths and M. Steyvers, “A probabilistic approach to
semantic representation,” in Proceedings of the Twenty-Fourth
Annual Conference of Cognitive Science Society, G. W. and
C. Schunn, Eds., 2002, pp. 381–386.
[6] T. Hofmann, “Probabilistic latent semantic analysis,” in In
Proc. of Uncertainty in Artificial Intelligence, UAI99, 1999,
pp. 289–296.
[7] D. M. Blei, N. Andrew, and M. I. Jordan, “Latent dirichlet
allocation,” Journal of Machine Learning Research, vol. 3,
pp. 993–1022, January 2003.
[8] T. L. Griffiths and M. Steyvers, Probabilistic Topic Models,
S. D. . W. K. T. Landauer, D. McNamara, Ed. Erlbaum,
2007.
[9] T. Hofmann, “Unsupervised learning by probabilistic
latent semantic analysis,” Machine Learning, vol. 42,
no. 1-2, pp. 177–196, 2001. [Online]. Available:
http://portal.acm.org/citation.cfm?id=599631
[10] T. L. Griffiths and M. Steyvers, “Finding scientific topics.”
Proc Natl Acad Sci U S A, vol. 101 Suppl 1, pp. 5228–5235,
April 2004.
[11] T. Joachims, Learning to Classify Text Using Support Vector
Machines: Methods, Theory and Algorithms. New York:
Springer, 2002.
[12] R. Cooley, “Classification of news stories using support vector
machines,” in Proc. 16th International Joint Conference on
Artificial Intelligence Text Mining Workshop, 1999.
[13] H. Lodhi, C. Saunders, N. Cristianini, C. Watkins, and
B. Scholkopf, “Text classification using string kernels,” Jour-
nal of Machine Learning Research, vol. 2, pp. 563–569, 2002.
[14] L. Chen, J. Huang, and Z.-H. Gong, “An anti-noise text
categorization method based on support vector machines,” in
AWIC, 2005, pp. 272–278.
[15] A. Lehmann and J. Shawe-Taylor, “A probabilistic model
for text kernels,” in ICML ’06: Proceedings of the 23rd
international conference on Machine learning. New York,
NY, USA: ACM, 2006, pp. 537–544.
[16] V. N. Vapnik, The Nature of Statistical Learning Theory
(Information Science and Statistics). Springer, 1999.
[17] T. Joachims, “Transductive inference for text classification
using support vector machines,” in Proceedings of ICML-
99, 16th International Conference on Machine Learning,
I. Bratko and S. Dzeroski, Eds. Morgan Kaufmann Pub-
lishers, San Francisco, US, 1999, pp. 200–209.
[18] S. Tong and D. Koller, “Support vector machine active
learning with applications to text classification,” Journal of
Machine Learning Research, vol. 2, pp. 45–66, November
2001.
[19] C. Xu and Y. Zhou, “Transductive support vector machine for
personal inboxes spam categorization,” Computational Intelli-
gence and Security Workshops, International Conference on,
vol. 0, pp. 459–463, 2007.
[20] A. Mccallum and K. Nigam, “A comparison of event models
for naive bayes text classification,” in In AAAI-98 Workshop
on Learning for Text Categorization. AAAI Press, 1998, pp.
41–48.
[21] D. D. Lewis, “Naive (bayes) at forty: The independence
assumption in information retrieval,” in ECML ’98: Proceed-
ings of the 10th European Conference on Machine Learning.
London, UK: Springer-Verlag, 1998, pp. 4–15.
[22] S. Dumais, J. Platt, D. Heckerman, and M. Sahami, “Inductive
learning algorithms and representations for text categoriza-
tion,” in CIKM ’98: Proceedings of the seventh international
conference on Information and knowledge management. New
York, NY, USA: ACM Press, 1998, pp. 148–155.
[23] Y. Yang and X. Liu, “A re-examination of text categorization
methods,” in SIGIR ’99: Proceedings of the 22nd annual
international ACM SIGIR conference on Research and de-
velopment in information retrieval. New York, NY, USA:
ACM Press, 1999, pp. 42–49.
[24] A. K. McCallum and T. Mitchell, “Text classification from
labeled and unlabeled documents using em,” in Machine
Learning, 2000, pp. 103–134.
[25] Y. Yang and C. G. Chute, “An example-based mapping
method for text categorization and retrieval,” ACM Trans. Inf.
Syst., vol. 12, no. 3, pp. 252–277, 1994.
[26] R. E. Schapire and Y. Singer, “Boostexter: a boosting-based
system for text categorization,” in Machine Learning, 2000,
pp. 135–168.
[27] H.-C. R. H. M. Sang-Bum Kim, Kyoung-Soo Han, “Some
effective techniques for naive bayes text classification,” IEEE
Transactions on Knowledge and Data Engineering, vol. 18,
no. 11, pp. 1457–1466, January 2006.
[28] G. Salton and M. J. Mcgill, Introduction to Modern Infor-
mation Retrieval. New York, NY, USA: McGraw-Hill, Inc.,
1986.