CS784: Recent Trends in Web Search
Sreenivasa Pavan Kuppili
May 17, 2008
Abstract
Web Search is an integral part of our lives today. The
advent of Google in the late 90’s revolutionalized search.
In this decade, there has been a lot of research on Web
search. The landscape is pretty diverse today, with people
exploring in various directions like social search, clus-
tering, personalized search, local search, vertical search,
object search, question-answering, etc. In this paper, we
look at some of the key ideas in these various fields (We
focus on text or information search, and not on audio or
video search). The goal is to give a flavor of these various
directions currently being explored in Web Search.
1 Introduction
The early Web Search engines have their origins in in-
formation retrieval research. However, the Web is far
more diverse and bigger than the specialized document
collections used in Information Retrieval research. Be-
sides, anyone is allowed to author articles on the Web.
Traditional information retrieval algorithms do not work
well on the Web. As a result, Web Search engines had to
evolve to meet the new requirements. One of the seminal
papers in this direction is [1] in which the link structure of
the Web was exploited to measure the global importance
of pages, independent of the query. The advent of Google
revolutionalized search. It made search ubiquitous. Per-
haps, more importantly, the advertising model adopted by
companies like Overture and Google proved to be very ef-
fective. The commercial success of Google made search
a very hot topic, both in academia and the industry. From
early 2000’s to date, there has been plenty of research
on Web Search, and there have been a plethora of Web
Search startups, trying to be the next Google or capture
1
niche markets. As can be expected, people have started
suggesting improvements to Web Search along many dif-
ferent dimensions. The Web Search landscape is pretty di-
verse today. There are several visions about the future of
Web Search, with no clear winner at this point. In this pa-
per, we will look at the various directions in which search
is evolving: some of these are social search, clustering
documents and results, personalized search, local search,
vertical search, object search, question-answering com-
munities, Semantic Web, etc. We look at the key ideas in
these different fields.
The paper is organized as follows. In section 2, we
look at the different classes of search queries and how the
search engines have been evolving to meet the require-
ments of these different classes of user queries. In sec-
tion 3, we examine the ideas in object-level search and
in vertical search. Web 2.0 and mass collaboration are
very popular today, and in section 4, we examine how
the user communities and social tagging can be used to
improve the quality of Web Search. In section 5, we ex-
amine how the search results can be personalized. Per-
sonalization can be seen as a way of biasing the search
results towards the user’s preferences. In section 6, we
examine how clustering of documents or queries might be
useful in the context of Web Search. In section 7, we see
how question-answering communities have the potential
to become good complements to Web search engines, and
examine current efforts of automatically answering ques-
tions by using the Web as a knowledge repository. In sec-
tion 8, we look at the Semantic Web and one of the first
generation Semantic Web search engines. In section 9, we
look at the different kinds of user interfaces, besides the
traditional ranked list of documents. Finally, we end with
our conclusions in section 10.
2 Taxonomy of Web Search queries
Let us look at how the web queries can be broadly clas-
sified, and how the search engines have evolved to meet
the requirements of these different kinds of queries. [2]
says that the queries can be broadly classified into three
categories: 1. Navigational: The main goal of the query
is to go to a site. (ex “Greyhound bus”). 2. Informational:
The goal is to acquire some information present on one
or more pages. 3. Transactional: The goal is to perform
some activity (like booking a flight). While the distinc-
tion can be blurred for queries, it is useful to see the evo-
lution of search engines as a response to meet these dif-
ferent kinds of queries. The first generation of search en-
gines (like AltaVista, Excite, Webcrawler) were primarily
focused on answering informational queries. Search en-
gines post Google (after 1998) are good at answering nav-
igational and informational queries. Link analysis and an-
chor text [3] are important aspects in doing a good job on
navigational queries. Today, we are seeing the emergence
of the third generation of search engines, where the search
engines are trying to help the user accomplish his task as
easily as possible. For instance, if you type “San Fran-
cisco” in a search engine today, you just do not get back a
bunch of pages. Instead you might get a map of San Fran-
cisco, weather information, possibly a site to book flight
(or even a form to automatically search for flight reser-
vations) and hotel reservations in San Francisco, etc. The
search engines today are trying to minimize the user effort
as the user goes about trying to finish his task. To achieve
this, search engines need to do semantic analysis of the
webpages in the background, and need to understand the
intent of the user query. This is a very important area
of current research: trying to extract information from
the webpages in the background (for instance, using In-
formation Extraction techqiues to extract information like
names, organizations, etc), and trying to understand the
user intent at query time. We will see in the next few
sections how the keyword search is often very ambigu-
ous, and how the search engines use additional informa-
tion to disambiguate the user intent. Search engines have
also started inspecting the so-called “deep web” which is
the information locked in various databases and available
through interfaces like web forms [4]. The Semantic Web
might also be viewed as an attempt to try to solve these
transactional queries. In the next few years, we might see
2
the search engines evolve to try to satisfy the transactional
queries in a better way.
3 Object-level Vertical Search
Vertical search is an important new paradigm. The ba-
sic premise is the search engine can be customized to a
particular domain like health, finance etc, and the vertical
search engine can perform better than a general purpose
search engine in that domain. Another new paradigm is
searching at an object level rather than at the webpage
level. Often times, users are interested in information
about a particular object, and in today’s search engine,
the user needs to extract this object information from the
returned pages. So, object-level search is an attempt to
bring the search engine closer to the user’s needs. For
instance, if you take a domain like academia, the vari-
ous objects can be researchers, papers, conferences etc.
A user might be interested in questions like who are the
most popular researchers in Machine Learning. It’s diffi-
cult for a search engine like Google to answer this query,
while this is a tailor-made query for an object-level verti-
cal search engine. The object-level vertical search archi-
tecture discussed in [11] conatins three main components:
1. Extracting the domain specific objects and relation-
ships from the crawled webpages. 2. Object Aggrega-
tion. 3. Object level ranking. The first component is the
process of looking at the crawled webpages, and extract-
ing objects. HMM’s were typically used for information
extraction. The current state of the art is to use condi-
tional random fields which typically outperform HMM’s
because of their ability to capture non-independent and
overlapping fields. This paper uses hierarchical condi-
tional random fields for object extraction. The second
component is the object aggregation stage, which includes
entity resolution (deciding that two objects point to the
same real world entity), and object disambiguation (two
objects with the same name refer to different real world
entities). In this component, they use the web as a repos-
itory and look for co-occurrences to decide on entity res-
olution and object disambiguation. For example, if two
papers appear in the website of a small research lab, and
if there is an author with the same name in the two papers,
it is highly likely that the two names refer to the same per-
son. The third component is deciding the popular objects.
Here, we can exploit the link structure to decide the pop-
ularity by a technique similar to page rank. The main dif-
ference is here the links might have different semantics,
while in deciding webpage popularity by PageRank, all
the links have the same semantics. As an example, one
link coming into a paper might be the “authored-by” link,
and a second link might be a “citation” link. A “citation”
link is definitely more important than an “authored-by”
link, because the importance of a paper is more dependent
on the number of citations, and it is almost independent
of the number of authors. So, in this paper, a popularity
propagation factor (PPF) is added to each type of link to
capture the importance of that type of link. So with this
architecture, we have an object-level vertical search en-
gine in place. Now the users can do both keyword kind of
searches, and the more advanced users can also do SQL-
styled queries to get more accurate and focussed answers.
(This paper is similar in spirit to the DBLife project [12],
an ongoing research project at UW-Madison, which is an
object-level community portal in the database field). This
work has been implemented in two working systems: Li-
bra Acdemic Search (http://libra.msra.cn) and Windows
Live Product Search (http://products.live.com). Vertical
search engines are definitely very popular today, and we
will surely see more of vertical search in the next few
years, as the different startups try to capture niche mar-
kets.
4 Social Search
For lack of better algorithms, image and video search is
still at a stage where we search by the metadata rather
than the actual content. It is quite common in image and
video search to search according to the annotations added
by users. There have also been efforts to automatically
get people to label images by representing the problem
as a game [13]. In [14], the authors consider the prob-
lem of using social annotations to improve the quality of
web search. Today, there are a lot of popular social book-
marking sites like de.li.ci.ous.com, and these annotations
can potentially be leveraged to improve the search qual-
ity. Two main ways of using social annotations are: 1.
Just like anchor text was utilized to provide additional in-
formation about webpages, the social annotations often
provide a multi-faceted description of the webpage, and
3
we can compute a similarity between the annotations and
the query terms. One can also exploit the relation that
annotations on the same webpage might be semantically
similar. The paper proposes Social Similarity Ranking
(SSR), an iterative algorithm to compute the similarities
among the various annotations. So, one can match the
query terms not only with the actual annotations, but also
annotations which are semantically similar to the actual
annotations of the page. 2. The annotations can also be
used as a measure of page importance. The paper pro-
poses SocialPageRank, an iterative algorithm which mu-
tually enhances relations among popular web-pages, upto
date users, and hot social annotations to measure the page
importance. The social page rank can be viewed as a com-
plement to traditional algorithms like PageRank. While
PageRank exploits the link structure of how the webpages
are linked by the page creators, social page rank captures
the importance of webpages from the annotator’s point of
view. The results in the paper show promise that similar-
ity ranking based on social annotations, and page ranking
by SocialPageRank, can be integrated with the current ap-
proaches to yield a better Web Search performance. The
authors do point out several challenges in this approach.
For instance, a lot of the webpages might have no social
annotations. Another major problem is malicious users.
As social annotations become an important part of Web
search, it will be accompanied by users who will try to
insert spurious annotations for their own ends. One can
monitor and identify the malicious users, and employ lin-
guistic and statistical techniques to filter out these spuri-
ous annotations, and this is an important direction of fu-
ture research.
[25] advocates combining social search and social nav-
igation among users of a particular community. The paper
argues that the resulting interface seamlessly integrates
the search and navigation tasks. Social navigation means
how the browsing behavior of past users guides the other
users towards particular websites. Social navigation in an
information space was originally introduced by Dourish
and Chalmers as moving towards cluster of people or se-
lecting objects because others have been examining them
[26]. The classic example of social navigation in a hy-
pertext environment was provided later by Wexelblat and
Maes in the Footprints system, which visualizes usage
paths in a Web site [27]. For instance, a community of
students searching for papers on a topic can follow the
trails of past students to locate papers. On the other hand,
Social search (in the context of community users) is the
way in which the ranking of search results can be modi-
fied to suit the particular needs of a community. This can
be viewed as another way of adding context to a query.
The query can be disambiguated by the interests of that
community. For instance, for a computer science commu-
nity, Sun probably means the computer science company,
and not the English newspapaer. This is essentially done
in this paper by keeping track of the community’s previ-
ous searches, and pushing ahead the previous user-clicked
results. This makes use of the fact that there can be a lot
of query repitions within a community [28]. The navi-
gation and search are integrated in the following ways:
While presenting search results, the interface can present
both previous navigation and search results as visual cues
to help the user choose the links to click among the re-
sults. For instance, the visual cues might indicate related
queries for the particular result (other queries which led to
the same result), the last time the result was encountered
(giving an idea of the freshness of the result), relevance of
the result for this query (how often the community users
select this result for the given query), the number of anno-
tations for the result by the community members, etc. Be-
sides, the browing interface is complemented with search
information. For example, when the user has navigated to
a document, the interface can give the list of queries by
the community users which lead them to this document.
Using these queries, the current user can find similar doc-
uments. Thus, This paper aims at bridging the domains of
navigation and search and allowing users to move more
fluidly and naturally between information sources via a
richer form of linkages. This paper talks about searches
of users within a single community. However, in social
networks like Facebook, a user might belong to multi-
ple communities, and the solution needs to be extended
to deal with this case. Nonetheless, the paper is a proof-
of-concept of how the searches and browsing by members
within a community might be leveraged to improve the
searching and browsing experience of other users of the
community.
4
5 Personalization
One of the problems with today’s Web Search is that the
user queries usually have a very small number of key-
words, and it is hard to ascertain the exact user intent. A
number of studies have shown that the search queries of-
ten tend to be short, underspecified [5], and consequently
ambiguous [6]. One way to add some context to the query
is personalization. The context of the query can help dis-
ambiguate the multiple interpretations of the query. There
are many ways in which personalization can be achieved.
One can imagine extracting information about a user’s
preferences from his search history (or other information
like the information on his local machine, his currently
opened files, etc), and focus on that part of the Web which
the user is possibly interested in. One way of achieving
personalization is by customizing the PageRank for each
user. For instance, [1] suggests that the random jumps
in the random surfer model can be limited to some sub-
set of pages (like a user’s bookmarks, or previously vis-
ited pages) to bias the Pagerank towards the user’s pref-
erences. For instance, if a person views a lot of Ma-
chine Learning pages, the personalized pagerank might
give higher weight to the homepage of Michael Jordan,
the researcher, than pages about Michael Jordan, the bas-
ketball player. If a personalized PageRank is computed
at the granularity of pages, it runs into problems of scal-
ability. [24] talks about how to scale personalized web
search using graph theoretic results, and how to decom-
pose a personalized page rank vector into multiple par-
tial vectors, and share these partial vectors across multiple
personalized views to achieve scalability.
[7] presents a way of analyzing a user’s click history
to improve the search performance. This work essen-
tially builds on Topic-Sensitive PageRank proposed in [8].
Topic-Sensitive PageRank computes multiple PageRank
values, one for each first-level topic in the Open Directory
[23]. These values are computed by changing the ran-
dom jump in PageRank computation to go to only pages
within the same topic. Once we have these topic specific
pagerank values, at search time, based on an estimate of
which topic the query is about, the sarch engine can use
the appropriate PageRank vector. In this paper [7], the
probability of the query belonging to a topic is computed
based on both the query and the user’s topic interest vec-
tor T (his interest in the different topics). It can be noted
that here the user’s preferences are being modelled at the
topic level rather than at the page level (this is computa-
tionally much more feasible). T is estimated by using the
user’s click history, and a topic-driven searcher model. In
the topic-driven searcher model, a user chooses a topic
t with probability according to his topic interest vector
T , and issues a query from that topic. The search en-
gine returns pages ranked by the Topic-sensitive Pager-
ank (T SP R), and the user clicks them according to the
model in [9]. Now, we can connect the vector V (V (p) is
the probability that a user clicks a page p) with the topic-
senstive pagerank and the topic interest vector by V (p) =
Σmi=1 T (i) ∗ [T SP Ri (p)]
9/4
. Vector V is known from the
user’s click history, and TSPR can be calculated as in [8].
The missing vector T is estimated using maximum likeli-
hood estimation. Now at search time, given a query q, we
need to compute which topic it belongs to. So, we need
to find P r(T (i)|q) which by Bayes rule is proportional to
P r(T (i)) ∗ P r(q|T (i)). So, the user’s topic interest will
serve as a prior in the Bayes model. Combining this with
TSPR, the personalized oage rank for a query q becomes
P P RT (p) = Σm i=1 T (i).P r(q|T (i)).T SP Ri (p). So, this
is a way of incorporating the user’s preferences to get a
personalized pagerank, and this will help bias the search
reuslts towards a user’s preferences.
6 Clustering
Clustering of web documents can potentially be used in
many ways. While its impact in improving the qual-
ity of search has been fairly limited, it has been pri-
marily used for improving browsing and improving the
user interface. For instance, a search engine like Clusty
[18], presents the results in the form of clusters. For in-
stance, on typing “Michael Schumacher” in Clusty, we get
back several clusters like “Formula One”, “Ferrari”, “Pic-
tures”, “Retirement”, etc, and the user can look at only
the documents in the clusters in which he is interested,
or jump to relevant clusters. Clusters can also be poten-
tially useful in improving the search quality. One area
in which clustering has been proposed as a useful tool
is in the question-answering community. Finding simi-
lar queries or questions can be very important in search
engines and question-answering communities. [10] ad-
dresses this problem. They combine two similarity fea-
5
tures - one is the standard keyword or string similarity
(augmented with phrase identification). The second sim-
ilarity metric is based on user logs. If two queries result
in similar document clicks, then the queries can be con-
sidered similar. This is an important complement to the
first similarity metric because now similar queries need
not even share words in common. For instance, if “atomic
bomb”, “Nagasaki”, “Manhattan Project”, “Hiroshima”,
all lead to a click on a document titled “Atomic Bomb”,
we can infer that the queries are similar despite having no
words in common. In this paper, they use a linear combi-
nation of the above two similarity metrics to get a similar-
ity between queries, and at least theoretically this should
perform better than either similarity metric in isolation.
However tuning the parameters is left as a manual knob,
and that is a disadvantage. Nonetheless, this is a good step
to identify similar queries and questions. Once we have
a good similarity metric between questions, one can clus-
ter the questions and identify the most frequently asked
questions (FAQ’s), and these can be better answered or
verified by human experts, or if possible, one can indicate
to the machine that these are important queries or ques-
tions. The second similarity metric of using document
clicks to find similarity between queries can also be used
to disambiguate queries with the same word, but different
intent. For instance, two users might have typed “crane”,
but based on the documents they look at, we can see that
one person intended crane-the bird, and another intended
crane-the machine. This kind of query disambiguation
can be useful in a search engine. For instance, the search
engine can provide clusters for a query - with each cluster
containing documents for one interpretation of the query.
7 Question-Answering
One could imagine an ideal search engine as an Omni-
scient Oracle. A user can plug in a question, and the
search engine would return the answer. Currently, there
are two kinds of question answering going on in the Web.
One is question-answering portals where the answers are
generated by humans. For instance, today we have sites
like Ask.com and answers.yahoo.com, and the many com-
munity specific forums which fit into this pattern. The
main advantage of these question-answering communi-
ties is they can answer questions which are difficult for
a search engine to answer. For instance, if something
is wrong with your computer, you can pose the specific
problem on a forum and get the answer, while it is very
hard for a search engine to diagnose your computer prob-
lem. The key problem here is how to find similar ques-
tions, so that when a user poses a question he can be di-
rected to other similar questions for which answers are
available. In section 6, we have seen one way of cluster-
ing similar questions. However, this is still an important
question with no satisfactory solution, and is a hurdle be-
fore the question-answering communities can really take
off. The second way of question-answering in today’s
web is doing it automatically. The process of retrieving
answers from questions is known as Natural Language
Question Answering (NLQA). [15] descibes a probabilis-
tic way of question-answering on the web. They propose
an algorithm which uses a combination of proximity and
question-type features, to answer the questions. The al-
gorithm proceeds in several phases. In the first phase,
the question is converted to an appropriate query, so that
this can be submitted to a search engine like Google. In
the second phase, the algorithm identifies the type of an-
swer expected (like person, place, distance, organization,
etc). The question type identification can be done by us-
ing either Machine Learning, or hand coded heuristics.
The paper claims that the heuristics are currently perform-
ing better than Machine Learning in identifying the ques-
tion types. In the third phase, the query is submitted to a
search engine which returns several documents, some of
which hopefully contain the answer to the question. The
fourth phase is identifying which passages or sentences
in the doucments contain the answers. The purpose of
identifying sentences is to reduce the computational com-
plexity of the next phrase ranking stage. The similarity
between a sentence and the query can be computed us-
ing standard measures like the N-gram approach (normal-
ized by the sentence length), or the vector space model
(for instance, a slight modification of the Okapi ranking
function [16]). In the fifth phase, the relevant passages
or sentences are split up into constituent phrases, each of
which is a potential answer candidate. This splitting into
phrases is done by using an off-the-shelf chunker [17].
Finally in the sixth phase, the algorithm ranks the phrases
extracted in the previous phase, and presents them to the
user. This is the most important phase. The ranking uses
two components. The first component is proximity be-
6
tween the text and the question. The phrase that con-
tains most query terms gets a high score. A phrase that
is near a phrase that contains most query terms will get
a slightly lower score, and phrases further away get lower
scores. The second score is using linguistic analysis. One
can use part-of-speech tagging to tag each word in the
phrase. Lets call that the signature of the phrase. Then
one can estimate the probability that the phrase is of the
same type as the question type, given the signature of the
phrase. For instance, suppose the question type is “Per-
son”, and the phrase is “Hugo Young” which has a sig-
nature “NNP, NNP”. We can then compute the probabil-
ity P r(P erson|N N P, N N P ). Lets call this probability
value the signature score. The total score is the product
of this signature score and the proximity score. While
automatic question-answering is not at the level of being
practically deployed, this is an interesting direction of fu-
ture research. Automatic question-answering might also
help in bootstarpping the question-answer communities.
8 Semantic Web Search
Ideally, we would like the search engine to be able to
perform tasks for us, or perfectly solve the transactional
queries. For instance, if a user wants to plan a trip to
New York, using today’s search, he gets multiple pages
back, and it involves sufficient effort on the part of the
user to compare various flight tickets, look at various ho-
tels etc. before he can finish his task. It would be cool
if the machine can automatically go to the websites, un-
derstand the information, process it, and reduce the user’s
effort. Semantic Web is an effort in this direction. Essen-
tially, each web page will be like a structured database,
and a machine can potentially read and process all this
information. Semantic Web is facing several significant
challenges like dealing with semantic heterogenity, in-
centives, efficient execution, etc. Nonetheless, it is start-
ing to gain momentum in some fields, and it might lead
to pockets of machine-readable Web. Swoogle [19] is
a search engine for the Semantic Web. Swoogle is not
meant to answer complex queries exploiting the semantic
structure and data in the Semantic Web. It is more like
a metadata search engine. It can be used in finding ap-
propriate ontologies, finding instance data corresponding
to an ontology, and finding the characteristics of the Se-
mantic Web like how is it connected, how are the ontolo-
gies referenced, what is the importance level of a semantic
web document (SWD), etc. Swoogle crawls the Web for
SWD’s using 1. specific queries (for instance type ex-
tensions like “.rdf”) on a search engine like Google, 2.
crawling user-supplied sites, and 3. following semantic
links from SWD’s. Once Swoogle crawls the SWD’s, it
extracts metadata from these SWD’s. Swoogle looks at
relationships at the SWD granularity, and not at the RDF
triple granularity (which is much harder and expensive).
It captures relationships like: 1. SWD A imports all the
content in SWD B 2. SWD A uses some of the terms
in SWD B without importing. 3. A extends the defini-
tions of terms defined by B. 4. A makes assertions about
the individuals defined by B. Using these various kinds of
links, Swoogle uses a rational random surfing model to
compute the importance of the SWD’s in a manner sim-
ilar to PageRank. The model is rational because we as-
sume that the user is rational: for instance if SWD A im-
ports content of B, it is expected that from A, the user
goes to B, because B is a part of A. Different weights are
assigned to the 4 kinds of links which are supposed to
reflect the probability of a rational user traversing those
links. Once, we have these weights in place, we com-
pute the rank of individual SWD’s in a manner similar to
PageRank. This ranking gives the importance of individ-
ual SWD’s. For indexing and keyword search, Swoogle
takes the SWD’s, reduces them to their rdf triples, extracts
the URIrefs, and treats an SWD as a bag of URIrefs (as
opposed to the bag of words approach in Web search en-
gines). Once we have these bag of URIrefs, traditional in-
formation retrieval techniques like N-gram approach are
used for keyword search to return ontologies and data in-
stances matching the keywords.
9 Search Interfaces
The user interface is certainly an important part of a
search engine. The clean, uncluttered interface of Google
was one of the reasons why it became popular. However,
the Google way of returning results as a ranked list of
documents is only one among several possible alterna-
tives. (Today, the interface of Google is actually a mix
of documents, images, videos, maps, book results, etc -
nonetheless, it is still a ranked list). Some other search
7
engines have alternative ways of presenting results. For
instance, the search results can be categorized by mean-
ingful and stable categories. The categories can be drawn
from thesaurai, glossaries, or ontologies. The categories
can even be simple categories like document type, size of
the document etc. Flamenco [20] uses multiple sets of
hierarchical categories to aid user browsing and search-
ing. Exalead [21] uses topical categories as well as cat-
egories based on document type and geography. Instead
of static categories, one can also dynamically categorize
the results. For instance, Clusty presents the results in
the form of a dynamic hierarchy of clusters. Findex [22]
presents results categorized by a flat set of clusters. There
are other ways of visually representing the results. For
insatnce, Themescape uses a topographic map metaphor
to plot keywords (shows how the keywords are related
pictorially) extracted from a corpus. Another interesting
way of presenting results is in the form of a map. This is
relevant in fields like academia. For instance, the results
can be presented in the form of a graph with nodes corre-
sponding to the papers, and edges corresponding to cita-
tions. These graphs also can visually help the user identify
the major topics and fields. A graph interface can be use-
ful in other contexts too. For instance, one can imagine
the search results of a general search engine being a small
subgraph, which connects two different webpages, which
together yield the information requested by the user. So,
there are multiple ways in which the results can be pre-
sented to the user, and probably the future search engines
will dynamically choose the most appropriate way of pre-
senting results, based on the query and the user.
10 Conclusions
We have looked at some of the important ways in which
Web Search is currently being explored. Vertical search
is going to be important in the next few years as the vari-
ous startups try to capture vertical markets. Clustering is
more of an academic topic at the moment. Social search
can improve the search quality, by using annotations as
additional metadata, or using annotations to create an al-
ternate view of the importance of pages, or by leveraging
community information to improve the search quality of
other members in the community. Social search has to
deal with the problems like spurious tagging. Personal-
ization is likely to be important in the near future. Person-
alization helps add context to the user queries and results
in better results. Semantic Web is still in a rudimentary
stage, and it faces several challenges before it can become
a useful tool at the Web level. However, the potential
rewards of a Semantic Web are huge. Question-answer
communities can turn out to be a good complement to
search engines. Humans can provide focused and accurate
answers to questions (which are difficult for a search en-
gine to answer). Automatic question-answering is still in
the academic realm at the moment. It is an interesting area
of research, and can also help bootstrap question-answer
communities. So, there are several interesting directions
in which the field of Web search is evolving. It is hard
to say what the next paradigm shift in search is going to
be. Probably in the next few years, the most important
improvement will be a better semantic understanding of
what the web pages mean, and understanding the user in-
tent. This should help the the search engines better an-
swer user queries, particularly transactional queries. The
ultimate search engine will be like an Oracle, which has
a complete understanding of the content on the Web, us-
ing which it can give accurate and focused answers to a
user query. Semantic Web is a small step in that direction.
Possibly, advances in AI, Machine Learning, and Natu-
ral Language Processing might help search engines move
towards that goal.
References
[1] L. Page, S. Brin, R. Motwani, and T. Winograd. “The PageRank
citation ranking: Bringing order to the Web.” Technical Report,
Stanfoed Digital Library Technologies Project, 1998.
[2] Andrei Broder. “A taxonomy of Web Search.” In SIGIR, 2002.
[3] Nick Craswell, David Hawking, and Stephen Robertson. “Effec-
tive site finding using link anchor information.” In SIGIR, 2001.
[4] http://googlewebmastercentral.blogspot.com/2008/04/crawling-
through-html-forms.html
[5] B. Jansen, A. Spink, and T. Saracevic. “Real life, real users, and
real needs: A study and analysis of user queries on the Web.” In-
formation Processing and Management, 2000.
[6] Robert Krovetz and W. Bruce Croft. “Lexical ambiguity and infor-
mation retreival.” In Information Systems, 1992.
[7] Feng Qiu and Junghoo Cho. “Automatic Identification of User In-
terest For Personalized Search.” In WWW, 2006.
[8] T. Haveliwala. “Topic-Sensitive PageRank.” In Proc of the
Eveleventh Intl. World Wide Web Conf, 2002.
[9] J. Cho and S. Roy. “Impact of Web search engines on page popu-
larity.” In Proc. of WWW, 2004.
[10] Ji-Rong Wen, Jian-Yun Nie, and Hong-Jian Zhang. “Clustering
User Queries of a Search Engine.” In WWW10, 2001.
[11] Zaiqing Nie, Ji-Rong Wen, and Wei-Ying Ma. “Object-level Verti-
cal Search.” In CIDR, 2007.
[12] Pedro DeRose, Warren Shen, Fei Chen, Yoonkyong Lee, Doug
Burdick, Anhai Doan, and Raghu Ramakrishna. “DBLife: A Com-
munity Information Management Platform for the Database Re-
search Community.” In CIDR, 2007.
[13] Luis von Ahn and Laura Dabbish. “Labeling images with a com-
puter game.” In SIGCHI, 2004.
[14] S. Bao, X. Wu, B. Fei, G. Xue, Z. Su, and Y. Yu. “Optimizing Web
Search using Social Annotations.” In WWW, 2007.
[15] D. Radev, W. Fan, H. Qi, H. Wu, A. Grewal. “Probabilistic Ques-
tion Answering on the Web” In WWW, 2002.
[16] S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and
M. Gatford. “Okapi at TREC-4.” In Proc. of the Fourth TREC,
1996.
[17] Andrei Mikheev. “Document centered approach to text normaliza-
tion.” In SIGIR, 2000.
[18] Clusty. http://www.clusty.com
[19] Li Ding, Tim Finin, Anupam Joshi, Yun Peng, R. Scott Cost, Joel
Sachs, Rong Pan, Pavan Reddivari, Vishal Doshi. “Swoogle: A
Semantic Web Search and Metadata Engine.” In CIKM, 2004.
[20] K. Yee, K. Swearingen, K. Li, and M. Hearst. “Faceted metadata
for image searching and browsing.” In SIGCHI, 2003
[21] Exalead. http://www.exalead.com
[22] Kaiki. “Findex: search result categories help when document rank-
ing fails.”’ In SIGCHI, 2005.
[23] http://dmoz.org
[24] Glen Jeh and Jennifer Widom. “Scaling Personalized Web Search.”
In WWW, 2003.
[25] Jill Freyne, Rosta Farzan, Peter Brusilovsky, Barry Smith, and
Maurice Coyle. “Collecting Community Wisdom: Integrating So-
cial Search and Social Navigation.” In IUI, 2007.
[26] P. Dourish and M. Chalmers “Running out of space: models of
information navigation.” Proceedings of HCI, 1994.
[27] A. Wexelblat and P. Maes “Footprints: History rich tools for infor-
mation foraging.” In CHI, 1999.
[28] B. Smyth, E. Balfe, J. Freyne, P. Briggs, M. Coyle, and O. Boy-
dell. “Exploiting Query Repitition and Regularity in an Adaptive
Community-based Web Seach Engine.” In User Modelling and
User-Adapted Interaction, 2004.