Page.1Q1.

Explain the meaning of data cleaning and data

formating.
Ans-Data cleaning
This step complements the previous one. It is also the most time
consuming due to a lot of possible techniques that can be
implemented so as to optimize data quality for future modeling stage.
Possible techniques for data cleaning include:
Data normalization. For example decimal scaling into the range
(0,1), or standard deviation normalization.
Data smoothing. Discretization of numeric attributes is one
example, this is helpful or even necessary for logic based methods.
Treatment of missing values. There is not simple and safe solution
for the cases where some of the attributes have significant number of
missing values. Generally, it is good to experiment with and without
these attributes in the modelling phase, in order to find out the
importance of the missing values.
Data formatting
Final data preparation step which represents syntactic modifications to the data that do
not change its meaning, but are required by the particular modelling tool chosen for the
DM task. These include:
reordering of the attributes or records: some modelling tools require reordering of the
attributes (or records) in the dataset: putting target attribute at the beginning or at the
end, randomizing order of records (required by neural networks for example)
changes related to the constraints of modelling tools: removing commas or tabs,
special characters, trimming strings to maximum allowed number of characters,
replacing special characters with allowed set of special characters.
Q2.What is metadata ? Explain various purpose in which
metadata is used.
Ans-Meta data is data about data. Since data in a dataware house is both voluminous
and dynamic, it needs constant monitoring. This can be done only if we have a separate
set of data about data is stored. This is the purpose of meta data.
Meta data is useful for data transformation and load data management and query
generation. This chapter introduces a few of the commonly used meta data functions
for each of them. Meta data, by definition is data about data or data that describes
the data. In simple terms, the data warehouse contains data that describes different
situations. But there should also be some data that gives details about the data stored in
a data warehouse. This data is metadata. Metadata, apart form other things, will be
used for the following purposes.
1. data transformation and loading
2. data management
3. query generation
Q3.Write the steps in designing of fact tables.
Ans-DESIGNING OF FACT TABLES
The above listed methods, when iterated repeatedly will help to finally arrive at a set of
entities that go into a fact table. The next question is how big a fact table can be? An
answer could be that it should be big enough to store all the facts, still making the task
of collecting data from this table reasonably fast. Obviously, this depends on the
hardware architecture as well as the design of the database. A suitable
hardware architecture can ensure that the cost of collecting data is reduced by the
inherent capability of the hardware on the other hand the database designed should
ensure that whenever a data is asked for, the time needed to search for the same is
minimum. In other words, the designer should be able to balance the value of
information made available by the database and cost of making the same data
available to the user. A larger database obviously stores more details, so is definitely
useful, but the cost of storing a larger database as well as the cost of searching and
evaluating the same becomes higher. Technologically, there is perhaps no limit on the
size of the database. How does one optimize the cost- benefit ratio? There are no
standard formulae, but some of the following facts can be taken not of.
i.
Understand the significance of the data stored with respect to
time. Only those data that are still needed for processing need to
be stored. For example customer details after a period of time
may become irrelevant. Salary details paid in 1980s may be of
little use in analyzing the employee cost of 21st century etc. As
and when the data becomes obsolete, it can be removed.
ii. Find out whether maintaining of statistical samples of each of the subsets could be
resorted to instead of storing the entire data. For example, instead of storing the sales
details of all the 200 towns in the last 5 years, one can store details of 10 smaller
towns, five metros, 10 bigger cities and 20 villages. After all data warehousing most
often is resorted to get trends and not the actual figures. The subsets of these individual
details can always be extrapolated
Q3.List and explain the aspects to be looked into while designing the summary
tables.
Ans-ASPECTS TO BE LOOKED INTO WHILE DESIGNING THE
SUMMARY TABLES
The main purpose of using summary tables is to cut down the time taken to execute a
specific query. The main methodology involves minimizing the volume of data being
scanned each time the query is to be answered. In other words, partial answers to the
query are already made available. For example, in the above cited example of mobile
market, if one expects
i) the citizens above 18 years of age
ii) with salaries greater than 15,000 and
iii) with professions that involve traveling are the potential customers, then, every time
the query is to be processed (may be every month or every quarter), one will have to
look at the entire data base to compute these values and then combine them suitably to
get the relevant answers. The other method is to prepare summary tables, which have
the values pertaining toe ach of these sub-queries, before hand, and then combine them
as and when the query is raised .
Itcan be noted that the summaries can be prepared in the background (or when the
number of queries running are relatively sparse) and only the aggregation can be done
on the fly. Summary table are designed by following the steps given below
i) Decide the dimensions along which aggregation is to be done.
ii) Determine the aggregation of multiple facts.
iii) Aggregate multiple facts into the summary table.

Q4.Explain the role of access control issues in data mart

design.
Ans-ROLE OF ACCESS CONTROL ISSUES IN DATA MART
DESIGN
This is one of the major constraints in data mart designs. Any data warehouse, with its
huge volume of data is, more often than not, subject to various access controls as to
who could access which part of data. The easiest case is where the data is partitioned so
clearly that a user of each partition cannot access any other data. In such cases, each of
these can be put in a data mart and the user of each can
access only his data . In the data ware house, the data pertaining to all these marts are
stored, but the partitioning are retained. If a super user wants to get an overall view of
the data, suitable aggregations can be generated. However, in certain other cases the
demarcation may not be so clear. In such cases, a judicious analysis of the privacy
constraints so as to optimize the privacy of each data mart is maintained.
Data marts, as described in the previous sections can be designed, based on several
splits noticeable either in the data or the organization or in privacy laws. They may also
be designed to suit the user access tools. In the latter case, there is not much choice
available for design parameters. In the other cases, it is always desirable to design the
data mart to suit the design of the ware house itself. This helps to maintain maximum
control on the data base instances, by ensuring that the same design is replicated in
each of the data marts. Similarly the summary informations on each of the data mart
can be a smaller replica of the summary of the data ware house it self.
Q5.List the application and reasons for the growing popularity of data mining.
Ans-REASONS FOR THE GROWING POPULARITY OF DATA
MINING
a) Growing Data Volume
The main reason for necessity of automated computer systems for intelligent data
analysis is the enormous volume of existing and newly appearing data that require
processing. The amount of data accumulated each day by various business, scientific,
and governmental organizations around the world is daunting. It becomes impossible
for human analysts to cope with such overwhelming amounts of data.
b) Limitations of Human Analysis
Two other problems that surface when human analysts process data are the inadequacy
of the human brain when searching for complex multifactor dependencies in data, and
the lack of objectiveness in such an analysis. A human expert is always a hostage of the
previous experience of investigating other systems. Sometimes this helps, sometimes
this hurts, but it is almost impossible to get rid of this fact.
c) Low Cost of Machine Learning
One additional benefit of using automated data mining systems is that this process has a
much lower cost than hiring an many highly trained professional statisticians. While
data mining does not eliminate human participation in solving the task completely, it
significantly simplifies the job and allows an analyst who is not a professional in
statistics and programming to manage the process of extracting knowledge from data.
Q6What is data mining ? What kind of data can be mined ?
Ans-There are many definitions for Data mining. Few important definitions are given
below.
Data mining refers to extracting or mining knowledge from large amounts of data.
Data mining is the process of exploration and analysis, by automatic or semiautomatic
means, of large quantities of data in order to discover meaningful patterns and rules.
WHAT KIND OF DATA CAN BE MINED?
In principle, data mining is not specific to one type of media or data. Data mining
should be applicable to any kind of information repository. However, algorithms and
approaches may differ when applied to different types of data. Indeed, the challenges
presented by different types of data vary significantly.
Data mining is being put into use and studied for databases, including relational
databases, object-relational databases and object oriented databases, data warehouses,
transactional databases, unstructured and semi structured repositories such as the World
Wide Web, advanced databases such as spatial databases,
multimedia databases, time-series databases and textual databases, and even flat files.
Here are some examples in more detail
Flat files: Flat files are actually the most common data source for data mining
algorithms, especially at the research level. Flat files are simple data files in text or
binary format with a structure known by the data mining algorithm to be applied. The
data in these files can be transactions, time-series data, scientific measurements.
Relational Databases: A relational database consists of a set of tables containing either
values of entity attributes, or values of attributes from entity relationships. Tables have
columns and rows, where columns represent attributes and rows represent tuples.
Q7.Give the top level syntax of the data mining query languages DMQL.
Ans-A data mining language helps in effective knowledge discovery from the data
mining systems. Designing a comprehensive data mining language is challenging
because data mining covers a wide spectrum of tasks from data characterization to
mining association rules, data classification and evolution analysis.
Each task has different requirements. The design of an effective data mining query
language requires a deep understanding of the power, limitation and underlying
mechanism of the various kinds of data mining tasks.
Q8. Explain the meaning of data mining with apriori algorithm
Ans-APriori algorithm data mining discovers items that are frequently associated
together. Let us look at the example of a store that sells DVDs, Videos, CDs, Books
and Games. The store owner might want to discover which of these items customers
are likely to buy together. This can be used to increase the stores cross sell and upsell
ratios. Customers in this particular store may like buying a DVD and a Game in 10 out
of every 100 transactions or the sale of Videos may hardly ever be associated
with a sale of a DVD. With the information above, the store could strive for more
optimum placement of DVDs and Games as the sale of one of them may improve the
chances of the sale of the other frequently associated item. On the other hand, the
mailing campaigns may be fine tuned to reflect the fact that offering discount
coupons on Videos may even negatively impact the sales of DVDs offered in the same
campaign. A better decision could be not to offer both DVDs and Videos in a
campaign. To arrive at these decisions, the store may have had to analyze 10,000 past
transactions of customers using calculations that seperate frequent and consequently
important associations from weak and unimportant associations.

iv) Determine the level of aggregation and the extent of embedding.

v) Design time into the table.
vi) Index the summary table.
2.Q9.Explain the working principle of decision tree used for data mining.
Ans-DATA MINING WITH DECISION TREES
Decision trees are powerful and popular tools for classification and prediction. The
attractiveness of tree-based methods is due in large part to the fact that, it is simple and
decision trees represent rules. Rules can readily be expressed so that we humans can
understand them or in a database access language like SQL so that records falling into
a particular category may be retrieved. In some applications, the accuracy of a
classification or prediction is the only thing that matters; if a direct mail firm obtains a
model that can accurately predict which members of a prospect pool are most
likely to respond to a certain solicitation, they may not care how or why the model
works.
Decision tree working concept
Decision tree is a classifier in the form of a tree structure where each node is either:
a leaf node, indicating a class of instances, or
a decision node that specifies some test to be carried out on a single attribute0value,
with one branch and sub-tree for each possible outcome of the test.
A decision tree can be used to classify an instance by starting at the root of the tree and
moving through it until a leaf node, which provides the classification of the instance.
Example: Decision making in the Bombay stock market
Suppose that the major factors affecting the Bombay stock market are:
what it did yesterday;
what the New Delhi market is doing today;
bank interest rate;
unemployment rate;
Indias prospect at cricket.
Q10. What is Bayes theorem ? Explain the working procedure of Bayesisan
classifier.
Ans-Bayes Theorem
Let X be a data sample whose class label is unknown. Let H be some hypothesis, such
as that the data sample X belongs to a specified class C. For classification problems,
we want to determine P (H/X), the probability that the hypothesis H holds given the
observed data sample X. P(HX) is the posterior probability, or a posteriori probability,
of H conditioned on X. For example, suppose the world of data samples consists of
fruits, described by their color and shape. Suppose that X is red and round, and that H
is the hypothesis that X is an apple. Then P(HX) reflects our confidence that
X is an apple given that we have seen that X is red and round. In contrast, P(H) is the
prior probability, or a priori probability, of H. For our example, this is the probability
that any given data sample is an apple, regardless of how the data sample looks. The
posterior probability, P(HX), is based on more information (Such as background
knowledge) than the prior probability, P(H), which is independent of X.
Similarly, P(XH) is the posterior probability of X conditioned on H. That is, it is the
probability that X is red and round given that we know that it is true that X is an apple.
P(X) is the prior probability of X. . P(X), P(H), and P(XH) may be estimated from the
given data, as we shall see below. Bayes theorem is useful in that it provides a way of
calculating the posterior probability,
P(HX), from P(H), P(X), and
P(XH). Bayes theorem is
P(HX) = P(XH) P(H)
P(X)
Q11.Explain how neural network can be used for data mining ?
Ams-A neural processing element receives inputs from other connected processing
elements. These input signals or values pass through weighted connections, which
either amplify or diminish the signals. Inside the neural processing element, all of these
input signals are summed together to give the total input to the unit. This total input
value is then passed through a mathematical function to produce an output or decision
value ranging from 0 to 1. Notice that this is a real valued (analog) output, not a digital
0/1 output. If the input signal matches the connection weights exactly, then the output
is close to 1. If the input signal totally mismatches the connection weights then the
output is close to 0. Varying degrees of similarity are represented by the intermediate
values. Now, of course, we can force the neural processing element to make a binary
(1/0) decision, but by using analog values ranging between 0.0 and 1.0 as the outputs,
we are retaining more information to pass on to the next layer of neural processing
units. In a very real sense, neural networks are analog computers.
Each neural processing element acts as a simple pattern recognition machine. It checks
the input signals against its memory traces (connection weights) and produces an
output signal that corresponds to the degree of match between those patterns. In typical
neural networks, there are hundreds of neural processing elements whose pattern
recognition and decision making abilities are harnessed together to
solve problems.
Backpropagation
Backpropagation learns by iteratively processing a set of training samples, comparing
the networks
prediction for each sample with the actual known class label. For each sample with the
actual known
class label. For each training sample, the weights are modified so as to minimize the
means squared error
between the networks prediction and the actual class. These modifications are made in
the backwards
direction, that is, from the output layer, through each hidden layer down to the first
hidden layer (hence the
name backpropagation). Although it is not guaranteed, in general the weights will
eventually coverage,
and the learning process stops. The algorithm is summarized in Figure each step is
described below.
Initialize the weights: The weights in the network are initialized to small random
numbers (e.g.,
ranging from 1.0 to 1.0, or -0.5 to 0.5). Each unit has a bias associated with it, as
explained below. The

K-means algorithm
This algorithm has as an input a predefined number of clusters, that is the k from its
name. Means stands for an average, an average location of all the members of a
particular cluster. When dealing with clustering techniques, one has to adopt a notion
of a high dimensional space, or space in which orthogonal dimensions are all attributes
from the table of data we are analyzing. The value of each attribute of an
example represents a distance of the example from the origin along the attribute axes.
The coordinates of this point are averages of attribute values of all examples that
belong to the cluster. The steps of the K-means algorithm are given below.
1. Select randomly k points (it can be also examples) to be the
seeds for the centroids of k clusters.
2. Assign each example to the centroid closest to the example,
forming in this way k exclusive clusters of examples.
3. Calculate new centroids of the clusters. For that purpose average
all attribute values of the examples belonging to the same cluster (centroid).
Q12.Explain the STAR-FLAKE schema in detail.
Ans-STAR FLAKE SCHEMAS
One of the key factors for a data base designer is to ensure that a database should be
able to answer all types of queries, even those that are not initially visualized by the
developer. To do this, it is essential to understand how the data within the database is
used. In a decision support system, which is what a data ware is supposed to provide
basically, a large number of different questions are asked about the same set of facts.
For example, given a sales data question like
i) What is the average sales quantum of a particular item?
ii) Which are the most popular brands in the last week?
iii) Which item has the least tumaround item.
iv) How many customers returned to procure the same item within one month. Etc.,.
Can be asked. They are all based on the sales data, but the method of viewing the data
to answer the question is different. The answers need to be given by rearranging or
cross referencing different facts.
Q13.Explain the method for designing dimension tables.
Ans-DESIGNING DIMENSION TABLES
After the fact tables have been designed, it is essential to design the dimension tables.
However, the design of dimension tables need not be considered a critical activity,
though a good design helps in improving the performance. It is also desirable to keep
the volumes relatively small, so that restructuring cost will be
less. Now we see some of the commonly used dimensions.
Star dimension
They speed up the query performance by denormalising reference information into a
single table. They presume that the bulk of queries coming are such that they analyze
the facts by applying a number of constraints to a single dimensioned data.
For example, the details of sales from a stores can be stored in horizontal rows and
select one/few of the attributes. Suppose a cloth store stores details of the sales one
below the other and questions like how many while shirts of size 85" are sold in one
week are asked. All that the query has to do is to put the relevant constraints to get the
information. This technique works well in solutions where there are a number of
entitles, all related to the key dimension entity.
Q14.Explain the Horizontal partioning in briefly.
Ans-Needless to say, the dataware design process should try to maximize the
performance of the system. One of the ways to ensure this is to try to optimize by
designing the data base with respect to specific hardware architecture. Obviously, the
exact details of optimization depends on the hardware platforms.
Normally the following guidelines are useful:
i. maximize the processing, disk and I/O operations.
ii. Reduce bottlenecks at the CPU and I/O
The following mechanisms become handly
4.3.1 Maximising the processing and avoiding bottlenecks
One of the ways of ensuring faster processing is to split the data query into several
parallel queries, convert them into parallel threads and run them parallel. This method
will work only when there are sufficient number of processors or sufficient processing
power to ensure that they can actually run in parallel. (again not that to run five threads,
it is not always necessary that we should have five processors. But to ensure optimality,
even a lesser number of processors should be able to do the job, provided they are able
to do it fast enough to avoid bottlenecks at eh processor).
Normalisation
The usual approach in normalization in database applications is to ensure that the data
is divided into two or more tables, such that when the data in one of them is updated, it
does not lead to anamolies of data (The student is advised to refer any book on data
base management systems for details, if interested).
The idea is to ensure that when combined, the data available is consistent.
However, in data warehousing, one may even tend to break the large table into several
denormalized smaller tables. This may lead to lots of extra space being used. But it
helps in an indirect way It avoids the overheads of joining the data during queries.
To make things clear consider the following table
Q16.Explain the need of data mart in detail.
Ans-THE NEED FOR DATA MARTS
In a crude sense, if you consider a data ware house as a store house of data, a data mart
is a retail outlet of data. Searching for any data in a huge store house is difficult, but if
the data is available, you should be positively able to get it. On the other hand, in a
retail out let, since the volume to be searched from is small, you can be able to access
the data fast. But it is possible that the data you are searching for may not be available
there, in which case you have to go back to your main store house to search for the
data. Coming back to technical terminology, one can say the following are the reasons
for which data marts are created.
i) Since the volume of data scanned is small, they speed up the query processing.
ii) Data can be structured in a form suitable for a user access too
iii) Data can be segmented or partitioned so that they can be used on different platforms
and
also different control strategies become applicable.

biases are similarly initialized to small random numbers.

Each training sample, X, is processed by the following steps.

3.IDENTIFY THE ACCESS TOOL REQUIREMENTS

Data marts are required to support internal data structures that support the user access
tools. Data within those structures are not actually controlled by the ware house, but
the data is to be rearranged and up dated by the ware house. This arrangement (called
populating of data) is suitable for the existing requirements of data analysis. While the
requirements are few and less complicated, any populating
method may be suitable, but as the demands increase (as it happens over a period of
time) the populating methods should match the tools used.
As a rule, this rearrangement (or populating) is to be done by the ware house after
acquiring the data from the source. In other words, the data received from the source
should not directly be arranged in the form of structures as needed by the access tools.
This is because each piece of data is likely to be used by several access tools which
need different populating methods. Also, additional requirements may
come up later. Hence each data mart is to be populated from the ware house based on
the access tool requirements of the data ware house.
Q17.Explain the Data warehouse process manager in detail.
Ans-DATAWARE HOUSE PROCESS MANAGERS
These are responsible for the smooth flow, maintainance and upkeep of data into and
out of the database. The main types of process managers are
Load manager
Warehouse manager and
Query manager
We shall look into each of them briefly. Before that, we look at a schematic diagram
that defines the boundaries of the three types of managers.
Load manager
This is responsible for any data transformations and for loading of data into the
database. They should
effect the following
Data source interaction
Data transformation
Data load.
The actual complexity of each of these modules depend on the size of the database.
It should be able to interact with the source systems to verify the received data. This is
a very important aspect and any improper operations leads to invalid data affecting the
entire warehouse. The concept is normally achieved by making the source and data
ware house systems compatible.
Ware house Manager
The warehouse manager is responsible for maintaining data of the ware house. It
should also create and maintain a layer of meta data. Some of the responsibilities of the
ware house manager are
o Data movement
Meta data management
o Performance monitoring
o Archiving.
Data movement includes the transfer of data within the ware house, aggregation,
creation and maintenance of tables, indexes and other objects of importance. It should
be able to create new aggregations as well as remove the old ones. Creation of
additional rows / columns, keeping track of the aggregation
processes and creating meta data are also its functions.
Query Manager
We shall look at the last of manager, but not of any less importance, the query
manager. The main responsibilities include the control of the following.
o Users access to data
o Query scheduling
Query Monitoring
These jobs are varied in nature and have not been automated as yet.
The main job of the query manager is to control the users access to data and also to
present the data as a result of the query processing in a format suitable to the user. The
raw data, often from different sources, need to be compiled in a format suitable for
querying. The query manager will have to act as a mediator between the user on one
hand and the meta data on the other.
Q18.Explain the Data warehouse Delivery process in detail.
Ans-THE DATA WAREHOUSE DELIVERY PROCESS
This section deals with the dataware house from a different view point - how the
different components that go into it enable the building of a data ware house. The study
helps us in two ways:
i) to have a clear view of the data ware house building process.
ii) to understand the working of the data ware house in the context of the components.
Now we look at the concepts in details.
i. IT Strategy
The company must and should have an overall IT strategy and the data ware housing
has to be a part of the overall strategy. This would not only ensure that adequate
backup in terms of data and investments are available, but also will help in integrating
the ware house into the strategy. In other words, a data ware house can not be
visualized in isolation.
ii. Business case analysis
This looks at an obvious thing, but is most often misunderstood. The overall
understanding of the
business and the importance of various components there in is a must. This will ensure
that one can clearly justify the appropriate level of investment that goes into the data
ware house design and also the amount of returns accruing.
Unfortunately, in many cases, the returns out of the ware housing activity are not
quantifiable. At the end of the year, one cannot say - I have saved / generated 2.5 crore
Rs. because of data ware housing - sort of statements. Data ware house affects the
business and strategy plans indirectly - giving scope for undue expectations on one
hand and total neglect on the other. Hence, it is essential that the designer
must have a sound understanding of the overall business, the scope for his concept

Q19.Briefly explain the system management tools.

Ans- SYSTEM MANAGEMENT TOOLS
The most important jobs done by this class of managers includes the following
1. Configuration managers
2. schedule managers
3. event managers
4. database mangers
5. back up recovery managers 6. resource and performance a monitors.
We shall look into the working of the first five classes, since last type of managers are
less critical in nature.
Configuration manager
This tool is responsible for setting up and configuring the hardware. Since several types
of machines are being addressed, several concepts like machine configuration,
compatibility etc. are to be taken care of, as also the platform on which the system
operates.
Schedule manager
The scheduling is the key for successful warehouse management. Almost all operations
in the ware house need some type of scheduling. Every operating system will have its
own scheduler and batch control mechanism. But these schedulers may not be capable
of fully meeting the requirements of a data warehouse.
Event manager
An event is defined as a measurable, observable occurrence of a defined action. If this
definition is quite vague, it is because it encompasses a very large set of operations.
The event manager is a software that continuously monitors the system for the
occurrence of the event and then take any action that is suitable (Note that the event is a
measurable and observable occurrence). The action to be taken is also normally
specific to the event.
Database manager
The database manger normally will also have a separate (and often independent)
system manager module. The purpose of these managers is to automate certain
processes and simplify the execution of others. Some of operations are listed as
follows.
Ability to add/remove users
o User management
o Manipulate user quotas
o Assign and deassign the user profiles
Q20.What is schema? Distinguish between facts and dimensions.
Ans- schema- A schema, by definition, is a logical arrangements of facts that facilitate
ease of storage and retrieval, as described by the end users. The end user is not
bothered about the overall arrangements of the data or the fields in it. For example, a
sales executive, trying to project the sales of a particular item is only interested in the
sales details of that item where as a tax practitioner looking at the same data will be
interested only in the amounts received by the company and the profits made.
Distinguish between facts and dimensions
The star schema looks a good solution to the problem of ware housing. It simply states
that one should identify the facts and store it in the read-only area and the dimensions
surround the area. Whereas the dimensions are liable to change, the facts are not. But
given a set of raw data from the sources, how does one identify the facts and the
dimensions? It is not always easy, but the following steps can help in that
direction.
i) Look for the fundamental transactions in the entire business process. These basic
entities are the facts.
ii) Find out the important dimensions that apply to each of these facts. They are the
candidates for dimension tables.
iii) Ensure that facts do not include those candidates that are actually dimensions, with
a set of facts attached to it.
iv) Ensure that dimensions do not include these candidates that are actually facts.
Q21.Explain how to categorize data mining system.
Ans- CATEGORIZE DATA MINING SYSTEMS
There are many data mining systems available or being developed. Some are
specialized systems dedicated to a given data source or are confined to limited data
mining functionalities, other are more versatile and comprehensive. Data mining
systems can be categorized according to various criteria among
other classification are the following
a) Classification according to the type of data source mined: this classification
categorizes data mining systems according to the type of data handled such as spatial
data, multimedia data, time-series data, text data, World Wide Web, etc.
b) Classification according to the data model drawn on: this classification
categorizes data mining systems based on the data model involved such as relational
database, object-oriented database, data warehouse, transactional, etc.
Classification according to the king of knowledge discovered: this classification
categorizes data mining systems based on the kind of knowledge discovered or data
mining functionalities, such as characterization, discrimination, association,
classification, clustering, etc. Some systems tend to be comprehensive systems offering
several data mining functionalities together.
Q22 ..A DATA MINING QUERY LANGUAGE
A data mining query language provides necessary primitives that allow users to
communicate with data mining systems. But novice users may find data mining query
language difficult to use and the syntax difficult to remember. Instead , user may prefer
to communicate with data mining systems through a graphical user interface (GUI). In
relational database technology , SQL serves as a standard core language
for relational systems , on top of which GUIs can easily be designed. Similarly, a data
mining query language may serve as a core language for data mining system
implementations, providing a basis for the development of GUI for effective data
mining. A data mining GUI may consist of the following functional components
a)
Data collection and data mining query composition - This component
allows the user to specify task-relevant data sets and to compose data
mining queries. It is similar to GUIs used for the specification of
relational queries.
b) Presentation of discovered patterns This component allows the display of the
discovered patterns in various forms, including tables, graphs, charts, curves and other
visualization techniques.

(data ware house) in the project, so that he can answer the probing questions.

4.Q.Enlist the desirable schemes required for a good architecture

of data mining systems.
Ans- ARCHITECTURES OF DATA MINING SYSTEMS
A good system architecture will enable the system to make best use of the software
environment , accomplish data mining tasks in an efficient and timely manner,
interoperate and exchange information with other information systems, be adaptable to
users different requirements and evolve with time. To know what are the desired
architectures for data mining systems, we view data mining is integrated
with database/data warehousing and coupling with the following schemes
a) no-coupling
b) loose coupling
c) semitight coupling
d) tight-coupling
No-coupling It means that data mining system will not utilize any function of a
database or data warehousing system. Here in this system , it fetches data from a
particular source such as a file , processes data using some data mining algorithms and
then store the mining result in another file. This system has some disadvantages
1) Database system provides a great deal of flexibility and efficiency at storing ,
organizing, accessing and processing data. Without this in a file, Data mining system
may spend a more amount of time finding, collecting , cleaning and transforming data.
Qno--CLUSTERING IN DATA MINING
Requirements for clustering
Clustering is a division of data into groups of similar objects. Each group, called
cluster, consists of objects that are similar between themselves and dissimilar to objects
of other groups. Representing data by fewer clusters necessarily loses certain fine
details (akin to lossy data compression), but achieves
simplification. It represents many data objects by few clusters, and hence, it models
data by its clusters. Data modeling puts clustering in a historical perspective rooted in
mathematics, statistics, and numerical analysis. From a machine learning perspective
clusters correspond to hidden patterns, the search for clusters is unsupervised learning,
and the resulting system represents a data concept. Therefore, clustering
is unsupervised learning of a hidden data concept. Data mining deals with large
databases that impose on clustering analysis additional severe computational
requirements.
Requirements for clustering
Clustering is a challenging and interesting field potential applications pose their own
special requirements.
The following are typical requirements of clustering in data mining.
Scalability: Many clustering algorithms work well on small data sets containing
fewer than 200 data objects However, a large database may contain millions of objects.
Clustering on a sample of a given large data set may lead to biased results. Highly
scalable clustering algorithms are needed.
Ability to deal with different types of attributed: Many algorithms are designed to
cluster interval-based (numerical) data. However, applications may require clustering
other types of data, such as binary, categorical (nominal), and ordinal data, or mixtures
of these data types.
Nominal, Ordinal and Ratio-Scaled Variables
Nominal Variables
A nominal variable is a generalization of the binary variable in that it can take on more
than two states. For example, map_color is a nominal variable that may have, say, five
states: red, yellow, green, pink and blue.
Nominal variables can be encoded by asymmetric binary variables by creating a new
binary variable for each of the M nominal states. For an object with a given state value,
the binary variable representing that state is set to 1, while variable map_color, a binary
variable can be created for each of the five colors listed above. For an object having the
color yellow, the yellow variable is set to 1, while the remaining four variables are set
to 0..
Ordinal Variables
A discrete ordinal variable resembles a nominal variable, except that the M states of the
ordinal value are ordered in a meaningful sequence. Ordinal variables are very useful
for registering subjective assessments of qualities that cannot be measured objectively.
For example professional ranks are often enumerated in a sequential order, such as
assistant, associate, and full. A continuous ordinal variable
looks like a set of continuous data of an unknown scale; that is, the relative ordering of
the values is essential but their actual magnitude is not. For example, the relative
ranking in a particular sport (e.g., gold, silver, bronze) is often more essential than the
actual values of a particular measure.
Ratio-Scaled Variables
A ratio-scaled variable makes a positive measurement on a nonlinear scale, such as an
exponential scale, approximately following the formula AeBt or Ae-Bt, Where A and
B are positive constants. Typical examples include the growth of a bacteria population,
or the decay of a radioactive element. To compute the dissimilarity between objects
described by ratio-scaled variable There are three methods to handle ratio-scaled
variables for computing the dissimilarity between objects.
Neural Network Topologies
The arrangement of neural processing units and their interconnections can have a
profound impact on the processing capabilities of the neural networks. In general, all
neural networks have some set of processing units that receive inputs from the outside
world, which we refer to appropriately as the input units. Many neural networks also
have one or more layers of hidden processing units that receive inputs only from
other processing units. A layer or slab of processing units receives a vector of data or
the outputs of a previous layer of units and processes them in parallel. The set of
processing units that represents the final result of the neural network computation is
designated as the output units. There are three major connection topologies that
define how data flows between the input, hidden, and output processing units.
Feed-Forward Networks
Feed-forward networks are used in situations when we can bring all of the information

Q24.With the help of a block diagram explain the typical process flow in a data
Warehouse.
Ans- TYPICAL PROCESS FLOW IN A DATA WAREHOUSE
Any data ware house must support the following activities
i) Populating the ware house (i.e. inclusion of data)
ii) day-to-day management of the ware house.
iii) Ability to accommodate the changes.
The processes to populate the ware house have to be able to extract the data, clean it
up, and make it available to the analysis systems. This is done on a daily / weekly basis
depending on the quantum of the data population to be incorporated.
The day to day management of data ware house is not to be confused with maintenance
and management of hardware and software. When large amounts of data are stored and
new data are being continually added at regular intervals, maintaince of the quality of
data becomes an important element. Ability to accommodate changes implies the
system is structured in such a way as to be able to cope with future changes without the
entire system being remodeled. Based on these, we can view the processes that a
typical data ware house scheme should support as follows.
Q25.How the Naive Bayesian classification works.
Ans- Naive Bayesian Classification
The nave Bayesian classifier, or simple Bayesian classifier, works as follows:
1. Each data sample is represented by an n-dimensional feature vector, X = (x1,
x2, . . . . xn), depicting n measurements made on the sample from n attributes,
respectively, A1, A2, . .An.
2. Suppose that there are m classes, C1, C2, . Cm. Given an unknown data sample, X
(i.e., having no class label), the classifier will predict that X belongs to the class having
the highest posterior probability, conditioned on X. That is, the nave Bayesian
classifier assigns an unknown sample X to the class Ci if and only if
P(CiX) > P(CjX) for 1 j m, j I Thus we maximize P(Ci/X). The class Ci for
which P(Ci/X) is maximized is called the maximum posteriori hypothesis. By Bayes
theorem P(CiX) = P(XCi) P(Ci)
P(X) 3. As P(X) is constant for all classes, only P(XCi) P(Ci) need be maximized. If
the class prior probabilities are not known, then it is commonly assumed that the
classes are equally likely,
that is, P(C1) = P(C2) = = P(Cm), and we would therefore maximize P(XCi).
Otherwise, we maximize P(XCi) P(Ci). Note that the class prior probabilities may be
estimated by P(Ci) = si/s where si is the number of training samples of class Ci, and s is
the total number of training samples.
Training Bayesian Belief Networks
In the learning or training of a belief network, a number of scenarios are possible. The
network structure may be given in advance or inferred from the data. The network
variables may be observable or hidden in all or some of the training samples. The case
of hidden data is also referred to as missing values or incomplete data.
If the network structure is known and the variables are observable, then training the
network is straightforward. It consists of computing the CPT entries, as is similarly
done when computing the probabilities involved in native Bayesian classification.
Neural Network Topologies
The arrangement of neural processing units and their interconnections can have a
profound impact on the processing capabilities of the neural networks. In general, all
neural networks have some set of processing units that receive inputs from the outside
world, which we refer to appropriately as the input units. Many neural networks also
have one or more layers of hidden processing units that receive
inputs only from other processing units. A layer or slab of processing units receives a
vector of data or the outputs of a previous layer of units and processes them in parallel.
The set of processing units that represents the final result of the neural network
computation is designated as the output units. There are three major connection
topologies that define how data flows between the input, hidden, and output
processing units.
Backpropagation
Backpropagation learns by iteratively processing a set of training samples, comparing
the networks prediction for each sample with the actual known class label. For each
sample with the actual known class label. For each training sample, the weights are
modified so as to minimize the means squared error between the networks prediction
and the actual class. These modifications are made in the backwards
direction, that is, from the output layer, through each hidden layer down to the first
hidden layer (hence the name backpropagation). Although it is not guaranteed, in
general the weights will eventually coverage, and the learning process stops. The
algorithm is summarized in Figure each step is described below.
Initialize the weights: The weights in the network are initialized to small random
numbers (e.g., ranging from 1.0 to 1.0, or -0.5 to 0.5). Each unit has a bias associated
with it, as explained below. The biases are similarly initialized to small random
numbers.
Nonlinear Regression
Polynomial regression can be modeled by adding polynomial terms to the basic linear
model. By applying transformations to the variables, we can convert the nonlinear
model into a linear one that can then be solved by the method of least squares.
Transformation of a polynomial regression model to a linear regression model.
Consider a cubic polynomial relationship given by
Y = + 1X1 + 2X2 + 3X3
To convert this equation to linear form, we define new variables:
X 1 = X X2 = X2 X3 = X3
Using the above Equation can then be converted to linear form by applying the above
assignments, resulting in the equation Y = + 1X1 + 2X2 + 3X3 which is solvable
by the method of least squares.

to bear on a problem at once, and we can present it to the neural network. It is like a
pop quiz, where the teacher walks in, writes a set of facts on the board, and says, OK,
tell me the answer. You must take the data, process it, and jump to a conclusion. In
this type of neural network, the data flows through the network in one
direction, and the answer is based solely on the current set of inputs.
5. Explain the concept of data warehousing and data mining.
Ans. A data warehouse is a collection of a large amount of data and these data is the
pieces of information Which is use to suitable managerial decisions. (a storehouse of
data) eg:- student data to the details of the citizens of a city or the sales of previous
years or the number of patients that came to a hospital with different ailments. Such
data becomes a storehouse of information.
Data mining is the process of exploration and analysis, by automatic or semiautomatic
means, of large quantities of data in order to discover meaningful patterns and rules.
The main concept of datamining using a variety of techniques to identify nuggets of
information or decision making knowledge in bodies of data, and extracting these in
such a way that they can be put to use in the areas such as decision support, prediction,
forecasting and estimation.
Q15. Define data mining query in term of primitives.
Ans: a) Growing Data Volume: The main reason for necessity of automated
computer systems for intelligent data analysis is the enormous volume of existing and
newly appearing data that require processing. The amount of data accumulated each
day by various business, scientific, and governmental organizations around the world is
daunting.
b) Limitations of Human Analysis: Two other problems that surface when human
analysts processdata are the inadequacy of the human brain when searching for
complex multifactor dependencies in data, and the lack of objectiveness in such an
analysis.
c) Low Cost of Machine Learning: While data mining does not eliminate human
participation in solving the task completely, it significantly simplifies the job and
allows an analyst who is not a professional in statistics and programming to manage
the process of extracting knowledge from data.
Qno-List various applications of Data mining in various fields.

YaExplain in brief the data mining applications

Ans: Data mining has many varied field of application which is listed below:
Retail/Marketing:
Identify buying patterns from customers.Find associations among customer
demographic characteristics
Predict response to mailing campaigns
Market basket analysis
Banking:
Detect patterns of fraudulent credit card useIdentity loyal customers
Predict customers, determine credit card spending
Identify stock trading
Insurance and Health Care:
Claims analysis
Identify behavior pattern of risky customers.Identify fraudulent behavior
Transportation:
Determine the distribution schedules among outlets.Analyze loading
Medicine:
Characterize patient behavior to predict office visits.
Identify successful medical therapies for different illnesses.
Q20.What are the guidelines for KDD environment.
Ans: The following are the guidelines for KDD environment are:1. Support extremely large data sets: Data mining deals with extremely large data
sets consisting of billions of records and without proper platforms to store and handle
these volumes of data, no reliable
data mining is possible. Parallel servers with databases optimized for decision support
system oriented queries are useful. Fast and flexible access to large data sets is of very
important.
2. Support hybrid learning: Learning tasks can be divided into three areas:
a.classification tasks b. knowledge engineering tasks c. problem-solving tasks. All
algorithms can not perform well in all the
above areas as discussed in previous chapters. Depending on our requirement one has
to choose the appropriate one.
3. Establish a data warehouse: A data warehouse contains historic data and is subject
oriented and static, that is, users do not update the data but it is created on a regular
time-frame on the basis of the operational data of an organization.
4. Introduce data cleaning facilities: Even when a data warehouse is in operation, the
data is certain to contain all sorts of heterogeneous mixture. Special tools for cleaning
data are necessary and some advanced tools are available, especially in the field of deduplication of client files.
5. Facilitate working with dynamic coding: Creative coding is the heart of the
knowledge discovery process. The environment should enable the user to experiment
with different coding schemes, store partial results make
attributes discrete, create time series out of historic data, select random sub-samples,
separate test sets and so on.
Q21. Explain data mining for financial data analysis.
Ans: Financial data collected in the banking and financial industries are often relatively
complete, reliable and of high quality, which facilitates systematic data analysis and
data mining. The various issues are
a) Design and construction of data warehouses for multidimensional data analysis
and data mining:
Data warehouses need to be constructed for banking and financial data.
Multidimensional data analysis methods should be used to analyze the general
properties of such data. Data warehouses, data cubes, multifeature and discoverydriven data cubes, characteristic and comparative analyses and outlier analyses all play
important roles in financial data analysis and mining.
b) Loan payment prediction and customer credit policy analysis: Loan payment
prediction and customer credit analysis are critical to the business of a bank. Many
factors can strongly or weakly influence loan payment performance and customer

Q23. What is the importance of period of retention of data?

Ans: A businessman says he wants to the data to be retained for as long as possible 5,
10, 15 years the longer the better. The more data we have, the better the information
generated. But such a view thing is unnecessarily simplistic. If a company wants to
have an idea of the recorder levels, details of sales for last 6 months to one year may be
enough. Sales pattern of 5 years is unlikely to be relevant today. So, It is important to
determine the retention period for each function but once it is drawn, it becomes easy to
decide on the optimum value of data to be stored.
Q25. Give the advantages and disadvantages of equal segment partitioning.
Ans: The advantage is that the slots are reusable. Suppose we are sure that we will no
more need the data of 10 years back, then we can simply delete the data of that slot and
use it again. Of course there is a serious draw back in the scheme if the partitions tend
to differ too much in size. The number of visitors visiting a till station, say in summer
months, will be much larger than in winter months and hence Purchase
recommendations can e advertised on the web, in weekly flyers or on the sales receipts
to help improve customer service, aid customers in selecting items and increase sales
37. Define aggregation. Explain steps require designing summary table.
Ans: Association: - A collection of items and a set of records, which containsome
number of items from the given collection, an association function is anoperation
against this set of records which return affinities or patterns that existamong the
collection of items. Summary table are designed by following the steps
given as follows: a) decide the dimensions along which aggregation is to be done.
b) Determine the aggregation of multiple facts. c) Aggregate multiple facts into
the summary table. d) Determine the level of aggregation and the extent of
embedding. e) Design time into the table. f) Index the summary table.
Q30.Explain horizontal and vertical partitioning and differentiate them.
Ans: HORIZONTAL PARTITIONING-This is essentially means that the table is
partitioned after the first few thousand entries, and the next few thousand entries etc.
This is because in most cases, not all the information in the fact table needed all the
time. Thus horizontal partitioning helps to reduce the query access time, by directly
cutting down the amount of data to be scanned by the queries.
a) Partition by time into equal segments : This is the most straight forward method of
partitioning by months or years etc. This will help if the queries often come regarding
the fortnightly or monthly performance / sales etc.
b) Partitioning by time into different sized segments: This is very useful technique
to keep the physical table small and also the operating cost low.
VERTICAL PARTITIONING- A vertical partitioning schema divides the table
vertically. Each row is divided into 2 or more partitions. i) We may not need to access
all the data pertaining to a student all the time.
For example, we may need either only his personal details like age, address etc. or only
the examination details of marks scored etc. Then we may choose to split them into
separate tables, each containing data only about the relevant fields. This will speed up
accessing.
Q27. Explain data mining for retail industry application.
Ans: The retail industry is a major application area for data mining since it collects
huge amount of data on sales, customer shopping history, goods transportation, and
consumption and service records and so on. The quantity of data collected continues to
expand rapidly, due to web or e-commerce.
a) Design and construction of data warehouses on the benefits of data mining: The
first aspect is to design a warehouse. Here it involves deciding which dimensions and
levels to include and what preprocessing to perform in order to facilitate quality and
efficient data mining.
b) Multidimensional analysis of sales, customers, products, time and region: The
retail industry requires timely information regarding customer needs, product sales,
trends and fashions as well as the quality, cost, profit and service of commodities. It is
therefore important to provide powerful multidimensional analysis and visualization
tools, including the construction of sophisticated data cubes according to the needs of
data analysis.
36.Explain multi dimensional schemas.
Ans: This is a very
convenient method of
analyzing data, when it
goes beyond the normal
tabular relations.
For example, a store
maintains a table of
each item it sells over a
month as a table, in each
of its 10 outlets. This is
a 2 dimensional table. One the other hand, if the company wants a data of all items
sold by its outlets, it can be done by simply by
superimposing the 2 dimensional table for each of
these items one behind the other. Then it
becomes a 3 dimensional view. Then the query,
instead of looking for a 2 dimensional rectangle of
data, will look for a 3 dimensional cuboid of data.
There is no reason why the dimensioning should
stop at 3 dimensions. In fact almost all queries can
be thought of as approaching a multi-dimensioned
unit of data from a multidimensional volume of the
schema. A lot of designing effort goes into optimizing such searches.
Q26. Explain the Query generation.
Ans: Meta data is also required to generate queries. The query manger uses the
metadata to build a history of all queries run and generator a query profile for each
user, or group of uses. We simply list a few of
the commonly used meta data for the query. The names are self explanatory. o QueryTable accessed- Column accessed, Name, Reference identifier. o Restrictions appliedColumn name, Table name, Reference identifier ,Restrictions. o Join criteria appliedColumn name, Table name, Reference Identifier, Column name, Table name,
Reference identifier. o Aggregate function used-Column name, Reference identifier,
Aggregate function. o Syntax o Resources o Disk

credit rating. Data mining methods, such as feature selection and attribute relevance
ranking may help identify important factors and eliminate irrelevant ones.
c) Classification and clustering of customers for targeted marketing: Classification
and clustering methods can be used for customer group identification and targeted
marketing..
Upload By:Abhimnayu kumar singh