Software Testing Basics and Quality Attributes

Software Testing-15IS63 Dept.
, of ISE Module -
I
Module1- BASICS OF SOFTWARE TESTING

BASIC DEFINITIONS:
Error: When people make mistakes while coding, we call these mistakes bugs. Errors tend to propagate;
a requirements error may be magnified during design and amplified still more during coding.
Fault: a fault is the result of an error. It is more precise to say that a fault is the representation of an
error, and can be represented as narrative text, data flow diagrams, hierarchy charts, source code, and so
on. Defect is a good synonym for fault, as is bug. When a designer makes an error of omission, the
resulting fault is that something is missing that should be present in the representation.
Types of Fault:
 Fault of Omission: Faults of omission occur when we fail to enter correct information. Of these
two types, faults of omission are more difficult to detect and resolve.
 Fault of Commission: A fault of commission occurs when we enter something into a

representation that is incorrect. Ex subtracting a figure that should have been added
Failure: a failure occurs when a fault executes and is usually represented to be source code, or loaded
object.
Incident: when a failure occurs, it may or may not be readily apparent to the user (or customer or tester).
An incident is the symptom associated with a failure that alerts the user to the occurrence of a failure.
Test: testing is obviously concerned with errors, faults, failures, and incidents. A test is the act of
exercising software with test cases. A test has two distinct goals: to find failures or to demonstrate
correct execution.
Test Case: test case has an identity and is associated with a program behavior. A test case also has a set
of inputs and a list of expected outputs.
Software Testing:
Definition: According to the definition given by Dave Gelperin and William C. Hetzel - Software testing
can be stated as the process of validating and verifying that a software:
 Meets the requirements that guided its design and development.
 Works as expected.
 Satisfies the needs of stakeholders.
Software testing is the process of analyzing a software item to detect the differences between existing
and required conditions (that is, bugs) and to evaluate the features of the software item. One of the test
Krupa K S, Asst. Professor 1

Global Academy of Technology
Software Testing-15IS63 Dept., of ISE Module -
I
techniques includes the process of executing a program or application with the intent of finding software
bugs (errors or other defects).
Necessity:
 To determine a set of test cases.

 To improve the quality of software.
 To build confidence in the software
 To demonstrate to the developer and the customer that the software meets its requirements.
 To discover faults or defects in the software.
Software Testing Life Cycle:
Significance of testing life cycle is that it depicts how and when bugs are introduced, tested and fixed in a
software development life cycle (SDLC).
Phases putting Bugs IN:

 Requirement specification: Errors are made during requirement collection and analysis.
 Design: Testers work with developers in determining what aspects of a design are testable and
under what parameter those testers work. Errors introduced in previous phase results into faults
and more errors are made.
 Coding: Design is implemented. Faults and errors introduced in previous phases propagate and
more errors are made.
Phase FINDING Bugs:

 Testing: Test strategy or planning is done to generate test cases or scenarios, which are executed
to find bugs or errors. Bugs found are reported in error logs.
Phases getting Bugs OUT:
 Fault classification: Reported faults are assigned to different severity classes like catastrophic,
serious, mild etc.
 Fault isolation: Causes and location of different faults are pinpointed.
 Fault resolution: Patch work is done to fix the faults which give another opportunity for error and
new faults to be made.

I
SOFTWARE QUALITY :-
Software quality is a multidimensional quantity and is measurable.
Quality Attributes: There exist several measures of software quality. These can be divided into static
and dynamic quality attributes.
 Static quality attributes refer to the actual code and related documentation.
 Dynamic quality attributes relate to the behavior of the application while in use.
Static quality attributes include structured, maintainable, testable code along with availability of correct
and complete documentation.

I
You might have come across complaints such as “Product X is excellent, I like the features it offers, but
its user manual stinks!” In this case, the user manual brings down the overall product quality.
As a part of corrective maintenance on any application code, we need to understand portions of the code
before we make any changes to it. This is where attributes such as code documentation, understandability,
and structure come into play. A poorly- structured and poorly documented piece of code will be harder to
understand, modify and difficult to test.
Dynamic quality attributes include software reliability, correctness, completeness, consistency,

usability, and performance. Dynamic quality attributes are generally determined through multiple
executions of a program.
 Reliability refers to the probability of failure-free operation.

 Correctness refers to the correct operation of an application and is always with reference to some
artifact. For a tester, correctness is w.r.t requirements; for a user, it is often with respect to a user
manual.
 Completeness refers to the availability of all features listed in the requirements, or user manual.
An incomplete software is one that does not fully implement all features required. Completeness
implements features that might itself be a subset of a larger set of features that are to be
implemented in some future version of the application. Therefore every piece of software that is
correct is also complete with respect to some feature set.
 Consistency refers to adherence to a common set of conventions and assumptions. For example,
to follow a common color coding convention in a user interface. An example of inconsistency is
to display date of birth of a person in different formats, without any regard for the user’s
preferences.
 Usability refers to the ease with which an application can be used. Usability testing refers to the
testing of a product by its potential users. Users in turn test for ease of use, functionality as
expected, performance, safety, and security. Users thus serve as an important source of tests that
developers or testers within the organization might not have conceived. Usability testing is
sometimes referred to as user-centric testing.
 Performance refers to the time the application takes to perform a requested task and is
considered as a non-functional requirement. It is specified in terms such as “This task must be
performed at the rate of X units of activity in one second on a machine running at speed Y, having
Z gigabytes of memory.” For example, the performance requirement for a compiler might be
stated in terms of the minimum average time to compile of a set of numerical applications.
 Reliability can be considered as a statistical measure of correctness.

I
Software reliability is the probability of failure-free operation of software over a given time
interval and under given conditions. This definition requires the user operational profile, which
is difficult or impossible to estimate accurately. However, if an operational profile can be
estimated for a given class of users, then an accurate estimate of the reliability can be found for
this class of users.
Software reliability is the probability of failure-free operation of software in its intended

environment. The term “environment” refers to the software and hardware elements needed to
execute the application, such as operating system, hardware requirements, and any other
applications needed for communication.
REQUIREMENTS, BEHAVIOR, AND CORRECTNESS:-
 Products, software in particular, are designed in response to requirements.

 Requirements specify the functions that a product is expected to perform.
 Once the product is ready, it is the requirements that determine the expected behavior.
 During development phase, the requirements might get changed from what was stated originally.
 Testers test the expected behavior by understanding the requirements.
Example Here are the two requirements, each of which leads to a different program.
Requirement 1: It is required to write a program that inputs two integers and outputs the maximum
of these.
Requirement 2: It is required to write a program that inputs a sequence of integers and outputs the
sorted version of this sequence.
The Requirement 1 as stated above fails to provide an answer if the tester wants to know if the two
integers are to be input to the program on one line followed by a carriage return, or on two separate lines
with a carriage return typed in after each number. This example illustrates the incompleteness of
Requirement 1.
The second requirement in the above example is ambiguous. It is not clear from this requirement whether
the input sequence is to be sorted in ascending or descending order. The behavior of the sort program,
written to satisfy this requirement, will depend on the decision taken by the programmer while
writing sort.
I
 Testers are often faced with incomplete and/or ambiguous requirements.

 In such situations, a tester may resort to a variety of ways to determine what behavior to expect
from the program under test.
 Regardless of the nature of the requirements, testing requires the determination of

the expected behavior of the program under test.
 The observed behavior of the program is compared with the expected behavior to determine if the
program functions as desired.
Input Domain and Program Correctness
 A program is considered correct if it behaves as desired on all possible test inputs.
 Testing a program on all possible inputs is known as exhaustive testing.
 If the requirements are complete and unambiguous, it should be possible to determine the set of
all possible inputs.
 The set of all possible inputs to a program Ρ is known as the input domain, or input space, of P.
 The set of all possible inputs is too large for the program to be executed on each input. For
example, suppose that the max program for Requirement 1 is to be tested for integers range from
–32,768 to 32,767, then a total of 232 executions of max is required.
 For Requirement1 the input domain of max to be the set of all pair of integers where each
elements in the pair is in the range –32,768 to 32,767.
 Requirement 2 can be modified with input domain as “ character ‘A’ to represent ascending and
‘D’ to represent descending order of sorting , followed by a sequence of zero or more integers
ending with a period ”
 For example, following are three elements in the input domain of sort:
< A 3 15 12 55.>
I
< D 23 78.>
< A .>
Correctness : A program is considered correct if it behaves as expected on each element of its input
domain.
Valid and invalid inputs:
In the examples above, the input domains are derived from the requirements. However, the requirements
are incomplete. The requirement mentions that the request characters can be “A” or “D”, but it fails to
answer the question “What if the user types a different character ?” other than “A” or “D” which is
considered as invalid input to sort. The requirement for sort does not specify what action it should
take when an invalid input is encountered.
Identifying the set of invalid inputs and testing the program against these inputs is an important part of
the testing activity.
Testing a program against invalid inputs might reveal errors in the program.
Example 1: Suppose that we are testing the sort program against the input: < Ε 7 19 . >. As input
character E is not in the input domain, the sort program enters into an infinite loop and neither asks the
user for any input nor responds to anything typed by the user. This observed behavior points to a possible
error in sort.
Example 2 the requirements for the sort program do not specify how the program should behave
if the sequence of integers to be sorted consists of any character, such as “*”instead of integers. Then the
program should inform the user that the input is invalid. But this expected behavior from sort needs to
be tested. This suggests that the input domain for sort should be modified.
In cases where the input to a program is not guaranteed to be correct, it is convenient to partition the
input domain into two subdomains. One subdomain consists of inputs that are valid and the other consists
of inputs that are invalid. A tester can then test the program on selected inputs from each subdomain.
CORRECTNESS VERSUS RELIABILITY:-

 Though correctness of a program is desirable, it is almost never the objective of testing.
 To establish correctness via testing on all elements in the input domain is impossible to
accomplish.
 Thus, correctness is established via mathematical proofs of programs. A proof uses the formal
specification of requirements and the program text to prove or disprove that the program will
behave as intended.

I
 While correctness attempts to establish that the program is error free, testing attempts to find if
there are any errors in it.
 Removal of errors from the program usually improves the chances, or the probability, of the
program executing without any failure.
 Testing, debugging, and the error removal processes together increase our confidence in the
correct functioning of the program under test.
Example 1.11 This example illustrates why the probability of program failure might not change upon
error removal. Consider the following program that inputs two integers x and y and prints the value
of f (x, y) or g(x, y) depending on the condition x < y.
The above program uses two functions f and g not defined here. Let us suppose that function fproduces
an incorrect result whenever it is invoked with x = y and that f (x, y) ≠ g(x, y), x = y. In its present form,
the program fails when tested with equal input values because function g is invoked instead of function f.
When the error is removed by changing the condition x < y to x ≤ y, the program fails again when the
input values are the same. The latter failure is due to the error in function f. In this program, when the
error in f is also removed, the program will be correct assuming that all other code is correct.
Reliability
 “The reliability of a program Ρ is the probability of its successful execution on a randomly
selected element from its input domain.”
 Reliability is a statistical attribute. According to one view, it is the probability of failure free
execution of a program under given conditions.
 A comparison of program correctness and reliability reveals that while correctness is a binary
metric, reliability is a continuous metric over a scale from 0 to 1. A program can be either correct
or incorrect; its reliability can be anywhere between 0 and 1.
 Example : Consider a program Ρ that takes a pair of integers as input. The input domain of this
program is the set of all pairs of integers. Suppose now that in actual use there are only three
pairs that will be input to P. These are {<(0, 0) (–l, l) (l, –l)>}

I
The above set of three pairs is a subset of the input domain of Ρ and is derived from a knowledge
of the actual use of P, and not solely from its requirements.
 Suppose also that each pair in the above set is equally likely to occur in practice. If it is known
that Ρ fails on exactly one of the three possible input pairs then the frequency with which Ρ will
function correctly is This number is an estimate of the probability of the successful operation
of Ρ and hence is the reliability of P.
Operational profiles
 An operational profile is a statistical summary of how a program would be used in practice.
 An operational profile is a numerical description of how a program is used.
 In accordance with the above definition, a program might have several operational profiles
depending on its users.
Example : Consider a sort program that, on any given execution, allows any one of two types of input
sequences. One sequence consists of numbers only and the other consists of alphanumeric strings. One
operational profile for sort is specified as follows:
Another operational profile for sort is specified as follows:
The two operational profiles above suggest significantly different uses of sort. In one case, it is used
mostly for sorting sequences of numbers, and in the other case, it is used mostly for sorting alphanumeric
strings.

I
TESTING AND DEBUGGING:-

 Testing is the process of determining if a program behaves as expected during which errors may
be discovered in the program under test.
 The process used to determine the cause of this error and remove it is known as debugging.
 As illustrated in Figure below, testing and debugging are often used as two related activities in a
cyclic manner.
Figure : A test and debug cycle.

I
 Preparing a test plan :
 A test cycle is often guided by a test plan.
 When relatively small programs are being tested, a test plan is usually informal and in the tester’s
mind, or there may be no plan at all.
 A sample test plan considers items such as the method used for testing, method for evaluating
the adequacy of test cases, and method to determine if a program has failed or not.
A sample test plan for testing the sort program is shown below
 Constructing Test Data

 A test case is a pair of input data and the corresponding program output.
 The test data are a set of values: one for each input variable.
 A test set is a collection of test cases.
 A test set/ test data is sometimes referred to as a test suite.

I
 Program requirements and the test plan help in the construction of test data.
 Execution of the program on test data might begin after all or a few test cases have been
constructed.
 While testing, relatively small programs testers often generate a few test cases and execute the
program against these. Based on the results obtained, the tester decides whether to continue the
construction of additional test cases or to enter the debugging phase.
Example The following test cases are generated for the sort program using the test plan above
Test cases 1 and 2 are derived in response to item 1 in the test plan; 3 and 4 are in response to item 2. Notice also
that the requirements for the sort program do not indicate what should be the output of sort when there is nothing
to be sorted. We therefore took an arbitrary decision while composing the “Expected output” for an input that has
no numbers to be sorted. Test cases 5 and 6 are in response to item 3 in the test plan.
 Executing the program

 Execution of a program under test is the next significant step in testing.
I
 The complexity of actual program execution is dependent on the program itself.
 Tester constructs a test harness to aid in program execution. The harness initializes any global
variables, inputs a test case, and executes the program.
 The output generated by the program may be saved in a file for subsequent examination by a
tester. The next example illustrates a simple test harness.
Example : The test harness in Figure below reads an input sequence, checks for its correctness, and then
calls sort. The sorted array sorted_sequence returned by sort is printed using the
print_sequence procedure. Test cases are assumed to be available in the Test pool file shown in
the figure. In some cases, the tests might be generated from within the harness.
Figure : A simple test harness to test the sort program.
In preparing this test harness we assume that :

 sort is coded as a procedure,
 the get_input procedure reads the request character and the sequence to be sorted into
variables request_char, num_items, and in_numbers, and
 the input is checked prior to calling sort by the check_input procedure.
 The test_setup procedure is usually invoked first to set up the test ( i.e identifying and opening the
file containing tests).
 The check_output procedure serves as the oracle that checks if the program under test behaves
correctly. The report_failure procedure is invoked in case the output from sort is incorrect.
 A failure might be reported via a message on the screen or saved in a test report file.
The print_sequence procedure prints the sequence generated by the sort program.

I
 The output generated by print_sequence can also be piped into a file for subsequent examination.
TEST CASES:-
Test case is a scenario made up of a sequence of steps and conditions or variables, where test inputs
are provided and the program is run using those inputs, to see how it performs.
An expected result is outlined and it is compared with result obtained. Certain, working conditions
present are also considered in the test cases so as to check how the program handles such conditions.
 Test cases are required to measure how program handles tricky situations or errors, and is
expected to showcase the hidden logical errors by different cases of input.
 A typical test case contains the following information:
Test case contains :
 Inputs of 2 types: Preconditions (prior to test case execution) and the actual inputs that were
identified by some testing methods.

I
 Expected outputs – 2 types : postconditions and actual outputs.
 Any test case should have an identity and a reason for being written.
 Required to record the execution history of a test case, including when and by whom it was run,
the pass/fail result of each execution, version of software on which it was run.
 Test cases are as valuable as source code.
 Test cases need to be developed, reviewed, used, managed and saved.
INSIGHT FROM VENN DIAGRAM:-
Testing is concerned with behaviour, and behaviour is orthogonal to the structural view common to
software developers.
Consider a universe of program behaviour. Given the program and its specification, consider the set S of
specified behaviors and set P of program behaviors. Figure above shows the relationship among our
universe of discourse as well as the specified and programmed behaviour.
Of all the program behaviors, the specified ones are in the circle labeled S, and all those behaviors
actually programmed are in P.
Figure below shows, specified, implemented and tested behaviors.
The new circle in the below figure is for test cases.

I
Region 1: Specified behaviors that are programmed and tested.

Region 2: Specified behaviors that are programmed but not tested.
Region 3: Unspecified behaviors that are programmed and tested.
Region 4: Specified behaviors that are tested but not programmed.
Region 5: Specified behaviors that are neither programmed nor tested.
Region 6: Unspecified behaviors that are programmed but not tested.
Region 7: Unspecified behaviors that are tested but not programmed.
Region 8: Behaviors that are neither specified and tested nor programmed.
IDENTIFYING THE TEST CASES:-

There are two different types of testing:
1. Functional Testing
2. Structural Testing
1. Black box/Functional testing:

Black box testing is a method of software testing that examines the functionality of an application
without peering into its internal structures or workings. Some functional testing methods are Boundary
Value Testing, Equivalence Class Testing and Decision Table-Based testing.
Advantages:
 Same Test Cases can be used in case of any changes in implementation i.e. Independent of
Software Implementation.
 Test case development can be done in parallel with the implementation, Hence reducing the
overall project development interval
I
Disadvantages:
 Significant Redundancies among test cases i.e. test cases similar to each other may repeat multiple
times.
 Possibility of gaps of untested software. This may result in a faulty product being delivered to the
customer.
2. White box/Structural/Clear box testing:
White- box testing (also known as clear box testing, glass box testing, and transparent box testing and
structural testing) is a method of testing software that tests internal structures or workings of an
application, as opposed to its functionality. Some structural testing methods are Path testing and dataflow
testing.
TEST- GENERATION STRATEGIES:-
 Test case generation - one of the key tasks in any software test activity.

I
 The program under test is executed against the test cases to determine whether or not it conforms
to the requirements.
 Test generation uses a source document. In informal test methods, the source document resides in
the mind of the tester who generates tests based on knowledge of the requirements.
 In some test processes, requirements document serve as a source for the development of formal
models used for test generation.
Figure below summarizes several strategies for test generation.
 The top row in this figure captures informal techniques that are applied directly to the
requirements. It assigns values to input variables without the use of any rigorous or formal
methods.
 These techniques identify input variables, capture the relationship amongst these variables, and
use formal techniques for test generation such as random test generation and cause–effect
graphing.
Figure: Requirements, models, and test generation algorithms.

I
 Model based test generation strategies require that a subset of the requirements be modeled
using a formal notation. Such a model is also known as a specification of a subset of
requirements.
 The tests are then generated with the specification serving as the source.
 Finite state machines, statecharts, Petri nets, and timed input–output automata are some of the
well-known and used formal notations for modeling various subsets of requirements.
 The graphical notations such as sequence and activity diagrams in UML are used as models of
subsets of requirements.
 Languages based on predicate logic as well as algebraic languages are also used to express
subsets of requirements in a formal manner.
 code-based test generation techniques generate tests directly from the code and are useful when
enhancing existing tests based on test adequacy criteria.
 These techniques are also used during regression testing when there is often a need to reduce the
size of the test suite, or prioritize tests, against which a regression test is to be performed.
 Such techniques take four inputs, the program to be regression tested P′, the program Ρ from
which P′ has been derived by making changes, an existing test suite Τ for P, and some run time
information obtained by executing Ρ against T. This run time information may include items such
as statement and branch coverage. The test generation algorithm then uses this information to
select tests from Τ that must be executed to test those parts of P′ that have changed or are affected
by the changes made to P. The resulting test suite is usually a subset of Τ.
TEST METRICS :-
 The term “metric” refers to a standard of measurement which measures some aspect of the test
process.

I
 In software testing, there exist a variety of metrics. Figure shows a classification of various types
of metrics.
 Metrics can be computed at the organizational, process, project, and product levels. Each set of
measurements has its value in monitoring, planning, and control.
Figure: Types of metrics used in software testing and their relationships.
 Regardless of the level at which metrics are defined and collected, there exist four general core
areas that assist in the design of metrics. These are schedule, quality, resources, and size.
 Schedule-related metrics measure actual completion times of various activities and compare these
with estimated time to completion.
 Quality-related metrics measure the quality of a product or a process.
 Resource-related metrics measure items such as cost in dollars, manpower, and tests executed.
 Size-related metrics measure the size of various objects such as the source code and the number of
tests in a test suite.
Organizational metrics :
 Organizational metrics are useful in overall project planning and management.
 They are obtained by aggregating compatible metrics across multiple projects. For example, the
number of defects reported after product release, averaged over a set of products developed and
marketed by an organization, is a useful metric of product quality at the organizational level.
 Quality in organization is ensured by Computing this metric at regular intervals and overall
products released over a given duration.
 Organizational level metrics allow senior management to monitor the overall strength of the
organization and points to areas of weaknesses.
I
 These metrics help the senior management in setting new goals and plan for resources needed to
realize these goals.
Example :The average defect density across all software projects in a company is 1.73 defects per KLOC.
Senior management has found that for the next generation of software products, which they plan to bid,
they need to show that product density can be reduced to 0.1 defects per KLOC. The management thus sets
a new goal.
Given the time frame from now until the time to bid, the management needs to do a feasibility analysis to
determine whether or not this goal can be met. If a preliminary analysis shows that it can be met, then a
detailed plan needs to be worked out and put into action. For example, management might decide to train
its employees in the use of new tools and techniques for defect prevention and detection using
sophisticated static analysis techniques.
Project Metrics :
 Project metrics relate to a specific project, for example, the I/O device testing project or a
compiler project.
 These are useful in the monitoring and control of a specific project.
 The ratio of actual to planned system test effort is one project metric. Test effort could be
measured in terms of the tester-man-months.
 At the start of the system test phase, for example, the project manager estimates the total system
test effort. The ratio of actual to estimated effort is zero prior to the system test phase. This ratio
builds up over time. Tracking the ratio assists the project manager in allocating testing resources.
 Another project metric is the ratio of the number of successful tests to the total number of tests in
the system test phase. At any time during the project, the evolution of this ratio from the start of
the project could be used to estimate the time remaining to complete the system test process.
Process Metrics :
 The purpose of a process metric is to assess the “goodness” of a process.
 Every project uses some test process. Ex: “big bang” approach is a process used in relatively
small single person projects.
 During the test process phases like unit test, integration test, system test etc, we can measure how
many defects were found in each phase.
 Defects found at later stage will be costlier to fix.

I
 Hence, a metric that classifies defects according to the phase in which they are found assists in
evaluating the process itself.
Example: In one software development project, it was found that 15% of the total defects were reported by
customers, 55% of the defects prior to shipping were found during system test, 22% during integration
test, and the remaining during unit test. The large number of defects found during the system test phase
indicates a possibly weak integration and unit test process. The management might also want to reduce the
fraction of defects reported by customers.
Product Metrics: generic

 Product metrics relate to a specific product Ex :a compiler for a programming language.
 These are useful in making decisions related to the product, for ex, “Should this product be
released for use by the customer ?”
 There are 2 types of Product metrics a) the Cyclomatic complexity and b) the Halstead metrics.
 The cyclomatic complexity proposed by Thomas McCabe in 1976 is based on the control flow of
a program.
 Given the CFG G of program Ρ containing Ν nodes, Ε edges, and p connected procedures, the
cyclomatic complexity V(G) is computed as follows:
V(G) = E – N + 2p
 It is a measure of program complexity; higher values imply higher complexity. This is a product
metric.
 V(G) is the complexity of a CFG G that corresponds to a procedure p reachable from the main
procedure. Also, V(G) is not the complexity of the entire program, instead it is the complexity of
a procedure in program Ρ that corresponds to G.
 Larger values of V(G) tend to imply higher program complexity and hence a program more
difficult to understand and test than one with a smaller values. V(G) is 5 or less are recommended.
 Halstead complexity measures were published by late Professor Maurice
 Table below lists some of the Halstead complexity metrics. Using program size (S) and effort (E),
the following estimator has been proposed for the number of errors (B) found during a software
development effort:
 B = 7.6E0.667S0.333

I
 An advantage of using an estimator such as Β is that it allows the management to plan for testing
resources. For example, a larger value of the number of expected errors will lead to a larger
number of testers and testing resources to complete the test process over a given duration.
Table : Halstead measures of program complexity and effort.
Product metrics: OO software :

 Table below lists a sample of product metrics for object-oriented and other applications.
 Product reliability is a quality metric and refers to the probability of product failure for a given
operational profile.
 Product reliability of software truly measures the probability of generating a failure causing test
input. If the probability is 0 for a given operational profile, then the program is perfectly reliable
despite the possible presence of errors.
 The OO metrics measure program or design complexity. A product with a complex design will
require more test effort to obtain a given level of defect density than a product with less
complexity.
Table : A Sample of Product Metrics

I
Progress monitoring and trends:

 Metrics are often used for monitoring progress where measurements are made on a regular basis
over time.
 For example, suppose that a browser has been coded, unit tested, and its components integrated. It
is now in the system testing phase. We can plot the measure of cumulative number of defects
found over time.
 Such a plot will rise over time. Eventually, it will likely show a saturation indicating that the
product is reaching a stability stage.
Figure: A sample plot of cumulative count of defects found over seven consecutive months in a software project.

I
Static and Dynamic Metrics :

 Product metrics could be classified as static or dynamic.
 Static metrics are those computed without having to execute the product. Ex: Number of testable
entities in an application.
 Dynamic metrics require code execution, ex. the number of testable entities actually covered by a
test suite is a dynamic product metric.
 In an organization and project, the average number of testers working on a project is a static
project metric.
 Number of defects remaining to be fixed could be treated as a dynamic metric as it can be

computed accurately only after a code change has been made and the product retested.
Testability :
 Testability is the “degree to which a system or component facilitates the establishment of test
criteria and the performance of tests to determine whether those criteria have been met.”
 2 ways to measure testability of a product:
 Static testability metrics. Ex Software complexity .The more complex an application, the lower
the testability, higher the effort required to test it.
 Dynamic testability metrics: ex code-based coverage criteria. When it is difficult to generate

tests that satisfy the statement coverage criterion is said to have low testability.
 High testability is a desirable goal and is achieved by knowing what needs to be tested and how,
well in advance.
 Features to be tested and how they are to be tested must be identified during the requirements
stage, further it is updated during the design phase and coding phase.
 A testability requirement can be met by writing additional code to a class, or by using special
hardware and probes.
 Testability is a concern in both hardware and software designs.
 In hardware design, testability focuses on verifying the correctness (no faults) of a finished
product.
 Testability in software focuses on the verification of design and implementation.

I
ERROR AND FAULT TAXONOMIES:
Basic definitions of Process and product: Process refers to how we do something, and product is end
result of process.
Software quality Assurance SQA tries to improve the product by improving the Process, and is concerned
with reducing errors endemic in the development process.
Testing is more concerned with discovering faults in a product.
Faults can be classified in several ways: development phase of fault occurance, consequences of
corresponding failure, difficulty to resolve, risk etc.
Fault classification by severity:
Various error and fault taxonomies based upon a)Input / Output Faults b) Logic Faults c)
Computation Faults d) Interface Faults e) Data Faults
Input:
 Correct input not accepted
 Incorrect input Accepted
 Description wrong or missing
 Parameters wrong or missing
Output:
 Wrong Format

I
 Wrong Result
 Correct result at wrong time
 Incomplete or missing result
 Spurious result
 Spelling / Grammar
Logical faults:
 Missing Case (s)
 Duplicate Case (s)
 Extreme Condition Neglected
 Misinterpretation
 Missing Condition
 Extraneous Condition (s)
 Test of wrong Variable
 Incorrect loop iteration
 Wrong operator (e.g., < instead of <=)
Computation Faults:
 Incorrect Algorithms
 Missing Computations
 Incorrect Operand
 Incorrect Operation
 Parenthesis error
 Insufficient Precision ( Round-off , Truncation)
 Wrong Built-in function
Interface Faults:
 Incorrect interrupt handling
 I/O Timing
 Call to wrong Procedure
 Call to Nonexistent Procedure
 Parameter mismatch (type, number)
 Incompatible types
 Superfluous types
Data Faults:
 Incorrect initialization
 Incorrect storage / access
 Wrong flag / index value
 Incorrect packing / unpacking
 Wrong variable used
 Wrong data reference
 Scaling or unit errors

I
 Incorrect data dimension

 Incorrect Subscript , Incorrect type
 Incorrect data scope
 Sensor data out of limits
LEVELS OF TESTING:-
Figure below shows the level of abstraction and testing in the waterfall model.
The levels are as follows:
 Requirement specification: This step involves taking down requirements from the customer and
making a complete sense of the specified requirements. It corresponds to System Testing.
 Preliminary design: A rough estimate (replica or sketch) of the product required by the customer is
made in this phase taking into account the requirements specified in the previous level. It corresponds to
Integration Testing
 Detailed design: A complete detailed design of the product is made in this phase taking into account
all the requirements specified in the first phase and the sketch made in the second phase. . It corresponds
to Unit Testing.
 Coding: The detailed design developed in the previous phase (i.e. detail design) is next coded and a
product is developed (in segments or subunits).
 Unit Testing: The developed subunits or segments are then tested for their specific functionality.
Structural testing is more appropriate at unit level testing.

I
 Integration Testing: The developed subunits in the coding phase are then integrated into a specific
module which is again tested to check the interactions between the subunits that are integrated.
Functional testing is more appropriate at Integration testing.
 System Testing: The integrated modules are then finally assembled into a finished product which is
tested again to check whether the product performs to the required standards as mentioned by the
customer. Functional testing is more appropriate at system level testing.
TESTING AND VERIFICATION :-
 Testing aims at uncovering errors in a program.

 Program verification aims at proving the correctness of programs by showing that it contains no
errors.
 Also verification aims at showing that a given program works for all possible inputs that satisfy a
set of conditions,
 Program verification and testing are best considered as complementary techniques.
 In the development of critical applications, such as smart cards, or control of nuclear plants, one
often makes use of verification techniques to prove the correctness of some artifact created during
the development cycle, not necessarily the complete program. Regardless of such proofs, testing
is used invariably to obtain confidence in the correctness of the application.
 Verification might appear to be a perfect process as it promises to verify that a program is free
from errors.
 However, a close look at verification reveals that it has its own weaknesses. The person who
verified a program might have made mistakes in the verification process; there might be an
incorrect assumption on the input conditions;
 Thus, neither verification nor testing are perfect techniques for proving the correctness of
programs.
STATIC TESTING:-
 Static testing is carried out without executing the application under test.
 In contrast, dynamic testing requires one or more executions of the application under test.
 Static testing is useful in that it may lead to the discovery of faults in the application, as well as
ambiguities and errors in requirements and other application relation documents, at a relatively
low cost.
I
 This is especially so when dynamic testing is expensive.
 Static testing is complementary to dynamic testing.
 Organizations often sacrifice static testing in favor of dynamic testing though this is not
considered a good practice.
 Static testing is best carried out by an individual who did not write the code, or by a team of
individuals.
 A sample process of static testing is illustrated in Figure. The test team responsible for static
testing has access to requirements documents, application, and all associated documents such as
design document and user manuals. The team also has access to one or more static testing tools. A
static testing tool takes the application code as input and generates a variety of data useful in the
test process.
Figure : Elements of static testing.
1.11.1 Walkthroughs
 Walkthroughs and inspections are an integral part of static testing.

 Walkthrough is an informal process to review any application-related document. For example,
requirements are reviewed using a process termed requirements walkthrough.
 Code walkthrough, also known as peer code review, is used to review code and may be
considered as a static testing technique
 A walkthrough begins with a review plan agreed upon by all members of the team. Each item of
the document, for example, a source code module, is reviewed with clearly stated objectives in
view. A detailed report is generated that lists items of concern regarding the document revied.
I
 In requirements walkthrough, the test team must review the requirements document to ensure that
the requirements match user needs, and are free from ambiguities and inconsistencies.
1.11.2 Inspections
 Inspection is a more formally defined process than a walkthrough.

 Several organizations consider formal code inspections as a tool to improve code quality at a
lower cost than incurred when dynamic testing is used.
 Organizations have reported significant increases in productivity and software quality due to the
use of code inspections.
 Code inspection is a rigorous process for assessing the quality of code.

 Code inspection is carried out by a team which works according to an inspection plan that
consists of the following elements: (a) statement of purpose, (b) work product to be inspected, this
includes code and associated documents needed for inspection, (c) team formation, roles, and
tasks to be performed, (d) rate at which the inspection task is to be completed, and (e) data
collection forms where the team will record its findings such as defects discovered, coding
standard violations, and time spent in each task.
 Members of the inspection team are assigned roles of moderator, reader, recorder, and author.
 The moderator is in charge of the process and leads the review.
 Actual code is read by the reader, perhaps with the help of a code browser and with monitors for
all in the team to view the code.
 The recorder records any errors discovered or issues to be looked into.
 The author is the actual developer whose primary task is to help others to understand the code.
1.11.3 Software complexity and static testing
 Often a team must decide which of the several modules should be inspected first.
 Several parameters enter this decision-making process—one of these being module complexity.
 A more complex module is likely to have more errors and must be accorded higher priority for
inspection than a module with lower complexity.

I
 Static analysis tools often compute complexity metrics using one or more complexity metrics,
could be used as a parameter in deciding which modules to inspect first. Certainly, the criticality
of the function a module serves in an application could override the complexity metric while
prioritizing modules.
PROBLEM STATEMENTS: GENERALIZED PSEUDO CODE:
Pseudo code provides a natural way to express program source code.

I
The Triangle Problem:
• The triangle program accepts 3 integers: a,b &c as sides of a triangle
• It outputs the type of the triangle from the length of the 3 sides – Equilateral (all sides are equal) –
Isosceles (2 sides are equal) – Scalene (all sides are not equal) – Not a triangle (if the sum of any 2 sides
<= the third)
• The triangle property: –
The sum of any pair of sides must be strictly greater than the third side ..(>) not (>=)
•a<b+c
•b<a+c
•c < a + b
• a, b,& c are integers from 1 to 200
Flow Chart for traditional triangle program implementation is as follows:

I
Generalized Pseudo code:
TRIANGLEPROBLEM:
READ VALUES A, B, AND C
SORT VALUES A, B, AND C (ASCENDING) TO GIVE AA, BB, AND CC
IF AA + BB >CC :
IF AA = BB = CC : PRINT "EQUILATERAL TRIANGLE"
ELSE
: IF AA = BB :: PRINT "ISOSCELES TRIANGLE"
: ELSE
:: PRINT "SCALENE TRIANGLE" : ENDIF
: ENDIF ELSE PRINT "TRIANGLE IMPOSSIBLE"
ENDIF
Structured Implementation:
Figure shows the dataflow diagram description of triangle program. We could implement it as a main
program with the three indicated procedures.

I
Note: For structured program: Refer Lab program code.
Next date Function:
Problem statement: NextDate is a function of three variables: month, day and year. It returns the date of
the day after the input date. The month, day and year variables have integer values subject to these
conditions:
C1. 1<=month<=12
C2. 1<=day<=31
C3 1812<=year<=2012
If any of the conditions c1, c2 or c3 fails, NextDate produces an output indicating the corresponding
variable is out-of-range value-for example, “Value of month not in range 1..12.” Because numerous
invalid day-month-year combinations exist, Next Date collapses these into one message: “Invalid Input
date.”
 Pseudo code: (Note: Both version1 and version2 have been specified here. However, from the exam
point of view, only version 1 is sufficient. Interested readers can study version 2 also.)

I

I

I

I
Commission problem:
Problem Statement: A rifle salesperson in the former Arizona Territory sold rifle locks, stocks, and
barrels made by a gunsmith in Missouri. Locks cost $45, stocks cost $30, and barrels cost $25. The
salesperson had to sell at least one complete rifle per month, and production limits were such that the
most the salesperson could sell in a month was 70 locks, 80 stocks, and 90 barrels. After each town visit,
the salesperson sent a telegram to the Missouri gunsmith with the number of locks, stocks, and barrels
sold in that town. At the end of a month, the salesperson sent a very short telegram showing –1 lock sold.
The gunsmith then knew the sales for the month were complete and computed the salesperson’s
commission as follows: 10% on sales up to (and including) $1000, 15% on the next $800, and 20% on
any sales in excess of $1800.
The commission program produced a monthly sales report on:
 The total number of locks,

 The total number of stocks,
 The total number of barrels sold,
 The salesperson’s total dollar sales, and,
 Finally, the commission for the total sales.
Example: Let us consider: Total number of locks sold=10, Total number of stocks sold=10, Total
number of Barrels sold=10,
We can calculate the sales using the formula:

Sales= (number of locks*45+number of stocks*30+number of barrels*25)
Therefore in our example sales would be 1000.0 We can calculate commission using the formula:
If sales>1 & sales<=1000
Then the commission will be= (0.1*sales)
If sales>1000 & sales<=1800
Then the commission will be= (0.1*sales) + (0.15*(sales-1000))
If sales>1800
Then the commission will be= (0.1*sales) + (0.15* (sales-1000)) + (0.2*(sales-1800))
In our Example commission will be=100.0
Structured Pseudo code:
Program commission (INPUT, OUTPUT)
Dim lockPrice, stockPrice, barrelPrice As Real

I
Dim locks, stocks, barrels As Integer

Dim totalLocks, totalStocks As Integer
Dim totalBarrels As Integer
Dim lockSales, stockSales As Real
Dim barrelSales As Real
Dim sales, commission As Real
lockPrice = 45.0, stockPrice = 30.0, barrelPrice = 25.0
totalLocks = 0, totalStocks = 0, totalBarrels = 0
Input (locks)
While NOT (locks = -1)
‘Input device uses -1 to indicate end of data
Input (stocks, barrels)
totalLocks = totalLocks + locks
totalStocks = totalStocks + stocks
totalBarrels = totalBarrels + barrels
Input (locks)
EndWhile
Output ("Locks sold: ", totalLocks)
Output ("Stocks sold: ", totalStocks)
Output ("Barrels sold: ", totalBarrels)
Sales = lockPrice*totalLocks + StockPrice*totalStocks + barrelPrice * totalBarrels
Output ("Total sales: ", sales)
If (sales > 1800.0)
Then commission = 0.10 * 1000.0
commission = commission + 0.15 * 800.0
commission = commission + 0.20*(sales-1800.0)
Else If (sales > 1000.0)
Then commission = 0.10 * 1000.0
commission = commission + 0.15*(sales-1000.0)
Else
commission = 0.10 * sales
EndIf
EndIf
Output ("Commission is $", commission)
End Commission
THE SATM System:-Simple automatic teller machine
The below figure shows SATM terminal

I
SATM system is actually used for communicating with bank customers. SATM customers can select any
of three transaction types:
 deposits
 withdrawals
 savings
When a bank customer arrives at an SATM station, screen 1 is displayed. The bank customer accesses the
SATM system with a plastic card encoded with a PAN, which is a key to an internal customer account file
containing ,among other things ,the customer name and account information. If customers PAN matches,
the screen 2 will be displayed to the customer else screen 4 will be displayed.
SATM communicate with bank customers via 15 screens as shown in below figure:

I
Thus SATM is just the day to day ATM services which we use in daily life.
And if there is any error then cancel the transaction and exit from the screen. At screen 2, customer is
asked to enter the PIN. If PIN is correct then screen 5 will be displayed, else screen 3 will be displayed
and user is given 3 chances before the card is blocked. On screen 5, the information provided is
customers account, current date, and increment to the number of ATM session. Customer selects the
transaction to be processed. If balance is requested then screen 14 is displayed. If deposit is requests then
screen 7 is displayed. Once deposit amount is entered then screen 13 is displayed. After this is over
screen 14 would be displayed. If withdrawal is requested screen 7 would be displayed or if it is jammed
then screen 10 is displayed. If the fund to be withdrawn is insufficient then screen 8 is displayed else
screen 11 is displayed and money is dispensed. After cash is removed, system displays screen 14. When
the “No” button is pressed in screen 10, 12 0r 14, then the system presents screen 15 and returns the
customer’s ATM card. Once the cared is removed from the card slot, screen 1 is displayed. When the
“yes” button is pressed in screen 10, 12, or 14, the system presents screen 5 so customer can select
additional transaction. Thus the basic SATM system along with SATM screens is explained.
The Currency Converter

An event-driven program with a graphical user interface (GUI), shown below
 The currency conversion program is another event driven program that emphasizes code
associated with GUI. The application converts U.S dollars to any of four currencies: Brazilian
reals, Canadian dollars, European euros and Japanese yen.
 Currency selection is governed by the radio buttons which are mutually exclusive. When a
country is selected, the system responds by completing the label. If Canadian button is clicked, a
I
small Canadian flag appears next to output position and the converted currency amount is
displayed. Either before or after currency selection, the user inputs an amount in US dollars. Once
tasks are accomplished, the user can click on the computer button, the clear button, or the quit
button. Clicking the computer button results in conversion of the US dollar amount to the
equivalent amount in the selected currency. Clicking on the clear button resets the currency
selection, the US dollar amount, and the equivalent currency amount and the associated label.
Clicking on the quit button ends the application.
Saturn Windshield Wiper Controller
• The windshield wiper on some Saturn cars is controlled by a lever with a dial • The lever has 4
positions: OFF, INT(intermittent), LOW and HIGH • The dial has three positions: 1,2,3 – The dial
positions indicate three intermittent speeds – It is relevant only when the lever is at the INT position
• The decision table below shows the windshield wiper speeds in wipes per minute for the lever and dial
positions.
Question bank:
1. Define the following : a) Error b) Fault c) Failure d) Incident e) Test f) Test case (6M)
July 2015, Jan 2015, July 2014
2. Difference between functional testing and structural testing (6M) Jan2015, July 2014
3.What is software testing? Why it is so important in SDLC? (5M)Jan 2016, Jan 2014
4. Explain the triangle problem statement along with flow chart for traditional implementation. (7M) July
2015, Jan 2014
5. Explain the IEEE error and fault taxonomy and IEEE standard anomaly process in software testing.
(8M) Jan 2016, Jan 2014
6. Explain error and fault taxonomies. (5M) July 2015, July 2013

I
7. Explain in detail various levels of software testing with embedded device like SATM as an example.
(15M) Jan 2015, July 2014, July 2013


Software Testing Basics and Quality Attributes

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Software Testing Basics and Quality Attributes

Uploaded by

Copyright:

Available Formats

Software Testing-15IS63 Dept.

Module1- BASICS OF SOFTWARE TESTING

 Fault of Commission: A fault of commission occurs when we enter something into a

Krupa K S, Asst. Professor 1

 To determine a set of test cases.

Software Testing Life Cycle:

Phases putting Bugs IN:

Phase FINDING Bugs:

Krupa K S, Asst. Professor 2

Software quality is a multidimensional quantity and is measurable.

Krupa K S, Asst. Professor 3

Dynamic quality attributes include software reliability, correctness, completeness, consistency,

 Reliability refers to the probability of failure-free operation.

 Reliability can be considered as a statistical measure of correctness.

Krupa K S, Asst. Professor 4

Software reliability is the probability of failure-free operation of software in its intended

REQUIREMENTS, BEHAVIOR, AND CORRECTNESS:-

 Products, software in particular, are designed in response to requirements.

 Testers test the expected behavior by understanding the requirements.

 Testers are often faced with incomplete and/or ambiguous requirements.

 Regardless of the nature of the requirements, testing requires the determination of

Input Domain and Program Correctness

 A program is considered correct if it behaves as desired on all possible test inputs.

 Testing a program on all possible inputs is known as exhaustive testing.

Valid and invalid inputs:

CORRECTNESS VERSUS RELIABILITY:-

Krupa K S, Asst. Professor 7

Krupa K S, Asst. Professor 8

Another operational profile for sort is specified as follows:

Krupa K S, Asst. Professor 9

TESTING AND DEBUGGING:-

Figure : A test and debug cycle.

Krupa K S, Asst. Professor 10

 Preparing a test plan :

 A test cycle is often guided by a test plan.

 Constructing Test Data

 A test set is a collection of test cases.

 A test set/ test data is sometimes referred to as a test suite.

 Executing the program

 The complexity of actual program execution is dependent on the program itself.

Figure : A simple test harness to test the sort program.

In preparing this test harness we assume that :

Krupa K S, Asst. Professor 13

 A typical test case contains the following information:

Test case contains :

Krupa K S, Asst. Professor 14

 Expected outputs – 2 types : postconditions and actual outputs.

 Test cases are as valuable as source code.

 Test cases need to be developed, reviewed, used, managed and saved.

INSIGHT FROM VENN DIAGRAM:-

Figure below shows, specified, implemented and tested behaviors.

The new circle in the below figure is for test cases.

Krupa K S, Asst. Professor 15

Region 1: Specified behaviors that are programmed and tested.

IDENTIFYING THE TEST CASES:-

1. Black box/Functional testing:

2. White box/Structural/Clear box testing:

TEST- GENERATION STRATEGIES:-

Krupa K S, Asst. Professor 17

Figure below summarizes several strategies for test generation.

Figure: Requirements, models, and test generation algorithms.

Krupa K S, Asst. Professor 18