jatinqaer@hotmail.

com Testing Document Software Testing Engineer

----------------------------------------------------------------------------------------------

Black box

Black-box Testing treats the system as a "black-box", so it doesn't use knowledge of

the internal structure explicitly. It is usually described as focusing on testing
functional requirements. Synonyms for black box include: behavioral, functional,
opaque-box, and closed-box. Black Box Testing is testing without knowledge of the
internal workings of the item being tested. For example, when black box testing is
applied to software engineering, the tester would only know the "legal" inputs and
what the expected outputs should be, but has no idea how the program actually
arrives at those outputs

It is because of this that black box testing can be considered testing with respect to
the specifications; no other knowledge of the program is necessary. For this reason,
the tester and the programmer can be independent of one another, avoiding
programmer bias toward his own work.

The so-called ``functionality testing'' is central to most testing exercises. Its primary
objective is to assess whether the program does what it is supposed to do, i.e. it
should meet user specified requirements. There are different approaches to
functionality testing. One is the testing of each program feature or function in
sequence. The other is to test module by module, i.e. each function where it is called
first.

Advantages:

• More effective on larger units of code than glass box testing

• Tester needs no knowledge of implementation, including specific programming
languages
• Tester and programmer are independent of each other
• Tests are done from a user's point of view

• Test cases can be designed as soon as the specifications are complete

Disadvantages

• Only a small number of possible inputs can actually be tested, to test every
possible input stream would take nearly forever
• Without clear and concise specifications, test cases are hard to design
• There may be unnecessary repetition of test inputs if the tester is not
informed of test cases the programmer has already tried

• May leave many program paths untested

• Cannot be directed toward specific segments of code which may be very

complex (and therefore more error prone)

Most testing related research has been directed toward glass box testing

1 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Techniques in Black Box

Black box testing attempts to derive sets of inputs that will fully exercise all the
functional requirements of a system. It is not an alternative to white box
testing. This type of testing attempts to find errors in the following categories:

• Incorrect or missing functions,

• Interface errors,
• Errors in data structures or external database access,
• Performance errors, and

• Initialization and termination errors.

• For input ranges bounded by a and b, test cases should include values a and
b and just above and just below a and b respectively.
• If an input condition specifies a number of values, test cases should be
developed to exercise the minimum and maximum numbers and values just
above and below these limits.
• Apply guidelines 1 and 2 to the output.
• If internal data structures have prescribed boundaries, a test case should be
designed to exercise the data structure at its boundary.

Cause-Effect Graphing Techniques

Cause-effect graphing is a technique that provides a concise representation of logical
conditions and corresponding actions. There are four steps:

• Causes (input conditions) and effects (actions) are listed for a module and an
identifier is assigned to each.
• A cause-effect graph is developed.
• The graph is converted to a decision table.

Decision table rules are converted to test cases.

Black Box Manual Testing

SQA team members upon receipt of the Development builds, walk through the GUI
and either update existing hard copy of the product Roadmaps, or create new hard
copy. This is then passed on to the Tools engineer to automate for new builds and
regression testing. Defects are entered into the bugs tracking database, for
investigation and resolution.

Features & Functions - SQA test engineers, swearing on the team definition,
exercise the product features and functions accordingly. Defects in feature/function
capability are entered into the defect tracking system and are communicated to the
team. Features are expected to perform as expected and their functionality should be
oriented toward ease of use and clarity of objective.

2 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Tests are planned around new features and regression tests are exercised to validate

Existing features and functions are enabled and performing in a manner consistent
with prior releases. SQA using the exploratory testing method manually tests and
then plans more exhaustive testing and automation. Regression tests are exercised
which consist of using developed test cases against the product to validate field
input, boundary conditions and so on... Automated tests developed for prior releases
are also used for regression testing.

Installation - Product is installed on each of the supported operating systems in

either default, flat file configuration, or with one of the supported databases. Every
operating system and database, supported by the product, is tested, though not in
all possible combinations. SQA is committed to executing, during the development
life cycle, the combinations most frequently used by the customers. Clean and
upgrade installations are the minimum requirements.

Documentation - All documentation, which is reviewed by Development prior to

Alpha, is reviewed by the SQA

Team prior to Beta. SQA not only verifies technical accuracy, clarity and
completeness, they also provide editorial input on consistency, style and
typographical errors.

Functionality Testing

Functional testing is validating an application or web site, conforms to its

specifications and correctly performs all its required functions.

This entails a series of tests, which perform a feature-by-feature validation of

behavior, using a wide range of normal and erroneous input data. This can involve
testing of the product's user interface, database management, security, installation,
networking, etc

The purpose of functionality testing is to reveal issues concerning the product’s

functionality and conformance to user requirement.

The first step in functionality testing is to become familiar with the program itself,
and with the program’s desired behavior. For this the tester should have clear idea
about the documentation such as the program’s functional specification or user
manual. Once a program’s expected functionality has been defined, test cases or test
procedures can be created that exercise the program in order to test actual behavior
against expected behavior. Testing the program’s functionality then involves the
execution of any test cases that have been created. Certain portions of a
functionality testing effort can also be automated, depends on several factors, and
should be discussed with a qualified engineer.

1. Range checking- minimum and maximum values should not be exceeded

(invalid values should not be accepted)
2. Check whether numeric fields accept only numeric values
3. Check ‘online Help’ feature (including buttons to open Help feature)
4. Check ‘Print’ feature

3 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

5. Check ‘Open file’ feature (must open correct file extensions and incorrect file
type should give error messages)
6. Check ‘Graph’ features
7. If there are logins, enter invalid login information for each field
8. Check for error messages for clarity and whether they come up when they are
supposed to.
9. In the presence of a database, check all connections through application are
valid when accessing data (error messages like “could not connect to
database” should not appear.
10. Modify data files (like add extra special characters) to make sure the
application gives correct error messages
11. For administrative features make sure only administrators of application may
access the features
12. Check by adding duplicate records
13. Delete all records to check whether such an action does not crash the
application
14. Check for compatibility using MS Office application (like copy and paste)
15. Click all buttons to make sure all of them are functioning appropriately
16. Click ‘save’ feature (should not be able to overwrite existing file without
permission), should save to correct directory, must create correct extension)
17. Check options/settings
18. Check international units are converted correctly
19. Make sure no spellings are incorrect
20. Check for valid date formats
21. Make sure windows are properly minimized, maximized and resized
22. Check whether keyboard shortcuts are working properly
23. Check that right mouse clicks show correct pop up menus
24. If hardware/software keys are present check if the application works as
intended with and without execution of keys

Compatibility Testing

A Testing to ensure compatibility of an application or Web site with different

browsers, OS and hardware platforms, Different versions, configurations, display
resolutions, and Internet connect speeds all can impact the behavior of the product
and introduce costly and embarrassing bugs. We test for compatibility using real test
environments. That is testing how will the system performs in the particular
software, hardware or network environment. Compatibility testing can be performed
manually or can be driven by an automated functional or regression test suite

The purpose of compatibility testing is to reveal issues related to the product’s

interaction with other software as well as hardware. The product compatibility is
evaluated by first identifying the hardware/software/browser components that the
product is designed to support. Then a hardware/software/browser matrix is
designed that indicates the configurations on which the product will be tested. Then,
with input from the client, a testing script is designed that will be sufficient to
evaluate compatibility between the product and the hardware/software/browser
matrix. Finally, the script is executed against the matrix, and any anomalies are
investigated to determine exactly where the incompatibility lies.

4 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Some typical compatibility tests include testing your application:

• On various client hardware configurations

• Using different memory sizes and hard drive space
• On various Operating Systems
• In different network environments
• With different printers and peripherals (i.e. zip drives, USB’s, etc.)

Regression Testing

Regression testing is testing the module in which a bug was identified earlier along
with the impacted areas to ensure that this fix has not introduced any further
defects.

The purpose of regression testing is to ensure that previously detected and fixed
issues really are fixed, they do not reappear, and new issues are not introduced into
the program as a result of the changes made to fix the issues.

Regression testing also referred to as verification testing, is initiated after a

programmer has attempted to fix a recognized problem or has added source code to
a program that may have inadvertently introduced errors. It is a quality control
measure to ensure that the newly modified code still complies with its specified
requirements and that unmodified code has not been affected by the maintenance
activity.

Regression Testing is in general a black box testing strategy where test case
execution of previously written test cases, that has exposed bugs, is done to check
whether previously fixed faults have reemerged. In a test suite, all the tests that has
caused bug are written and are re-tested whenever changes are made to the
program to fix any bug. But this is a tedious process as after every compilation it is
difficult to go through the process of retesting all the test cases repeatedly. To make
this process simpler regression testing is automated using some testing tools?

Typically regression testing should be performed on a daily basis. Once an issue in

the defect

Tracking database has been fixed it is reassigned back for final resolution. Now it can
be either reopens the issue, if it has not been satisfactorily addressed, or close the
issue if it has, indeed, been fixed.

Performance Testing

Performance testing is a rigorous usability evaluation of a working system under

realistic conditions to identify usability problems and to compare measures such as
success rate, task time and user satisfaction with requirements.

The goal of performance testing is not to find bugs, but to eliminate bottlenecks and
establish a baseline for future regression testing

5 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

To conduct performance testing is to engage in a carefully controlled process of

measurement and analysis. Ideally, the software under test is already stable enough
so that this process can proceed smoothly.

A clearly defined set of expectations is essential for meaningful performance testing.

For example, for a Web application, you need to know at least two things:

• Expected load in terms of concurrent users or HTTP connections

• Acceptable response time

Load testing:
Load testing is usually defined as the process of exercising the system under test by
feeding it the largest tasks it can operate with. Load testing is sometimes called
volume testing, or longevity/endurance testing
Examples of volume testing:

• Testing a word processor by editing a very large document

• Testing a printer by sending it a very large job
• Testing a mail server with thousands of users mailboxes

Examples of longevity/endurance testing:

• Testing a client-server application by running the client in a loop against the

server over an extended period of time

Goals of load testing:

• Expose bugs that do not surface in cursory testing, such as memory

management bugs, memory leaks, buffer overflows, etc.

• Ensure that the application meets the performance baseline established

during performance testing. This is done by running regression tests against
the application at a specified maximum load.

Although performance testing and load testing can see similar, their goals are
different. On one hand, performance testing uses load testing techniques and tools
for measurement and benchmarking purposes and uses various load levels whereas
load testing operates at a predefined load level, the highest load that the system can
accept while still functioning properly.
Stress testing:
Stress testing is a form of testing that is used to determine the stability of a given
system or entity. This is designed to test the software with abnormal situations.
Stress testing attempts to find the limits at which the system will fail through
abnormal quantity or frequency of inputs.
Stress testing tries to break the system under test by overwhelming its resources or
by taking resources away from it (in which case it is sometimes called negative
testing). The main purpose behind this madness is to make sure that the system fails
and recovers gracefully -- this quality is known as recoverability.

6 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Stress testing does not break the system but instead it allows observing how the
system reacts to failure. Stress testing observes for the following.

• Does it save its state or does it crash suddenly?

• Does it just hang and freeze or does it fail gracefully?
• Is it able to recover from the last good state on restart? Etc.

Web Testing

During testing the websites the following scenarios should be considered.

• Functionality
• Performance
• Usability
• Server side interface
• Client side compatibility
• Security

Functionality:

In testing the functionality of the web sites the following should be tested.

• Links

 Internal links
 External links
 Mail links
 Broken links

• Forms

 Field validation
 Functional chart
 Error message for wrong input
 Optional and mandatory fields

• Database

 Testing will be done on the database integrity.

• Cookies

 Testing will be done on the client system side, on the temporary

Internet files.

Performance:
Performance testing can be applied to understand the web site's scalability, or to
benchmark the performance in the environment of third party products such as
servers and middleware for potential purchase.

7 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

• Connection speed:

o Tested over various Networks like Dial up, ISDN etc

• Load

o What is the no? Of users per time?

o Check for peak loads & how system behaves.
o Large amount of data accessed by user.

• Stress

o Continuous load
o Performance of memory, CPU, file handling etc.

Usability:
Usability testing is the process by which the human-computer interaction
characteristics of a system are measured, and weaknesses are identified for
correction.
Usability can be defined as the degree to which a given piece of software assists the
person sitting at the keyboard to accomplish a task, as opposed to becoming an
additional impediment to such accomplishment. The broad goal of usable systems is
often assessed using several criteria:

• Ease of learning
• Navigation
• Subjective user satisfaction
• General appearance

Server side interface:

In web testing the server side interface should be tested. This is done by
Verify that communication is done properly.
Compatibility of server with software, hardware, network and database should be
tested.
The client side compatibility is also tested in various platforms, using various
browsers etc.
Security:
The primary reason for testing the security of a web is to identify potential
vulnerabilities and subsequently repair them.
The following types of testing are described in this section:

• Network Scanning
• Vulnerability Scanning
• Password Cracking
• Log Review
• Integrity Checkers
• Virus Detection

8 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Black Box Testing Skills

Essential Testing Skills needed for Testers:

Test Planning: Analyzing a project to determine the kinds of testing needed, the
kinds of people needed, the scope of testing needed, the kinds of people needed, the
scope of testing (including what should and should not be tested), the time available
for testing activities, the initiation criteria for testing, the completion criteria and the
critical success factors of testing

Test Tool Usage: Knowing which tools are most appropriate in a given testing
situation, how to apply the tools to solve testing problems effectively, how to
organize automated testing, and how to integrate test tools into an organization

Test Execution: Performing various kinds of tests, such as unit testing, system
testing, UAT, stress testing and regression testing. This can also include how to
determine which conditions to test and how to evaluate whether the system under
test passes or fails. Test execution can often be dependent on your unique
environment and project needs, although basic testing principles can be adopted to
test most projects

Defect Management: Understanding the nature of defects, how to report defects,

how to track defects and how to use the information gained from defects to improve
the development and testing processes

Risk analysis: Understanding the nature of risk, how to assess project and software
risks, how to use the results of a risk assessment to prioritize and plan testing, and
how to use risk analysis to prevent defects and project failure.

Test Measurement: Knowing what to measure during a test, how to use the
measurements to reach meaningful conclusions and how to use measurements to
improve the testing and development processes

Black Box Test Approach

Design Validation

Statements regarding coverage of the feature design - including both specification

and development documents. Will testing review design? Is design an issue on this
release? How much concern does testing have regarding design, etc. etc.

Data Validation
What types of data will require validation? What parts of the feature will use what
types of data? What are the data types that test cases will address? Etc.

9 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

API Testing
What level of API testing will be performed? What is justification for taking this
approach (only if none is being taken)

Content Testing

Is your area/feature/product content based? What is the nature of the content? What
strategies will be employed in your feature/area to address content related issues?

Low-Resource Testing

What resources does your feature use? Which are used most, and are most likely to
cause problems? What tools/methods will be used in testing to cover low resource
(memory, disk, etc.) issues?

Setup Testing
How is your feature affected by setup? What are the necessary requirements for a
successful setup of your feature? What is the testing approach that will be employed
to confirm valid setup of the feature?

Modes and Runtime Options

What are the different run time modes the program can be in? Are there views that
can be turned off and on? Controls that toggle visibility states? Are there options a
user can set which will affect the run of the program? List here the different run time
states and options the program has available. It may be worthwhile to indicate here
which ones demonstrate a need for more testing focus.

Interoperability

How will this product interact with other products? What level of knowledge does it
need to have about other programs -- “good neighbor”, program cognizant, program
interaction, and fundamental system changes? What methods will be used to verify
these capabilities?

Integration Testing

Go through each area in the product and determine how it might interact with other
aspects of the project. Start with the ones that are obviously connected, but try
every area to some degree. There may be subtle connections you do not think about
until you start using the features together. The test cases created with this approach
may duplicate the modes and objects approaches, but there are some areas, which
do not fit in those categories and might be missed if you do not check each area.

Compatibility: Clients

Is your feature a server-based component that interacts with clients? Is there a

standard protocol that many clients are expected to use? How many and which
clients are expected to use your feature? How will you approach testing client
compatibility? Is your server suited to handle ill-behaved clients? Are there subtleties
in the interpretation of standard protocols that might cause incompatibilities? Are

10 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

there non-standard, but widely practiced use of your protocols that might cause
incompatibilities?

Compatibility: Servers

Is your feature a client-based component that interacts with servers? Is there a

standard protocol supported by many servers that your client speaks? How many
different servers will your client program need to support? How will you approach
testing server compatibility? Is your client suited to handle ill-behaved or non-
standard servers? Are there subtleties in the interpretation of standard protocols that
might cause incompatibilities? Are there non-standard, but widely practiced use of
protocols that might cause incompatibilities?

Beta Testing

What is the beta schedule? What is the distribution scale of the beta? What are the
entry criteria for beta? How is testing planning on utilizing the beta for feedback on
this feature? What problems do you anticipate discovering in the beta? Who is
coordinating the beta, and how?

Environment/System – General

Are there issues regarding the environment, system, or platform that should get
special attention in the test plan? What are the run time modes and options in the
environment that may cause difference in the feature? List the components of critical
concern here. Are there platform or system specific compliance issues that must be
maintained?

Configuration

Are their configuration issues regarding hardware and software in the environment
that may get special attention in the test plan? Some of the classical issues are
machine and bios types, printers, modems, video cards and drivers, special or
popular TSR’s, memory managers, networks, etc. List those types of configurations
that will need special attention.

User Interface

List the items in the feature that explicitly require a user interface. Is the user
interface designed such that a user will be able to use the feature satisfactorily?
Which part of the user interface is most likely to have bugs? How will the interface
testing be approached?

Performance & Capacity Testing

How fast and how much can the feature do? Does it do enough fast enough? What
testing methodology will be used to determine this information? What criterion will
be used to indicate acceptable performance? If modifications of an existing product,
what are the current metrics? What are the expected major bottlenecks and
performance problem areas on this feature?

11 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Scalability
Is the ability to scale and expand this feature a major requirement? What parts of
the feature are most likely to have scalability problems? What approach will testing
use to define the scalability issues in the feature?

Stress Testing

How does the feature do when pushed beyond its performance and capacity limits?
How is its recovery? What is its breakpoint? What is the user experience when this
occurs? What is the expected behavior when the client reaches stress levels? What
testing methodology will be used to determine this information? What area is
expected to have the most stress related problems?

Volume Testing

Volume testing differs from performance and stress testing in so much as it focuses
on doing volumes of work in realistic environments, durations, and configurations.
Run the software, as expected user will - with certain other components running, or
for so many hours, or with data sets of a certain size, or with certain expected
number of repetitions.

International Issues

Confirm localized functionality that strings are localized and that code pages are
mapped properly. Assure program works properly on localized builds, and that
international settings in the program and environment do not break functionality.
How is localization and internationalization being done on this project? List those
parts of the feature that are most likely to be affected by localization. State
methodology used to verify International sufficiency and localization.

Robustness
How stable is the code base? Does it break easily? Are there memory leaks? Are
there portions of code prone to crash, save failure, or data corruption? How good is
the program’s recovery when these problems occur? How is the user affected when
the program behaves incorrectly? What is the testing approach to find these problem
areas? What is the overall robustness goal and criteria?

Error Testing
How does the program handle error conditions? List the possible error conditions.
What testing methodology will be used to evoke and determine proper behavior for
error conditions? What feedback mechanism is being given to the user, and is it
sufficient? What criteria will be used to define sufficient error recovery?

Usability
What are the major usability issues on the feature? What is testing’s approach to
discover more problems? What sorts of usability tests and studies have been
performed, or will be performed? What is the usability goal and criteria for this
feature?

12 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Accessibility
Is the feature designed in compliance with accessibility guidelines? Could a user with
special accessibility requirements still be able to utilize this feature? What are the
criteria for acceptance on accessibility issues on this feature? What is the testing
approach to discover problems and issues? Are there particular parts of the feature
that are more problematic than others?

User Scenarios

What real world user activities are you going to try to mimic? What classes of users
(i.e. secretaries, artist, writers, animators, construction worker, airline pilot,
shoemaker, etc.) are expected to use this program, and doing which activities? How
will you attempt to mimic these key scenarios? Are there special niche markets that
your product is aimed at (intentionally or unintentionally) where mimic real user
scenarios are critical?

Boundaries and Limits

Are their particular boundaries and limits inherent in the feature or area that deserve
special mention here? What is the testing methodology to discover problems
handling these boundaries and limits?

Operational Issues

If your program is being deployed in a data center, or as part of a customer's

operational facility, then testing must, in the very least, mimic the user scenario of
performing basic operational tasks with the software.

Backup
Identify all files representing data and machine state, and indicate how those will be
backed up. If it is imperative that service remains running, determine whether or not
it is possible to backup the data and still keep services or code running.

Recovery
If the program goes down, or must be shut down, are there steps and procedures
that will restore program state and get the program or service operational again? Are
there holes in this process that may make a service or state deficient? Are there
holes that could provide loss of data? Mimic as many states of loss of services that
are likely to happen, and go through the process of successfully restoring service.

Archiving
Archival is different from backup. Backup is when data is saved in order to restore
service or program state. Archive is when data is saved for retrieval later. Most
archival and backup systems piggyback on each other's processes.
Is archival of data going to be considered a crucial operational issue on your feature?
If so, is it possible to archive the data without taking the service down? Is the data,
once archived, readily accessible?

Monitoring
Does the service have adequate monitoring messages to indicate status,
performance, or error conditions? When something goes wrong, are messages

13 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

sufficient for operational staff to know what to do to restore proper functionality? Are
the "heartbeat" counters that indicate whether or not the program or service is
working? Attempt to mimic the scenario of an operational staff trying to keep a
service up and running.

Upgrade
Does the customer likely have a previous version of your software, or some other
software? Will they be performing an upgrade? Can the upgrade take place without
interrupting service? Will anything be lost (functionality, state, data) in the upgrade?
Does it take unreasonably long to upgrade the service?

Migration
Is there data, script, code or other artifacts from previous versions that will need to
be migrated to a new version? Testing should create an example of installation with
an old version, and migrate that example to the new version, moving all data and
scripts into the new format.
List here all data files, formats, or code that would be affected by migration, the
solution for migration, and how testing will approach each.

Special Code Profiling and Other Metrics

How much focus will be placed on code coverage? What tools and methods will be
used to measure the degree to which testing coverage is sufficiently addressing all of
the code?

Black Box Test Metrics

A Metric is a quantitative measure of the degree to which a system, component or

process possesses a given attribute. Software metrics are measures that are used to
quantify the software, software development resources and software development
process. A metric is defined to be the name of a mathematical function used to
measure some attribute of a product or process. The actual numerical value
produced by a metric is a measure. For example, cyclomatic complexity is a metric;
when applied to program code, the number yielded by the formula is the cyclomatic
complexity measure.

Two general classes of metrics include the following:

• Management metrics, which assist in the management of the software

development process.
• Quality metrics, which are predictors or indicators of the product qualities.

Metrics related to software error detection ("Testing") in the broad sense, grouped
into the following categories:
General metrics that may be captured and analyzed throughout the product life
cycle

14 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Software Requirements metrics, which may give early warning of quality

problems in requirements specifications
Software Design metrics, which may be used to assess the status of software
designs;
Code metrics reveal properties of the program source code;
Test metrics can be used to control the testing process, to assess its effectiveness,
and to set improvement targets;
Software Installation metrics, which are applicable during the installation
process;
Software Operation and Maintenance metrics, including those used in providing
software product support.
Test Metrics
The following are the metrics collected in the testing process.
1.Defect age.
Defect age is the time from when a defect is introduced to when it is detected (or
fixed). Assign the numbers 1 through 6 to each of the software development
activities from software requirements to software operation and maintenance. The
defect age is computed as shown.
(Activity Detected - Activity Introduced)
Average Defect Age = –——————————————————
Number of Defects
2. Defect response time.
This measure is the time between when a defect is detected to when it is fixed or
closed.
3. Defect cost ($ d)
The cost of a defect may be computed as:
$ D = (cost to analyze the defect) + (cost to fix it)
+ (cost of failures already incurred due to it)
4. Defect removal efficiency (DRE).
The DRE is the percentage of defects that have been removed during an activity,
computed with the equation below. The DRE can also be computed for each software
development activity and plotted on a bar graph to show the relative defect removal
efficiencies for each activity. Or, the DRE may be computed for a specific task or
technique (e.g., design inspection, code walkthrough, unit test, 6 month operation,
etc.). [SQE]
Number Defects Removed
DRE = –—————————————————— * 100
Number Defects At Start Of Process
5 Mean time to failure (MTTF).
Gives an estimate of the mean time to the next failure, by accurately recording
failure times t i, the elapsed time between the ith and the (i-1) st failures, and
computing the average of all the failure times. This metric is the basic parameter
required by most software reliability models. High values imply good reliability.
MMTF should be corrected by a weighted scheme similar to that used for computing
Fault density (see under Test Metrics).
6. Fault density (FD).
This measure is computed by dividing the number of faults by the size (usually in
KLOC, thousands of lines of code).

15 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Black Box Test Plan

Test planning is one of the keys to successful software testing. Test plan can be
defined as a document that describes the scope, approach, resources and schedule
of intended test activities. The main purpose of preparing test plan is that every one
concerned with the project are in synchronized with regards to scope, deliverables,
deadlines and response for the project.

The complete document will help the people outside the test group understand the
“WHY” and “HOW” of the product validation.

Test planning can and should occur at several levels. The first plan to consider is the
Master Test Plan. The purpose of the Master Test Plan is to orchestrate testing at all
levels (unit, integration, system, acceptance, beta, etc.). The Master Test Plan is to
testing what the Project Plan is to the entire development effort.

The goal of test planning is not to create a long list of test cases, but rather to deal
with the important issues of testing strategy, resource utilization, responsibilities,
risks, and priorities.

Contents of test plan:

Purpose:

This section should contain the purpose of preparing the test plan.

Scope:

This section should talk about the areas of the application, which are to be tested by
the QA team and specify those areas, which are definitely out of the scope.

Test approach:

This would contain details on how the testing is to perform and whether any specific
strategy is to be followed.

Entry criteria:

This section explains the various steps to be performed before the start of test (i.e.)
pre-requisites.

E.g. Environment setup, starting web server/ application server, successful

implementation of latest build etc.

Resources:

This list out the people who would be involved in the project and their designation
etc

Tasks and responsibilities:

16 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

This talk about the tasks to be performed and the responsibilities assigned to the
various members in the project.

Exit criteria:

This contains tasks like bringing down the system or server, restoring system to pre-
test environment, database, refresh etc.

Schedules/ Milestones:

This section deals with the final delivery date and the various milestone dates to be
met in the course of project.

Hardware/ software requirements:

This section contains the details of system/server required to install the application
or perform the testing, specific s/w that needs to be installed on the system to get
the application running or to connect to the database, connectivity related issues etc.

Risks and mitigation process:

This section should list out all the possible risks that can arise during the testing and
mitigation plans that the QA team plans to implement incase the risk actually turns
into a reality.

Tools to be used:

This would list out the testing tools or utilities that are to be used in the project.

E.g. Winrunner, QTP, Test Director PCOM etc.

Deliverables:

This section contains various deliverables that are due to the client at various points
of time. I.e. daily, weekly, start of project, end of project etc. these could include test
plans, test procedures, test matrices, status reports, test scripts etc. templates for
all these also be attached.

Annexure:

This section contains the embedded documents or links to document, which have
been/will be used in the course of testing. E.g. Templates used for reports, test cases
etc. reference documents can also be attached here.

Sign off:

This section contains the mutual agreement between the client and QA team with
both leads/ -managers signing off their agreement on the test plan.

17 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

SOFTWARE TESTING TYPES:

Acceptance Testing:

Testing conducted to enable a user/customer to determine whether to accept a

software product. Normally performed to validate the software meets a set of agreed
acceptance criteria.

Accessibility Testing:

Verifying a product is accessible to the people having disabilities (deaf, blind,

mentally disabled etc.).

Ad Hoc Testing:

Ad-hoc testing is the interactive testing process where developers invoke application
units explicitly,

And individually compare execution results to expected results.

Agile Testing:

Testing practice for projects using agile methodologies, treating development as the
customer of testing and emphasizing a test-first design paradigm. See also Test
Driven Development.

Alpha Testing: Early testing of a software product conducted by selected

customers.

Automated Testing:

• Testing employing software tools which execute tests without manual intervention.
Can be applied in GUI, performance, API, etc. testing.

• The use of software to control the execution of tests, the comparison of actual
outcomes to predicted outcomes, the setting up of test preconditions, and other test
control and test reporting functions.

Basis Path Testing:

A white box test case design technique that uses the algorithmic flow of the program
to design tests.

Beta Testing:

Testing of a re-release of a software product conducted by customers.

Binary Portability Testing:

18 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Testing an executable application for portability across system platforms and

environments, usually for conformation to an ABI specification.

Black Box Testing:

Testing based on an analysis of the specification of a piece of software without

reference to its internal workings. The goal is to test how well the component
conforms to the published requirements for the component.

Bottom Up Testing:

An approach to integration testing where the lowest level components are tested
first, then used to facilitate the testing of higher-level components. The process is
repeated until the component at the top of the hierarchy is tested.

Boundary Testing:

Test which focus on the boundary or limit conditions of the software being tested.
(Some of these tests are stress tests).

Branch Testing:

Testing in which all branches in the program source code are tested at least once.

Breadth Testing:

A test suite that exercises the full functionality of a product but does not test
features in detail.

Compatibility Testing:

Testing whether software is compatible with other elements of a system with which it
should operate, e.g. browsers, Operating Systems, or hardware.

Concurrency Testing:

Multi-user testing geared towards determining the effects of accessing the same
application code, module or database records. Identifies and measures the level of
locking, deadlocking and use of single thread code and locking semaphores.

Conversion Testing:

Testing of programs or procedures used to convert data from existing systems for
use in replacement systems.

Data Driven Testing :

Testing in which the action of a test case is parameterized by externally defined data
values, maintained as a file or spreadsheet. A common technique in Automated
Testing.

19 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Dependency Testing:

Examines an application's requirements for pre-existing software, initial states and

configuration in order to maintain proper functionality.

Depth Testing:

A test that exercises a feature of a product in full detail.

Dynamic Testing :

Testing software through executing it. See also Static Testing.

Endurance Testing :

Checks for memory leaks or other problems that may occur with prolonged
execution.

End-to-End testing :

Testing a complete application environment in a situation that mimics real-world use,

such as interacting with a database, using network communications, or interacting
with other hardware, applications, or systems if appropriate.

Exhaustive Testing :

Testing which covers all combinations of input values and preconditions for an
element of the software under test.

Gorilla Testing :

Testing one particular module, functionality heavily.

Gray Box Testing :

A combination of Black Box and White Box testing methodologies: testing a piece of
software against its specification but using some knowledge of its internal workings.

Integration Testing :

Testing of combined parts of an application to determine if they function together

correctly. Usually performed after unit and functional testing. This type of testing is
especially relevant to client/server and distributed systems.

Installation Testing :

Confirms that the application under test recovers from expected or unexpected
events without loss of data or functionality. Events can include shortage of disk
space, unexpected loss of communication, or power out conditions.

20 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Localization Testing :

This term refers to making software specifically designed for a specific locality.

Loop Testing :

A white box testing technique that exercises program loops.

Monkey Testing :

Testing a system or an Application on the fly, i.e just few tests here and there to
ensure the system or an application does not crash out.

Negative Testing :

Testing aimed at showing software does not work. Also known as "test to fail".

N+1 Testing:

A variation of Regression Testing. Testing conducted with multiple cycles in which

errors found in test cycle N is resolved and the solution is retested in test cycle N+1.
The cycles are typically repeated until the solution reaches a steady state and there
are no errors.

Path Testing:

Testing in which all paths in the program source code are tested at least once.

Performance Testing:

Testing conducted to evaluate the compliance of a system or component with

specified performance requirements. Often this is performed using an automated test
tool to simulate large number of users. Also know as "Load Testing".

Positive Testing:

Testing aimed at showing software works. Also known as "test to pass". See also
Negative Testing.

Recovery Testing:

Confirms that the program recovers from expected or unexpected events without
loss of data or functionality. Events can include shortage of disk space, unexpected
loss of communication, or power out conditions.

Regression Testing:

Retesting a previously tested program following modification to ensure that faults

have not been introduced or uncovered as a result of the changes made.

21 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Sanity Testing:

Brief test of major functional elements of a piece of software to determine if it’s

basically operational.

Scalability Testing:

Performance testing focused on ensuring the application under test gracefully

handles increases in workload.

Security Testing:

Testing which confirms that the program can restrict access to authorized personnel
and that the authorized personnel can access the functions available to their security
level.

Smoke Testing:

A quick-and-dirty test that the major functions of a piece of software work.

Soak Testing:

Running a system at high load for a prolonged period of time. For example, running
several times more transactions in an entire day (or night) than would be expected in
a busy day, to identify and performance problems that appear after a large number
of transactions have been executed.

Static Testing:

Analysis of a program carried out without executing the program.

Storage Testing:

Testing that verifies the program under test stores data files in the correct directories
and that it reserves sufficient space to prevent unexpected termination resulting
from lack of space. This is external storage as opposed to internal storage.

Stress Testing:

Testing conducted to evaluate a system or component at or beyond the limits of its

specified requirements to determine the load under which it fails and how. Often this
is performance testing using a very high level of simulated load.

Structural Testing:

Testing based on an analysis of internal workings and structure of a piece of

software.

System Testing:

22 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Testing that attempts to discover defects that are properties of the entire system
rather than of its individual components.

Thread Testing:

A variation of top-down testing where the progressive integration of components

follows the implementation of subsets of the requirements, as opposed to the
integration of components by successively lower levels.

Top Down Testing:

An approach to integration testing where the component at the top of the component
hierarchy is tested first, with lower level components being simulated by stubs.
Tested components are then used to test lower level components. The process is
repeated until the lowest level components have been tested.

Usability Testing:

Testing the ease with which users can learn and use a product.

User Acceptance Testing:

A formal product evaluation performed by a customer as a condition of purchase.

Unit Testing:

The testing done to show whether a unit satisfies its functional specification or its
implemented structure matches the intended design structure.

Volume Testing:

Testing which confirms that any values that may become large over time can be
accommodated by the program and will not cause the program to stop working or
degrade its operation in any manner.

White Box Testing:

Testing based on an analysis of internal workings and structure of a piece of

software. Includes techniques such as Branch Testing and Path Testing. Also known
as Structural Testing and Glass Box Testing.

Workflow Testing:

Scripted end-to-end testing which duplicates specific workflows, which are expected
to be utilized by the end-user.

23 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Black Box Auto Tools

Nowadays automated testing tools more often used than ever before to ensure that
their applications are working properly prior to deployment. That's particularly
important today, because more applications are written for use on the Web—the
most public of venues. If a browser-based application crashes or performs
improperly, it can cause more problems than a smaller, local application. But for
many IT and quality assurance managers, the decision of which testing tools to use
can cause confusion

The first decision is which category of tool to use— one that tests specific units of
code before the application is fully combined, one that tests how well the code is
working as envisioned, or one that tests how well the application performs under
stress. And once that decision is made, the team must wade through a variety of
choices in each category to determine which tool best meets its needs.

Functional-Testing Tools

Automated Testing is automating the manual testing process currently in use. The
real use and purpose of automated test tools is to automate regression testing. This
means that you must have or must develop a database of detailed test cases that
are repeatable, and this suite of tests is run every time there is a change to the
application to ensure that the change does not produce unintended consequences.

At a functional level, they provide record/playback capabilities, which allow

developers to record an existing application and modify scripts to meet changes in an
upcoming release. Tools in this category include:

Win Runner provides a relatively simple way to design tests and build reusable
scripts without extensive programming knowledge. Win Runner captures, verifies,
and replays user interactions automatically, so you can identify defects and ensure
that business processes work flawlessly upon deployment and remain reliable. Win
Runner supports more than 30 environments, including Web, Java, and Visual Basic.
It also provides solutions for leading ERP and Customer Relationship Management
(CRM) applications.

Astra Quick Test: This functional-testing tool is built specifically to test Web-based
applications. It helps ensure that objects, images, and text on Web pages function
properly and can test multiple browsers. Astra Quick Text provides record and
playback support for every ActiveX control in a Web browser and uses checkpoints to
verify specific information during a test run.

Silk Test tool is specifically designed for doing regression and FUNCTIONALITY
testing. This tool tests both mainframe and client/server applications. This is the
leading functional testing product foe e-business application. It also provides facilities
for rapid test customization and automated infrastructure development.

Rational Suite Test Studio is a full suite of testing tools. Its functional-testing
component, called Rational Robot uses automated regression as the first step in the

24 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

functional testing process. The tool records and replays test scripts that recognize
objects through point-and-click processes. The tool also tracks, reports, and charts
information about the developer's quality assurance testing process and can view
and edit test scripts during the recording process. It also enables the developer to
use the same script to test an application in multiple platforms without modifications.

QACenter is a full suite of testing tools. One functional tool considered part of
QACenter is QARun, which automates the creation and execution of test scripts,
verifies tests, and analyzes test results. A second functional tool under the QACenter
is Test Partner, which uses visual scripting and automatic wizards to help test
applications based on Microsoft, Java, and Web-based technologies. Test Partner
offers fast record and playback of application test scripts, and provides facilities for
testers without much programming experience to create and execute tests.

White Box Testing

Software testing approaches that examine the program structure and derive test
data from the program logic. White-box testing strategies include designing tests
such that every line of source code is executed at least once, or requiring every
function to be individually tested.

Test coverage is an important component of white-box testing. The goal is to try to

execute (that is, test) all lines in an application at least once

Because white-box testing tools can individually or collectively instrument source

lines, it is straightforward to determine which lines in a host program have or have
not been executed without modifying source

Synonyms for white box testing:

• Glass Box testing

• Structural testing
• Clear Box testing
• Open Box Testing

The purpose of white box testing:

Build quality throughout the life cycle of a software product or service.
Provide a complementary function to black box testing.
Perform complete coverage at the component level.
Improve quality by optimizing performance.
Code Coverage Analysis:
Basis Path Testing
A testing mechanism that derive a logical complexity measure of a procedural design
and use this as a guide for defining a basic set of execution paths. These are test
cases that exercise basic set will execute every statement at least once.
Flow Graph Notation: A notation for representing control flow similar to flow charts
And UML activity diagrams.
Cyclomatic Complexity: The Cyclomatic complexity gives a quantitative measure of
4th logical complexity. This value gives the number of independent paths in the
Basis set, and an upper bound for the number of tests to ensure that each statement
Is executed at least once. An independent path is any path through a program that

25 of 26
jatinqaer@hotmail.com Testing Document Software Testing Engineer
----------------------------------------------------------------------------------------------

Introduces at least one new set of processing statements or a new condition.

Cyclomatic complexity provides upper bound for number of tests required
To guarantee coverage of all program statements
Control Structure testing:
Conditions Testing
Condition testing aims to exercise all logical conditions in a program module. They
May define:
Relational expression: (E1 op E2), where E1 and E2 are arithmetic expressions.
Simple condition: Boolean variable or relational expression, possibly proceeded by a
NOT operator.
Compound condition: composed of two or more simple conditions, Boolean operators
And parentheses.
Boolean expression: Condition without Relational expressions.
Data Flow Testing
Selects test paths according to the location of definitions and use of variables
Loop Testing
Loops fundamental to many algorithms. Can define loops as simple, concatenated,
Nested, and unstructured.
Advantages of White Box Testing

• Forces test developer to reason carefully about implementation

• Approximate the partitioning done by execution equivalence
• Reveals errors in "hidden" code
• Beneficent side-effects

Disadvantages of White Box Testing

• Expensive
• Cases omitted in the code could be missed out.

26 of 26