You are on page 1of 122

Testing Manual

CONTENTS 1

Testing Manual

I. The Software life cycle 1.1 1.2 1.3 1.4 1.5 1.6 Feasibility Study Systems Analysis Systems Design Coding Testing Installation & Maintenance

2. SDLC Models 2.1 2.2 2.3 2.4 2.5 2.6 2.7 Code-and-fix model Waterfall model Prototyping model Incremental V-Model Spiral model RAD model

3. Testing Life Cycle 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 System Study Scope/ Approach/ Estimation Test Plan Design Test Cases Design Test Case Review Test Case Execution Defect Handling Gap Analysis Deliverables

4 Testing Phases The V Model 4.1 4.2 4.3 4.4 4.5 4.6 Requirement Analysis Testing Design Testing Unit Testing Integration Testing System Testing Acceptance Testing

Testing Manual

5 Testing Methods FURRPSC Model 5.1 Functionality Testing 5.2 Usability Testing 5.3 Reliability Testing 5.4 Regression Testing 5.5 Performance Testing 5.6 Scalability Testing 5.7 Compatibility Testing 5.8 Security Testing 5.9 Installation Testing 5.10 Adhoc Testing 5.11 Exhaustive Testing 6 Performance Life Cycle 6.1 What is Performance Testing 6.2 Why Performance Testing 6.3 Performance Testing 6.4 Load Testing 6.5 Stress Testing 6.6 When should we start Performance Testing 6.7 Popular tools used to conduct Performance Testing 6.8 Performance Test Process 7 Life Cycle of Automation 7.1 7.2 7.3 7.4 What is Automation? Benefits of Test Automation False Benefits What are the different tools available in the market?

8 Testing 8.1 Test Strategy 8.2 Testing Approach 8.3 Test Environment 8.4 Risk Analysis 8.5 Testing Limitations 8.6 Testing Objectives 8.7 Testing Metrics 8.7 Test Stop Criteria: 8.8 Six Essentials of Software Testing 8.9 Five common problems in s/w development process? 8.10 Solution for common problems that occur during software development. 3

Testing Manual

8.11 What Should be done no enough time for testing 8.12 How do you know when to stop testing? 8.14 Why does the software have Bugs? 9 Tester Responsibilities 9.1 9.2 9.3 9.4 Test Manager Test Lead Tester Engineer How to Prioritize Tests

10 How can we improve the efficiency in testing? 11 CMM and ISO Standards 12 Bug Report Template 13 Testing Glossary 14 Interview Questions

Testing Manual

What Manual Testing?


Testing activities performed by people without the help of Software Testing Tools.

What is Software Quality?


It is reasonably bug free delivered on time with in the budget, meets all requirements and it is maintainable.

1. The Software life cycle


All the stages from start to finish that take place when developing a new Software. Feasibility Study Analysis Feasibility Study What exactly is this system supposed to do? Analysis Determine and list out the details of the problem. Design How will the system solve the problem? Coding Translating the design into the actual system. Testing Does the system solve the problem? Have the requirements been satisfied? Does the system work properly in all situations? Maintenance Bug fixes

Design Coding Testing Installation & Maintenance

The software life-cycle is a description of the events that occur between the birth and death of a software project inclusively. SDLC is separated into phases (steps, stages)

SLDC also determines the order of the phases, and the criteria for transitioning from phase to phase

The Analyst conducts an initial study of the problem and asks is the solution Change Requests on Requirement Specifications Technologically possible? Why Customer ask Change Requests Economically possible? Legally possible? Different users/customers have different requirements. Operationally possible? Requirements get clarified/ known at a later date

Testing Manual

s i Changes to business environment b l Technology changes e Misunderstanding of the stated requirements due to lack of domain ? knowledge O p How to Communicate the Change Requests to team e Formal indication of Changes to requirements r a Joint Review Meetings t Regular Daily Communication i o Queries n Defects reported by Clients during testing. a l Client Reviews of SRS, SDD, Test plans etc. l Across the corridor/desk (Internal Projects) y Presentations/ Demonstrations Classification Specific Generic Categorization Bug Enhancement Clarification etc. Impact Analysis Identify the Items that will be effected Time estimations Any other clashes / open issues raised due to this? Analyzing the Changes p o s s i b l e ? S c h e d u l e d t i m e 6 s c a

e
Testing Manual

Benefits of accepting Change Requests 1. Direct Benefits Facilitates Proper Control and Monitoring Metrics Speak for themselves You can buy more time. You may be able to bill more. Builds Customer Confidence. st e m s a n a l y s t

p o s s i b l e ?

2. Indirect Benefits: What can be done if the requirements are changing

continuously?
Work with project stakeholders early on to understand how the requirements might change. So that alternative test plans and strategies can be worked out in advance.

It is helpful if the application initial design has some adaptability. So c that later changes do not require redoing the application from scratch.o n to If the code is well commented and well documented. Then it is easy d make changes for developers. u Use rapid prototyping whenever possible to help customers feel sure c of their requirements and minimize changes. t s the Negotiate to allow only easily-implemented new requirements into project, while moving more difficult new requirements into future versions of the application. a n i n i t i a l s t u d y o f 7 t h e

Testing Manual

1.1 Feasibility Study:


Feasibility Study Analysis (Requirements) Design

r o b l e m a n d a s k s i s t h e s o l u t i o n T e c h n o l o g i c a l l y p o s s i b l e ?

Coding Testing Installation & Maintenance

The feasibility report


BRS (Business Requirement Document) Applications areas to be considered (Stock control, banking, Accounts etc) System investigations and System requirements for each application Cost estimates Timescale for implementation Expected benefits

c o n Testing Manual o m i 1.2 Systems Analysis: c a The process of identifying problems, resources opportunities, requirements l and constraints l System analysis is the process of Investigating a business with a y view to determining how best to manage the various procedures and information p processing tasks that it involves. o s Feasibility Study s 1.2.1 The Systems Analyst i Analysis Performs the investigation andb might l recommend the use e of a computer to improve the efficiency of the ? information system. Design L e 1.2.2 Systems Analysis g a Coding The intention to determine how well a l business copes l with its current information processing needs y Testing Whether it is possible to improve the p procedures in order o to make it more efficient or profitable. Installation & s Maintenance s The (BRS, FRS and SRS) Documents Bridge the communication i Gap between the client, user, developer and Tester b The System Analysis Report l SRS(Software Requirement Specification) e ? Use Cases( User action and system Response) O FRS(Functional Requirement Document) Or Functional p e specifications r a [These 3 are the Base documents for writing Test Cases] t Documenting the results i o Systems flow charts n Data flow diagrams a l Organization charts l Report y Note p FRS contains Input, Output, process but no format. o Use Cases contains user action and system response with fixed s 9 s i

l e ? S Testing Manual c h e 1.3 Systems Design: d u The business of finding a way to meet the functional requirements l within the specified constraints using the available technology e Planning the structure of the information system to be implemented. d Systems analysis determines what the system should do Design determines how it should be done. t i m e s c a l e User interface design Design of output reports Input screens Data storage i .e files, database Tables System security Backups, validation, passwords Test plan p o s s i b l e ?

Feasibility Study Analysis

Design

Coding

Testing Installation & Maintenance

System Design Report consist of Architectural Design Database Design Interface Design High Level Design Low Level Design

Design Phases

10

Testing Manual

High Level Design


1. 2. 3. 4. 5. 6. List of modules and a brief description of each Brief functionality of each module Interface relationship among modules Dependencies between modules Database tables identified with key elements Overall architecture diagrams along with technology

details

Low Level Design


1. Detailed functional logic of the module, in pseudo code 2. Database tables, with all elements, including their type and size 3. All interface details 4. All dependency issues 5. Error MSG listing 6. Complete input and output format of a module Note: HLD and LLD phases put together called Design phase

1.4 Coding:
Feasibility Study Analysis Translating the design into the actual system Program development Unite testing by Development Tem (Dev Tem)

Design

Coding Testing Installation & Maintenance

Coding Report All the programs, Functions, Reports that related to System

11

Testing Manual

1.5 Testing:
Feasibility Study Analysis Design Coding Testing 1.5.1 What Is Software Testing? IEEE Terminology : An examination of the behavior of the program by executing on sample data sets. TestingisExecutingaprogramwithanintentionoffindingdefects Testing is executing a program with an indent of finding Error/Fault and Failure.

Fault : It is a condition that causes the software to fail to perform its required function.

Error : Refers to difference between Actual Output and Expected output. Installation & Maintenance F F Failure : It is the inability of a system or component to perform required function according to its specification . IEEE Definitions Failure: External behavior is incorrect Fault: Discrepancy in code that causes a failure. Error: Human mistake that caused fault Note: Error is terminology of Developer Bug is terminology of Tester Why is Software Testing? 1. 2. 3. 4. 5. 6. 7. 8. To discover defects. To avoid user detecting problems To prove that the software has no faults To learn about the reliability of the software. To avoid being sued by customers To ensure that product works as user expected. To stay in business To detect defects early, which helps in reducing the cost of defect fixing?

12

Testing Manual

Cost of Defect Repair Phase Requirements Design Coding Testing Customer Site % Cost 0 10 20 50 100 Cost of Defect Repair
100 80 Cost 60 40 20 0 % Cost Requirem 0 Design 10 Coding 20 SDLC Phase Testing 50 Customer 100

How exactly Testing is different from QA/QC Testing is often confused with the processes of quality control and quality assurance.

Testing

It is the process of Creating, Implementing and Evaluating tests. Testing measures software quality Testing can find faults. When they are removed, software quality is

improved. Quality Control (QC)


It is the process of Inspections, Walk-troughs and Reviews. Measures the quality of the product It is a Detection Process

13

Testing Manual

14

Testing Manual

Quality Analysis (QA )


Monitoring and improving the entire SDLC process Make sure that all agreed-upon standards and procedures are followed Ensuring that problems are found and addressed. Measures the quality of the process used to create good quality Product It is a Prevention Process Yes, We definitely need an approach for testing. To over come following problems, we need a formal approach for Testing.

Why should we need an approach for testing?

Incomplete functional coverage: Completeness of testing is difficult task for testing team with out a formal approach. Team will not be in a position to announce the percentage of testing completed. No risk management -- this is no way to measure overall risk issues regarding code coverage and quality metrics. Effective quality assurance measures quality over time and starting from a known base of evaluation. Too little emphasis on user tasks -- because testers will focus on ideal paths instead of real paths. With no time to prepare, ideal paths are defined according to best guesses or developer feedback rather than by careful consideration of how users will understand the system or how users understand real-world analogues to the application tasks. With no time to prepare, testers will be using a very restricted set input data, rather than using real data (from user activity logs, from logical scenarios, from careful consideration of the concept domain). Inefficient over the long term -- quality assurance involves a range of tasks. Effective quality assurance programs expand their base of documentation on the product and on the testing process over time, increasing the coverage and granularity of tests over time. Great testing requires good test setup and preparation, but success with the kind Test planless approach described in this essay may reinforce bad project and test methodologies. A continued pattern of quick-and-dirty testing like this is a sign that the product or application is unsustainable in the long run.

Areas of Testing: Black Box Testing White Box Testing Gray Box Testing 15

Testing Manual

Black Box Testing

Test the correctness of the functionality with the help of Inputs and User doesnt require the knowledge of software code. Black box testing is also called as Functionality Testing. Incorrect or missing functions. Interface errors. Errors in data structures or external data base access. Behavior or performance based errors. Initialization or termination errors. Equivalence Class:

Outputs.

It attempts to find errors in the following categories:

Approach: Class Label the classes as Valid or Invalid Generate one test case for each Invalid Equivalence class Generate a test case that covers as many Valid Equivalence Classes For each piece of the specification, generate one or more equivalence

as possible

An input condition for Equivalence Class A specific numeric value A range of values A set of related values 16

Testing Manual

A Boolean condition

17

Testing Manual

Equivalence classes can be defined using the following guidelines If an input condition specifies a range, one valid and two invalid If an input condition requires a specific value, one valid and two If an input condition specifies a member of a set, one valid and one If an input condition is Boolean, one valid and one invalid classes are Boundary Value Analysis Generate test cases for the boundary values. Minimum Value, Minimum Value + 1, Minimum Value -1 Maximum Value, Maximum Value + 1, Maximum Value - 1 Error Guessing. Generating test cases against to the specification. White Box Testing Testing the Internal program logic White box testing is also called as Structural testing. User does require the knowledge of software code. Purpose Testing all loops Testing Basis paths Testing conditional statements Testing data structures Testing Logic Errors Testing Incorrect assumptions Structure = 1 Entry + 1 Exit with certain Constraints, Conditions and Loops.

equivalence class are defined. invalid equivalence classes are defined. invalid equivalence classes are defined. defined.

18

Testing Manual

Logic Errors and incorrect assumptions most are likely to be made while coding for special cases. Need to ensure these execution paths are tested.

19

Testing Manual

Approach Basic Path Testing (Cyclomatic Complexity(Mc Cabe Method) Measures the logical complexity of a procedural design. Provides flow-graph notation to identify independent paths of Once paths are identified - tests can be developed for - loops, Process guarantees that every statement will get executed at least Structure Testing: Condition Testing All logical conditions contained in the program module should be tested. Data Flow Testing Selects test paths according to the location of definitions and use of variables.

processing conditions once.

Loop Testing Simple Loops Nested Loops Concatenated Loops Unstructured Loops

Gray Box Testing. It is just a combination of both Black box & white box testing. It is platform independent and language independent. Used to test embedded systems. Functionality and behavioral parts are tested.

Tester should have the knowledge of both the internals and externals of the function If you know something about how the product works on the inside, you can test it better from the outside.

20

Testing Manual

Gray box testing is especially important with Web and Internet applications, because the Internet is built around loosely integrated components that connect via relatively well-defined interfaces. Unless you understand the architecture of the Net, your testing will be skin deep.

21

Testing Manual

1.6 Installation & Maintenance


Feasibility Study Analysis

Design Coding Testing Installation & Maintenance Installation File conversion New system becomes operational Staff training Corrective maintenance A type of maintenance performed to correct a defect Perfective maintenance Reengineering include enhancement

Maintenance

Adaptive maintenance To change software so that it will work in an altered environment, such as when an operating system, hardware platform, compiler, software library or database structure changes

Table format of all the phases in SDLC: PHASE Analysis Design Coding Testing INPUT BRS FRS and SRS Design Doc All the above Docs OUTPUT FRS and SRS Design Doc .exe File/Application/Website Defect Report

22

Testing Manual

2. Software Development Life Cycles


Life cycle: Entire duration of a project, from inception to termination

Different life cycle models


2.1. Code-and-fix model: Earliest software development approach (1950s) Iterative, programmers' approach Two phases: 1. coding, 2. fixing the code Project planning Analysis Design Testing Maintenance

No provision for:

Problems with code-and-fix model: 1. After several iterations, code became very poorly structured; subsequent fixes became very expensive 2. Even well-designed software often very poorly matched users requirements: were rejected or needed to be redeveloped (expensively!) 3. Changes to code were expensive, because of poor testing and maintenance practices Solutions: 1. Design before coding 2. Requirements analysis before design 3. Separate testing and maintenance phases after coding 2.2. Waterfall model:

Also called the classic life cycle Introduced in 1956 to overcome limitations of code-and-fix model Very structured, organized approach and suitable for planning
Waterfall model is a linear approach, quite inflexible

23

Testing Manual

At each phase, feedback to previous phases is possible (but is discouraged in practice)

Still is the most widespread model today


Main phases: 1. Requirements 2. Analysis 3. Design (overall design & detailed design) 4. Coding 5. Testing (unit test, integration test, acceptance test) 6. Maintenance


Requirements Analysis Design Coding Testing Maintenance

24

Testing Manual

Approaches The standard waterfall model for systems development is an approach that goes through the following steps: 1. Document System Concept 2. Identify System Requirements and Analyze them 3. Break the System into Pieces (Architectural Design) 4. Design Each Piece (Detailed Design) 5. Code the System Components and Test Them Individually (Coding, Debugging, and Unit Testing) 6. Integrate the Pieces and Test the System (System Testing) 7. Deploy the System and Operate It Waterfall Model Assumption The requirements are knowable in advance of implementation. The requirements have no unresolved, high-risk implications -- e.g., risks due to COTS choices, cost, schedule, performance, safety, security, user interface, organizational impacts The nature of the requirements are compatible with all the key system stakeholders expectations -- e.g., users, customer, developers, maintainers, investors The right architecture for implementing the requirements is well understood. There is enough calendar time to proceed sequentially. Conversion of existing projects in to new projects. For proven platforms and technologies, it works fine. Suitable for short duration projects. The waterfall model is effective when there is no change in the requirements, and the requirements are fully known . If there is no Rework, this model build a high quality product. The stages are clear cut All R&D done before coding starts, implies better quality program design

Advantages of Waterfall Model

25

Testing Manual

Disadvantages with Waterfall Model: Testing is postponed to later stage till coding completes. Not suitable for large projects It assumes uniform and orderly sequence of steps. Risk in certain project where technology itself is a risk. Correction at the end of phase need correction to the previous phase, So rework is more. Real projects rarely flow in a sequential process. It is difficult to define all requirements at the beginning of a project. The model has problems adapting to change. A working version of the system is not seen until late in the project's life. Errors are discovering later (repairing problem further along the lifecycle becomes progressively more expensive). Maintenance cost can be as much as 70% of system costs. Delivery only at the end (long wait) Introduced to overcome shortcomings of waterfall model Suitable to overcome problem of requirements definition Prototyping builds an operational model of the planned system, which the customer can evaluate

2.3. Prototyping model:

Main phases: 1. Requirements gathering 2. Quick design 3. Build prototype 4. Customer evaluation of prototype 5. Refine prototype 6. Iterate steps 4. and 5. to "tune" the prototype 7. Engineer product

26

Testing Manual

Prototyping
Engineer Product No Changes? Refine Prototype Evaluate Prototype
Note: Mostly, the prototype is discarded after step 5. and the actual system is built from scratch in step 6. (throw-away prototyping) Possible problems: Customer may object to prototype being thrown away and may demand "a few changes" to make it working (results in poor software quality and maintainability) Inferior, temporary design solutions may become permanent after a while, when the developer has forgotten that they were only intended to be temporary (results in poor software quality) Helps counter the limitations of waterfall model After prototype is developed, the end user and the client are permitted to use the application and further modifications are done based on their feedback. User oriented What the user sees Not enigmatic diagrams 27

Requirements Quick Design Build Prototype

Yes

Advantages

Testing Manual

Quicker error feedback Earlier training Possibility of developing a system that closely addresses users' needs and expectations

Disadvantages Development costs are high. User expectations Bypass analysis Documentation Never ending Managing the prototyping process is difficult because of its rapid, iterative nature Requires feedback on the prototype Incomplete prototypes may be regarded as complete systems

28

Testing Manual

2.4 Incremental: During the first one-month phase, the development team worked from static visual designs to code a prototype. In focus group meetings, the team discussed users needs and the potential features of the product and then showed a demonstration of its prototype. The excellent feedback from these focus groups had a large impact on the quality of the product. Main phases: 1. Define outline Requirements 2. Assign requirements to increments 3. Design system architecture

4. 5.

Develop Integrate 6. Validate

Incremental
Define outline requirements Assign requirements to increments Design system architecture

Develop system increment

Validate increment

Integrate increment

Validate Final system System

System incomplete
After the second group of focus groups, the feature set was frozen and the product definition complete. Implementation consisted of four-tosix-week cycles, with software delivered for beta use at the end of each cycle. The entire release took 10 months from definition to manufacturing release. Implementation lasted 4.5 months. The result was a world-class product that has won many awards and has been easy to support.

29

Testing Manual

2.5 V-Model: Verification (Static System Doing Right Job) To test the system correctness as to whether the system is being functioning as per specifications. Typically involves in Reviews and Meetings to evaluate documents, plans, code, requirements and specifications. This can be done with checklists, walkthroughs and inspection meetings.

issue

lists,

Validation (Dynamic System - Job Right) Testing the system in a real environment i.e, whether software is catering the customers requirements. Typically involves in actual testing and take place after verifications are completed V E R I F I C A T I O N

V A L I D ATI O N

Advantages
Reduces the cost of defect repair (. Every document is verified by tester )

No Ideal time for Testers


Efficiency of V-model is more when compare to Waterfall Model Change management can be effected in V-model Risk management is not possible 30

Disadvantages

Testing Manual

Applicable of medium sized projects

2.6 Spiral model: Objective: overcome problems of other models, while combining their advantages Key component: risk management (because traditional models often fail when risk is neglected) Development is done incrementally, in several cycles _ Cycle as often as necessary to finish

Main phases: 1. Determine objectives, alternatives for development, and constraints for the portion of the whole system to be developed in the current cycle 2. Evaluate alternatives, considering objectives and constraints; identify and resolve risks 3. Develop the current cycle's part of the system, using evolutionary or conventional development methods (depending on remaining risks); perform validation at the end 4. Prepare plans for subsequent phases

31

Testing Manual

Spiral Model

This model is very appropriate for large software projects. The model consists of four main parts, or blocks, and the process is shown by a continuous loop going from the outside towards the inside. This shows the progress of the project. Planning This phase is where the objectives, alternatives, and constraints are determined. Risk Analysis What happens here is that alternative solutions and constraints are defined, and risks are identified and analyzed. If risk analysis indicates uncertainty in the requirements, the prototyping model might be used to assist the situation. Engineering Here the customer decides when the next phase of planning and risk analysis occur. If it is determined that the risks are to high, the project can be terminated. Customer Evaluation In this phase, the customer will assess the engineering results and make changes if necessary.

32

Testing Manual

Spiral model flexibility Well-understood systems (low technical risk) - Waterfall model. Risk analysis phase is relatively cheap Stable requirements and formal specification. Safety criticality Formal transformation model High UI risk, incomplete specification - prototyping model Hybrid models accommodated for different parts of the project Good for large and complex projects Customer Evaluation allows for any changes deemed necessary, or would allow for new technological advances to be used Allows customer and developer to determine and to react to risks at each evolutionary level Direct consideration of risks at all levels greatly reduces problems Difficult to convince customer that this approach is controllable Requires significant risk assessment expertise to succeed Not yet widely used efficacy not yet proven If a risk is not discovered, problems will surely occur

Advantages of spiral model:

Problems with spiral model:

2.7 RAD Model RAD refers to a development life cycle designed to give much faster development and higher quality systems than the traditional life cycle. It is designed to take advantage of powerful development software like CASE tools, prototyping tools and code generators. The key objectives of RAD are: High Speed, High Quality and Low Cost. RAD is a approach. people-centered and incremental development

Active user involvement, as well as collaboration and cooperation between all stakeholders are imperative. Testing is integrated throughout the development life cycle so that the system is tested and reviewed by both developers and 33

Testing Manual

users incrementally.

34

Testing Manual

Problem Addressed By RAD With conventional methods, there is a long delay before the customer gets to see any results. With conventional methods, development can take so long that the customer's business has fundamentally changed by the time the system is ready for use. With conventional methods, there is nothing until 100% of the process is finished, then 100% of the software is delivered

Bad Reasons For Using RAD To prevent cost overruns (RAD needs a team already disciplined in cost management) To prevent runaway schedules (RAD needs a team already disciplined in time management) Good Reasons for using RAD To converge early toward a design acceptable to the customer and feasible for the developers To limit a project's exposure to the forces of change To save development time, possibly at the expense of economy or product quality Mapping between System Development Life Cycle (SDLC) of ITSD and RAD stages is depicted as follows. SDLC Project Request & Maintenance System Analysis & Design SA&D) Requirements Planning User Design RAD Construction Implementation Post Implementation 35 Review Transition RAD

RAD in SDLC

Testing Manual

Essential Ingredients of RAD RAD has four essential ingredients:


Tools Methodology People Management.

The following benefits can be realized in using RAD: High quality system will be delivered because of methodology, tools and user involvement; Business benefits can be realized earlier; Capacity will be utilized to meet a specific and urgent business need; Standards and consistency can be enforced through the use of CASE tools. Time required to get system developed will be reduced; Productivity of developers will be increased.

In the long run, we will also achieve that:

36

Testing Manual

37

Testing Manual

Advantages of RAD Buying may save money compared to building Deliverables sometimes easier to port Early visibility Greater flexibility (because developers can redesign almost at will) Greatly reduced manual coding (because of wizards, code generators, code reuse) Increased user involvement (because they are represented on the team at all times) Possibly reduced cost (because time is money, also because of reuse) Buying may not save money compared to building Cost of integrated toolset and hardware to run it Harder to gauge progress (because there are no classic milestones) Less efficient (because code isn't hand crafted) More defects Reduced features Requirements may not converge Standardized appearance) look and feel (undistinguished, lackluster

Disadvantages of RAD

Successful efforts difficult to repeat Unwanted features

38

Testing Manual

Test cases

Test d ata

Test results

Test repo rts

Desig n test cases

Prepare test d ata

Ru n p rog ram with test data

Co mpare r esu lts to test cases

39

Testing Manual

3. Testing Life Cycle


A systemic approach for Testing System Study

Scope/ Approach/ Estimations

Test Plan Design

Test Case Design

Test Case Review

Test Case Execution

Defect Handling

Gap Analysis

Deliverables

40

Testing Manual

3.1 System Study 1. Domain Knowledge :- Used to know about the client business Banking / Finance / Insurance / Real-estates / ERP / CRM / Others 2. Software : Front End Process programmes Back End 3. Hardware: install. (GUI) VB / JAVA/ FORMS / Browser Language witch we want to write Database like Oracle, SQL Server etc. Internet/ Intranet/ Servers which you want to Ten Lines Of Code (LOC) = 1 Functional The document which you want to

4. Functional Points: Point. 5. Number of Pages: prepare.

6. Number of Resources : -Like Programmers, Designers, and Managers. 7. Number of Days: - For actual completion of the Project. 8. Numbers of Modules 9. Priority:- High/ Medium/ Low importance for Modules 3.2 Scope/ Approach/ Estimation: Scope What to test What not to test Methods, tools and techniques used to accomplish test objectives. Estimation should be done based on LOC/ FP/Resources 1000 LOC = 100 FP (by considering 10 LOC = 1 FP) 100 x 3 = 300 (FP x 3 Tech. = Test Cases)

Approach

Estimation

The 3 Tech are (Equivalence Class, Boundary Value Analysis, Error Guessing)

41

Testing Manual

30 TC Par Day => 300/30 = 10 Days to Design Test Cases Test Case Review => of Test Case Design (5 Days) Test Case Execution = 1 of Test Case Design(15 Days) Defect Headlining = Test Case Design (5 Days) Test Plan = 5 days ( 1 week ) Buffer Time = 25% of Estimation

3.3 Test Plan Design: A test plan prescribes the scope, approach, resources, and schedule of testing activities.

Why Test Plan? 1. Repeatable 2. To Control 3. Adequate Coverage Importance of Test Plan Test planning process is a critical step in the testing process. Without a documented test plan, the test itself cannot be verified, coverage cannot be analyzed and the test is not repeatable The Test Plan Design document helps in test execution it contain 1. About the client and company 2. Reference document (BRS, FRS and UI etc.) 3. Scope (What to be tested and what not to be ) 4. Overview of Application 5. Testing approach (Testing strategy) 6. For each testing Definition Technique Start criteria Stop criteria 7. Resources and there Roles and Responsibilities

8. Defect definition 9. Risk / Contingency / Mitigation Plan 10. Training Required 11. Schedules 42

Testing Manual

12. Deliverables To support testing, a plan should be there, which specifies What to Do? How to Do? When to Do?

43

Testing Manual

3.4 Test Cases Design: What is a test case? Test case is a description of what to be tested, what data to be given and what actions to be done to check the actual result against the expected result. What are the items of test case? 1. Test Case Number 2. Pre-Condition 3. Description 4. Expected Result 5. Actual Result 6. Status (Pass/Fail) 7. Remarks. Test Case Template TC ID PreConditi on Conditio n to satisfied Description Expec ted Result As pear FSR Actual Result Statu s Remark s If any

Unique Test Case number

Yahoo001

Yahoo web page should displaye d

1. What to be tested 2. what data to provide d 3. what action to be done 1. Check inbox is display ed 2. User ID/PW 3. Click on Submit

System Pass respon or se Fail

System System should respon mail se box

44

Testing Manual

Testcase Development process Identify all potential Test Cases needed to fully test the business and technical requirements Document Test Procedures and Test Data requirements Prioritize test cases Identify Test Automation Candidates Automate designated test cases Source Specifications Logical system Code Existing files or test cases Limits and boundary conditions

Types of Test Cases Type 1. Requirement Based 2. Design based 3. Code based 4. Extracted 5. Extreme

Can this test cases reusable? Test cases developed for functionality testing and can be reusable for Integration System Regression Performance Testing with few modifications.

What are the characteristics of good test case? A good test case should have the following: TC should start with what you are testing. TC should be independent. TC should not contain If statements. TC should be uniform. (Convention should be followed same across the Project Eg. <Action Buttons> , Links

45

Testing Manual

The following issues should be considered while writing the test cases All the TCs should be traceable. There should not be any duplicate test cases. Out dated test cases should be cleared off. All the test cases should be executable. Developed to verify that specific requirements or design are satisfied Each component must be tested with at least two test cases: Positive and Negative Real data should be used to reality test the modules after successful test data is used

Test case Guidelines

3.5 Test Case Review: 1. Peer to peer Reviews 2. Team Lead Review 3. Team Manager Review Review Process Take checklist

Take a demo of functionally Go through the Use cases & Functional Spec Try to find the gap between TC & Use cases

Submit the Review Report 3.6 Test Case Execution:

46

Testing Manual

Test execution is completion of testing activities, which involves executing the planned test cases and conducting of the tests. Test execution phase broadly involves execution and reporting. Execution and execution results plays a vital role in the testing.

47

Testing Manual

Test execution consists of following activities to be performed 1. Creation of test setup or Test bed 2. Execution of test cases on the setup 3. Test Methodology used 4. Collection of Metrics 5. Defect Tracking and Reporting 6. Regression Testing The following activities should be taken care: 1. Number of test cases executed. 2. Number of defects found 3. Screen shoots of failure executions should be taken in word document. 4. Time taken to execute. 5. Time wasted due to the unavailability of the system. Test Case Execution Process: Take the Test Case document

Check the availability of application

Implement the Test Cases

Raise the Defects

Test Case Test Data


INPUT 3.7 Defect Handling
Test Case Execution

Raise the Defect Screen shot


OUTPUT Installation &

PROCESS

48

Testing Manual

What is Defect?

Defect is a coding error in a computer program. A software error is present when the program does not do what
its end user expects it to do. Who can report a Defect? Anyone who has involved in software development life cycle and who is using the software can report a Defect. In most of the cases defects are reported by Testing Team. A short list of people expected to report bugs: 1. 2. 3. 4. 5. Testers / QA Engineers Developers Technical Support End Users Sales and Marketing Engineers Defect or Bug Report is the medium of communication between the tester and the programmer Provides clarity to the management, particularly at the summary level Defect Report should be accurate, concise, thoroughly-edited, well conceived, high-quality technical document The problem should be described in a way that maximizes the probability that it will be fixed Defect Report should be non-judgmental and should not point finger at the programmer Crisp Defect Reporting process improves the test teams communications with the senior and peer management

Defect Reporting

49

Testing Manual

Defect Life Cycle Defect Life Cycle helps in handling defects efficiently. This DLC will help the users to know the status of the defect.

Defect Raised

Internal Review

Defect

Defect Submitted to Dev Team

Defect Rejected

No

Valid Yes s Defect Accepted

Defect Postponed

Defect Fixed

No
Valid

Yes s

Close the Defect

50

Testing Manual

Types of Defects 1. Cosmetic flaw 2. Data corruption 3. Data loss 4. Documentation Issue 5. Incorrect Operation 6. Installation Problem 7. Missing Feature 8. Slow Performance 9. System Crash 10. 11. Unexpected Behavior Unfriendly behavior

How do u decide the Severity of the defect Severity Level High Description A defect occurred due to the inability of a key function to perform. This problem causes the system hang it halts (crash), or the user is dropped out of the system. An immediate fix or work around is needed from development so that testing can continue. A defect occurred which severely restricts the system such as the inability to use a major function of the system. There is no acceptable workaround but the problem does not inhibit the testing of other functions Response Time or Turn-around Time Defect should be responded to within 24 hours and the situation should be resolved test exit

A response or action plan should be provided within 3 working days and the situation should be resolved before test exit.

51

Testing Manual

Severity Description Level Low A defect is occurred which places minor restrict on a function that is not critical. There is an acceptable workaround for the defect. Others An incident occurred which places no restrictions on any function of the system. No immediate impact to testing. A Design issue or Requirements not definitively detailed in project. The fix dates are subject to negotiation.

Response Time or Turnaround Time A response or action plan should be provided within 5 working days and the situation should be resolved before test exit. An action plan should be provided for next release or future enhancement

Defect Severity VS Defect Priority The General rule for the fixing the defects will depend on the Severity. All the High Severity Defects should be fixed first. This may not be the same in all cases some times even though severity of the bug is high it may not be take as the High priority. At the same time the low severity bug may be considered as high priority.

Defect Tracking Sheet Defect No Unique No Descripti on Dec of Bug Origin Birth place of the Bug Severity Critical Major Medium Minor Cosmetic Priority High Medium Low Status Submitte d Accepted Fixed Rejected Postpone d Closed

52

Testing Manual

53

Testing Manual

Defect Tracking Tools Bug Tracker -- BSL Proprietary Tools Rational Clear Quest Test Director

3.8 Gap Analysis: 1. BRS Vs SRS BRS01 SRS01 -SRS02 -SRS03 2. SRS Vs TC SRS01 TC01 - TC02 - TC03 3. TC Vs Defects TC01 Defects01 -- Defects02 3.9 Deliverables: All the documents witch are prepared in each and every stage. FRS SRS Use Cases Test Plain Defect Report Review Report etc.,
BRS001 SRS002 SRS003 BRS SRS SRS001 Test Case TC001 TC002 TC003 Defects Defect001 Defect002

54

Testing Manual

4 Testing Phases

Requirement Analysis Testing Design Testing Unit Testing Integration Testing System Testing Acceptance Testing

4.1 Requirement Analysis Testing Objective The objective of Requirement AnalysisTesting is to ensure software quality by eradicating errors as earlier as possible in the developement process If the errors noticed at the end of the software life cycle are more costly compared to that of early ones, and there by validating each of the Outputs.

The objective can be acheived by three basic issues: 1. Correctness 2. Completeness 3. Consistency Types of requirements Functional Requirements Data Requirements Look and Feel requirements Usability requirements Performance Requirements Operational requirements Maintainability requirements Security requirements Scalability requirements

55

Testing Manual

Difficulties in conducting requirements analysis: Analyst not prepared Customer has no time/interest Incorrect customer personnel involved Insufficient time allotted in project schedule

What constitutes good requirements?


Clear Concise Unambiguous terminology No unnecessary narrative or non-relevant facts

Consistent Requirements that are similar are stated in similar terms. Requirements do not conflict with each other. Complete All functionality needed to satisfy the goals of the system is specified to a level of detail sufficient for design to take place Testing related activities during Requirement phase 1. 2. Creation and finalization of testing templates Creation of over-all Test Plan and Test Strategy

3. Capturing Acceptance criteria and preparation of Acceptance Test Plan 4. Capturing Performance criteria of the software requirements 4.2 Design Testing Objective The objective of the design phase testing is to generate a complete specifications for implementing a system using a set of tools and languages

Design objective is fulfilled by five issues 1. Consistency 2. Completeness 3. Correctness 4. Feasibility 5. Tractability

56

Testing Manual

57

Testing Manual

Testing activities in Design phase 1. Develop Test cases to ensure that product is on par with Requirement Specification document. 2. Verify Test Cases & test scripts by peer reviews. 3. Preparation of traceability matrix from system requirements 4.3 Unit Testing Objective In Unit testing user is supposed to check each and every micro function. All field level validations are expected to test at the stage of testing. In most of the cases Developer will do this. The objective can be achieved by the following issues 1. Correctness 2. Completeness 3. Early Testing 4. Debugging 4.4 Integration Testing: Objective The primary objective of integration testing is to discover errors in the interfaces between Modules/Sub-Systems (Host & Client Interfaces). Minimizing the errors which include internal and external Interface errors

Approach: Top-Down Approach

58

Testing Manual

The integration process is performed in a series of 5 steps 1. The main control module is used as a test driver, and stubs are substituted for all modules directly subordinate to the main control module. 2. Depending on the integration approach selected (depth or breadth-first) subordinate stubs are replaced at a time with actual modules. 3. Tests are conducted as each module is module is integrated. 4. One completion of each set of tests, another stub is replaced with the real-module. 5. Regression testing may be conducted to ensure that new errors have not been introduced. Advantages We can verify the major controls early in the testing Process Disadvantage: Stubs are required. Very difficult to develop stubs Bottom-Up Approach.

59

Testing Manual

A bottom-up integration strategy may be implemented with the following steps: 1. Low level modules are combined into clusters (Some times called builds) that perform a specific software sub function. 2. A driver (control program for testing) is written to coordinate test case input and output. 3. The cluster is tested. 4. Drivers are removed and clusters are combined upward in the program structure Advantages Easy to Develop the drivers than stubs The need of test drivers Late detection of interface problems Disadvantage:

An integration testing is conducted, the tester should identify critical modules. A critical module has one or more of the following characteristics: 1. Address several software requirements. 2. Has a high-level of control. (resides relatively high in the program structure) 3. Complex & Error-Phone. 4. Have definite performance requirements. Testing activities in Integration Testing Phase 1. This testing is conducted in parallel with integration of various applications (or components) 60

Testing Manual

2. Testing the product with its external and internal interfaces without using drivers and stubs. 3. Incremental approach while integrating the interfaces. 4.5 System Testing: The primary objective of system testing is to discover errors when the system is tested as a hole. System testing is also called as End-End Testing. User is expected to test from Login-To-Logout by covering various business functionalities. Recovery Testing. Security Testing. Load & Stress Testing. Functional Testing

The following Tests will be conducted in Systemtesting

Testing activities in System Testing phase 1. System test is done for validating the product with respect to client requirements 2. Testing can be in multiple rounds 3. Defect found during system test should be logged into Defect Tracking System for the purpose of tracking. 4. Test logs and defects are captured and maintained. 5. Review of all the test documents Approach: IDO Model Identifying the End-End/Business Life Cycles. Design the test and data. Optimize the End-End/Business Life Cycles. The primary objective of acceptance testing is to get the acceptance from the client. Testing the system behavior against customers requirements Customers undertake typical tasks to check their requirements Done at the customers premises on the user environment Alpha Testing 61

4.6 Acceptance Testing:

Acceptance Testing Types

Testing Manual

Testing the application on the developers premises itself in a controlled environment. Generally, the Quality Assurance cell is the body that is responsible for conducting the test. On successful completion of this phase, the software is ready to migrate outside the developers premises. Beta Testing It is carried out at one or more users premises using their infrastructure in an uncontrolled manner. It is the customer or his representative that conducts the test, with/without the developer around. As and bugs are uncovered, the developer is notified about the same. This phase enables the developer to modify the code so as to alleviate any remaining bugs before the final official release. Building a team with real-time user, functional users and developers. Execution of business Test Cases.

Approach: BE

62

Testing Manual

When Should we start writing Test Cases/ Testing V Model is the most suitable way to start writing Test Cases and conduct Testing. SDLC Phase Business Requirements Docs Software Requirements Docs Design Requirements Docs Code Requirements Freeze Acceptance Test Cases System Test Cases Integration test Cases Unit Test Cases Requirements Build Acceptance Testing System testing Integration Testing Unit Testing

63

Testing Manual

5 Testing Methods
5.1 Functionality Testing: Objective: Testing the functionality of the application with the help of input and out put Test against system requirements. To confirm all the requirements are covered.

Approach: 1. Equivalence Class 2. Boundary Value Analysis 3. Error Guessing. 5.2 Usability Testing:

To test the Easiness and User-friendliness of the system.


Approach: 1. Qualitative & Quantitative 2. Qualitative Approach: Qualitative Approach Each and every function should available from all the pages of the site. User should able to submit each and every request with in 4-5 actions. Confirmation message should be displayed for each and every submit. Quantitative Approach: Heuristic Checklist should be prepared with all the general test cases that fall under the classification of checking. This generic test cases should be given to 10 different people and ask to execute the system to mark the pass/fail status. The average of 10 different people should be considered as the final result.

Example: Some people may feel system is more users friendly, If the submit is button on the left side of the screen. At the same time some other may feel its better if the submit button is placed on the right side. 64

Testing Manual

65

Testing Manual

Classification of Checking:

Clarity of communication. Accessibility Consistency Navigation Design & Maintenance Visual Representation.
5.3 Reliability Testing: Objective Reliability is considered as the probability of failure-free operation for a specified time in a specified environment for a given purpose To find Mean Time between failure/time available under specific load pattern. Mean time for recovery. By performing the continuous hours of operation. More then 85% of the stability is must. Business logic performs as expected Active buttons are really active Correct menu options are available Reliable hyper links

Approach

Reliability Testing helps you to confirm:

Note: This should be done by using performance testing tools 5.4 Regression Testing: Objective is to check the new functionalities has incorporated correctly with out failing the existing functionalities. RAD In case of Rapid Application development Regression Test plays a vital role as the total development happens in bits and pieces. Testing the code problems have been fixed correctly or not. Manual Testing (By using impact Analysis)

Approach

66

Testing Manual

Automation tools

67

Testing Manual

5.5 Performance Testing: Primary objective of the performance testing is to demonstrate the system works functionally as per specifications with in given response time on a production sized database. Assessing the system capacity for growth. Identifying weak points in the architecture Detect obscure bugs in software Tuning the system Verify resilience & reliability

Objectives

Performance Parameters Request-Response Time Transactions per Second Turn Around time Page down load time Through Put

Approach Usage of Automation Tools Load Test Volume Test Stress Test Classification of Performance Testing:

Load Testing Estimating the design capacity of the system within the resources limit Approach is Load Profile Is the process of feeding a program with heavy volume of data. Approach is data profile Estimating the breakdown point of the system beyond the resources limit.

Volume Testing

Stress Testing

68

Testing Manual

Repeatedly working on the same functionality Critical Query Execution (Join Queries) To Emulate peak load. With the Simple Scenario (Functional Query), N number of people working on it will not enforce stress on the server. A complex scenario with even one less number of users will stress the server. Objective is to find the maximum number of user system can handle.

Load Vs Stress:

5.6 Scalability Testing:

Classification: Approach Performance Tools Network Scalability Server Scalability Application Scalability

5.7 Compatibility Testing: Compatibility testing provides a basic understanding of how a product will perform over a wide range of hardware, software & network configuration and to isolate the specific problems. Approach

Environment Selection. Understanding the end users Importance of selecting both old browser and new browsers Selection of the Operating System Test Bed Creation Partition of the hard disk. Creation of Base Image

69

Testing Manual

5.8 Security Testing:

Testing how well the system protects against unauthorized


internal or external access. Verify how easily a system is subject to security violations under different conditions and environments During Security testing, password cracking, unauthorized entry into the software, network security are all taken into consideration. 5.8 Installation Testing: Installation testing is performed to ensure that all Install features and options function properly and to verify that all necessary components of the application are installed.

The uninstallation of the product also needs to be tested to ensure that all data, executables, and .DLLs are removed. The uninstallation of the application is tested using DOS command line, Add/Remove programs, and manual deletion of files 5.9 Adhoc Testing

Testing carried out using no recognized test case design


technique. 5.10 Exhaustive Testing Testing the application with all possible combinations of values for program variables. Feasible only for small, simple programs.

70

Testing Manual

6 Performance Life Cycle


6.1 What is Performance Testing? Primary objective of the performance testing is to demonstrate the system works functionally as per specifications with in given response time on a production sized database 6.2 Why Performance Testing: To assess the system capacity for growth The load and response data gained from the tests can be used to validate the capacity planning model and assist decision making. To identify weak points in the architecture The controlled load can be increased to extreme levels to stress the architecture and break it bottlenecks and weak components can be fixed or replaced To detect obscure bugs in software Tests executed for extended periods can cause failures caused by memory leaks and reveal obscure contention problems or conflicts To tune the system Repeat runs of tests can be performed to verify that tuning activities are having the desired effect improving performance. To verify resilience & reliability Executing tests at production loads for extended periods is the only way to access the systems resilience and reliability to ensure required service levels are likely to be met. 6.3 Performance-Tests: Used to test each part of the web application to find out what parts of the website are slow and how we can make them faster. 6.4 Load-Tests: This type of test is done to test the website using the load that the customer expects to have on his site. This is something like a real world test of the website. First we have to define the maximum request times we want the customers to experience, this is done from the business and usability point of view, not from a technical point of view. At

71

Testing Manual

this point we need to calculate the impact of a slow website on the company sales and support costs. Then we have to calculate the anticipated load and load pattern for the website (Refer Annexure I for details on load calculation) which we then simulate using the Tool. At the end we compare the test results with the requests times we wanted to achieve. 6.5 Stress-Tests: They simulate brute force attacks with excessive load on the web server. In the real world situations like this can be created by a massive spike of users far above the normal usage e.g. caused by a large referrer (imagine the website being mentioned on national TV). The goals of stress tests are to learn under what load the server generates errors, whether it will come back online after such a massive spike at all or crash and when it will come back online. It is even a good idea to start performance testing before a line of code is written at all! Early testing the base technology (network, load balancer, application-, database- and webservers) for the load levels can save a lot of money when you can already discover at this moment that your hardware is to slow. Also the first stress tests can be a good idea at this point. The costs for correcting a performance problem rise steeply from the start of development until the website goes productive and can be unbelievable high for a website already online. As soon as several web pages are working the first load tests should be conducted and from there on should be part of the regular testing routine each day or week or for each build of the software. LoadRunner from Mercury Interactive AstraLoad from Mercury Interactive Silk Performer from Segue Rational Suite Test Studio from Rational Rational Site Load from Rational Webload from Radview

6.6 When should we start Performance Testing:

6.7 Popular tools used to conduct Performance Testing:

72

Testing Manual

RSW eSuite from Empirix MS Stress tool from Microsoft

6.8 Performance Test Process: This is a general process for performance Testing. This process can be customized according to the project needs. Few more process steps can be added to the existing process, deleting any of the steps from the existing process may result in Incomplete process. If Client is using any of the tools, In this case one can blindly follow the respective process demonstrated by the tool. General Process Steps:

Setting up of the Environment

Record & Playback in the standby mode

Enhancement of the script to support multiple users

Configure the scripts

Execution for fixed users and reporting the status to the developers

Re-execution of the scenarios after the developers fine-tune the code

73

Testing Manual

Setting up of the test environment The installation of the tool, agents Directory structure creation for the storage of the scripts and results Installation of additional software if essential to collect the server statistics It is also essential to ensure the correctness of the environment by implementing the dry run. The scripts are generated using the script generator and played back to ensure that there are no errors in the script. Enhancement of the script to support multiple users The variables like logins, user inputs etc. parameterised to simulate the live environment. should be

Record & playback in the stand by mode

It is also essential since in some of the applications no two users can login with the same id. Scenarios should be configured to run the scripts on different agents, schedule the scenarios Distribute the users onto different scripts, collect the data related to database etc. Hosts The next important step in the testing approach is to run the virtual users on different host machines to reduce the load on the client machine by sharing the resources of the other machines. Users The number of users who need to be activated during the execution of the scenario. Scenarios A scenario might either comprise of a single script or multiple scripts. The main intention of creating a scenario to simulate load on the server similar to the live/production environment.

Configuration of the scenarios

74

Testing Manual

Ramping

In the live environment not all the users login to the application simultaneously. At this stage we can simulate the virtual users similar to the live environment by deciding 1. How many users should be activated at a particular point of time as a batch? 2. What should be the time interval between every batch of users? Execution for fixed users and reporting the status to the developers The script should be initially executed for one user and the results/inputs should be verified to check it out whether the server response time for a transaction is less than or equal to the acceptable limit (bench mark). If the results are found adequate the execution should be continued for different set of users. At the end of every execution the results should be analysed. If a stage reaches when the time taken for the server to respond to a transaction is above the acceptable limit, then the inputs should be given to the developers.

Re-execution of the scenarios after the developers fine tune the code After the fine-tuning, the scenarios should be re-executed for the specific set of users for which the response was inadequate. If found satisfactory, then the execution should be continued until the decided load.

Final report At the end of the performance testing, final report should be generated which should comprise of the following Introduction about the application. Objectives set / specified in the test plan. Approach summary of the steps followed in conducting the test Analysis & Results is a brief explanation about the results and the analysis of the report. Conclusion the report should be concluded by telling whether the objectives set before the test is met or not.

75

Testing Manual

Annexure can consist of graphical representation of the data with a brief description, comparison statistics if any etc.

76

Testing Manual

7 Life Cycle of Automation


Analyze Application the

Select the Tool

Identify the Scenarios

Design / Record Test Scripts Reporting the Defect

Finding & Reporting the Modify the Test Defect Scripts Finding Defect & Reporting the

Run the Test Scripts

Finding Defect

&

Reporting

the

Finding Defect

&

Reporting

the

sdasdadadasdadafhgfdgdf Finding Defect & Reporting the

7.1 What is Automation?

77

Finding Defect

&

Reporting

the

Testing Manual

A software program that is used to test another software program, This is referred to as automated software testing. 7.2 Why Automation Avoid the errors that humans make when they get tired after multiple repetitions. The test program wont skip any test by mistake. Each future test cycle will take less time and require less human intervention. Required for regression testing. 7.3 Benefits of Test Automation: Allows more testing to happen Tightens / Strengthen Test Cycle Testing is consistent, repeatable Useful when new patches released Makes configuration testing easier Test battery can be continuously improved. Fewer tests will be needed It will be easier if it is automate Compensate for poor design No more manual testing. Rational Robot WinRunner SilkTest QA Run WebFT

7.4 False Benefits:

7.5 What are the different tools available in the market?

8 Testing
78

Testing Manual

8.1 Test Strategy Test strategy is statement of overall approach of testing to meet the business and test objectives. It is a plan level document and has to be prepared in the requirement stage of the project. It identifies the methods, techniques and tools to be used for testing . It can be a project or an organization specific. Developing a test strategy which effectively meets the needs of the organization/project is critical to the success of the software development An effective strategy has to meet the project and business objectives Defining the strategy upfront before the actual testing helps in planning the test activities Definition of test objective Strategy to meet the specified objective Overall testing approach Test Environment Test Automation requirements Metric Plan Risk Identification, Mitigation and Contingency plan Details of Tools usage Specific Document templates used in testing Test approach will be based on the objectives set for testing Test approach will detail the way the testing to be carried out Types of testing to be done viz Unit, Integration and system testing The method of testing viz Blackbox, White-box etc., Details of any automated testing to be done

A test strategy will typically cover the following aspects

8.2 Testing Approach

79

Testing Manual

8.3 Test Environment All the Hardware and Software requirements for carrying out testing shall be identified in detail. Any specific tools required for testing will also be identified If the testing is going to be done remotely, then it has to be considered during estimation Risk analysis should carried out for testing phase The risk identification will be accomplished by identifying causesand-effects or effects-and-causes The identified Risks are classified into to Internal and External Risks. The internal risks are things that the test team can control or influence. The external risks are things beyond the control or influence of the test team Once Risks are identified and classified, the following activities will be carried out Identify the probability of occurrence Identify the impact areas if the risk were to occur Risk mitigation plan how avoid this risk? Risk contingency plan if the risk were to occur what do we do? You cannot test a program completely We can only test against system requirements May not detect errors in the requirements. Incomplete or ambiguous requirements inadequate or incorrect testing. may lead to

8.4 Risk Analysis

8.5 Testing Limitations

Exhaustive (total) testing is impossible in present scenario. Time and budget constraints normally require very careful planning of the testing effort. Compromise between thoroughness and budget.

80

Testing Manual

Test results are used to make business decisions for release dates. Even if you do find the last bug, youll never know it You will run out of time before you run out of test cases You cannot test every path You cannot test every valid input You cannot test every invalid input You cannot prove a program correct (because it isnt!) The purpose of testing is to find problems The purpose of finding problems is to get them corrected

8.6 Testing Objectives

8.7 Testing Metrics Time Time per test case Time per test script Time per unit test Time per system test Sizing Function points Lines of code Defects Numbers of defects Defects per sizing measure Defects per phase of testing Defect origin Defect removal efficiency Number of defects found in producer testing Defect Removal Efficiency = Number of defects during the life of the product Actual Size-Planed Size Size Variance = Planed Size Actual end date Planed end date Delivery Variance = Planed end date Planed start date Actual effort Planed effort

81

Testing Manual

Effort = Planed effort

82

Testing Manual

Effort Productivity = Size No defect found during the review time Review efficiency = Effort 8.7 Test Stop Criteria: Minimum number of test cases successfully executed. Uncover minimum number of defects (16/1000 stm) Statement coverage Testing uneconomical Reliability model

8.8 Six Essentials of Software Testing 1. The quality of the test process determines the success of the test effort. 2. Prevent defect migration by using early life-cycle testing techniques. 3. The time for software testing tools is now. 4. A real person must take responsibility for improving the testing process. 5. Testing is a professional discipline requiring trained, skilled people. 6. Cultivate a positive team attitude of creative destruction. 8.9 What are the five common problems in s/w development process? Poor Requirements: If the requirements are unclear, incomplete, too general and not testable there will be problem. Unrealistic Schedules: If too much work is creamed in too little time. Problems are inventible. Inadequate Testing: No one will know weather the system is good or not. Until the complains system crash Futurities: Requests to pile on new features after development is underway. Extremely common Miscommunication: If the developers dont know what is needed (or) customers have erroneous expectations, problems are guaranteed. 83

Testing Manual

8.10 Give me five common problems that occur during software development. Solid requirements :Ensure the requirements are solid, clear, complete, detailed, cohesive, attainable and testable. Realistic Schedules: Have schedules that are realistic. Allow adequate time for planning, design, testing, bug fixing, re-testing, changes and documentation. Personnel should be able to complete the project without burning out. Adequate Testing: Do testing that is adequate. Start testing early on, re-test after fixes or changes, and plan for sufficient time for both testing and bug fixing. Firm Requirements: Avoid new requirements as much as possible. Communication. Communicate inspections when appropriate features. Stick to initial and

Require

walk-thorough

8.11 What Should be done no enough time for testing Risk analysis to determine where testing should be focused Which functionality is most important to the project's intended purpose? Which functionality is most visible to the user? Which functionality has the largest safety impact? Which functionality has the largest financial impact on users? Which aspects of the application are most important to the customer? Which aspects of the application can be tested early in the development cycle? Which parts of the code are most complex and thus most subject to errors? Which parts of the application were developed in rush or panic mode? Which aspects of problems? similar/related previous projects caused

Which aspects of similar/related previous projects had large maintenance expenses? Which parts of the requirements and design are unclear or poorly thought out?

84

Testing Manual

What do the developers think are the highest-risk aspects of the application? What kinds of problems would cause the worst publicity? What kinds of problems would cause the most customer service complaints? What kinds of tests could easily cover multiple functionalities? Which tests will have the best high-risk-coverage to timerequired ratio?

8.12 How do you know when to stop testing? Common factors in deciding when to stop are... Deadlines, e.g. release deadlines, testing deadlines; Test cases completed with certain percentage passed; Test budget has been depleted; Coverage of code, functionality, or requirements reaches a specified point; Bug rate falls below a certain level; or Beta or alpha testing period ends. Miscommunication or No communication Software Complexity Programming Errors Changing Requirements Time Pressures Poorly Documented Code User Interface Errors Error Handling Boundary related errors Calculation errors Initial and Later states Control flow errors Errors in Handling or Interpreting Data

8.14 Why does the software have Bugs?

8.15 Different Type of Errors in Software

85

Testing Manual

Race Conditions Load Conditions Hardware

Source, Version and ID Control Testingand Errors Responsibilities 9 Roles 9.1 Test Manager Single point contact between Wipro onsite and offshore team Prepare the project plan Test Management Test Planning Interact with Wipro onsite lead, Client QA manager Team management Work allocation to the team Test coverage analysis Co-ordination with onsite for issue resolution. Monitoring the deliverables Verify readiness of the product for release through release review Obtain customer acceptance on the deliverables Performing risk analysis when required Reviews and status reporting Authorize intermediate deliverables and patch releases to customer. Resolves technical issues for the product group Provides direction to the team members Performs activities for the respective product group Review and Approve of Test Plan / Test cases Review Test Script / Code Approve completion of Integration testing Conduct System / Regression tests 86

9.2 Test Lead

Testing Manual

Ensure tests are conducted as per plan Reports status to the Offshore Test Manager

87

Testing Manual

9.3 Tester Engineer


Development of Test cases and Scripts Test Execution Result capturing and analysing Follow the test plans, scripts etc. as documented. Check tests are correct before reporting s/w faults Defect Reporting and Status reporting Assess risk objectively Prioritize what you report Communicate the truth. We cant test every thing. There is never enough time to do all testing you would like. So Prioritize Tests. Possible ranking criteria ( all risk based) Test where a failure would be most serve. Test where failures would be most visible. Take the help of customer in understanding what is most important to him. What is most critical to the customers business. Areas changed most often. Areas with most problems in the past. Most complex areas, or technically critical.

9.4 How to Prioritize Tests:

Tips

Note: If you follow above, whenever you stop testing, you have done the best testing in the time available.

88

Testing Manual

10 How can we improve the efficiency in testing?

In the recent year it has show lot of outsourcing in testing area. Its right time to think and create process to improve the efficiency of testing projects. The best team will result in the efficient deliverables. The team should contain 55% hard core test engineers, 30 domain knowledge engineers and 15% technology engineers.

How did we arrive to this figures? The past projects has shown 50-60 percent of the test cases are written on the basis of testing techniques, 28-33% of test cases are resulted to cover the domain oriented business rules and 15-18% technology oriented test cases.

Testing Vs Domain Vs Tech

Technology 15%

Domain 30%

Testing 55%

Software testability is simply how easily a computer program can be tested. A set of program characteristics that lead to testable software:

89

Testing Manual

11. CMM Levels


CMM = 'Capability Maturity Model', developed by the SEI. It's a model of 5 levels of organizational 'maturity' that determine effectiveness in delivering quality software. It is geared to large organizations such as large U.S. Defense Department contractors. However, many of the QA processes involved are appropriate to any organization, and if reasonably applied can be helpful. Organizations can receive CMM assessments by qualified auditors. ratings by undergoing

The Software Engineering Institute uses a conceptual framework based on industry best practices to assess the process maturity, capability and performance of a software development organization. This framework is called the Capability Maturity Model "CMM".

The extent of implementation for a specific key Process Area is evaluated by assessing: 1. 2. 3. 4. Commitment to perform (policies and leadership) Ability to perform (resources and training) Activities performed (plans and procedures) Measurement and analysis (measures and status) of implementation (oversight and quality

5. Verification assurance)

The Capability Maturity Model defined five levels of process maturity: 1. 2. 3. 4. 5. Initial (worship the hero) Repeatable (Plan the work) Defined (Work the plan) Managed (measure the work) Optimized (work the measures)

Software Development Process Maturity levels


The Initial Process (Level 1) The Initial (ad hoc) process level is unpredictable and often very chaotic. At this stage, the organization typically operates

90

Testing Manual

without formalized procedures, cost estimates, and project plans. Organizations at the ad hoc process level can improve their performance by instituting basic project controls. The most important are project management, management oversight, quality assurance, and change control. The fundamental role of the project management system is to ensure effective control of commitments. This requires adequate preparation, clear responsibility, a public declaration, and a dedication to performance. The Repeatable Process (Level 2) The repeatable process has one important strength that the ad hoc process does not: It provides control over the way the organization establishes its plans and commitments. This control provides such an improvement over the ad hoc process level that the people in the organization tend to believe they have mastered the software problem The Repeatable level having 6 key processing areas

Requirements management Software Project planning Software project tracking and oversight Software subcontract management Software Quality Assurance Software configuration management A suitably disciplined software development organization must have senior management oversight. This includes review and approval of all major development plans prior to their official commitment. Also, a quarterly review should be conducted of facility wide process compliance, installed quality performance, schedule tracking, cost trends, computing service. A quality assurance group is charged with assuring management that software work is done the way it is supposed to be done. Sufficient resources to monitor performance of all key planning, implementation, and verification activities.

The Defined Process (Level 3) With the defined process, the organization has achieved the foundation for major and continuing progress. The procedure for establishing a software development process architecture, or development life cycle, that describes the technical and management activities required for proper execution.

91

Testing Manual

The Defined level having 7 key processing areas. Organization process focus Organization process definition Training program Integrated software management Software product Engineering Inter-group coordination Peer reviews Metrics are used to track productivity, processes, and products. Project performance is predictable, and quality is consistently high. The Managed level having 2 key processing areas

Level 4: Managed

Quantitative process management Software process management

Level 5: Optimized The focus is on continuous process improvement. The impact of new processes and technologies can be predicted and effectively implemented when required. The Optimized level having3 key processing areas.

Defect prevention Technology change management Process change management Perspective on CMM ratings: During 1997-2001, 1018 organizations were assessed. Level 1 Level 2 Level 3 Level 4 Level 5 27 % 39 % 23 % 4% 5%

92

Testing Manual

CMM-I(Capability Maturity Model-Integration): CMM-I is an advanced version of capability Maturity Model(CMM). It has four components. They are

1. 2. 3. 4.

Software Engineering Software Integrated Product process development Software Acquisition

There is a significant difference between CMM and CMM-I . Two major factors 1. 2. Risk Identification Decision Analysis

Note: CMM is being used in over 5000 organizations worldwide. Only 120 organizations worldwide are at CMM level5. SEI-CMM (Software Engineering Institute-Capability Maturity Model): It is a frame work for Software Development P-CMM(People-Capability Maturity Model): Focus on People-Related process. Such as performance management, training and development, recruitment, staffing and the interoperability between different roles with in a services Organization.

ISO(International Organisation for Standardization)


The ISO 9001:2000 standard concerns quality systems that are assessed by outside auditors, and it applies to many kinds of production and manufacturing organizations, not just software. It covers documentation, design, development, production, testing, installation, servicing, and other processes.

The full set of standards consists of Q9001-2000 - Quality Management Systems: Requirements Q9000-2000 - Quality Management Systems: Fundamentals and Vocabulary Q9004-2000 - Quality Management Systems: Guidelines for Performance Improvements.

IEEE (Institute of Electrical and Electronics Engineers')

93

Testing Manual

IEEE/ANSI Standard 829: IEEE Standard for Software Test Documentation IEEE/ANSI Standard 1008: IEEE Standard of Software Unit Testing IEEE/ANSI Standard 730: IEEE Standard for Software Quality Assurance Plans.

94

Testing Manual

Testing Types

Term

Definition

Acceptance Testing

Testing the system with the intent of confirming readiness of the product and customer acceptance.

Ad Hoc Testing

Testing without a formal test plan or outside of a test plan. With some projects this type of testing is carried out as an adjunct to formal testing. If carried out by a skilled tester, it can often find problems that are not caught in regular testing. Sometimes, if testing occurs very late in the development cycle, this will be the only kind of testing that can be performed. Sometimes ad hoc testing is referred to as exploratory testing.

Alpha Testing

Testing after code is mostly complete or contains most of the functionality and prior to users being involved. Sometimes a select group of users are involved. More often this testing will be performed in-house or by an outside testing firm in close cooperation with the software engineering department.

Automated Testing

Software testing that utilizes a variety of tools to automate the testing process and when the importance of having a person manually testing is diminished. Automated testing still requires a skilled quality assurance professional with knowledge of the automation tool and the software being tested to set up the tests.

Beta Testing

Testing after the product is code complete. Betas are often widely distributed or even distributed to the public at large in hopes that they will buy the final product when it is released.

Black Box Testing

Testing software without any knowledge of the inner workings, structure or language of the module being tested. Black box tests, as most other kinds of tests, must be written from a definitive source document, such as a specification or requirements document..

Compatibility Testing browsers, utilities, and competing software will conflict with the software being

Testing used to determine whether other system software components such as tested.

Configuration Testing

Testing to determine how well the product works with a broad range of

95

Testing Manual
hardware/peripheral equipment configurations as well as on different operating systems and software.

Functional Testing

Testing two or more modules together with the intent of finding defects, demonstrating that defects are not present, verifying that the module performs its intended functions as stated in the specification and establishing confidence that a program does what it is supposed to do.

Independent Verification and Validation (IV&V)

The process of exercising software with the intent of ensuring that the software system meets its requirements and user expectations and doesn't fail in an unacceptable manner. The individual or group doing this work is not part of the group or organization that developed the software. A term often applied to government work or where the government regulates the products, as in medical devices.

Installation Testing

Testing with the intent of determining if the product will install on a variety of platforms and how easily it installs.

Integration Testing

Testing two or more modules or functions together with the intent of finding interface defects between the modules or functions. Testing completed at as a part of unit or functional testing, and sometimes, becomes its own standalone test phase. On a larger level, integration testing can involve a putting together of groups of modules and functions with the goal of completing and verifying that the system meets the system requirements. (see system testing)

Load Testing

Testing with the intent of determining how well the product handles competition for system resources. The competition may come in the form of network traffic, CPU utilization or memory allocation.

Performance Testing

Testing with the intent of determining how quickly a product handles a variety of events. Automated test tools geared specifically to test and fine-tune performance are used most often for this type of testing.

Pilot Testing

Testing that involves the users just before actual release to ensure that users become familiar with the release contents and ultimately accept it. Often is considered a Move-to-Production activity for ERP releases or a beta test for commercial products. Typically involves many users, is conducted over a short period of time and is tightly controlled. (see beta testing)

Regression Testing

Testing with the intent of determining if bug fixes have been successful and have not created any new problems. Also, this type of testing is done to ensure that no

96

Testing Manual
degradation of baseline functionality has occurred.

Security Testing

Testing of database and network software in order to keep company data and resources secure from mistaken/accidental users, hackers, and other malevolent attackers.

Software Testing

The process of exercising software with the intent of ensuring that the software system meets its requirements and user expectations and doesn't fail in an unacceptable manner. The organization and management of individuals or groups doing this work is not relevant. This term is often applied to commercial products such as internet applications. (contrast with independent verification and validation)

Stress Testing

Testing with the intent of determining how well a product performs when a load is placed on the system resources that nears and then exceeds capacity.

System Integration Testing

Testing a specific hardware/software installation. This is typically performed on a COTS (commerical off the shelf) system or any other system comprised of disparent parts where custom configurations and/or unique installations are the norm.

User Acceptance Testing

See Acceptance Testing.

White Box Testing

Testing in which the software tester has knowledge of the inner workings, structure and language of the software, or at least its purpose.

97

Testing Manual

12. Bug Report Template

Sample Bug Report


Bug Report: Bug ID: Auto generated Title: Password field is not Encrypted Area: Login Window Type: Code defect Environment:OS: Windows2000 Browser: IE6.0 Test Case ID: 1103 Build: build001 Opened By: Abc Opened on: 2.2.22 Assigned to : Active Attachment: screensnap.doc Steps to Repro: .Open Internet Explorer window .Enter www.hotmail.com in the address bar .Click on Go button .Enter login(say xyz@hotmail.com) in the login Text box .Enter text in the password text box Expected Result: Password field should be encrypted and text should be displayed as * Actual Result: Text appeared in the password Text box.

Other Items of Bug Templete:


Resolved by Resolve Date Build Resolution Closed by Closed date Attachements Documents Related Bugs

98

Testing Manual

Status Report:

Test Case Status:


Total Test Cases 50 Test Cases Executed Passed 34 Failed 6 Blocked 0 40

Bug Statistics:
Sev1 Active Resolved Closed Total 1 Sev2 0 0 1 7 Sev3 2 2 3 5 Sev4 0 1 4 2 Total 1 3 0 3 1 9 15

99

Testing Manual

13 Testing Glossary
Acceptance Testing: Formal testing conducted to determine whether or not a system satisfies its acceptance criteriaenables an end user to determine whether or not to accept the system. Affinity Diagram: A group process that takes large amounts of language data, such as a list developed by brainstorming, and divides it into categories. Alpha Testing: Testing of a software product or system conducted at the developers site by the end user. Audit: An inspection/assessment activity that verifies compliance with plans, policies, and procedures, and ensures that resources are conserved. Audit is a staff function; it serves as the eyes and ears of management. Automated Testing: That part of software testing that is assisted with software tool(s) that does not require operator input, analysis, or evaluation. Beta Testing: Testing conducted at one or more end user sites by the end user of a delivered software product or system. Black-box Testing: Functional testing based on requirements with no knowledge of the internal program structure or data. Also known as closed-box testing. Black box testing indicates whether or not a program meets required specifications by spotting faults of omission -places where the specification is not fulfilled. Bottom-up Testing: An integration testing technique that tests the low-level components first using test drivers for those components that have not yet been developed to call the low-level components for test. Boundary Value Analysis: A test data selection technique in which values are chosen to lie along data extremes. Boundary values include maximum, mini-mum, just inside/outside boundaries, typical values, and error values. Brainstorming: A group process for generating creative and diverse ideas. Branch Coverage Testing: A test method satisfying coverage criteria that requires each decision point at each possible branch to be executed at least once. Bug: A design flaw that will result in symptoms exhibited by some object (the object under test or some other object) when an object is subjected to an appropriate test.

100

Testing Manual

Cause-and-Effect (Fishbone) Diagram: A tool used to identify possible causes of a problem by representing the relationship between some effect and its possible cause.

101

Testing Manual

Cause-effect Graphing: A testing technique that aids in selecting, in a systematic way, a high-yield set of test cases that logically relates causes to effects to produce test cases. It has a beneficial side effect in pointing out incompleteness and ambiguities in specifications. Checksheet: A form used to record data as it is gathered. Clear-box Testing: Another term for white-box testing. Structural testing is sometimes referred to as clear-box testing, since white boxes are considered opaque and do not really permit visibility into the code. This is also known as glass-box or open-box testing. Client: The end user that pays for the product received, and receives the benefit from the use of the product. Control Chart: A statistical method for distinguishing between common and special cause variation exhibited by processes. Customer (end user): The individual or organization, internal or external to the producing organization, that receives the product. Cyclomatic Complexity: A measure of the number of linearly independent paths through a program module. Data Flow Analysis: Consists of the graphical analysis of collections of (sequential) data definitions and reference patterns to determine constraints that can be placed on data values at various points of executing the source program. Debugging: The act of attempting to determine the cause of the symptoms of malfunctions detected by testing or by frenzied user complaints. Defect: NOTE: Operationally, it is useful to work with two definitions of a defect: 1) From the producers viewpoint: a product requirement that has not been met or a product attribute possessed by a product or a function performed by a product that is not in the statement of requirements that define the product. 2) From the end users viewpoint: anything that causes end user dissatisfaction, whether in the statement of requirements or not. Defect Analysis: Using defects as data for continuous quality improvement. Defect analysis generally seeks to classify defects into categories and identify possible causes in order to direct process improvement efforts. Defect Density: Ratio of the number of defects to program length (a relative number). Desk Checking: A form of manual static analysis usually performed by the originator. Source code documentation, etc., is visually checked against requirements and standards. 102

Testing Manual

Dynamic Analysis: The process of evaluating a program based on execution of that program. Dynamic analysis approaches rely on executing a piece of software with selected test data. Dynamic Testing: Verification or validation performed which executes the systems code. Error: 1) A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition; and 2) a mental mistake made by a programmer that may result in a program fault. Error-based Testing: Testing where information about programming style, error-prone language constructs, and other programming knowledge is applied to select test data capable of detecting faults, either a specified class of faults or all possible faults. Evaluation: The process of examining a system or system component to determine the extent to which specified properties are present. Execution: The process of a computer carrying out an instruction or instructions of a computer. Exhaustive Testing: Executing the program combinations of values for program variables. with all possible

Failure: The inability of a system or system component to perform a required function within specified limits. A failure may be produced when a fault is encountered. Failure-directed Testing: Testing based on the knowledge of the types of errors made in the past that are likely for the system under test. Fault: A manifestation of an error in software. A fault, if encountered, may cause a failure. Fault Tree Analysis: A form of safety analysis that assesses hardware safety to provide failure statistics and sensitivity analyses that indicate the possible effect of critical failures. Fault-based Testing: Testing that employs a test data selection strategy designed to generate test data capable of demonstrating the absence of a set of pre-specified faults, typically, frequently occurring faults. Flowchart: A diagram showing the sequential steps of a process or of a workflow around a product or service. Formal Review: A technical review conducted with the end user, including the types of reviews called for in the standards.

103

Testing Manual

Function Points: A consistent measure of software size based on user requirements. Data components include inputs, outputs, etc. Environment characteristics include data communications, performance, reusability, operational ease, etc. Weight scale: 0 = not present; 1 = minor influence, 5 = strong influence. Functional Testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as black-box testing. Heuristics Testing: Another term for failure-directed testing. Histogram: A graphical description of individual measured values in a data set that is organized according to the frequency or relative frequency of occurrence. A histogram illustrates the shape of the distribution of individual values in a data set along with information regarding the average and variation. Hybrid Testing: A combination of top-down testing combined with bottom-up testing of prioritized or available components. Incremental Analysis: Incremental analysis occurs when (partial) analysis may be performed on an incomplete product to allow early feedback on the development of that product. Infeasible Path: Program statement sequence that can never be executed. Inputs: Products, services, or information needed from suppliers to make a process work. Inspection: 1) A formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems. 2) A quality improvement process for written material that consists of two dominant components: product (document) improvement and process improvement (document production and inspection). Instrument: To install or insert devices or instructions into hardware or software to monitor the operation of a system or component. Integration: The process of combining software components or hardware components, or both, into an overall system. Integration Testing: An orderly progression of testing in which software components or hardware components, or both, are combined and tested until the entire system has been integrated. Interface: A shared boundary. An interface might be a hardware component to link two devices, or it might be a portion of storage or registers accessed by two or more computer programs.

104

Testing Manual

Interface Analysis: Checks the interfaces between program elements for consistency and adherence to predefined rules or axioms. Intrusive Testing: Testing that collects timing and processing information during program execution that may change the behavior of the software from its behavior in a real environment. Usually involves additional code embedded in the software being tested or additional processes running concurrently with software being tested on the same platform. IV&V: Independent verification and validation is the verification and validation of a software product by an organization that is both technically and managerially separate from the organization responsible for developing the product. Life Cycle: The period that starts when a software product is conceived and ends when the product is no longer available for use. The software life cycle typically includes a requirements phase, design phase, implementation (code) phase, test phase, installation and checkout phase, operation and maintenance phase, and a retirement phase. Manual Testing: That part of software testing that requires operator input, analysis, or evaluation. Mean: A value derived by adding several qualities and dividing the sum by the number of these quantities. Measurement: 1) The act or process of measuring. A figure, extent, or amount obtained by measuring. Metric: A measure of the extent or degree to which a product possesses and exhibits a certain quality, property, or attribute. Mutation Testing: A method to determine test set thoroughness by measuring the extent to which a test set can discriminate the program from slight variants of the program. Non-intrusive Testing: Testing that is transparent to the software under test; i.e., testing that does not change the timing or processing characteristics of the software under test from its behavior in a real environment. Usually involves additional hardware that collects timing or processing information and processes that information on another platform. Operational Requirements: Qualitative and quantitative parameters that specify the desired operational capabilities of a system and serve as a basis for deter-mining the operational effectiveness and suitability of a system prior to deployment. Operational Testing: Testing performed by the end user on software in its normal operating environment.

105

Testing Manual

Outputs: Products, services, or information supplied to meet end user needs. Path Analysis: Program analysis performed to identify all possible paths through a program, to detect incomplete paths, or to discover portions of the program that are not on any path. Path Coverage Testing: A test method satisfying coverage criteria that each logical path through the program is tested. Paths through the program often are grouped into a finite set of classes; one path from each class is tested. Peer Reviews: A methodical examination of software work products by the producers peers to identify defects and areas where changes are needed. Policy: Managerial desires and intents concerning either process (intended objectives) or products (desired attributes). Problem: Any deviation from defined standards. Same as defect. Procedure: The step-by-step standards are met. method followed to ensure that

Process: The work effort that produces a product. This includes efforts of people and equipment guided by policies, standards, and procedures. Process Improvement: To change a process to make the process produce a given product faster, more economically, or of higher quality. Such changes may require the product to be changed. The defect rate must be maintained or reduced. Product: The output of a process; the work product. There are three useful classes of products: manufactured products (standard and custom), administrative/ information products (invoices, letters, etc.), and service products (physical, intellectual, physiological, and psychological). Products are defined by a statement of requirements; they are produced by one or more people working in a process. Product Improvement: To change the statement of requirements that defines a product to make the product more satisfying and attractive to the end user (more competitive). Such changes may add to or delete from the list of attributes and/or the list of functions defining a product. Such changes frequently require the process to be changed. NOTE: This process could result in a totally new product. Productivity: The ratio of the output of a process to the input, usually measured in the same units. It is frequently useful to compare the value added to a product by a process to the value of the input resources required (using fair market values for both input and output).

106

Testing Manual

Proof Checker: A program that checks formal proofs of program properties for logical correctness. Prototyping: Evaluating requirements or designs at the conceptualization phase, the requirements analysis phase, or design phase by quickly building scaled-down components of the intended system to obtain rapid feedback of analysis and design decisions. Qualification Testing: Formal testing, usually conducted by the developer for the end user, to demonstrate that the software meets its specified requirements. Quality: A product is a quality product if it is defect free. To the producer a product is a quality product if it meets or conforms to the statement of requirements that defines the product. This statement is usually shortened to quality means meets requirements. NOTE: Operationally, the work quality refers to products. Quality Assurance (QA): The set of support activities (including facilitation, training, measurement, and analysis) needed to provide adequate confidence that processes are established and continuously improved in order to produce products that meet specifications and are fit for use. Quality Control (QC): The process by which product quality is compared with applicable standards; and the action taken when nonconformance is detected. Its focus is defect detection and removal. This is a line function, that is, the performance of these tasks is the responsibility of the people working within the process. Quality Improvement: To change a production process so that the rate at which defective products (defects) are produced is reduced. Some process changes may require the product to be changed. Random Testing: An essentially black-box testing approach in which a program is tested by randomly choosing a subset of all possible input values. The distribution may be arbitrary or may attempt to accurately reflect the distribution of inputs in the application environment. Regression Testing: Selective retesting to detect faults introduced during modification of a system or system component, to verify that modifications have not caused unintended adverse effects, or to verify that a modified system or system component still meets its specified requirements. Reliability: The probability of failure-free operation for a specified period. Requirement: A formal statement of: 1) an attribute to be possessed by the product or a function to be performed by the product; the performance standard for the attribute or function; or 3) the measuring process to be used in verifying that the standard has been met. 107

Testing Manual

Review: A way to use the diversity and power of a group of people to point out needed improvements in a product or confirm those parts of a product in which improvement is either not desired or not needed. A review is a general work product evaluation technique that includes desk checking, walkthroughs, technical reviews, peer reviews, formal reviews, and inspections. Run Chart: A graph of data points in chronological order used to illustrate trends or cycles of the characteristic being measured for the purpose of suggesting an assignable cause rather than random variation. Scatter Plot (correlation diagram): A graph designed to show whether there is a relationship between two changing factors. Semantics: 1) The relationship of characters or a group of characters to their meanings, independent of the manner of their interpretation and use. 2) The relationships between symbols and their meanings. Software Characteristic: An inherent, possibly accidental, trait, quality, or property of software (for example, functionality, performance, attributes, design constraints, number of states, lines of branches). Software Feature: A software characteristic specified or implied by requirements documentation (for example, functionality, performance, attributes, or design constraints). Software Tool: A computer program used to help develop, test, analyze, or maintain another computer program or its documentation; e.g., automated design tools, compilers, test tools, and maintenance tools. Standards: The measure used to evaluate products and identify nonconformance. The basis upon which adherence to policies is measured. Standardize: Procedures are implemented to ensure that the output of a process is maintained at a desired level. Statement Coverage Testing: A test method satisfying coverage criteria that requires each statement be executed at least once. Statement of Requirements: The exhaustive list of requirements that define a product. NOTE: The statement of requirements should document requirements proposed and rejected (including the reason for the rejection) during the requirements determination process. Static Testing: Verification performed without executing the systems code. Also called static analysis.

108

Testing Manual

Statistical Process Control: The use of statistical techniques and tools to measure an ongoing process for change or stability. Structural Coverage: This requires that each pair of module invocations be executed at least once. Structural Testing: A testing method where the test data is derived solely from the program structure. Stub: A software component that usually minimally simulates the actions of called components that have not yet been integrated during top-down testing. Supplier: An individual or organization that supplies inputs needed to generate a product, service, or information to an end user. Syntax: 1) The relationship among characters or groups of characters independent of their meanings or the manner of their interpretation and use; 2) the structure of expressions in a language; and 3) the rules governing the structure of the language. System: A collection of people, machines, and methods organized to accomplish a set of specified functions. System Simulation: Another name for prototyping. System Testing: The process of testing an integrated hardware and software system to verify that the system meets its specified requirements. Technical Review: A review that refers to content of the technical material being reviewed. Test Bed: 1) An environment that contains the integral hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test of a logically or physically separate component. 2) A suite of test programs used in conducting the test of a component or system. Test Case: The definition of test case differs from company to company, engineer to engineer, and even project to project. A test case usually includes an identified set of information about observable states, conditions, events, and data, including inputs and expected outputs. Test Development: The development of anything required to conduct testing. This may include test requirements (objectives), strategies, processes, plans, software, procedures, cases, documentation, etc. Test Executive: Another term for test harness.

109

Testing Manual

Test Harness: A software tool that enables the testing of software components that links test capabilities to perform specific tests, accept program inputs, simulate missing components, compare actual outputs with expected outputs to determine correctness, and report discrepancies. Test Objective: An identified set of software features to be measured under specified conditions by comparing actual behavior with the required behavior described in the software documentation. Test Plan: A formal or informal plan to be followed to assure the controlled testing of the product under test. Test Procedure: The formal or informal procedure that will be followed to execute a test. This is usually a written document that allows others to execute the test with a minimum of training. Testing: Any activity aimed at evaluating an attribute or capability of a program or system to determine that it meets its required results. The process of exercising or evaluating a system or system component by manual or automated means to verify that it satisfies specified requirements or to identify differences between expected and actual results. Top-down Testing: An integration testing technique that tests the high-level components first using stubs for lower-level called components that have not yet been integrated and that stimulate the required actions of those components. Unit Testing: The testing done to show whether a unit (the smallest piece of software that can be independently compiled or assembled, loaded, and tested) satisfies its functional specification or its implemented structure matches the intended design structure. User: The end user that actually uses the product received. V- Diagram (model): a diagram that visualizes the order of testing activities and their corresponding phases of development Validation: The process of evaluating compliance with specified requirements. software to determine

Verification: The process of evaluating the products of a given software development activity to determine correctness and consistency with respect to the products and standards provided as input to that activity. Walkthrough: Usually, a step-by-step simulation of the execution of a procedure, as when walking through code, line by line, with an imagined set of inputs. The term has been extended to the review of material that is not procedural, such as data descriptions, reference manuals, specifications, etc.

110

Testing Manual

White-box Testing: Testing approaches that examine the program structure and derive test data from the program logic. This is also known as clear box testing, glass-box or open-box testing. White box testing determines if program-code structure and logic is faulty. The test is accurate only if the tester knows what the program is supposed to do. He or she can then see if the program diverges from its intended goal. White box testing does not account for errors caused by omission, and all visible code must also be readable.

111

Testing Manual

TATA CONSULTANCY SERVICES (TCS) Technical Interview (1 Round)


1. Tell me about yourself? 2. What kind of projects did u handle? 3. How big is your organization? 4. How much QA Strength in your organization? 5. Have u worked on Winrunner? 6. Winrunner which version u used in your project? 7. How many months of experience in Winrunner? 8. Winrunner works on IE or NETSCAPE? 9. What u used in your project IE or NETSCAPE? 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. What r the different types of add-ins in Winrunner? How u worked on JAVA Project, which add-ins u used? For JAVA how u load add-ins and what is the command for Can u write SQL Queries? How u comfortable in SQL Server? What is Defect Removal Efficiency? What is Defect Density? How u measure Quality? What is traceability matrix? What r the acceptance testcases? After completing Acceptance Testing what kind of bugs u What is Data Driven Test? Can u write script for DDT? Write a script using DDT for a Screen having 3 fileds ID, Can u write SQL Query? There is an EMP Table

loading Java add-ins?

faced?

NAME, SAL and 2 buttons SAVE , CANCEL?

112

Testing Manual

Eno 10 20 30 40 50 60 70 80 90 95 in italics.

Ename Purna

Sal 6000 7000 8000 9000 10000 20000 25000 30000 35000

Saibaba Lakshmi Vijay Feroz Kala Sushma Kalyan Dwaraka

Narendra 15000

Write a query for retrieving the records in the above table which are 25.What is bug life cycle? 26.What is your development team size? 27. Testing team size?( I told Dev. Team size lessthan Testing team) 28. Why Dev. Team size is less? 29. Can u write VB coding? 30. Do u have any Questions?

HR Interview (2 Round)
1. About yourself? 2. About your company profile like company strength, Team Size? 3. What is your family? 4. Why are you looking for a change? 5. Why u choose TCS? 6. What you know about TCS? 7. How much time require to join? 8. Cant you join before 1 month? 9. Why you choose Computers Subject? 10. 113 Why u opting only TCS?

Testing Manual

11. 12. 13. 14. 15.

Have u appeared before to the TCS? Can u bring Payslip? What is your current salary? What is your expected salary? Who is CEO of TCS, do u know?

Management Review (3 Round)


1. About yourself? 2. What types of testing u have done? 3. Do u know Winrunner? 4. Do u know Loadrunner, Rational? 5. What is testing life cycle? 6. Explain each of the phases in SDLC? 7. Models of SDLC? 8. CMM explain? 9. Difference between regression testing and acceptance testing? 10. 11. 12. 13. 14. 15. 16. Difference between integration and system testing? Difference between functionality and performance testing? Have u done any mistakes in your previous projects? What r the characterstics of test engineer as a team How much time u require to join? What r your strengths? What do u mean by learning new techniques very quickly?

member?

( I told my strength as ability to learn new techniques very easily) 17. 18. Cant u join tomorrow? Can u relocate in Chennai or Bangalore? SATYAM COMPUTER SERVICES

114

Testing Manual

Personal Interview(Screening 1 Round)


1. What is your company name? 2. Strength of your company? 3. What tools u know? 4. Your background education? 5. What is your previous company? 6. What is current project? 7. What r different types of testing? 8. How can u rate in Winrunner? 9. Which testing u r familiar? 10. 11. Your company is training institute or S/W company? How do u tested your project?

Technical Interview (2 Round)


1. Difference between S/W Testing and QA? 2. Testing comes under QA or not? 3. About your self? 4. Your current project? 5. Test plan document consists of what? 6. How do u derive testcases? 7. Difference between Use Cases and Functional Specifications? 8. How do u justify your testcases are correct when your PL comes to you? (you derived 270 testcases and your colleague derived 800 testcases how can u justify your testcases are correct ?hereit is a small discussion actually here u need to say about traceability matrix) 9. What is system testing? 10. 11. What r the techniques for writing Testcases? What is equivalence class?

115

Testing Manual

12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 116

Give me example of equivalence class? One Testcase is enough for valid equivalence class? XML do u know? Regarding SQL Server how much you r comfortable? SQL Database concepts , R you comfortable? What is Referential Integrity Constraint? What happens if we delete a record in child table? What happens if we delete a record in parent table? What r your strengths? What is primary key? What is Unique key? Difference between Unique and What is stored procedure? Why we use Stored Procedure? Ondelete Cascade? What r different Joins? Main attributes of XML? What is constraint? What r your academic %? What r your achievements? What is the best bug? What is bug life cycle? TestDirector do u know? Winrunner u know? What is a checkpoint? Definition? Cant we use checkpoint if we r having only one version of How do u say a Test is pass/fail? Shall we use a check point in single version or not? Tell about Winrunner what u know? When we go for automation?

Primary keys?

application?

Testing Manual

41.

What is the Stop Criteria of testing?

HR Interview (3 Round)
1. About yourself? 2. When we go for automation? 3. Cant we use tool for testing? 4. Family background? 5. Are u single? 6. TestDirector u know ? Explain? 7. Explain Loadrunner? 8. What is Performace Testing? 9. Have u used Loadrunner? 10. 11. 12. 13. 14. 15. 16. Do u know Rational Robot? What you are expecting from Satyam? What is your CTC? What is Expected Salary? Cant u work for 2.2 Package if u want career growth? How do u rate in WinRunner? When you are expecting your marriage?

117

Testing Manual

VIRTUSA

Technical Interview
1. About Yourself and Jobprofile? 2. What is the difference between QA & QC? 3. What is the difference between Verification & Validation? 4. Which model u follow in your company? 5. Draw the Structure of V-model? 6. Testplan why? 7. What it contains? 8. How u prepare Testcases? 9. What is difference between Usecases & Functional Specifications? 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. Why u derive Testcases from both Usecases and Functional What r the types of testing? What is greybox testing? If the tester does not know VB.net cant he test ISO & CMM ..abbreviations? What is CMM-i? What r the Steps in CMM? What r the testing types? What is the limitation of V-model? What is Functionality testing? What is difference between Client/Server and Web Cant we test VB Application in all the Platforms? Is VB browser compatability? Write the test plan for the following screen? Location: combobox 118 specs not from one?

application?

Application?

Testing Manual

Date: Testbox Time: Textbox 24. 25. 26. 27. 28. 29. 30. 31. Difference between Unit Testing & Functionality? What is the High level Architechture for farmers auto How do u submit a defect in your company? What is bug life cycle? How do u know test is pass/fail? What is start criteria? What r the techniques u know for preparing testcases Tell me testcases for the following ..like N can take numbers from 1 to 100. A can take numbers from 1 to 50 B can take numbers from 50 to 100 OK 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. How u submit a defect using tool? How u maintain bug tracking sheet? Compile Module? What is the use of Compile Module? Do u know Oracle? What r the types of Joins? What is outer join? What is self join? When u go for automation? When u stop testing? Difference between Retesting and Regression Testing?

insurance?

.explain the techniques?

43. Why V-Model is called V 119

Testing Manual

COGNIZANT TECHNOLOGY SOLUTIONS

Technical Interview
1. What r the Phases of SDLC? 2. Explain Waterfall model? 3. What is Testing? 4. Describe V-Model? 5. What Testpaln Document Consists? 6. Who will prepare Testplan in your company? 7. What is Testcase? 8. What is template for testcase? 9. Explain Bug lifecycle? 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 120 What r the deliverables? What is Functionality Testing? What is Regression Testing? What is Compatability Testing? Write the Testcases for the Telephone? What is stop criteria for testing? Without any specifications can you test the application? If What r your responsibilities as a tester? Why automation testing? What is the difference between Verification and Validation? CMM Levels------Explain each? What is difference between Quality control and Quality What is the quality approach for your company? What r the Recording modes in Winrunner? What r the Execution modes of Winrunner? What is the definition of Winrunner?

so that is called what type of testing?

Assurance?

Testing Manual

26. 27. 28. 29. Point) 30.

What r the different types of checkpoints? What r the different types of GUI Checkpoints? What files will be created when you use GUI Check Points What is Synchronization Point?(Syntax of Synchronization When Requirements are changed what u would do

with what extension?

whether you read all the requirements again or you execute all the testcases. What is the easiest way? (Exactly I dont remember but the concept is ..U need to tell about Why Automation) 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. GUI Map Editor is for what? What is Compile Module? What is the Difference Between Compile Module and Test Have you ever used Compile Module in your project? What is the Primary Key, Foreign Key? What is Stored Procedure and syntax? Have u ever tested Stored Procedure in your Project? Can you add 0 in a table having Primary Key? What r the different Metrics in your Project? Can u write VB Program now itself? APPLABS

Script?

Aptitude and English Test (1 Round)

Technical Interview (2 Round)


1. Tell me about yourself? 2. Company Strength how many members?

121

Testing Manual

3. Testing team size? 4. What is your role in that company? 5. How will you test the application? 6. Write a C- Program to sum 1 to 100 numbers? 7. What is Regression Testing? 8. Tell me about Cyber Cops Project? 9. Tell me about FARMERS Project? 10. 11. 12. 13. 14. Do u know VC++? Do u know VB? Write the testcases for the following screen? ( Some screen What is bug lifecycle? How do u execute a testcase?

is given)

122