TESTING CONCEPTS

Beginners Guide to Software Testing

By M.Saravanan (Software Testing Engineer)

TESTING CONCEPTS

FOREWORD Beginners Guide to Software Testing introduces a practical approach to testing software. It bridges the gap between theoretical knowledge and real world implementation. This article helps you gain an insight to Software Testing understand technical aspects and the processes followed in a real working environment. Who will benefit? Beginners: For those of you who wish to mould your theoretical software engineering knowledge into practical approach to working in the real world. Those who wish to take up Software Testing as a profession. Developers! This is an era where you need to be an All rounder. It is advantageous for developers to posses testing capabilities to test the application before hand. This will help reduce overhead on the testing team. Already a Tester! You can refresh all your testing basics and techniques and gear up for Certifications in Software Testing An earnest suggestion: No matter which profession you choose, it is advisable that you posses the following skills: Good communication skills oratory and writing Fluency in English Good Typing skills By the time you finish reading this article, you will be aware of all the techniques and processes that improves your efficiency, skills and confidence to jump start into the field of Software Testing. Beginners Guide To Software Testing is our sincere effort to educate and create awareness among people, the growing importance of software quality. With the advent of globalization and increase in market demand for software with good

TESTING CONCEPTS quality, we see the need for all Software Engineers to know more about Software Testing. We believe that this article helps serve our motto Adding Values to Lives Around Us. A brief history of Software Engineering Being a Software Test Professional, you must know a brief history of Software Engineering. Software Testing comes into picture in every phase of Software Engineering. The software industry has evolved through 4 eras, 50s 60s, mid 60s late 70s, mid 70s- mid 80s, and mid 80s-present. Each era has its own distinctive characteristics, but over the years the softwares have increased in size and complexity. Several problems are common to almost all of the eras and are discussed below. The Software Crisis dates back to the 1960s when the primary reasons for this situation were less than acceptable software engineering practices. In the early stages of software there was a lot of interest in computers, a lot of code written but no established standards. Then in early 70s a lot of computer programs Started failing and people lost confidence and thus an industry crisis was declared. Various reasons leading to the crisis included: - Hardware advances outpacing the ability to build software for this hardware. - The ability to build in pace with the demands. - Increasing dependency on softwares - Struggle to build reliable and high quality software - Poor design and inadequate resources. This crisis though identified in the early years, exists to date and we have examples of software failures around the world. Software is basically considered a failure if the project is terminated because of costs or overrun schedules, if the project has experienced overruns in excess of 50% of the original or if the software results in client lawsuits. Some examples of failures include failure of Air traffic control systems, failure of medical software, and failure in telecommunication

TESTING CONCEPTS software. The primary reason for these failures other than those mentioned above is due to bad software engineering practices adopted. Some of the worst software practices include: No historical software-measurement data. Rejection of accurate cost estimates. Failure to use automated estimating and planning tools. Excessive, irrational schedule pressure and creep in user requirements. Failure to monitor progress and to perform risk management. Failure to use design reviews and code inspections. To avoid these failures and thus improve the record, what is needed is a better understanding of the process, better estimation techniques for cost time and quality measures. But the question is, what is a process? Process transform inputs to outputs i.e. a product. A software process is a set of activities, methods and practices involving transformation that people use to develop and maintain software. At present a large number of problems exist due to a chaotic software process and the occasional success depends on individual efforts. Therefore to be able to deliver successful software projects, a focus on the process is essential since a focus on the product alone is likely to miss the scalability issues, and improvements in the existing system. This focus would help in the predictability of outcomes, project trends, and project characteristics. The process that has been defined and adopted needs to be managed well and thus process management comes into play. Process management is concerned with the knowledge and management of the software process, its technical aspects and also ensures that the processes are being followed as expected and improvements are shown. From this we conclude that a set of defined processes can possibly save us from software project failures. But it is nonetheless important to note that the process alone cannot help us avoid all the problems, because with varying circumstances the need varies and the process has to be adaptive to these varying needs. Importance needs to be given to the human aspect of software development

TESTING CONCEPTS since that alone can have a lot of impact on the results, and effective cost and time estimations may go totally waste if the human resources are not planned and managed effectively. Secondly, the reasons mentioned related to the software engineering principles may be resolved when the needs are correctly identified. Correct identification would then make it easier to identify the best practices that can be applied because one process that might be suitable for one organization may not be most suitable for another. Therefore to make a successful product a combination of Process and Technicalities will be required under the umbrella of a well-defined process. Having talked about the Software process overall, it is important to identify and relate the role software testing plays not only in producing quality software but also maneuvering the overall process. The computer society defines testing as follows: Testing -- A verification method that applies a controlled set of conditions and stimuli for the purpose of finding errors. This is the most desirable method of verifying the functional and performance requirements. Test results are documented proof that requirements were met and can be repeated. The resulting data can be reviewed by all concerned for confirmation of capabilities. There may be many definitions of software testing and many which appeal to us from time to time, but its best to start by defining testing and then move on depending on the requirements or needs.

TESTING CONCEPTS

Content:

1. TESTING. 2. TESTING TYPES.. 2.1Static Testing. 2.2Dynamic Testing.4 3TESTING TECHNIQUES.. 3.1 White box Testing. White Box Techniques.. 3.1.1 Method Coverage. 3.1.2 Statement Coverage.. 3.1.3 Branch Coverage.. 3.1.4 Condition Coverage. Types of White Box Testing Advantages Disadvantages 3.2 Black box Testing Black Box Techniques.. 3.2.1 Boundary value Analysis 3.2.2 Equivalence Partitioning 3.2.3 Decision Table Testing.. 3.3 Gray box Testing.. 4. Testing Levels . a. Unit Testing.. b. Integration Testing. Types of Integration Testing. 1. Top down Integration Testing.. 2. Bottom up Integration Testing. 3. System Testing. 4. User Acceptance Testing.. Types of user acceptance Testing. 1. Alpha Testing..

1 2 3 5 X X X X X X X X X X X X X X X X X X X X X X X X

TESTING CONCEPTS 2. Beta Testing.. .................... 3. Gamma Testing Smoke Testing...... Sanity Testing Performance Testing X a. Load Testing.. b. Stress Testing Functional Testing Exploratory Testing. Ad-hoc Testing.. Regression Testing.. Retesting. Usability Testing Install / Uninstall Testing Security Testing Compatibility Testing Compression Testing Incremental Integration Testing. End-to-End Testing Recovery Testing 5. TESTING METHODOLOGY 5.1 V model.. 5.2 Waterfall model. 5.3 The Spiral Model 5.4 Iterative Model 6. STLC.. 7. TESTING DOCUMENTS Test Plan X Software Requirement Specification Test Case 8. BUG TRACKING 9. BUG LIFE CYCLE . 10. TEST SUMMARY RESULT. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

TESTING CONCEPTS What is a good test case Severity Priority. Client server testing.. Web testing. Configuration management.. Test deliverables Why do we go for automation testing. GUI TESTING.. . Windows Compliance Testing.. Application.. X X X X X X X X X X X

Text Boxes.. X Option (Radio Buttons) X Check Boxes.. X Command Buttons. X Drop Down List Boxes.. X Combo Boxes. X List Boxes.. X Screen Validation Checklist

TESTING CONCEPTS Aesthetic Conditions. X Validation Conditions. Navigation Conditions.. Usability Conditions. Data Integrity Conditions Modes (Editable Read-only) Conditions General Conditions.. Specific Field Tests.. Testing Interview Questions What is Regression Testing.. CLIENT / SERVER TESTING... WEB TESTING.. Black Box Testing: Types and techniques of BBT.. BVA(Boundary Value Analysis) techniques. Types of Risks in Software Projects Categories of risks How Domain knowledge is Important for testers How to get your all bugs resolved without any Invalid bug label. What troubleshooting you need to perform before reporting any bug X How to write a good bug report? Tips and Tricks. How to Report a Bug Some Bonus tips to write a good bug report X X X X X X X X X X X X X X X X X X X X

TESTING CONCEPTS How to write software Testing Weekly Status Report. How to hire the right candidates for software testing positions X X

Website Cookie Testing, Test cases for testing web application cookies? Applications where cookies can be used.. Drawbacks of cookies. Some Major Test cases for web application cookie testing. Software Installation/Uninstallation Testing. What are the Quality attributes. Developers are not good testers. What you say Living life as a Software Tester.. How to be a good tester. Need of Skilled Testers Steps to Effective Software Testing. Unit Testing: Why? What? & How. Integration Testing: Why? What? And how. Metrics Used In Testing.. Metrics for Evaluating Application System Testing... Positive and Negative Testing.. Table of Content.. Foreword........................................................................................... 1. Introduction................................................................................. 2. Scope .......................................................................................... 3. Arrangement ............................................................................... 4. Normative references................................................................... 5. Definitions................................................................................... A...................................................................................................... B...................................................................................................... X X X X X X X X X X X X X X X X X X X X X

10

TESTING CONCEPTS C....................................................................................................... D....................................................................................................... E........................................................................................................ F........................................................................................................ G....................................................................................... . H....................................................................................................... I........................................................................................................ K................................................................................... .. L................................................................................. .. M....................................................................................................... N........................................................................................................ O................................................................................. .. P........................................................................................ . R........................................................................................................ S .................................................................................................... .. T.................................................................................................... U................................................................................................... V...................................................................................... .. W................................................................................................... .. Annex A (Informative).......................................................... .. Annex B (Method of commenting on this glossary)...........................

11

TESTING CONCEPTS

TESTING: Testing is the process of executing a program with the intent of finding errors TESTING TYPES There are two types of testing. Static Testing Dynamic Testing Static Testing: Verifying the document alone is called static testing. Dynamic Testing Testing the functionality is called dynamic Testing. Difference b/w static testing and dynamic testing: There are many approaches to software testing. Reviews, walkthroughs or inspections are considered as static testing, whereas actually executing programmed code with a given set of test cases is referred to as dynamic testing. The former can be, and unfortunately in practice often is, omitted, whereas the latter takes place when programs begin to be used for the first time - which is normally considered the beginning of the testing stage. This may actually begin before the program is 100% complete in order to test particular sections of code (modules or discrete functions). For example, Spreadsheet programs are, by their very nature, tested to a large

12

TESTING CONCEPTS extent "on the fly" during the build process as the result of some calculation or text manipulation is shown interactively immediately after each formula is entered. TESTING TECHNIQUES: White box testing OR (clear box testing, glass box testing, and transparent box testing, translucent box testing or structural testing) The process of Checking the program coding or source coding of the application is called as whit box testing.

WHITE BOX TESTING TECHNIQUES: Testing the application with the knowledge of coding to examine output is called white box testing. 1. Method Coverage: Method coverage is a measure of the percentage of methods that have been executed by test cases. Undoubtedly, your tests should call 100% of your methods. It seems irresponsible to deliver methods in your product when your testing never used these methods. As a result, you need to ensure you have 100% method coverage. 2. Statement Coverage: Statement coverage is a measure of the percentage of statements that have been executed by test cases. Your objective should be to achieve 100% statement coverage through your testing. Identifying your cyclomatic number and executing this minimum set of test cases will make this statement coverage achievable. 3. Branch Coverage: Branch coverage is a measure of the percentage of the decision points (Boolean expressions) of the program have been evaluated as both true and false in test cases. 4. Condition Coverage:

13

TESTING CONCEPTS We will go one step deeper and examine condition coverage. Condition coverage is a Measure of percentage of Boolean sub-expressions of the program that have been evaluated as either true or false outcome [applies to compound predicate] in test cases.

Types of white box testing The following types of white box testing exist: 1. Code coverage: Creating tests to satisfy some criteria of code coverage. For example, the test designer can create tests to cause all statements in the program to be executed at least once. Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing. Currently, the use of code coverage is extended to the field of digital hardware, the contemporary design methodology of which relies on Hardware description languages (HDLs). Code coverage techniques were amongst the first techniques invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM in 1963. Code coverage is one consideration in the safety certification of avionics equipment. The standard by which avionics gear is certified by the Federal Aviation Administration (FAA) is documented in DO-178(b) 2. Mutation testing methods:

14

TESTING CONCEPTS Mutation testing (or Mutation analysis) is a method of software testing, which involves modifying program's source code in small ways.[1] These, so-called mutations, are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as driving each expression to zero). The purpose is to help the tester develop effective tests or locate weaknesses in the test data used for the program or in sections of the code that are seldom or never accessed during execution.

3. Fault injection methods: In software testing, fault injection is a technique for improving the coverage of a test by introducing faults in order to test code paths, in particular error handling code paths, which might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part of developing robust software. The propagation of a fault through to an observable failure follows a well defined cycle. When executed, a fault may cause an error, which is an invalid state within a system boundary. An error may cause further errors within the system boundary, therefore each new error acts as a fault, or it may propagate to the system boundary and be observable. When error states are observed at the system boundary they are termed failures. This mechanism is termed the fault-error-failure cycle
[2]

and is a key mechanism in dependability.

4. Static testing: White box testing includes all static testing. Static testing is a form of software testing where the software isn't actually used. This is in contrast to dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code or and manually reading of the code or document to find errors. This type of testing can be used by the developer who wrote the code, in isolation. Code reviews, inspections and walkthroughs are also used.

15

TESTING CONCEPTS From the black box testing point of view, static testing involves review of requirements or specifications. This is done with an eye toward completeness or appropriateness for the task at hand. This is the verification portion of Verification and Validation. Even static testing can be automated. A static testing test suite consists in programs to be analyzed by an interpreter or a compiler that asserts the programs syntactic validity. Bugs discovered at this stage of development are less expensive to fix than later in the development cycle. Advantages: As the knowledge of internal coding structure is precondition, it becomes very easy to find out which type of input/data can help in testing the application effectively. The other advantage of white box testing is that it helps in optimizing the code It helps in removing the extra lines of code, which can bring in hidden defects. Forces test developer to reason carefully about implementation Disadvantages: As knowledge of code and internal structure is a prerequisite, a skilled tester is needed to carry out this type of testing, which increases the cost. And it is nearly impossible to look into every bit of code to find out hidden errors, which may create problems, resulting in failure of the application. Not looking at the code in a runtime environment. Thats important for a number of reasons. Development of vulnerability is dependent upon all aspects of the platform being targeted and source code is just of those components. The underlying operating system, the backend database being used, third party security tools, dependent libraries, etc. must all be taken into account when determining exploitability. A source code review is not able to take these factors into account. Very few white-box tests can be done without modifying the program, changing values to force different execution paths, or to generate a full range of inputs to test a particular function. Miss cases omitted in the code a. Unit Testing

16

TESTING CONCEPTS Unit testing is defines as Testing the individual module. The Tools used in Unit Testing are debuggers, tracers and is done by Programmers. b. Static and Dynamic Analysis Static analysis involves going through the code in order to find out any possible defect in the code. Dynamic analysis involves executing the code and analyzing the output.

c. Statement coverage In this type of testing the code is executed in such a manner that every statement of the application is executed at least once. It helps in assuring that all the statements execute without any side effect. d. Branch coverage No software application can be written in a continuous mode of coding, at some point we need to branch out the code in order to perform a particular functionality. Branch coverage testing helps in validating of all the branches in the code and making sure that no branching leads to abnormal behavior of the application. e. Security Testing Security Testing is carried out in order to find out how well the system can protect itself from unauthorized access, hacking cracking, any code damage etc. which deals with the code of application. This type of testing needs sophisticated testing techniques. f. Mutation Testing A kind of testing in which, the application is tested for the code that was modified after fixing a particular bug/defect. It also helps in finding out which code and which strategy of coding can help in developing the functionality effectively. BLACK BOX TESTING: (Testing of application without the knowledge of coding) Black box testing treats the software as a black box without any knowledge of internal implementation. Black box testing methods include equivalence partitioning,

17

TESTING CONCEPTS boundary value analysis, all-pairs testing, fuzz testing, model-based testing, traceability matrix, exploratory testing and specification-based testing. a. Boundary value Analysis Boundary value is defined as a data value that corresponds to a minimum or maximum input, internal, or output value specified for a system or component. b. Equivalence Partitioning Equivalence partitioning is a strategy that can be used to reduce the number of test cases that need to be developed. Equivalence partitioning divides the input domain of a program into classes. For each of these equivalence classes, the set of data should be treated the same by the module under test and should produce the same answer. Test cases should be designed so the inputs lie within these equivalence classes. c. Decision Table Testing Decision tables are used to record complex business rules that must be implemented in the program, and therefore tested. GRAY BOX TESTING: Gray box testing is a software testing technique that uses a combination of black box testing and white box testing. Gray box testing is not black box testing, because the tester does know some of the internal workings of the software under test. In gray box testing, the tester applies a limited number of test cases to the internal workings of the software under test. In the remaining part of the gray box testing, one takes a black box approach in applying inputs to the software under test and observing the outputs.

Gray box testing is a powerful idea. The concept is simple; if one knows something about how the product works on the inside, one can test it better, even from the outside. Gray box testing is not to be confused with white box testing; i.e. a testing approach that attempts to cover the internals of the product in detail. Gray box testing is a test strategy based partly on internals. The testing approach is known as gray box testing, when one does have some knowledge, but not the full knowledge of the internals of the product one is testing.

In gray box testing, just as in black box testing, you test from the outside of a product, just as you do with black box, but you make better-informed testing choices

18

TESTING CONCEPTS because you're better informed; because you know how the underlying software components operate and interact. TESTING LEVELS There are four testing level such as; Unit Testing Integration Testing System Testing User acceptance Testing

Unit Testing: The testing done to a unit or to a smallest piece of software. Done to verify if it satisfies its functional specification or its intended design structure. Integration Testing: Testing which takes place as sub elements are combined (i.e., integrated) to form higher-level elements Or Testing the related module/ one or more module together is called as integration testing. Types of Integrated Testing: There are two types of integrated Testing such as; 1. Top down Integration Testing. 2. Bottom up Integration Testing. 1. Top down Integration Testing: Checking the high level module to low level module of the application. Low level module normally simulated stubs. 2.Bottom up Integration Testing: Checking the low level module to high level module of the application. High level module normally simulated main. 3.System Testing: Testing the software for the required specifications on the intended hardware. OR

19

TESTING CONCEPTS Several modules constitute a project, once all this modules is integrated several error may occur and this type of testing is done in system testing. 4.User acceptance Testing: Formal testing conducted to determine whether or not a system satisfies its acceptance criteria, which enables a customer to determine whether to accept the system or not. OR It is the testing with intent of confirming readiness of the product and customer acceptance.

Types of User Acceptance Testing There are three types of user acceptance Testing such as; 1. Alpha testing 2. Beta testing 3. Gamma Testing 1. Alpha testing: Alpha testing is a part of the user acceptance testing. Before releasing the application the customer test that application in developer sight. 2. Beta testing Beta testing is also part of the user acceptance testing. Before releasing the application testers and customers combined check the application inn customer sight. Gamma Testing: Gamma testing is testing of software that has all the required features, but it did not go through all the in-house quality checks. Some of the Testing are mentioned below are: Smoke testing Smoke testing is a safe test. Smoke testing is a set of test cases, which we execute when we getting a new build. Smoke testing is verify the build is testable or not to based on smoke test cases. We can reject the build. It covers all the functionality. Sanity testing

20

TESTING CONCEPTS Sanity testing is test critical and major functionality of application. It is a one time application. For example cursor navigation. Performance testing We have putting under heavy load of the application. Checking the multi task performance of the application. Two types of performance testing. Load testing Stress testing a. Load testing We have putting under heavy load of the application. Checking the within a limit of the application. Load testing success is criteria. b. Stress testing We have putting under heavy load of the application. Checking the beyond the limit of the application. Stress testing failure is criteria. Functional Testing: Testing the application, whether the application is functioning as per the requirements is called Functional testing. Exploratory testing Checking the application without coding knowledge. Testing is done by giving random input. Ad-hoc testing It is a part of the exploratory testing. It is a random testing it means testing a application proper without test plans. Its carried out at the end of the project at the all test cases are executed. OR Testing with out a formal test plan or outside of a test plan. Regression testing Testing the application to find whether the change in code affects anywhere in the application. Checking Re executing the modification of the application. It should affect old build. not

21

TESTING CONCEPTS

Retesting: We check for the particular bug and its dependencies after it is said to be fixed. OR Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions. Usability testing - User-friendliness check. Application flow is tested, Can new user understand the application easily, Proper help documented whenever user stuck at any point. Basically system navigation is checked in this testing. Install/uninstall testing - Tested for full, partial, or upgrade install/uninstall processes on different operating systems under different hardware, software environment. Compatibility testing - Testing how well software performs in a particular hardware/software/operating system/network environment and different combination s of above. Comparison testing - Comparison of product strengths and weaknesses with previous versions or other similar products. Incremental integration testing - Bottom up approach for testing i.e continuous testing of an application as new functionality is added; Application functionality and modules should be independent enough to test separately. done by programmers or by testers. End-to-end testing - Similar to system testing, involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate. Recovery testing - Testing how well a system recovers from crashes, hardware failures, or other catastrophic problems. TESTING METHODOLOGY

22

TESTING CONCEPTS Testing methodology means what kind of the methods to be selected that suits for our applications. It has v & v model, waterfall model., etc. 5.1 V model The V-model is a software development model which can be accepted to be the extension of the waterfall model. Instead of moving down in a linear way, the process steps are bent upwards after the coding phase, to form the typical V shape. The V-Model demonstrates the relationships between each phase of the development life cycle and its associated phase of testing.

Verification User acceptance testing technique Requirement

Validation

User acceptance testing

Design

System testing technique

System testing

Integration testing technique Coding Integration testing

Unit testing 5.2 Waterfall model:

23

TESTING CONCEPTS Waterfall approach was first Process Model to be introduced and followed widely in software testing to ensure success of the project. In "The Waterfall" approach, the whole process of software development is divided into separate process phases. The phases in Waterfall model are: Requirement Specifications phase, Software Design, Implementation and Testing & Maintenance. All these phases are cascaded to each other so that second phase is started as and when defined set of goals are achieved for first phase and it is signed off, so the name "Waterfall Model". All the methods and processes undertaken in Waterfall Model are more visible.

Water fall model

Requirement

Design

Coding

Unit testing

Integration testing

24

TESTING CONCEPTS

System testing

User acceptance testing

5.3 The Spiral Model The spiral model, also known as the spiral lifecycle model, is a systems development method (SDM) used in information technology (IT). This model of development combines the features of the prototyping model and the waterfall model. The spiral model is intended for large, expensive, and complicated projects. The steps in the spiral model can be generalized as follows: 1.The new system requirements are defined in as much detail as possible. This usually involves interviewing a number of users representing all the external or internal users and other aspects of the existing system. 2.A preliminary design is created for the new system. 3.A first prototype of the new system is constructed from the preliminary characteristics of the final product. 4.A second prototype is evolved by a fourfold procedure: (1) evaluating the first prototype in terms of its strengths, weaknesses, and risks; (2) defining the design.

This is usually a scaled-down system, and represents an approximation of the

25

TESTING CONCEPTS requirements of the second prototype; (3) planning and designing the second prototype; (4) constructing and testing the second prototype. 5.At the customer's option, the entire project can be aborted if the risk is deemed too great. Risk factors might involve development cost overruns, operating-cost miscalculation, or any other factor that could, in the customer's judgment, result in a less-than-satisfactory final product. 6.The existing prototype is evaluated in the same manner as was the previous prototype, and, if necessary, another prototype is developed from it according to the fourfold procedure outlined above. 7.The preceding steps are iterated until the customer is satisfied that the refined prototype represents the final product desired. 8.The final system is constructed, based on the refined prototype. 9.The final system is thoroughly evaluated and tested. Routine maintenance is carried out on a continuing basis to prevent large-scale failures and to minimize downtime. Applications For a typical shrink-wrap application, the spiral model might mean that you have a rough-cut of user elements (without the polished / pretty graphics) as an operable application, add features in phases, and, at some point, add the final graphics. The spiral model is used most often in large projects. For smaller projects, the concept of agile software development is becoming a viable alternative. The US military has adopted the spiral model for its Future Combat Systems program. Advantages 1.Estimates (i.e. budget, schedule, etc.) become more realistic as work progresses, because important issues are discovered earlier. 2.It is more able to cope with the (nearly inevitable) changes that software development generally entails.

26

TESTING CONCEPTS 3.Software engineers (who can get restless with protracted design processes) can get their hands in and start working on a project earlier. Disadvantages 1.Highly customized limiting re-usability 2.Applied differently for each application 3.Risk of not meeting budget or schedule 4.Risk of not meeting budget or schedule 5.4 Iterative Model An iterative lifecycle model does not attempt to start with a full specification of requirements. Instead, development begins by specifying and implementing just part of the software, which can then be reviewed in order to identify further requirements. This process is then repeated, producing a new version of the software for each cycle of the model. Consider an iterative lifecycle model which consists of repeating the following four phases in sequence:

27

TESTING CONCEPTS A Requirements phase, in which the requirements for the software are gathered and analyzed. Iteration should eventually result in a requirements phase that produces a complete and final specification of requirements. - A Design phase, in which a software solution to meet the requirements is designed. This may be a new design, or an extension of an earlier design. An Implementation and Test phase, when the software is coded, integrated and tested. A Review phase, in which the software is evaluated, the current requirements are reviewed, and changes and additions to requirements proposed. For each cycle of the model, a decision has to be made as to whether the software produced by the cycle will be discarded, or kept as a starting point for the next cycle (sometimes referred to as incremental prototyping). Eventually a point will be reached where the requirements are complete and the software can be delivered, or it becomes impossible to enhance the software as required, and a fresh start has to be made. The iterative lifecycle model can be likened to producing software by successive approximation. Drawing an analogy with mathematical methods that use successive approximation to arrive at a final solution, the benefit of such methods depends on how rapidly they converge on a solution. The key to successful use of an iterative software development lifecycle is rigorous validation of requirements, and verification (including testing) of each version of the software against those requirements within each cycle of the model. The first three phases of the example iterative model is in fact an abbreviated form of a sequential V or waterfall lifecycle model. Each cycle of the model produces software that requires testing at the unit level, for software integration, for system integration and for acceptance. As the software evolves through successive cycles, tests have to be repeated and extended to verify each version of the software. 1.STLC:

Requirement

28

TESTING CONCEPTS

Test plan

Test case design

Test case execution (manual or automation)

Defect tracking

TESTING DOCUMENTS TEST PLAN Test plan is document. It is a set of activities of the application. Test plan based when, who, what and how. Test plan involves following details. Objective Aim of the project. Introduction, overview of the project. Scope 1. Features to be tested 2. Features not to be tested Approach 1. Write a high level scenario. 2. Write a flow graph

29

TESTING CONCEPTS Testing functionality Functionality of the testing, what we done test functionality of the application. Assumption Risk analysis Analyzing risks Backup plan Effort estimation Estimate a effort . ex., Time , cost Roles and responsibilities Roles and responsible of the tester. Entry and exit criteria Entry and exit criteria of the tester. Templates Test automation Using automation tools. Environment Software and hardware requirements. Defect tracking Deliverables Approvals Purpose of test plan : Preparing it helps us to think through the efforts needed to validate the acceptability of a software product. It can and will help people outside the test group to understand the why and how of the product validation. In regulated, environments, we have to have written test plan We want a document that describes the objectives, scope, approach and focus of the software testing effort. It includes test cases, conditions, the test environment, a list of related tasks, pass/fail criteria and risk assessment. One of the outputs for creating a test strategy is an approved and signed off test plan. Test plans should be documented. So that they are repeatable.

30

TESTING CONCEPTS SOFTWARE REQUIREMENT SPECIFICATION (SRS) : A Software Requirements Specification (SRS) is a complete description of the behavior of the system to be developed. It includes a set of use cases that describe all the interactions the users will have with the software. Use cases are also known as functional requirements. In addition to use cases, the SRS also contains nonfunctional (or supplementary) requirements. Non-functional requirements are requirements which impose constraints on the design or implementation 9 TEST CASE Test case is a document. It is a set of the activities of the application. Here, we compare the expected result to actual result. Test case design Test case id Description Procedure Input Expected result Actual result Status. Error programmatically mistake leads to error Bug - Deviation from the expected result Defect - Problem in algorithm leads to failure. Failure - Result of any of the above BUG TRACKING: A bug tracking system is a software application that is designed to help quality assurance and programmers keep track of reported software bugs in their work. It may be regarded as a sort of issue tracking system. Many bug-tracking systems, such as those used by most open source software projects, allow users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other software project management applications Bug reporting:

31

TESTING CONCEPTS Design of the bug reporting, S. no Link Bug id Description Priority Severity Status 1.BUG LIFE CYCLE: In software development process, the bug has a life cycle. The bug should go through the life cycle to be closed. A specific life cycle ensures that the process is standardized. The bug attains different states in the life cycle. The life cycle of the bug can be shown diagrammatically as follows:

The different states of a bug can be summarized as follows: 1.New: bugs posted on first time. 2.Open: After a tester has posted a bug, the lead checks if it is genuine then it is called open. 3.Assign: Once the lead changes the state as OPEN, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to

32

TESTING CONCEPTS ASSIGN. 4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to TEST. It specifies that the bug has been fixed and is released to testing team. 5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software. 6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to REJECTED. 7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to DUPLICATE. 8. Verified: Once the bug is fixed and the status is changed to TEST, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to VERIFIED. 9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to REOPENED. 10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to CLOSED. 10 TEST SUMMARY RESULTS: A management report providing any important information uncovered by the tests accomplished, and including assessments of the quality of the testing effort, the quality of the software system under test, and statistics derived from Incident Reports. The report also records what testing was done and how long it took, in order to improve any future test planning. This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders. Format: Objective

33

TESTING CONCEPTS Scope Testing methodology Expecting date of testing Actual date of testing Release Date Number of days tested Release to Production Roles and Responsibilities Number of Modules Tested Number of Test cases executed Number of Pass cases executed Number of Fail cases executed Pass percentage Fail percentage Bug status New Fixed Reopen Closed Priority bugs Severity bugs Conclusion What is a good test case? Designing good test cases is a complex art. The complexity comes from three Sources: Test cases help us discover information. Different types of tests are more effective for different classes of information. Test cases can be good in a variety of ways. No test cases will be good in all of them. People tend to create test cases according to certain testing styles, such as domain testing or riskSeverity:

34

TESTING CONCEPTS This is assigned by the tester. Severity of a defect is set based on the issue's seriousness. It can be stated as mentioned It contain, Major Medium Minor Priority: This will be set by the team lead or the project lead. Based on the severity and the time constraint that the module has the priority will be set. It contain, High Medium Low

Client server testing: This type of testing usually done for 2 tier applications (usually developed for LAN) Here we will be having front-end and backend. The application launched on front-end will be having forms and reports which will be monitoring and manipulating data Web testing: This is done for 3 tier applications (developed for Internet / intranet / extranet). Here we will be having Browser, web server and DB server. Configuration management: Configuration management (CM) is the detailed recording and updating of information that describes an enterprise's computer systems and networks, including

35

TESTING CONCEPTS all hardware and software components. Such information typically includes the versions and updates that have been applied to installed software packages and the locations and network addresses of hardware devices. Special configuration management software is available. When a system needs a hardware or software upgrade, a computer technician can accesses the configuration management program and database to see what is currently installed. An advantage of a configuration management application is that the entire collection of systems can be reviewed to make sure any changes made to one system do not adversely affect any of the other systems Configuration management is also used in software development, where it is called Unified Configuration Management (UCM). Using UCM, developers can keep track of the source code, documentation, problems, changes requested, and changes made.

Test deliverables: During the testing what are the documents we delivered, in all the documents are test deliverables. Like, Test plan, test case, bug report, RTM, SRS, test summary report. Why do we go for automation testing? In automation testing, user actions are simulated using a testing tool. The actions a manual tester performs are recorded, and then played back to execute the same test case. The benefits of automating software testing are many: Providing more coverage of regression testing. Reducing the elapsed time for testing, getting your product to market faster.

36

TESTING CONCEPTS Improving productivity of human testing. Improving the re-usability of tests. Providing a detailed test log.

37

TESTING CONCEPTS

GUI Testing

What is GUI Testing? GUI is the abbreviation for Graphic User Interface. It is absolutely essential that any application has to be user-friendly. The end user should be comfortable while using all the components on screen and the components should also perform their functionality with utmost clarity. Hence it becomes very essential to test the GUI components of any application. GUI Testing can refer to just ensuring that the look-and-feel of the application is acceptable to the user, or it can refer to testing the functionality of each and every component involved. The following is a set of guidelines to ensure effective GUI Testing and can be used even as a checklist while testing a product / application.

Section 1 - Windows Compliance Testing Application Start Application by Double Clicking on its ICON. The Loading message should show the application name, version number, and a bigger pictorial representation of the icon. No Login is necessary. The main window of the application should have the same caption as the caption of the icon in Program Manager. Closing the application should result in an "Are you Sure" message box Attempt to start application twice. This should not be allowed - you should be returned to main window. Try to start the application twice as it is loading. On each window, if the application is busy, then the hour glass should be displayed. If there is no hour glass, then some enquiry in progress message should be displayed. All screens should have a Help button (i.e.) F1 key should work the same. If Window has a Minimize Button, click it. Window should return to an icon on the bottom of the screen. This icon should correspond to the Original Icon under Program Manager. Double Click the Icon to return the Window to its original size. The window caption for every application should have the name of the application and the window name - especially the error messages. These should be checked for spelling,

38

TESTING CONCEPTS English and clarity, especially on the top of the screen. Check does the title of the window make sense. If the screen has a Control menu, then use all un-grayed options. Check all text on window for Spelling/Tense and Grammar. Use TAB to move focus around the Window. Use SHIFT+TAB to move focus backwards. Tab order should be left to right, and Up to Down within a group box on the screen. All controls should get focus - indicated by dotted box, or cursor. Tabbing to an entry field with text in it should highlight the entire text in the field. The text in the Micro Help line should change - Check for spelling, clarity and non-updateable etc. If a field is disabled (grayed) then it should not get focus. It should not be possible to select them with either the mouse or by using TAB. Try this for every grayed control. Never updateable fields should be displayed with black text on a gray background with a black label. All text should be left justified, followed by a colon tight to it. In a field that may or may not be updateable, the label text and contents changes from black to gray depending on the current status. List boxes are always white background with black text whether they are disabled or not. All others are gray. In general, double-clicking is not essential. In general, everything can be done using both the mouse and the keyboard. All tab buttons should have a distinct letter. Text Boxes Move the Mouse Cursor over all Enterable Text Boxes. Cursor should change from arrow to Insert Bar. If it doesn't then the text in the box should be gray or nonupdateable. Refer to previous page. Enter text into Box Try to overflow the text by typing to many characters - should be stopped Check the field width with capitals W. Enter invalid characters - Letters in amount fields, try strange characters like + , - * etc. in All fields. SHIFT and Arrow should Select Characters. Selection should also be possible with mouse. Double Click should select all text in box. Option (Radio Buttons) Left and Right arrows should move 'ON' Selection. So should Up and Down. Select with mouse by clicking.

39

TESTING CONCEPTS Check Boxes Clicking with the mouse on the box, or on the text should SET/UNSET the box. SPACE should do the same. Command Buttons If Command Button leads to another Screen, and if the user can enter or change details on the other screen then the Text on the button should be followed by three dots. All Buttons except for OK and Cancel should have a letter Access to them. This is indicated by a letter underlined in the button text. Pressing ALT+Letter should activate the button. Make sure there is no duplication. Click each button once with the mouse - This should activate Tab to each button - Press SPACE - This should activate Tab to each button - Press RETURN - This should activate The above are VERY IMPORTANT, and should be done for EVERY command Button. Tab to another type of control (not a command button). One button on the screen should be default (indicated by a thick black border). Pressing Return in ANY no command button control should activate it. If there is a Cancel Button on the screen, then pressing <Esc> should activate it. If pressing the Command button results in uncorrectable data e.g. closing an action step, there should be a message phrased positively with Yes/No answers where Yes results in the completion of the action. Drop Down List Boxes Pressing the Arrow should give list of options. This List may be scrollable. You should not be able to type text in the box. Pressing a letter should bring you to the first item in the list with that start with that letter. Pressing Ctrl - F4 should open/drop down the list box. Spacing should be compatible with the existing windows spacing (word etc.). Items should be in alphabetical order with the exception of blank/none, which is at the top or the bottom of the list box. Drop down with the item selected should be display the list with the selected item on the top. Make sure only one space appears, shouldn't have a blank line at the bottom. Combo Boxes Should allow text to be entered. Clicking Arrow should allow user to choose from list

40

TESTING CONCEPTS List Boxes Should allow a single selection to be chosen, by clicking with the mouse, or using the Up and Down Arrow keys. Pressing a letter should take you to the first item in the list starting with that letter. If there is a 'View' or 'Open' button besides the list box then double clicking on a line in the List Box, should act in the same way as selecting and item in the list box, then clicking the command button. Force the scroll bar to appear, make sure all the data can be seen in the box.

Section 2 - Screen Validation Checklist Aesthetic Conditions: 1. Is the general screen background the correct color? 2. Are the field prompts the correct color? 3. Are the field backgrounds the correct color? 4. In read-only mode, are the field prompts the correct color? 5. In read-only mode, are the field backgrounds the correct color? 6. Are all the screen prompts specified in the correct screen font? 7. Is the text in all fields specified in the correct screen font? 8. Are all the field prompts aligned perfectly on the screen? 9. Are all the field edit boxes aligned perfectly on the screen? 10. Are all group boxes aligned correctly on the screen? 11. Should the screen be resizable? 12. Should the screen be allowed to minimize? 13. Are all the field prompts spelt correctly? 14. Are all character or alphanumeric fields left justified? This is the default unless otherwise specified. 15. Are all numeric fields right justified? This is the default unless otherwise specified. 16. Is all the micro-help text spelt correctly on this screen? 17. Is all the error message text spelt correctly on this screen? 18. Is all user input captured in UPPER case or lowercase consistently? 19. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact.

41

TESTING CONCEPTS 20. Assure that all windows have a consistent look and feel. 21. Assure that all dialog boxes have a consistent look and feel. Validation Conditions: 1. Does a failure of validation on every field cause a sensible user error message? 2. Is the user required to fix entries, which have failed validation tests? 3. Have any fields got multiple validation rules and if so are all rules being applied? 4. If the user enters an invalid value and clicks on the OK button (i.e. does not TAB off the field) is the invalid entry identified and highlighted correctly with an error message? 5. Is validation consistently applied at screen level unless specifically required at field level? 6. For all numeric fields check whether negative numbers can and should be able to be entered. 7. For all numeric fields check the minimum and maximum values and also some mid-range values allowable? 8. For all character/alphanumeric fields check the field to ensure that there is a character limit specified and that this limit is exactly correct for the specified database size? 9. Do all mandatory fields require user input? 10. If any of the database columns don't allow null values then the corresponding screen fields must be mandatory. (If any field, which initially was mandatory, has become optional then check whether null values are allowed in this field.)

Navigation Conditions: 1. Can the screen be accessed correctly from the menu? 2. Can the screen be accessed correctly from the toolbar? 3. Can the screen be accessed correctly by double clicking on a list control on the previous screen? 4. Can all screens accessible via buttons on this screen be accessed correctly?

42

TESTING CONCEPTS 5. Can all screens accessible by double clicking on a list control be accessed correctly? 6. Is the screen modal? (i.e.) Is the user prevented from accessing other functions when this screen is active and is this correct? 7. Can a number of instances of this screen be opened at the same time and is this correct? Usability Conditions: 1. Are all the dropdowns on this screen sorted correctly? Alphabetic sorting is the default unless otherwise specified. 2. Is all date entry required in the correct format? 3. Have all pushbuttons on the screen been given appropriate Shortcut keys? 4. Do the Shortcut keys work correctly? 5. Have the menu options that apply to your screen got fast keys associated and should they have? 6. Does the Tab Order specified on the screen go in sequence from Top Left to bottom right? This is the default unless otherwise specified. 7. Are all read-only fields avoided in the TAB sequence? 8. Are all disabled fields avoided in the TAB sequence?

9. Can the cursor be placed in the micro help text box by clicking on the text box
with the mouse? 10. Can the cursor be placed in read-only fields by clicking in the field with the mouse? 11. Is the cursor positioned in the first input field or control when the screen is opened? 12. Is there a default button specified on the screen? 13. Does the default button work correctly? 14. When an error message occurs does the focus return to the field in error when the user cancels it? 15. When the user Alt+Tab's to another application does this have any impact on the screen upon return to the application? 16. Do all the fields edit boxes indicate the number of characters they will hold by there length? e.g. a 30 character field should be a lot longer

43

TESTING CONCEPTS Data Integrity Conditions: 1. Is the data saved when the window is closed by double clicking on the close box? 2. Check the maximum field lengths to ensure that there are no truncated characters? 3. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact. 4. Check maximum and minimum field values for numeric fields? 5. If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers? 6. If a set of radio buttons represents a fixed set of values such as A, B and C then what happens if a blank value is retrieved from the database? (In some situations rows can be created on the database by other functions, which are not screen based, and thus the required initial values can be incorrect.) 7. If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values. Modes (Editable Read-only) Conditions: 1. Are the screen and field colors adjusted correctly for read-only mode? 2. Should a read-only mode be provided for this screen? 3. Are all fields and controls disabled in read-only mode? 4. Can the screen be accessed from the previous screen/menu/toolbar in readonly mode? 5. Can all screens available from this screen be accessed in read-only mode? 6. Check that no validation is performed in read-only mode. General Conditions: 1. Assure the existence of the "Help" menu. 2. Assure that the proper commands and options are in each menu. 3. Assure that all buttons on all tool bars have a corresponding key commands. 4. Assure that each menu command has an alternative (hot-key) key sequence, which will invoke it where appropriate.

44

TESTING CONCEPTS 5. In drop down list boxes, ensure that the names are not abbreviations / cut short 6. In drop down list boxes, assure that the list and each entry in the list can be accessed via appropriate key / hot key combinations. 7. Ensure that duplicate hot keys do not exist on each screen 8. Ensure the proper usage of the escape key (which is to undo any changes that have been made) and generates a caution message "Changes will be lost - Continue yes/no" 9. Assure that the cancel button functions the same as the escape key. 10. Assure that the Cancel button operates, as a Close button when changes have been made that cannot be undone. 11. Assure that only command buttons, which are used by a particular window, or in a particular dialog box, are present. (i.e) make sure they don't work on the screen behind the current screen. 12. When a command button is used sometimes and not at other times, assures that it is grayed out when it should not be used. 13. Assure that OK and Cancel buttons are grouped separately from other command buttons. 14. Assure that command button names are not abbreviations. 15. Assure that all field labels/names are not technical labels, but rather are names meaningful to system users. 16. Assure that command buttons are all of similar size and shape, and same font & font size. 17. Assure that each command button can be accessed via a hot key combination. 18. Assure that command buttons in the same window/dialog box do not have duplicate hot keys. 19. Assure that each window/dialog box has a clearly marked default value (command button, or other object) which is invoked when the Enter key is pressed - and NOT the Cancel or Close button 20. Assure that focus is set to an object/button, which makes sense according to the function of the window/dialog box. 21. Assure that all option buttons (and radio buttons) names are not abbreviations.

45

TESTING CONCEPTS 22. Assure that option button names are not technical labels, but rather are names meaningful to system users. 23. If hot keys are used to access option buttons, assure that duplicate hot keys do not exist in the same window/dialog box. 24. Assure that option box names are not abbreviations. 25. Assure that option boxes, option buttons, and command buttons are logically grouped together in clearly demarcated areas "Group Box" 26. Assure that the Tab key sequence, which traverses the screens, does so in a logical way. 27. Assure consistency of mouse actions across windows. 28. Assure that the color red is not used to highlight active objects (many individuals are red-green color blind). 29. Assure that the user will have control of the desktop with respect to general color and highlighting (the application should not dictate the desktop background characteristics). 30. Assure that the screen/window does not have a cluttered appearance 31. Ctrl + F6 opens next tab within tabbed window 32. Shift + Ctrl + F6 opens previous tab within tabbed window 33. Tabbing will open next tab within tabbed window if on last field of current tab 34. Tabbing will go onto the 'Continue' button if on last field of last tab within tabbed window 35. Tabbing will go onto the next editable field in the window 36. Banner style & size & display exact same as existing windows 37. If 8 or less options in a list box, display all options on open of list box - should be no need to scroll 38. Errors on continue will cause user to be returned to the tab and the focus should be on the field causing the error. (i.e the tab is opened, highlighting the field with the error on it) 39. Pressing continue while on the first tab of a tabbed window (assuming all fields filled correctly) will not open all the tabs. 40. On open of tab focus will be on first editable field 41. All fonts to be the same 42. Alt+F4 will close the tabbed window and return you to main screen or previous screen (as appropriate), generating "changes will be lost" message if necessary.

46

TESTING CONCEPTS

43. Micro help text for every enabled field & button
44. Ensure all fields are disabled in read-only mode 45. Progress messages on load of tabbed screens 46. Return operates continue 47. If retrieve on load of tabbed window fails window should not open Specific Field Tests Date Field Checks 1. Assure that leap years are validated correctly & do not cause errors/miscalculations. 2. Assure that month code 00 and 13 are validated correctly & do not cause errors/miscalculations. 3. Assure that 00 and 13 are reported as errors. 4. Assure that day values 00 and 32 are validated correctly & do not cause errors/miscalculations. 5. Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations. 6. Assure that Feb. 30 is reported as an error. 7. Assure that century change is validated correctly & does not cause errors/ miscalculations. 8. Assure that out of cycle dates are validated correctly & do not cause errors/miscalculations. Numeric Fields 1. Assure that lowest and highest values are handled correctly. 2. Assure that invalid values are logged and reported. 3. Assure that valid values are handles by the correct procedure. 4. Assure that numeric fields with a blank in position 1 are processed or reported as an error. 5. Assure that fields with a blank in the last position are processed or reported as an error an error. 6. Assure that both + and - values are correctly processed. 7. Assure that division by zero does not occur. 8. Include value zero in all calculations.

47

TESTING CONCEPTS 9. Include at least one in-range value. 10. Include maximum and minimum range values. 11. Include out of range values above the maximum and below the minimum. 12. Assure that upper and lower values in ranges are handled correctly. Alpha Field Checks 1. Use blank and non-blank data. 2. Include lowest and highest values. 3. Include invalid characters & symbols. 4. Include valid characters. 5. Include data items with first position blank. 6. Include data items with last position blank.

48

TESTING CONCEPTS

Interview Questions What's Ad Hoc Testing? A testing where the tester tries to break the software by randomly trying functionality of software. What's the Accessibility Testing? Testing that determines if software will be usable by people with disabilities. What's the Alpha Testing? The Alpha Testing is conducted at the developer sites and in a controlled environment by the end user of the software What's the Beta Testing? Testing the application after the installation at the client place. What is Component Testing? Testing of individual software components (Unit Testing). What's Compatibility Testing? In Compatibility testing we can test that software is compatible with other elements of system. What is Concurrency Testing? Multi-user testing geared towards determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores.

49

TESTING CONCEPTS

What is Conformance Testing? The process of testing that an implementation conforms to the specification on which it is based. Usually applied to testing conformance to a formal standard. What is Context Driven Testing? The context-driven school of software testing is flavor of Agile Testing that advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now. What is Data Driven Testing? Testing in which the action of a test case is parameterized by externally defined data values, maintained as a file or spreadsheet. A common technique in Automated Testing. What is Conversion Testing? Testing of programs or procedures used to convert data from existing systems for use in replacement systems. What is Dependency Testing? Examines an application's requirements for pre-existing software, initial states and configuration in order to maintain proper functionality. What is Depth Testing? A test that exercises a feature of a product in full detail. What is Dynamic Testing? Testing software through executing it. See also Static Testing.

50

TESTING CONCEPTS

What is Endurance Testing? Checks for memory leaks or other problems that may occur with prolonged execution. What is End-to-End testing? Testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate. What is Exhaustive Testing? Testing which covers all combinations of input values and preconditions for an element of the software under test. What is Gorilla Testing? Testing one particular module, functionality heavily. What is Installation Testing? Confirms that the application under test recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions. What is Localization Testing? This term refers to making software specifically designed for a specific locality. What is Loop Testing? A white box testing technique that exercises program loops.

51

TESTING CONCEPTS What is Mutation Testing? Mutation testing is a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources What is Monkey Testing? Testing a system or an Application on the fly, i.e. just few tests here and there to ensure the system or an application does not crash out. What is Positive Testing? Testing aimed at showing software works. Also known as "test to pass". See also Negative Testing. What is Negative Testing? Testing aimed at showing software does not work. Also known as "test to fail". See also Positive Testing. What is Path Testing? Testing in which all paths in the program source code are tested at least once. What is Performance Testing? Testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users. Also know as "Load Testing". What is Ramp Testing? Continuously raising an input signal until the system breaks down.

52

TESTING CONCEPTS

What is Recovery Testing? Confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions. What is the Re-testing testing? Retesting- Again testing the functionality of the application. What is the Regression testing? Regression- Check that change in code have not effected the working functionality What is Sanity Testing? Brief test of major functional elements of a piece of software to determine if its basically operational. What is Scalability Testing? Performance testing focused on ensuring the application under test gracefully handles increases in work load. What is Security Testing? Testing which confirms that the program can restrict access to authorized personnel and that the authorized personnel can access the functions available to their security level. What is Stress Testing? Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a

53

TESTING CONCEPTS breaking point, in order to observe the results. What is Smoke Testing? A quick-and-dirty test that the major functions of a piece of software work. Originated in the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch on fire. What is Soak Testing? Running a system at high load for a prolonged period of time. For example, running several times more transactions in an entire day (or night) than would be expected in a busy day, to identify and performance problems that appear after a large number of transactions have been executed. What's the Usability testing? Usability testing is for user friendliness. What's the User acceptance testing? User acceptance testing is determining if software is satisfactory to an end-user or customer. What's the Volume Testing? We can perform the Volume testing, where the system is subjected to large volume of data.

Difference b/w high severity and low severity? When the application has critical problem and it has to be solved after a month then we can say it as high severity and low priority.

54

TESTING CONCEPTS When the application has trivial problem (less affected) and it has to be solved with in a day then we can say it as low severity with high priority. When the testing should be ended? Testing is a never ending process. Because of some factors testing May terminates. The factors may be most of the tests are executed, project deadline, test budget depletion, bug rate falls down below the criteria. Is regression testing performed manually? It depends on the initial testing approach. If the initial testing approach was manual testing, then the regression testing is normally performed manually. Conversely, if the initial testing approach was automated testing, then the regression testing is normally performed by automated testing. Give me others' FAQs on testing. Here is what you can do. You can visit my web site, and on http://robdavispe.com/free http://robdavispe.com/free2, you can findanswers to the vast majority of others' questions on testing, from a tester's point of view. As to questions and answers that are not on my web site now, please be patient, as I am going to add more FAQs and answers, as soon as time permits. Can you share with me your knowledge of software testing? Surely I can. Visit my web site, http://robdavispe.com/free and http://robdavispe.com/free2, and you will find my knowledge on software testing, from a tester's point of view. As to knowledge that is not on my web site now, please be patient, as I am going to add more answers, as soon as time permits.

55

TESTING CONCEPTS

How can I learn software testing? I suggest you visit my web site, especially http://robdavispe.com/free and http://robdavispe.com/free2, and you will find answers to most questions on software testing. As to questions and answers that are not on my web site at the moment, please be patient. I will add more questions and answers, as soon as time permits. I also suggest you get a job in software testing. Why? Because you can get additional, free education, on the job, by an employer, while you are being paid to do software testing. On the job, you will be able to use some of the more popular software tools, including Win Runner, Load Runner, Lab View, and the Rational Toolset. The tools you use will depend on the end client, their needs and preferences. I also suggest you sign up for courses at nearby educational institutions. Classroom education, especially non degree courses in local community colleges, tends to be inexpensive. What is your view of software QA/testing? Software QA/testing is easy, if requirements are solid, clear, complete, detailed, cohesive, attainable and testable, and if schedules are realistic, and if there is good communication. Software QA/testing is a piece of cake, if project schedules are realistic, if adequate time is allowed for planning, design, testing, bug fixing, retesting, changes, and documentation. Software QA/testing is relatively easy, if testing is started early on, and if fixes or changes are re-tested, and if sufficient time is planned for both testing and bug fixing. Software QA/testing is easy, if new features are avoided, and if one sticks to initial requirements as much as possible. How can I be a good tester? We, good testers, take the customers' point of view. We are tactful and diplomatic. We have a "test to break" attitude, a strong desire for quality, an attention to detail, and good communication skills, both oral and written. Previous software development experience is also helpful as it provides a deeper understanding of the software development process. What is the difference between software bug and software defect? A 'software bug' is a nonspecific term that means an inexplicable defect, error, flaw, mistake, failure, fault, or unwanted behavior of a computer program. Other

56

TESTING CONCEPTS terms, e.g. software defect and software failure, are more specific. While there are many who believe the term 'bug' is a reference to insects that caused malfunctions in early electromechanical computers (1950-1970), the term 'bug' had been a part of engineering jargon for many decades before the 1950s; even the great inventor, Thomas Edison (1847-1931), wrote about a 'bug' in one of his letters. How can I improve my career in software QA/testing? Invest in your skills! Learn all you can! Visit my web site, and on http://robdavispe.com/free and http://robdavispe.com/free2, you will find answers to the vast majority of questions on testing, from software QA/testers' point of view. Get additional education, on the job. Free education is often provided by employers, while you are paid to do the job of a tester. On the job, often you can use many software tools, including WinRunner, LoadRunner, LabView, and Rational Toolset. Find an employer whose needs and preferences are similar to yours. Get an education! Sign up for courses at nearby educational institutes. Take classes! Classroom education, especially non-degree courses in local community colleges, tends to be inexpensive. Improve your attitude! Become the best software QA/tester! Always strive to exceed the expectations of your customers!

How do you compare two files? Use PVCS, SCCS, or "diff". PVCS is a document version control tool, a competitor of SCCS. SCCS is an original UNIX program, based on "diff". Diff is a UNIX utility that compares the difference between two text files. What do we use for comparison? Generally speaking, when we write a software program to compare files, we compare two files, bit by bit. For example, when we use "diff", a UNIX utility, we compare two text files.

What is the reason we compare files? We compare files because of configuration management, revision control, requirement version control, or document version control. Examples are Rational ClearCase, DOORS, PVCS, and CVS. CVS, for example, enables several, often distant, developers to work together on the same source code.

57

TESTING CONCEPTS When is a process repeatable? If we use detailed and well-written processes and procedures, we ensure the correct steps are being executed. This facilitates a successful completion of a task. This is a way we also ensure a process is repeatable. What is test methodology? One test methodology is a three-step process. Creating a test strategy, creating a test plan/design, and executing tests. This methodology can be used and molded to your organization's needs. Rob Davis believes that using this methodology is important in the development and ongoing maintenance of his customers' applications. What does a Test Strategy Document contain? The test strategy document is a formal description of how a software product will be tested. A test strategy is developed for all levels of testing, as required. The test team analyzes the requirements, writes the test strategy and reviews the plan with the project team. The test plan may include test cases, conditions, the test environment, and a list of related tasks, pass/fail criteria and risk assessment. Additional sections in the test strategy document include: A description of the required hardware and software components, including test tools. This information comes from the test environment, including test tool data. A description of roles and responsibilities of the resources required for the test and schedule constraints. This information comes from man-hours and schedules. Testing methodology. This is based on known standards. Functional and technical requirements of the application. This information comes from requirements, change request, technical, and functional design documents. Requirements that the system cannot provide, e.g. system limitations. What is monkey testing? "Monkey testing" is random testing performed by automated testing tools. These automated testing tools are considered "monkeys", if they work at random. We call them "monkeys" because it is widely believed, if we allow six monkeys to pound on six typewriters at random, for a million years, they will recreate all the works of Isaac Asimov. There are "smart monkeys" and "dumb monkeys". "Smart

58

TESTING CONCEPTS monkeys" are valuable for load and stress testing, and will find a significant number of bugs, but they're also very expensive to develop. "Dumb monkeys", on the other hand, are inexpensive to develop, are able to do some basic testing, but they will find few bugs. However, the bugs "dumb monkeys" do find will be hangs and crashes, i.e. the bugs you least want to have in your software product. "Monkey testing" can be valuable, but they should not be your only testing. What is stochastic testing? Stochastic testing is the same as "monkey testing", but stochastic testing is a more technical sounding name for the same testing process. Stochastic testing is black box testing, random testing, performed by automated testing tools. Stochastic testing is a series of random tests over time. The software under test typically passes the individual tests, but our goal is to see if it can pass a large series of the individual tests. What is mutation testing? In mutation testing, we create mutant software, we make mutant software to fail, and thus demonstrate the adequacy of our test case. When we create a set of mutant software, each mutant software differs from the original software by one mutation, i.e. one single syntax change made to one of its program statements, i.e. each mutant software contains only one single fault. When we apply test cases to the original software and to the mutant software, we evaluate if our test case is adequate. Our test case is inadequate, if both the original software and all mutant software generate the same output. Our test case is adequate, if our test case detects faults, or, if, at least one mutant software generates a different output than does the original software for our test case. What is PDR? PDR is an acronym. In the world of software QA/testing, it stands for "peer design review", or "peer review". What is is good about PDRs? PDRs are informal meetings, and I do like all informal meetings. PDRs make perfect sense, because they're for the mutual benefit of you and your end client.

59

TESTING CONCEPTS Your end client requires a PDR, because they work on a product, and want to come up with the very best possible design and documentation. Your end client requires you to have a PDR, because when you organize a PDR, you invite and assemble the end client's best experts and encourage them to voice their concerns as to what should or should not go into the design and documentation, and why. When you're a developer, designer, author, or writer, it's also to your advantage to come up with the best possible design and documentation. Therefore you want to embrace the idea of the PDR, because holding a PDR gives you a significant opportunity to invite and assemble the end client's best experts and make them work for you for one hour, for your own benefit. To come up with the best possible design and documentation, you want to encourage your end client's experts to speak up and voice their concerns as to what should or should not go into your design and documentation, and why. Why is that my company requires a PDR? Your company requires a PDR, because your company wants to be the owner of the very best possible design and documentation. Your company requires a PDR, because when you organize a PDR, you invite, assemble and encourage the company's best experts to voice their concerns as to what should or should not go into your design and documentation, and why. Remember, PDRs are not about you, but about design and documentation. Please don't be negative; please do not assume your company is finding fault with your work, or distrusting you in any way. There is a 90+ per cent probability your company wants you, likes you and trust you, because you're a specialist, and because your company hired you after a long and careful selection process. Your company requires a PDR, because PDRs are useful and constructive. Just about everyone - even corporate chief executive officers (CEOs) - attend PDRsfrom time to time. When a corporate CEO attends a PDR, he has to listen for "feedback" from shareholders. When a CEO attends a PDR, the meeting is called the "annual shareholders' meeting".

60

TESTING CONCEPTS

Give me a list of ten good things about PDRs! 1. PDRs are easy, because all your meeting attendees are your coworkers and friends. 2. PDRs do produce results. With the help of your meeting attendees, PDRs help you produce better designs and better documents than the ones you could come up with, without the help of your meeting attendees. 3. Preparation for PDRs helps a lot, but, in the worst case, if you had no time to read every page of every document, it's still OK for you to show up at the PDR. 4. It's technical expertise that counts the most, but many times you can influence your group just as much, or even more so, if you're dominant or have good acting skills. 5. PDRs are easy, because, even at the best and biggest companies, you can dominate the meeting by being either very negative, or very bright and wise. 6. It is easy to deliver gentle suggestions and constructive criticism. The brightest and wisest meeting attendees are usually gentle on you; they deliver gentle suggestions that are constructive, not destructive. 7. You get many-many chances to express your ideas, every time a meeting attendee asks you to justify why you wrote what you wrote. 8. PDRs are effective, because there is no need to wait for anything or anyone; because the attendees make decisions quickly (as to what errors are in your document). There is no confusion either, because all the group's recommendations are clearly written down for you by the PDR's facilitator. 9. Your work goes faster, because the group itself is an independent decision making authority. Your work gets done faster, because the group's decisions are subject to neither oversight nor supervision. 10. At PDRs, your meeting attendees are the very best experts anyone can find, and they work for you, for FREE! What is the Exit criteria? "Exit criteria" is a checklist, sometimes known as the "PDR sign-off sheet", i.e. a list of peer design review related tasks that have to be done by the facilitator or other attendees of the PDR, during or near the conclusion of the PDR. By having a checklist, and by going through a checklist, the facilitator can...

61

TESTING CONCEPTS 1. Verify that the attendees have inspected all the relevant documents and reports, and 2. Verify that all suggestions and recommendations for each issue have been recorded, and 3. Verify that all relevant facts of the meeting have been recorded. The facilitator's checklist includes the following questions: 1. "Have we inspected all the relevant documents, code blocks, or products?" 2. "Have we completed all the required checklists?" 3. "Have I recorded all the facts relevant to this peer review?" 4. "Does anyone have any additional suggestions, recommendations, or comments?" 5. "What is the outcome of this peer review?" At the end of the peer review, the facilitator asks the attendees of the peer review to make a decision as to the outcome of the peer review. I.e., "What is our consensus?" "Are we accepting the design (or document or code)?" Or, "Are we accepting it with minor modifications? "Or, "Are we accepting it, after it is modified, and approved through e-mails to the participants?" Or, "Do we want another peer review?" This is a phase, during which the attendees of the PDR work as a committee, and the committee's decision is final. What is the Entry criteria? The entry criteria is a checklist, or a combination of checklists that includes the "developer's checklist", "testing checklist", and the "PDR checklist". Checklists are list of tasks that have to be done by developers, testers, or the facilitator, at or before the start of the peer review. Using these checklists, before the start of the peer review, the developer, tester and facilitator can determine if all the documents, reports, code blocks or software products are ready to be reviewed, and if the peer review's attendees are prepared to inspect them. The facilitator can ask the peer review's attendees if they have been able to prepare for the peer review, and if they're not well prepared, the facilitator can send them back to their desks, and even ask the task lead to reschedule the peer review. The facilitator's script for the entry criteria includes the following questions: 1. Are all the required attendees present at the peer review? 2. Have all the attendees received all the relevant documents and reports? 3. Are all the attendees well prepared for this peer review?

62

TESTING CONCEPTS 4. Have all the preceding life cycle activities been concluded? 5. Are there any changes to the baseline? What are the parameters of peer reviews? By definition, parameters are values on which something else depends. Peer reviews depend on the attendance and active participation of several key people; usually the facilitator, task lead, test lead, and at least one additional reviewer. The attendance of these four people are usually required for the approval of the PDR. According to company policy, depending on your company, other participants are often invited, but generally not required for approval. Peer reviews depend on the facilitator, sometimes known as the moderator, who controls the meeting, keeps the meeting on schedule, and records all suggestions from all attendees. Peer reviews greatly depend on the developer, also known as the designer, author, or task lead -- usually a software engineer -- who is most familiar with the project, and most likely able to answer any questions or address any concerns that may come up during the peer review. Peer reviews greatly depend on the tester, also known as test lead, or bench test person -- usually another software engineer -- who is also familiar with the project, and most likely able to answer any questions or address any concerns that may come up during the peer review. Peer reviews greatly depend on the participation of additional reviewers and additional attendees who often make specific suggestions and recommendations, and ask the largest number of questions. Have you attended any review meetings? Yes, in the last 10+ years I have attended many review meetings; mostly peer reviews. In today's corporate world, the vast majority of review meetings are peer review meetings. In my experience, the most useful peer reviews are the ones where you're the author of something. Why? Because when you're the author, then it's you who decides what to do and how, and it's you who receives all the free help. In my experience, on the long run, the inputs of your additional reviewers and additional attendees can be the most valuable to you and your company. But, in your own best interest, in order to expedite things, before every peer review it is a good idea to get

63

TESTING CONCEPTS together with the additional reviewer and additional attendee, and talk with them about issues, because if you don't, they will be the ones with the largest number of questions and usually negative feedback. When a PDR is done right, it is useful, beneficial, pleasant, and friendly. Generally speaking, the fewer people show up at the PDR, the easier it tends to be, and the earlier it can be adjourned. When you're an author, developer, or task lead, many times you can relax, because during your peer review your facilitator and test lead are unlikely to ask any tough questions from you. Why? Because, the facilitator is too busy taking notes, and the test lead is kind of bored (because he had already asked his toughest questions before the PDR). When you're a facilitator, every PDR tends to be a pleasant experience. In my experience, one of the easiest review meetings are PDRs where you're the facilitator (whose only job is to call the shots and make notes). What types of review meetings can you tell me about? Of review meetings, peer design reviews are the most common. Peer design reviews are so common that they tend to replace both inspections and walk-troughs. Peer design reviews can be classified according to the 'subject' of the review. I.e., "Is this a document review, design review, or code review?" Peer design reviews can be classified according to the 'role' you play at the meeting. I.e., "Are you the task lead, test lead, facilitator, moderator, or additional reviewer?" Peer design reviews can be classified according to the 'job title of attendees. I.e., "Is this a meeting of peers, managers, systems engineers, or system integration testers?" Peer design reviews can be classified according to what is being reviewed at the meeting. I.e., "Are we reviewing the work of a developer, tester, engineer, or technical document writer?" Peer design reviews can be classified according to the 'objective' of the review. I.e., "Is this document for the file cabinets of our company, or that of the government (e.g. the FAA or FDA)?" PDRs of government documents tend to attract the attention of managers, and the meeting quickly becomes a meeting of managers. How can I shift my focus and area of work from QC to QA? Number one, focus on your strengths, skills, and abilities! Realize that there are MANY similarities between Quality Control and Quality Assurance! Realize that you have MANY transferable skills! Number two, make a plan! Develop a belief that getting a job in QA is easy!

64

TESTING CONCEPTS HR professionals cannot tell the difference between quality control and quality assurance! HR professionals tend to respond to keywords (i.e. QC and QA), without knowing the exact meaning of those keywords! Number three, make it a reality! Invest your time! Get some hands-on experience! Do some QA work! Do any QA work, even if, for a few months, you get paid a little less than usual! Your goals, beliefs, enthusiasm, and action will make a huge difference in your life! Number four, I suggest you read all you can, and that includes reading product pamphlets, manuals, books, information on the Internet, and whatever information you can lay your hands on! If there is a will, there is a way! You CAN do it, if you put your mind to it! You CAN learn to do QA work, with little or no outside help! Click on a link! What techniques and tools can enable me to migrate from QC to QA?

Technique number one is mental preparation. Understand and believe what you want is not unusual at all! Develop a belief in yourself! Start believing what you want is attainable! You can change your career! Every year, millions of men and women change their careers successfully!

Number two, make a plan! Develop a belief that getting a job in QA is easy! HR professionals cannot tell the difference between quality control and quality assurance! HR professionals tend to respond to keywords (i.e. QC and QA), without knowing the exact meaning of those keywords!

Number three, make it a reality! Invest your time! Get some hands-on experience! Do some QA work! Do any QA work, even if, for a few months, you get paid a little less than usual! Your goals, beliefs, enthusiasm, and action will make a huge difference in your life!

Number four, I suggest you read all you can, and that includes reading product pamphlets, manuals, books, information on the Internet, and whatever information you can lay your hands on! If there is a will, there is a way! You CAN do it, if you put your mind to it! You CAN learn to do QA work, with little or no outside help! Click on a link!

65

TESTING CONCEPTS

What is the difference between build and release? A: Builds and releases are similar, because both builds and releases are end products of software development processes. Builds and releases are similar, because both builds and releases help developers and QA teams to deliver reliable software. Build means a version of a software, typically one that is still in testing. Usually a version number is given to a released product, but, sometimes, a build number is used instead. Difference number one: Builds refer to software that is still in testing, release refers to software that is usually no longer in testing. Difference number two: Builds occur more frequently; releases occur less frequently. Difference number three: Versions are based on builds, and not vice versa. Builds, or usually a series of builds, are generated first, as often as one build per every morning, depending on the company, and then every release is based on a build, or several builds, i.e. the accumulated code of several builds. What is CMM? A: CMM is an acronym that stands for Capability Maturity Model. The idea of CMM is, as to future efforts in developing and testing software, concepts and experiences do not always point us in the right direction, therefore we should develop processes, and then refine those processes. There are five CMM levels, of which Level 5 is the highest... CMM Level 1 is called "Initial". CMM Level 2 is called "Repeatable". CMM Level 3 is called "Defined". CMM Level 4 is called "Managed". CMM Level 5 is called "Optimized". There are not many Level 5 companies; most hardly need to be. Within the United States, fewer than 8% of software companies are rated CMM Level 4, or higher. The U.S. government requires that all companies with federal government contracts to maintain a minimum of a CMM Level 3 assessment. CMM assessments take two weeks. They're conducted by a nine-member team led by a SEI-certified lead assessor.

66

TESTING CONCEPTS

What are CMM levels and their definitions? There are five CMM levels of which level 5 is the highest.

CMM level 1 is called "initial". The software process is at CMM level 1, if it is an ad hoc process. At CMM level 1, few processes are defined, and success, in general, depends on individual effort and heroism.

CMM level 2 is called "repeatable". The software process is at CMM level 2, if the subject company has some basic project management processes, in order to track cost, schedule, and functionality. Software processes are at CMM level 2, if necessary processes are in place, in order to repeat earlier successes on projects with similar applications. Software processes are at CMM level 2, if there are requirements management, project planning, project tracking, subcontract management, QA, and configuration management.

CMM level 3 is called "defined". The software process is at CMM level 3, if the software process is documented, standardized, and integrated into a standard software process for the subject company. The software process is at CMM level 3, if all projects use approved, tailored versions of the company's standard software process for developing and maintaining software. Software processes are at CMM level 3, if there are process definition, training programs, process focus, integrated software management, software product engineering, inter group coordination, and peer reviews.

CMM level 4 is called "managed". The software process is at CMM level 4, if the subject company collects detailed data on the software process and product quality, and if both the software process and the software products are quantitatively understood and controlled. Software processes are at CMM level 4, if there are software quality management (SQM) and quantitative process management.

CMM level 5 is called "optimized". The software process is at CMM level 5, if there is continuous process improvement, if there is quantitative feedback from the process and from piloting innovative ideas and technologies. Software processes are at CMM level 5, if there are process change management, and defect prevention technology change management.

What is the difference between bug and defect in software testing?

67

TESTING CONCEPTS In software testing, the difference between bug and defect is small, and depends on your company. For some companies, bug and defect are synonymous, while others believe bug is a subset of defect. Generally speaking, we, software test engineers, discover BOTH bugs and defects, before bugs and defects damage the reputation of our company. We, QA engineers, use the software much like real users would, to find BOTH bugs and defects, to find ways to replicate BOTH bugs and defects, to submit bug reports to the developers, and to provide feedback to the developers, i.e. tell them if they've achieved the desired level of quality. Therefore, we, software A engineers, do not differentiate between bugs and defects. In our bug reports, we include BOTH bugs and defects, and any differences between them are minor. Difference number one: In bug reports, the defects are usually easier to describe. Difference number two: In bug reports, it is usually easier to write the descriptions on how to replicate the defects. Defects tend to require brief explanations only. What is grey box testing? Grey box testing is a software testing technique that uses a combination of black box testing and white box testing. Gray box testing is not black box testing, because the tester does know some of the internal workings of the software under test. In grey box testing, the tester applies a limited number of test cases to the internal workings of the software under test. In the remaining part of the grey box testing, one takes a black box approach in applying inputs to the software under test and observing the outputs. Gray box testing is a powerful idea. The concept is simple; if one knows something about how the product works on the inside, one can test it better, even from the outside. Grey box testing is not to be confused with white box testing; i.e. a testing approach that attempts to cover the internals of the product in detail. Grey box testing is a test strategy based partly on internals. The testing approach is known as gray box testing, when one does have some knowledge, but not the full knowledge of the internals of the product one is testing. In gray box testing, just as in black box testing, you test from the outside of a product, just as you do with black box, but you make better-informed testing choices because you're better informed; because you know how the underlying software components operate and interact.

68

TESTING CONCEPTS

What is the difference between version and release? Both version and release indicate a particular point in the software development life cycle, or in the lifecycle of a document. The two terms, version and release, are similar (i.e. mean pretty much the same thing), but there are minor differences between them. Version means a VARIATION of an earlier, or original, type; for example, "I've downloaded the latest version of the software from the Internet. The latest version number is 3.3." Release, on the other hand, is the ACT OR INSTANCE of issuing something for publication, use, or distribution. Release is something thus released. For example, "A new release of a software program." What is data integrity? Data integrity is one of the six fundamental components of information security. Data integrity is the completeness, soundness, and wholeness of the data that also complies with the intention of the creators of the data. In databases, important data including customer information, order database, and pricing tables -- may be stored. In databases, data integrity is achieved by preventing accidental, or deliberate, or unauthorized insertion, or modification, or destruction of data. How do you test data integrity? Data integrity testing should verify the completeness, soundness, and wholeness of the stored data. Testing should be performed on a regular basis, because important data can and will change over time. Data integrity tests include the followings: 1. Verify that you can create, modify, and delete any data in tables. 2. Verify that sets of radio buttons represent fixed sets of values. 3. Verify that a blank value can be retrieved from the database. 4. Verify that, when a particular set of data is saved to the database, each value gets saved fully, and the truncation of strings and rounding of numeric values do not occur. 5. Verify that the default values are saved in the database, if the user input is not specified.

69

TESTING CONCEPTS 6. Verify compatibility with old data, old hardware, versions of operating systems, and interfaces with other software. What is data validity? Data validity is the correctness and reasonableness of data. Reasonableness of data means, for example, account numbers falling within a range, numeric data being all digits, dates having a valid month, day and year, spelling of proper names. Data validity errors are probably the most common, and the most difficult to detect, data-related errors. What causes data validity errors? Data validity errors are usually caused by incorrect data entries, when a large volume of data is entered in a short period of time. For example, 12/25/2005 is entered as 13/25/2005 by mistake. This date is therefore invalid. How can you reduce data validity errors? Use simple field validation rules. Technique 1: If the date field in a database uses the MM/DD/YYYY format, then use a program with the following two data validation rules: "MM should not exceed 12, and DD should not exceed 31". Technique 2: If the original figures do not seem to match the ones in the database, then use a program to validate data fields. Compare the sum of the numbers in the database data field to the original sum of numbers from the source. If there is a difference between the figures, it is an indication of an error in at least one data element. What is the difference between data validity and data integrity?

Difference number one: Data validity is about the correctness and reasonableness of data, while data integrity is about the completeness, soundness, and wholeness of the data that also complies with the intention of the creators of the data.

Difference number two: Data validity errors are more common, while data integrity errors are less common. Difference number three: Errors in data validity are caused by HUMANS -- usually data entry personnel who enter, for example, 13/25/2005, by mistake, while errors in data integrity are caused by BUGS in computer programs that, for example, cause the overwriting of some of the data in the database, when one attempts to retrieve a blank value from the database.

What is Test Director? Test Director, also known as Mercury Test Director, is a software tool made for software QA professionals. Mercury Test Director, as the name implies, is the product

70

TESTING CONCEPTS of Mercury Interactive Corporation, located at 379 North Whisman Road, Mountain View, California 94043 USA. Mercury's products include the Mercury TestDirector, Mercury QuickTest Professional, Mercury WinRunner, and Mercury Business Process Testing. 139. Tell me about 'TestDirector'. Made by Mercury Interactive, 'Test Director' is a single browser-based application that streamlines the software QA process. It is a software tool that helps software QA professionals to gather requirements, to plan, schedule and run tests, and to manage and track defects/issues/bugs. Test Director's Requirements Manager links test cases to requirements, ensures traceability, and calculates what percentage of the requirements are covered by tests, how many of these tests have been run, and how many have passed or failed. As to planning, test plans can be created, or imported, for both manual and automated tests. The test plans can then be reused, shared, and preserved. As to running tests, the Test Directors Test Lab Manager allows you to schedule tests to run unattended, or run even overnight. The Test Director's Defect Manager supports the entire bug life cycle, from initial problem detection through fixing the defect, and verifying the fix. Additionally, the Test Director can create customizable graphs and reports, including test execution reports and release status assessments. What is structural testing? Structural testing is also known as clear box testing, glass box testing. Structural testing is a way to test software with knowledge of the internal workings of the code being tested. Structural testing is white box testing, not black box testing, since black boxes are considered opaque and do not permit visibility into the code. What is the difference between static and dynamic testing? The differences between static and dynamic testing are as follows:

Difference number 1: Static testing is about prevention, dynamic testing is about cure. Difference number 2: She static tools offer greater marginal benefits. Difference number 3: Static testing is many times more cost-effective than dynamic testing.

71

TESTING CONCEPTS

Difference number 4: Static testing beats dynamic testing by a wide margin. Difference number 5: Static testing is more effective! Difference number 6: Static testing gives you comprehensive diagnostics your code. Difference number 7: Static testing achieves 100% statement coverage in a relatively short time, while dynamic testing often often achieves less than 50% statement coverage, because dynamic testing finds bugs only in parts of the code that are actually executed. for

Difference number 8: Dynamic testing usually takes longer than static testing. Dynamic testing may involve running several test cases, each of which may take longer than compilation.

Difference number 9: Dynamic testing finds fewer bugs than static testing. Difference number 10: Static testing can be done before compilation, while dynamic testing can take place only after compilation and linking. Difference number 11: Static testing can find all of the following that dynamic testing cannot find: syntax errors, code that is hard to maintain, code that is hard to test, code that does not conform to coding standards, and ANSI violations.

What testing tools should I use? Ideally, you should use both static and dynamic testing tools. To maximize software reliability, you should use both static and dynamic techniques, supported by appropriate static and dynamic testing tools. Static and dynamic testing are complementary. Static and dynamic testing find different classes of bugs. Some bugs are detectable only by static testing, some only by dynamic. Dynamic testing does detect some errors that static testing misses. To eliminate as many errors as possible, both static and dynamic testing should be used. All this static testing (i.e. testing for syntax errors, testing for code that is hard to maintain, testing for code that is hard to test, testing for code that does not conform to coding standards, and testing for ANSI violations) takes place before compilation. Static testing takes roughly as long as compilation and checks every statement you have written. Why should I use static testing techniques?

72

TESTING CONCEPTS You should use static testing techniques because static testing is a bargain, compared to dynamic testing. Static testing is up to 100 times more effective. Even in selective testing, static testing may be up to 10 times more effective. The most pessimistic estimates suggest a factor of 4. Since static testing is faster and achieves 100% coverage, the unit cost of detecting these bugs by static testing is many times lower than that of by dynamic testing.About half of the bugs, detectable by dynamic testing, can be detected earlier by static testing. If you use neither static nor dynamic test tools, the static tools offer greater marginal benefits. If urgent deadlines loom on the horizon, the use of dynamic testing tools can be omitted, but tool-supported static testing should never be omitted. How can I get registered and licensed as a professional engineer? To get registered and licensed as a professional engineer, generally you have to be a legal resident of the jurisdiction where you submit your application. You also have to be at least 18 years of age, trustworthy, with no criminal record. You also have to have a minimum of a bachelor's degree in engineering, from an established, recognized, and approved university. Usually you have to provide two references, from licensed and professional engineers, and work for a few years as an engineer, as an "engineer in training", under the supervision of a registered and licensed professional engineer. You have to pass a test of competence in your engineering discipline as well as in professional ethics. For many candidates, the biggest two hurdles of getting a license seem to be the lack of a university degree in engineering, or the lack of an acceptable, verifiable work experience, under the supervision of a licensed, professional engineer. What is the definition of top down design? Top down design progresses from simple design to detailed design. Top down design solves problems by breaking them down into smaller, easier to solve sub problems. Top down design creates solutions to these smaller problems, and then tests them using test drivers. In other words, top down design starts the design process with the main module or system, and then progresses down to lower level modules and subsystems. To put it differently, top down design looks at the whole system, and then explodes it into subsystems, or smaller parts. A systems engineer or systems

73

TESTING CONCEPTS analyst determines what the top level objectives are, and how they can be met. He then divides the system into subsystems, i.e. breaks the whole system into logical, manageable-size modules, and deals with them individually. What are the future prospects of software QA/testing? In many IT-related occupations, employers want to see an increasingly broader range of skills; often non technical skills. In software QA/testing, for example, employers want us to have a combination of technical, business, and personal skills. Technical skills mean skills in IT, quantitative analysis, data modeling, and technical writing. Business skills mean skills in strategy and business writing. Personal skills mean personal communication, leadership, teamwork, and problemsolving skills. We, employees, on the other hand, want increasingly more autonomy, better lifestyle, increasingly more employee oriented company culture, and better geographic location. We will continue to enjoy relatively good job security and, depending on the business cycle, many job opportunities as well. We realize our skills are important, and have strong incentives to upgrade our skills, although sometimes lack the information on how to do so. Educational institutions are increasingly more likely to ensure that we are exposed to real-life situations and problems, but high turnover rates and a rapid pace of change in the IT industry will often act as strong disincentives for employers to invest in our skills, especially non-company specific skills. Employers will continue to establish closer links with educational institutions, both through in-house educationprograms and human resources.The share of IT workers with IT degrees will keep increasing. Certification will continue to keep helping employers to quickly identify us with the latest skills. During boom times, smaller and younger companies will continue to be the most attractive to us, especially those companies that offer stock options and performance bonuses in order to retain and attract those of us who are most skilled. High turnover rates will continue to be the norm, especially during boom. Software QA/Testing will continue to be outsourced to offshore locations. Software QA/testing will continue to be performed by a disproportionate share of men, but the share of women will increase.

74

TESTING CONCEPTS

How can I be effective and efficient, when I do black box testing of ecommerce web sites? When you're doing black box testing of e-commerce web sites, you're most efficient and effective when you're testing the sites' Visual Appeal, Contents, and Home Pages. When you want to be effective and efficient, you need to verify that the site is well planned. Verify that the site is customer-friendly. Verify that the choices of colors are attractive. Verify that the choices of fonts are attractive. Verify that the site's audio is customer friendly. Verify that the site's video is attractive. Verify that the choice of graphics is attractive. Verify that every page of the site is displayed properly on all the popular browsers. Verify the authenticity of facts. Ensure the site provides reliable and consistent information. Test the site for appearance. Test the site for grammatical and spelling errors. Test the site for visual appeal, choice of browsers, consistency of font size, download time, broken links, missing links, incorrect links, and browser compatibility. Test each toolbar, each menu item, every window, every field prompt, every pop-up text, and every error message. Test every page of the site for left and right justifications, every shortcut key, each control, each push button, every radio button, and each item on every drop-down menu. Test each list box, and each help menu item. Also check, if the command buttons are grayed out when they're not in use. What is the difference between top down and bottom up design? Top down design proceeds from the abstract (entity) to get to the concrete (design). Bottom up design proceeds from the concrete (design) to get to the abstract (entity). Top down design is most often used in designing brand new systems, while bottom up design is sometimes used when one is reverse engineering a design; i.e. when one is trying to figure out what somebody else designed in an existing system. Bottom up design begins the design with the lowest level modules or subsystems, and progresses upward to the main program, module, or subsystem. With bottom up design, a structure chart is necessary to determine the order of execution, and the development of drivers is necessary to complete the bottom up approach. Top down design, on the other hand, begins the design with the main or toplevel module, and progresses downward to the lowest level modules or

75

TESTING CONCEPTS subsystems. Real life sometimes is a combination of top down design and bottom up design. For instance, data modeling sessions tend to be iterative, bouncing back and forth between top down and bottom up modes, as the need arises. What is the definition of bottom up design? Bottom up design begins the design at the lowest level modules or subsystems, and progresses upward to the design of the main program, main module, or main subsystem. To determine the order of execution, a structure chart is needed, and, to complete the bottom up design, the development of drivers is needed. In software design - assuming that the data you start with is a pretty good model of what you're trying to do - bottom up design generally starts with the known data (e.g. customer lists, order forms), then the data is broken into into chunks (i.e. entities) appropriate for planning a relational database. This process reveals what relationships the entities have, and what the entities' attributes are. In software design, bottom up design doesn't only mean writing the program in a different order, but there is more to it. When you design bottom up, you often end up with a different program. Instead of a single, monolithic program, you get a larger language, with more abstract operators, and a smaller program written in it. Once you abstract out the parts which are merely utilities, what is left is much shorter program. The higher you build up the language, the less distance you will have to travel down to it, from the top. Bottom up design makes it easy to reuse code blocks. For example, many of the utilities you write for one program are also useful for programs you have to write later. Bottom up design also makes programs easier to read. What is smoke testing? Smoke testing is a relatively simple check to see whether the product "smokes" when it runs. Smoke testing is also known as ad hoc testing, i.e. testing without a formal test plan. With many projects, smoke testing is carried out in addition to formal testing. If smoke testing is carried out by a skilled tester, it can often find problems that are not caught during regular testing. Sometimes, if testing occurs very early or very late in the software development cycle, this can be the only kind of testing that can be performed. Smoke tests are, by definition, not exhaustive, but, over time, you can increase your coverage of smoke testing. A common practice

76

TESTING CONCEPTS at Microsoft, and some other software companies, is the daily build and smoke test process. This means, every file is compiled, linked, and combined into an executable file every single day, and then the software is smoke tested. Smoke testing minimizes integration risk, reduces the risk of low quality, supports easier defect diagnosis, and improves morale. Smoke testing does not have to be exhaustive, but should expose any major problems. Smoke testing should be thorough enough that, if it passes, the tester can assume the product is stable enough to be tested more thoroughly. Without smoke testing, the daily build is just a time wasting exercise. Smoke testing is the sentry that guards against any errors in development and future problems during integration. At first, smoke testing might be the testing of something that is easy to test. Then, as the system grows, smoke testing should expand and grow, from a few seconds to 30 minutes or more. What is the difference between monkey testing and smoke testing?

Difference number 1: Monkey testing is random testing, and smoke testing is a nonrandom check to see whether the product "smokes" when it runs. testing is nonrandom testing that deliberately exercises the entire from end to end, with the goal of exposing any major problems. Smoke system

Difference number 2: Monkey testing is performed by automated testing tools. On the other hand, smoke testing, more often than not, is a manual check to see whether the product "smokes" when it runs.

Difference number 3: Monkey testing is performed by "monkeys", while smoke testing is performed by skilled testers (to see whether the product "smokes" when it runs).

Difference number 4: "Smart monkeys" are valuable for load and stress testing, but not very valuable for smoke testing, because they are too expensive for smoke testing.

Difference number 5: "Dumb monkeys" are inexpensive to develop, are able to do some basic testing, but, if we use them for smoke testing, they find few bugs during smoke testing.

Difference number 6: Monkey testing is not a thorough testing, but smoke testing is thorough enough that, if the build passes, one can assume that the program is stable enough to be tested more thoroughly.

77

TESTING CONCEPTS

Difference number 7: Monkey testing does not evolve. Smoke testing, on other hand, evolves as the system evolves from something simple to more thorough.

the

something

Difference number 8: Monkey testing takes "six monkeys" and a "million years" to run. Smoke testing, on the other hand, takes much less time to run, i.e. anywhere from a few seconds to a couple of hours.

Tell me about the process of daily builds and smoke tests. The idea behind the process of daily builds and smoke tests is to build the product every day, and test it every day. The software development process at Microsoft and many other software companies requires daily builds and smoke tests. According to their process, every day, every single file has to be compiled, linked, and combined into an executable program. And, then, the program has to be "smoke tested". Smoke testing is a relatively simple check to see whether the product "smokes" when it runs. You should add revisions to the build only when it makes sense to do so. You should to establish a Build Group, and build *daily*; set your *own standard* for what constitutes "breaking the build", and create a penalty for breaking the build, and check for broken builds *every day*. In addition to the daily builds, you should smoke test the builds, and smoke test them Daily. You should make the smoke test Evolve, as the system evolves. You should build and smoke test Daily, even when the project is under pressure. Think about the many benefits of this process! The process of daily builds and smoke tests minimizes the integration risk, reduces the risk of low quality, supports easier defect diagnosis, improves morale, enforces discipline, and keeps pressure-cooker projects on track. If you build and smoke test *daily*, success will come, even when you're working on large projects! What is the purpose of test strategy?

Reason number 1: The number one reason of writing a test strategy document is to "have" a signed, sealed, and delivered, FDA (or FAA) approved document, where the document includes a written testing methodology, test plan, and test cases.

Reason number 2: Having a test strategy does satisfy one important step in the software testing process. Reason number 3: The test strategy document tells us how the software product will be tested.

78

TESTING CONCEPTS

Reason number 4: The creation of a test strategy document presents an opportunity to review the test plan with the project team. Reason number 5: The test strategy document describes the roles, responsibilities, and the resources required for the test and schedule constraints. Reason number 6: When we create a test strategy document, we have to put into writing any testing issues requiring resolution (and usually this means additional negotiation at the project management level).

Reason number 7: The test strategy is decided first, before lower level decisions are made on the test plan, test design, and other testing issues.

What do you mean by 'the process is repeatable'? A process is repeatable, whenever we have the necessary processes in place, in order to repeat earlier successes on projects with similar applications. A process is repeatable, if we use detailed and well-written processes and procedures. A process is repeatable, if we ensure that the correct steps are executed. When the correct steps are executed, we facilitate a successful completion of the task. Documentation is critical. A software process is repeatable, if there are requirements management, project planning, project tracking, subcontract management, QA, and configuration management. Both QA processes and practices should be documented, so that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals should all be documented, so that they are repeatable. Document files should be well organized. There should be a system for easily finding and obtaining documents, and determining what document has a particular piece of information. We should use documentation change management, if possible. Once Rob Davis has learned and reviewed a customer's business processes and procedures, he will follow them. He will also recommend improvements and/or additions. What is the purpose of a test plan?

Reason number 1: We create a test plan because preparing it helps us to think through the efforts needed to validate the acceptability of a software product. Reason number 2: We create a test plan because it can and will help people outside the test group to understand the why and how of product validation.

79

TESTING CONCEPTS

Reason number 3: We create a test plan because, in regulated environments, we have to have a written test plan. Reason number 4: We create a test plan because the general testing process includes the creation of a test plan. Reason number 5: We create a test plan because we want a document that describes the objectives, scope, approach and focus of the software testing effort. Reason number 6: We create a test plan because it includes test cases, conditions, the test environment, a list of related tasks, pass/fail criteria, and risk assessment.

Reason number 7: We create test plan because one of the outputs for creating a test strategy is an approved and signed off test plan document. Reason number 8: We create a test plan because the software testing methodology a three step process, and one of the steps is the creation of a test plan.

Reason number 9: We create a test plan because we want an opportunity to review the test plan with the project team. Reason number 10: We create a test plan document because test plans should be documented, so that they are repeatable.

Give me one test case that catches all the bugs! If there is a "magic bullet", i.e. the one test case that has a good possibility to catch ALL the bugs, or at least the most important bugs, it is a challenge to find it, because test cases depend on requirements; requirements depend on what customers need; and customers can have great many different needs. As software systems are getting increasingly complex, it is increasingly more challenging to write test cases. It is true that there are ways to create "minimal test cases" which can greatly simplify the test steps to be executed. But, writing such test cases is time consuming, and project deadlines often prevent us from going that route. Often the lack of enough time for testing is the reason for bugs to occur in the field. However, even with ample time to catch the "most important bugs", bugs still surface with amazing spontaneity. The challenge is, developers do not seem to know how to avoid providing the many opportunities for bugs to hide, and testers do not seem to know where the bugs are hiding.

80

TESTING CONCEPTS What is the difference between a test plan and a test scenario? Difference number 1: A test plan is a document that describes the scope, approach, resources, and schedule of intended testing activities, while a test scenario is a document that describes both typical and atypical situations that may occur in the use of an application. Difference number 2: Test plans define the scope, approach, resources, and schedule of the intended testing activities, while test procedures define test conditions, data to be used for testing, and expected results, including database updates, file outputs, and report results. Difference number 3: A test plan is a description of the scope, approach, resources, and schedule of intended testing activities, while a test scenario is a description of test cases that ensure that a business process flow, applicable to the customer, is tested from end to end. What is a test scenario? The terms "test scenario" and "test case" are often used synonymously. Test scenarios are test cases, or test scripts, and the sequence in which they are to be executed. Test scenarios are test cases that ensure that business process flows are tested from end to end. Test scenarios are independent tests, or a series of tests, that follow each other, where each of them dependent upon the output of the previous one. Test scenarios are prepared by reviewing functional requirements, and preparing logical groups of functions that can be further broken into test procedures. Test scenarios are designed to represent both typical and unusual situations that may occur in the application. Test engineers define unit test requirements and unit test scenarios. Test engineers also execute unit test scenarios. It is the test team that, with assistance of developers and clients, develops test scenarios for integration and system testing. Test scenarios are executed through the use of test procedures or scripts. Test procedures or scripts define a series of steps necessary to perform one or more test scenarios. Test procedures or scripts may cover multiple test scenarios. Give me some sample test cases you would write! For instance, if one of the requirements is, "Brake lights shall be on, when the brake pedal is depressed", then, based on this one simple requirement, for starters, I would write all of the following test cases: Test case number 101: "Inputs: The headlights are on. The brake pedal is depressed. Expected result: The brake lights

81

TESTING CONCEPTS are on. Verify that the brake lights are on, when the brake pedal is depressed." Test case number 102: "Inputs: The left turn lights are on. The brake pedal is depressed. Expected result: The brake lights are on. Verify that the brake lights are on, when the brake pedal is depressed." Test case number 103: "Inputs: The right turn lights are on. The brake pedal is depressed. Expected result: The brake lights are on. Verify that the brake lights are on, when the brake pedal is depressed." As you might have guessed, in the work place, in real life, requirements are more complex than this one; and, just to verify this one, simple requirement, there is a need for many more test cases. How do you write test cases? When I write test cases, I concentrate on one requirement at a time. Then, based on that one requirement, I come up with several real life scenarios that are likely to occur in the use of the application by end users. When I write test cases, I describe the inputs, action, or event, and their expected results, in order to determine if a feature of an application is working correctly. To make the test case complete, I also add particulars e.g. test case identifiers, test case names, objectives, test conditions (or setups), input data requirements (or steps), and expected results. If I have a choice, I prefer writing test cases as early as possible in the development life cycle. Why? Because, as a side benefit of writing test cases, many times I am able to find problems in the requirements or design of an application. And, because the process of developing test cases makes me completely think through the operation of the application. You can learn to write test cases! If there is a will, there is a way! You CAN do it, if you put your mind to it! You CAN learn to write test cases, with little or no outside help. Click on a link! What is a parameter? A parameter is an item of information - such as a name, a number, or a selected option - that is passed to a program by a user or another program. By definition, a parameter is a value on which something else depends. Any desired numerical value may be given as a parameter. We use parameters when we want to allow a specified range of variables. We use parameters when we want to differentiate behavior or pass input data to computer programs or their subprograms. Thus, when we are testing, the parameters of the test can be varied to produce different results, because parameters do affect the operation of the program receiving them. Example 1: We use a parameter, such as temperature, that defines a

82

TESTING CONCEPTS system. In this definition, it is temperature that defines the system and determines its behavior. Example 2: In the definition of function f(x) = x + 10, x is a parameter. In this definition, x defines the f(x) function and determines its behavior. Thus, when we are testing, x can be varied to make f(x) produce different values, because the value of x does affect the value of f(x).When parameters are passed to a function subroutine, they are called arguments. What is a constant? In software or software testing, a constant is a meaningful name that represents a number, or string, that does not change. Constants are variables whose value remain the same, i.e. constant, throughout the execution of a program. Why do developers use constants? Because if we have code that contains constant values that keep reappearing, or, if we have code that depends on certain numbers that are difficult to remember, we can improve both the readability and maintainability of our code, by using constants. To give you an example, let's suppose we declare a constant and we call it Pi. We set it to 3.14159265 and use it throughout our code. Constants, such as Pi, as the name implies, store values that remain constant throughout the execution of our program. Keep in mind that, unlike variables which can be read from and written to, constants are read-only variables. Although constants resemble variables, we cannot modify or assign new values to them, as we can to variables. But we can make constants public, or private. We can also specify what data type they are. What is a requirements test matrix? The requirements test matrix is a project management tool for tracking and managing testing efforts, based on requirements, throughout the project's life cycle. The requirements test matrix is a table, where requirement descriptions are put in the rows of the table, and the descriptions of testing efforts are put in the column headers of the same table. The requirements test matrix is similar to the requirements traceability matrix, which is a representation of user requirements aligned against system functionality. The requirements traceability matrix ensures that all user requirements are addressed by the system integration team and implemented in the system integration effort. The requirements test matrix is a representation of user requirements aligned against system testing. Similarly to the requirements traceability matrix, the requirements test matrix ensures that all user

83

TESTING CONCEPTS requirements are addressed by the system test team and implemented in the system testing effort. Give me a requirements test matrix template! For a simple requirements test matrix template, you want a basic table that you would like to use for cross referencing purposes. How do you create one? You can create a requirements test matrix template in the following six steps: Step 1: Find out how many requirements you have. Step 2: Find out how many test cases you have. Step 3: Based on these numbers, create a basic table. Let's suppose you have a list of 90 requirements and 360 test cases. Based on these numbers, you want to create a table of 91 rows and 361 columns. Step 4: Focus on the first column of your table. One by one, copy all your 90 requirement numbers, and paste them into rows 2 through 91 of your table. Step 5: Focus on the first row of your table. One by one, copy all your 360 test case numbers, and paste them into columns 2 through 361 of your table. Step 6: Examine each of your 360 test cases, and, one by one, determine which of the 90 requirements they satisfy. If, for the sake of this example, test case 64 satisfies requirement 12, then put a large "X" into cell 13-65 of your table... and then you have it; you have just created a requirements test matrix template that you can use for cross-referencing purposes. What is reliability testing? Reliability testing is designing reliability test cases, using accelerated reliability techniques (e.g. step-stress, test/analyze/fix, and continuously increasing stress testing techniques), AND testing units or systems to failure, in order to obtain raw failure time data for product life analysis. The purpose of reliability testing is to determine product reliability, and to determine whether the software meets the customer's reliability requirements. In the system test phase, or after the software is fully developed, one reliability testing technique we use is a test/analyze/fix technique, where we couple reliability testing with the removal of faults. When we identify a failure, we send the software back to the developers, for repair. The developers build a new version of the software, and then we do another test iteration. We track failure intensity (e.g. failures per transaction, or failures per hour) in order to guide our test process, and to determine the feasibility of the software

84

TESTING CONCEPTS release, and to determine whether the software meets the customer's reliability requirements. Give me an example on reliability testing. For example, our products are defibrillators. From direct contact with customers during the requirements gathering phase, our sales team learns that a large hospital wants to purchase defibrillators with the assurance that 99 out of every 100 shocks will be delivered properly. In this example, the fact that our defibrillator is able to run for 250 hours without any failure, in order to demonstrate the reliability, is irrelevant to these customers. In order to test for reliability we need to translate terminology that is meaningful to the customers into equivalent delivery units, such as the number of shocks. We describe the customer needs in a quantifiable manner, using the customers terminology. For example, our of quantified reliability testing goal becomes as follows: Our defibrillator will be considered sufficiently reliable if 10 (or fewer) failures occur from 1,000 shocks. Then, for example, we use a test/analyze/fix technique, and couple reliability testing with the removal of errors. When we identify a failed delivery of a shock, we send the software back to the developers, for repair. The developers build a new version of the software, and then we deliver another 1,000 shocks into dummy resistor loads. We track failure intensity (i.e. number of failures per 1,000 shocks) in order to guide our reliability testing, and to determine the feasibility of the software release, and to determine whether the software meets our customers' reliability requirements. What is verification? Verification ensures the product is designed to deliver all functionality to the customer; it typically involves reviews and meetings to evaluate documents, plans, code, requirements and specifications; this can be done with checklists, issues lists, walk-through and inspection meetings. You CAN learn to do verification, with little or no outside help. Get CAN get free information. Click on a link! What is validation? Validation ensures that functionality, as defined in requirements, is the intended behavior of the product; validation typically involves actual testing and takes place after verifications are completed.

85

TESTING CONCEPTS

What is a walk-through? A walk-through is an informal meeting for evaluation or informational purposes. A walk-through is also a process at an abstract level. It's the process of inspecting software code by following paths through the code (as determined by input conditions and choices made along the way). The purpose of code walk-through is to ensure the code fits the purpose. Walk-through also offer opportunities to assess an individual's or team's competency. What is an inspection? An inspection is a formal meeting, more formalized than a walk-through and typically consists of 3-10 people including a moderator, reader (the author of whatever is being reviewed) and a recorder (to make notes in the document). The subject of the inspection is typically a document, such as a requirements document or a test plan. The purpose of an inspection is to find problems and see what is missing, not to fix anything. The result of the meeting should be documented in a written report. Attendees should prepare for this type of meeting by reading through the document, before the meeting starts; most problems are found during this preparation. Preparation for inspections is difficult, but is one of the most cost effective methods of ensuring quality, since bug prevention is more cost effective than bug detection. What is quality? Quality software is software that is reasonably bug-free, delivered on time and within budget, meets requirements and expectations and is maintainable. However, quality is a subjective term. Quality depends on who the customer is and their overall influence in the scheme of things. Customers of a software development project include end-users, customer acceptance test engineers, testers, customer contract officers, customer management, the development organization's management, test engineers, testers, salespeople, software engineers, stockholders and accountants. Each type of customer will have his or her own slant on quality. The accounting department might define quality in terms of profits, while an end-user might define quality as user friendly and bug free.

86

TESTING CONCEPTS

What is good code? A good code is code that works, is free of bugs and is readable and maintainable. Organizations usually have coding standards all developers should adhere to, but every programmer and software engineer has different ideas about what is best and what are too many or too few rules. We need to keep in mind that excessive use of rules can stifle both productivity and creativity. Peer reviews and code analysis tools can be used to check for problems and enforce standards. What is good design? Design could mean to many things, but often refers to functional design or internal design. Good functional design is indicated by software functionality can be traced back to customer and end-user requirements. Good internal design is indicated by software code whose overall structure is clear, understandable, easily modifiable and maintainable; is robust with sufficient error handling and status logging capability; and works correctly when implemented. What is software life Software life cycle begins when a software product is first conceived and ends when it is no longer in use. It includes phases like initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting and phase-out. How do you introduce a new software QA process? It depends on the size of the organization and the risks involved. For large organizations with high-risk projects, a serious management buy-in is required and a formalized QA process is necessary. For medium size organizations with lower risk projects, management and organizational buy-in and a slower, step-by-step process is required. Generally speaking, QA processes should be balanced with productivity, in order to keep any bureaucracy from getting out of hand. For smaller groups or projects, an ad-hoc process is more appropriate. A lot depends on team leads and managers, feedback to developers and good communication is essential among customers, managers, developers, test engineers and testers. Regardless the size of

87

TESTING CONCEPTS the company, the greatest value for effort is in managing requirement processes, where the goal is requirements that are clear, complete and testable. What is the role of documentation in QA? Documentation plays a critical role in QA. QA practices should be documented, so that they are repeatable. Specifications, designs, business rules, inspection reports, configurations, code changes, test plans, test cases, bug reports, user manuals should all be documented. Ideally, there should be a system for easily finding and obtaining of documents and determining what document will have a particular piece of information. Use documentation change management, if possible. Why are there so many software bugs? Generally speaking, there are bugs in software because of unclear requirements, software complexity, programming errors, changes in requirements, errors made in bug tracking, time pressure, poorly documented code and/or bugs in tools used in software development. _ There are unclear software requirements because there is miscommunication as to what the software should or shouldn't do. _ Software complexity. All of the followings contribute to the exponential growth in software and system complexity: Windows interfaces, client-server and distributed applications, data communications, enormous relational databases and the sheer size of applications. _ Programming errors occur because programmers and software engineers, like everyone else, can make mistakes. _ As to changing requirements, in some fast-changing business environments, continuously modified requirements are a fact of life. Sometimes customers do not understand the effects of changes, or understand them but request them anyway. And the changes require redesign of the software, rescheduling of resources and some of the work already completed have to be redone or discarded and hardware requirements can be effected, too. _ Bug tracking can result in errors because the complexity of keeping track of changes can result in errors, too. _ Time pressures can cause problems, because scheduling of software projects is not easy and it often requires a lot of guesswork and when deadlines loom and the crunch comes, mistakes will be made. _ Code documentation is tough to maintain and it is also tough to modify code that is poorly documented. The result is bugs. Sometimes there is no incentive for programmers and software engineers to document their code and write clearly documented, understandable code. Sometimes developers get kudos for quickly turning out code, or programmers

88

TESTING CONCEPTS and software engineers feel they cannot have job security if everyone can understand the code they write, or they believe if the code was hard to write, it should be hard to read. _ Software development tools , including visual tools, class libraries, compilers, scripting tools, can introduce their own bugs. Other times the tools are poorly documented, which can create additional bugs. Give me five common problems that occur during software development. Poorly written requirements, unrealistic schedules, inadequate testing, adding new features after development is underway and poor communication. 1. Requirements are poorly written when requirements are unclear, incomplete, too general, or not testable; therefore there will be problems. 2. The schedule is unrealistic if too much work is crammed in too little time. 3. Software testing is inadequate if none knows whether or not the software is any good until customers complain or the system crashes. 4. It's extremely common that new features are added after development is underway. 5. Miscommunication either means the developers don't know what is needed, or customers have unrealistic expectations and therefore problems are guaranteed. 179. What is a backward compatible design? The design is backward compatible, if the design continues to work with earlier versions of a language, program, code, or software. When the design is backward compatible, the signals or data that had to be changed, did not break the existing code. For instance, our mythical web designer decides that the fun of using Java script and Flash is more important than backward compatible design, or, he decides that he doesn't have the resources to maintain multiple styles of backward compatible web design. This decision of his will inconvenience some users, because some of the earlier versions of Internet Explorer and Netscape will not display his web pages properly, as there are some serious improvements in the newer versions of Internet Explorer and Netscape that make the older versions of these browsers incompatible with, for example, DHTML. This is when we say, "This design doesn't continue to work with earlier versions of browser software. Therefore our mythical designer's web design is not backward compatible". On the other hand, if the same mythical web designer decides that backward compatibility is more important than fun, or, if he decides that he has the resources to maintain multiple styles of

89

TESTING CONCEPTS backward compatible code, then no user will be inconvenienced. No one will be inconvenienced, even when Microsoft and Netscape make some serious improvements in their web browsers. This is when we can say, "Our mythical web designer's design is backward compatible". What is Regression Testing? Introduction: This article attempts to take a close look at the process and techniques in Regression Testing. What is Regression Testing? If a piece of Software is modified for any reason testing needs to be done to ensure that it works as specified and that it has not negatively impacted any functionality that it offered previously. This is known as Regression Testing. Regression Testing attempts to verify: That the application works as specified even after the changes/additions/modification were made to it The original functionality continues to work as specified even after changes/additions/modification to the software application The changes/additions/modification to the software application have not introduced any new bugs When is Regression Testing necessary? Regression Testing plays an important role in any Scenario where a change has been made to a previously tested software code. Regression Testing is hence an important aspect in various Software Methodologies where software changes enhancements occur frequently. Any Software Development Project is invariably faced with requests for changing Design, code, features or all of them.

90

TESTING CONCEPTS Some Development Methodologies embrace change. For example Extreme Programming Methodology advocates applying small incremental changes to the system based on the end user feedback. Each change implies more Regression Testing needs to be done to ensure that the System meets the Project Goals. Why is Regression Testing important? Any Software change can cause existing functionality to break. Changes to a Software component could impact dependent Components. It is commonly observed that a Software fix could cause other bugs. All this affects the quality and reliability of the system. Hence Regression Testing, since it aims to verify all this, is very important. Making Regression Testing Cost Effective: Every time a change occurs one or more of the following scenarios may occur: More Functionality may be added to the system More complexity may be added to the system New bugs may be introduced New vulnerabilities may be introduced in the system System may tend to become more and more fragile with each change After the change the new functionality may have to be tested along with all the original functionality. With each change Regression Testing could become more and more costly. To make the Regression Testing Cost Effective and yet ensure good coverage one or more of the following techniques may be applied: Test Automation: If the Test cases are automated the test cases may be executed using scripts after each change is introduced in the system. The execution of test cases in this way helps eliminate oversight, human errors,. It may also result

91

TESTING CONCEPTS in faster and cheaper execution of Test cases. However there is cost involved in building the scripts. Selective Testing: Some Teams choose execute the test cases selectively. They do not execute all the Test Cases during the Regression Testing. They test only what they decide is relevant. This helps reduce the Testing Time and Effort. Regression Testing What to Test? Since Regression Testing tends to verify the software application after a change has been made everything that may be impacted by the change should be tested during Regression Testing. Generally the following areas are covered during Regression Testing: Any functionality that was addressed by the change Original Functionality of the system Performance of the System after the change was introduced Regression Testing How to Test? Like any other Testing Regression Testing Needs proper planning. For an Effective Regression Testing to be done the following ingredients are necessary: Create a Regression Test Plan: Test Plan identified Focus Areas, Strategy, Test Entry and Exit Criteria. It can also outline Testing Prerequisites, Responsibilities, etc. Create Test Cases: Test Cases that cover all the necessary areas are important. They describe what to Test, Steps needed to test, Inputs and Expected Outputs. Test Cases used for Regression Testing should specifically cover the functionality addressed by the change and all components affected by the change. The Regression Test case may also include the testing of the performance of the components and the application after the change(s) were done. Defect Tracking: As in all other Testing Levels and Types It is important Defects are tracked systematically, otherwise it undermines the Testing Effort.

92

TESTING CONCEPTS Summary: In this article we studied the importance of Regression Testing, its role and how it is done. What is client-server and web based testing and how to test these applications What is the difference between client-server testing and web based testing and what are things that we need to test in such applications? Ans: Projects are broadly divided into two types of: 2 tier applications 3 tier applications

CLIENT / SERVER TESTING This type of testing usually done for 2 tier applications (usually developed for LAN) Here we will be having front-end and backend. The application launched on front-end will be having forms and reports which will be monitoring and manipulating data E.g: applications developed in VB, VC++, Core Java, C, C++, D2K, PowerBuilder etc., The backend for these applications would be MS Access, SQL Server, Oracle, Sybase, Mysql, Quadbase The tests performed on these types of applications would be User interface testing Manual support testing Functionality testing Compatibility testing & configuration testing Intersystem testing

93

TESTING CONCEPTS WEB TESTING This is done for 3 tier applications (developed for Internet / intranet / xtranet) Here we will be having Browser, web server and DB server. The applications accessible in browser would be developed in HTML, DHTML, XML, JavaScript etc. (We can monitor through these applications) Applications for the web server would be developed in Java, ASP, JSP, VBScript, JavaScript, Perl, Cold Fusion, PHP etc. (All the manipulations are done on the web server with the help of these programs developed) The DB server would be having oracle, sill server, sybase, mysql etc. (All data is stored in the database available on the DB server) The tests performed on these types of applications would be User interface testing Functionality testing Security testing Browser compatibility testing Load / stress testing Interoperability testing/intersystem testing

Storage and data volume testing

A web-application is a three-tier application. This has a browser (monitors data) [monitoring is done using html, dhtml, xml, javascript]-> webserver (manipulates data) [manipulations are done using programming languages or scripts like adv java, asp, jsp, vbscript, javascript, perl, coldfusion, php] -> database server (stores data) [data storage and retrieval is done using databases like oracle, sql server, sybase, mysql]. The types of tests, which can be applied on this type of applications, are: 1. User interface testing for validation & user friendliness 2. Functionality testing to validate behaviors, i/p, error handling, o/p, manipulations, services levels, order of functionality, links, content of web page & backend coverages 3. Security testing 4. Browser compatibility

94

TESTING CONCEPTS 5. Load / stress testing 6. Interoperability testing 7. Storage & data volume testing A client-server application is a two tier application. This has forms & reporting at front-end (monitoring & manipulations are done) [using vb, vc++, core java, c, c++, d2k, power builder etc.,] -> database server at the backend [data storage & retrieval) [using ms access, sql server, oracle, sybase, mysql, quadbase etc.,] The tests performed on these applications would be 1. User interface testing 2. Manual support testing 3. Functionality testing 4. Compatibility testing 5. Intersystem testing

Some more points to clear the difference between client server, web and desktop applications: Desktop application: 1. Application runs in single memory (Front end and Back end in one place) 2. Single user only Client/Server application: 1. Application runs in two or more machines 2. Application is a menu-driven 3. Connected mode (connection exists always until logout) 4. Limited number of users 5. Less number of network issues when compared to web app. Web application: 1. Application runs in two or more machines 2. URL-driven 3. Disconnected mode (state less) 4. Unlimited number of users

95

TESTING CONCEPTS 5. Many issues like hardware compatibility, browser compatibility, version compatibility, security issues, performance issues etc. As per difference in both the applications come where, how to access the resources. In client server once connection is made it will be in state on connected, whereas in case of web testing http protocol is stateless, then there comes logic of cookies, which is not in client server. For client server application users are well known, whereas for web application any user can login and access the content, he/she will use it as per his intentions. So, there are always issues of security and compatibility for web application. Black Box Testing: Types and techniques of BBT I have covered what is White box Testing in previous article. Here I will concentrate on Black box testing. BBT advantages, disadvantages and and How Black box testing is performed i.e. the black box testing techniques. Black box testing treats the system as a black-box, so it doesnt explicitly use Knowledge of the internal structure or code. Or in other words the Test engineer need not know the internal working of the Black box or application. Main focus in black box testing is on functionality of the system as a whole. The term behavioral testing is also used for black box testing and white box testing is also sometimes called structural testing. Behavioral test design is slightly different from black-box test design because the use of internal knowledge isnt strictly forbidden, but its still discouraged. Each testing method has its own advantages and disadvantages. There are some bugs that cannot be found using only black box or only white box. Majority of the application are tested by black box testing method. We need to cover majority of test cases so that most of the bugs will get discovered by black box testing. Black box testing occurs throughout the software development and Testing life cycle i.e. in Unit, Integration, System, Acceptance and regression testing stages.

96

TESTING CONCEPTS Tools used for Black Box testing: Black box testing tools are mainly record and playback tools. These tools are used for regression testing that to check whether new build has created any bug in previous working application functionality. These record and playback tools records test cases in the form of some scripts like TSL, VB script, Java script, Perl. Advantages of Black Box Testing - Tester can be non-technical. - Used to verify contradictions in actual system and the specifications. - Test cases can be designed as soon as the functional specifications are complete Disadvantages of Black Box Testing - The test inputs needs to be from large sample space. - It is difficult to identify all possible inputs in limited testing time. So writing test cases is slow and difficult - Chances of having unidentified paths during this testing Methods of Black box Testing: Graph Based Testing Methods: Each and every application is build up of some objects. All such objects are identified and graph is prepared. From this object graph each object relationship is identified and test cases written accordingly to discover the errors. Error Guessing: This is purely based on previous experience and judgment of tester. Error Guessing is the art of guessing where errors can be hidden. For this technique there are no specific tools, writing the test cases that cover all the application paths. Boundary Value Analysis: Many systems have tendency to fail on boundary. So testing boundry values of application is important. Boundary Value Analysis (BVA) is a test Functional Testing technique where the extreme boundary values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. Extends equivalence partitioning Test both sides of each boundary

97

TESTING CONCEPTS Look at output boundaries for test cases too Test min, min-1, max, max+1, typical values BVA techniques: 1. Number of variables For n variables: BVA yields 4n + 1 test cases. 2. Kinds of ranges Generalizing ranges depends on the nature or type of variables Advantages of Boundary Value Analysis 1. Robustness Testing - Boundary Value Analysis plus values that go beyond the limits 2. Min - 1, Min, Min +1, Nom, Max -1, Max, Max +1 3. Forces attention to exception handling Limitations of Boundary Value Analysis Boundary value testing is efficient only for variables of fixed values i.e boundary. Equivalence Partitioning: Equivalence partitioning is a black box testing method that divides the input domain of a program into classes of data from which test cases can be derived. How is this partitioning performed while testing: 1. If an input condition specifies a range, one valid and one two invalid classes are defined. 2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined. 3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined. 4. If an input condition is Boolean, one valid and one invalid class is defined. Comparison Testing: Different independent versions of same software are used to compare to each other for testing in this method. What you need to know about BVT (Build Verification Testing) What is BVT?

98

TESTING CONCEPTS Build Verification test is a set of tests run on every new build to verify that build is testable before it is released to test team for further testing. These test cases are core functionality test cases that ensure application is stable and can be tested thoroughly. Typically BVT process is automated. If BVT fails that build is again get assigned to developer for fix. BVT is also called smoke testing or build acceptance testing (BAT) New Build is checked mainly for two things: Build validation Build acceptance

Some BVT basics: It is a subset of tests that verify main functionalities. The BVTs are typically run on daily builds and if the BVT fails the build is rejected and a new build is released after the fixes are done. The advantage of BVT is it saves the efforts of a test team to setup and test a build when major functionality is broken. Design BVTs carefully enough to cover basic functionality. Typically BVT should not run more than 30 minutes. BVT is a type of regression testing, done on each and every new build.

BVT primarily checks for the project integrity and checks whether all the modules are integrated properly or not. Module integration testing is very important when different teams develop project modules. I heard many cases of application failure due to improper module integration. Even in worst cases complete project gets scraped due to failure in module integration.

What is the main task in build release?

99

TESTING CONCEPTS Obviously file check in i.e. to include all the new and modified project files associated with respective builds. BVT was primarily introduced to check initial build health i.e. to check whether - all the new and modified files are included in release, all file formats are correct, every file version and language, flags associated with each file. These basic checks are worth before build release to test team for testing. You will save time and money by discovering the build flaws at the very beginning using BVT. Which test cases should be included in BVT? This is very tricky decision to take before automating the BVT task. Keep in mind that success of BVT depends on which test cases you include in BVT. Here are some simple tips to include test cases in your BVT automation suite: Include only critical test cases in BVT. All test cases included in BVT should be stable. All the test cases should have known expected result. Make sure all included critical functionality test cases are sufficient for application test coverage. Also do not includes modules in BVT, which are not yet stable. For some underdevelopment features you cant predict expected behavior as these modules are unstable and you might know some known failures before testing for these incomplete modules. There is no point using such modules or test cases in BVT. You can make this critical functionality test cases inclusion task simple by communicating with all those involved in project development and testing life cycle. Such process should negotiate BVT test cases, which ultimately ensure BVT success. Set some BVT quality standards and these standards can be met only by analyzing major project features and scenarios. Example: Test cases to be included in BVT for Text editor application (Some sample tests only): 1) Test case for creating text file. 2) Test cases for writing something into text editor

100

TESTING CONCEPTS 3) Test case for copy, cut, paste functionality of text editor 4) Test case for opening, saving, deleting text file. These are some sample test cases, which can be marked as critical and for every minor or major changes in application these basic critical test cases should be executed. This task can be easily accomplished by BVT. BVT automation suits needs to be maintained and modified time-to-time. E.g. include test cases in BVT when there are new stable project modules available. What happens when BVT suite run: Say Build verification automation test suite executed after any new build. 1) The result of BVT execution is sent to all the email IDs associated with that project. 2) The BVT owner (person executing and maintaining the BVT suite) inspects the result of BVT. 3) If BVT fails then BVT owner diagnose the cause of failure. 4) If the failure cause is defect in build, all the relevant information with failure logs is sent to respective developers. 5) Developer on his initial diagnostic replies to team about the failure cause. Whether this is really a bug? And if its a bug then what will be his bug-fixing scenario. 6) On bug fix once again BVT test suite is executed and if build passes BVT, the build is passed to test team for further detail functionality, performance and other testes. This process gets repeated for every new build. Why BVT or build fails? BVT breaks sometimes. This doesnt mean that there is always bug in the build. There are some other reasons to build fail like test case coding error, automation suite error, infrastructure error, hardware failures etc. You need to troubleshoot the cause for the BVT break and need to take proper action after diagnosis. Tips for BVT success: 1) Spend considerable time writing BVT test cases scripts. 2) Log as much detailed info as possible to diagnose the BVT pass or fail result. This

101

TESTING CONCEPTS will help developer team to debug and quickly know the failure cause. 3) Select stable test cases to include in BVT. For new features if new critical test case passes consistently on different configuration then promote this test case in your BVT suite. This will reduce the probability of frequent build failure due to new unstable modules and test cases. 4) Automate BVT process as much as possible. Right from build release process to BVT result - automate everything. 5) Have some penalties for breaking the build Some chocolates or team coffee party from developer who breaks the build will do. Types of Risks in Software Projects Are you developing any Test plan or test strategy for your project? Have you addressed all risks properly in your test plan or test strategy? As testing is the last part of the project, its always under pressure and time constraint. To save time and money you should be able to prioritize your testing work. How will prioritize testing work? For this you should be able to judge more important and less important testing work. How will you decide which work is more or less important? Here comes need of risk-based testing. What is Risk? Risk are future uncertain events with a probability of occurrence and a potential for loss Risk identification and management are the main concerns in every software project. Effective analysis of software risks will help to effective planning and assignments of work. In this article I will cover what are the types of risks. In next articles I will try to focus on risk identification, risk management and mitigation. Risks are identified, classified and managed before actual execution of program. These risks are classified in different categories.

102

TESTING CONCEPTS Categories of risks: Schedule Risk: Project schedule get slip when project tasks and schedule release risks are not addressed properly. Schedule risks mainly affect on project and finally on company economy and may lead to project failure. Schedules often slip due to following reasons: Wrong time estimation Resources are not tracked properly. All resources like staff, systems, skills of individuals etc. Failure to identify complex functionalities and time required to develop those functionalities. Unexpected project scope expansions.

Budget Risk: Wrong budget estimation. Cost overruns Project scope expansion

Operational Risks: Risks of loss due to improper process implementation, failed system or some external events risks. Causes of Operational risks: Failure to address priority conflicts Failure to resolve the responsibilities Insufficient resources No proper subject training No resource planning No communication in team.

103

TESTING CONCEPTS Technical risks: Technical risks generally leads to failure of functionality and performance. Causes of technical risks are: Continuous changing requirements No advanced technology available or the existing technology is in initial stages. Product is complex to implement. Difficult project modules integration.

Programmatic Risks: These are the external risks beyond the operational limits. These are all uncertain risks are outside the control of the program. These external events can be: Running out of fund. Market development Changing customer product strategy and priority Government rule changes.

How Domain knowledge is Important for testers?

Looking at the current scenario from the industry it is seen that the testers are expected to have both technical testing skills as well either need to be from the domain background or have gathered domain knowledge mainly for BFSI is commonly seen. I would like to know why and when is this domain knowledge imparted to the tester during the testing cycle? First of all I would like to introduce three dimensional testing career mentioned by Danny R. Faught. There are three categories of skill that need to be judged before hiring any software tester. What are those three skill categories? 1) Testing skill 2) Domain knowledge 3) Technical expertise.

104

TESTING CONCEPTS No doubt that any tester should have the basic testing skills like Manual testing and Automation testing. Tester having the common sense can even find most of the obvious bugs in the software. Then would you say that this much testing is sufficient? Would you release the product on the basis of this much testing done? Certainly not. You will certainly have a product look by the domain expert before the product goes into the market. While testing any application you should think like a end-user. But every human being has the limitations and one cant be the expert in all of the three dimensions mentioned above. (If you are the experts in all of the above skills then please let me know ;-)) So you cant assure that you can think 100% like how the end-user going to use your application. User who is going to use your application may be having a good understanding of the domain he is working on. You need to balance all these skill activities so that all product aspects will get addressed. Nowadays you can see the professional being hired in different companies are more domain experts than having technical skills. Current software industry is also seeing a good trend that many professional developers and domain experts are moving into software testing. We can observe one more reason why domain experts are most wanted! When you hire fresh engineers who are just out of college you cannot expect them to compete with the experienced professionals. Why? Because experienced professional certainly have the advantage of domain and testing experience and they have better understandings of different issues and can deliver the application better and faster. Here are some of the examples where you can see the distinct edge of domain knowledge: 1) Mobile application testing. 2) Wireless application testing 3) VoIP applications 4) Protocol testing 5) Banking applications 6) Network testing How will you test such applications without knowledge of specific domain? Are you going to test the BFSI applications (Banking, Financial Services and

105

TESTING CONCEPTS Insurance) just for UI or functionality or security or load or stress? You should know what are the user requirements in banking, working procedures, commerce background, exposure to brokerage etc and should test application accordingly, then only you can say that your testing is enough - Here comes the need of subjectmatter experts. Lets take example of my current project: I am currently working on search engine application. Where I need to know the basic of search engine terminologies and concepts. Many times I see some other team testers asking me questions like what is publishers and advertisers, what is the difference and what they do? Do you think they can test the application based on current online advertising and SEO? Certainly not. Unless and until they get well familiar with these terminologies and functionalities. When I know the functional domain better I can better write and execute more test cases and can effectively simulate the end user actions which is distinctly a big advantage. Here is the big list of the required testing knowledge: Testing skill Bug hunting skill Technical skill Domain knowledge Communication skill Automation skill Some programming skill Quick grasping Ability to Work under pressure

That is going to be a huge list. So you will certainly say, do I need to have these many skills? Its depends on you. You can stick to one skill or can be expert in one skill and have good understanding of other skills or balanced approach of all the skills. This is the competitive market and you should definitely take advantage of it. Make sure to be expert in at least one domain before making any move.

106

TESTING CONCEPTS What if you dont have enough domain knowledge? You will be posted on any project and company can assign any work to you. Then what if you dont have enough domain knowledge of that project? You need to quickly grasp as many concepts as you can. Try to understand the product as if you are the customer and what customer will do with application. Visit the customer site if possible know how they work with the product, Read online resources about the domain you want to test the application, participate in events addressing on such domain, meet the domain experts. Or either company will provide all this in-house training before assigning any domain specific task to testers. There is no specific stage where you need this domain knowledge. You need to apply your domain knowledge in each and every software testing life cycle. How to get your all bugs resolved without any Invalid bug label? I hate Invalid bug label from developers for the bugs reported by me, do you? I think every tester should try to get his/her 100% bugs resolved. This requires bug reporting skill. See my previous post on How to write a good bug report? Tips and Tricks to report bugs professionally and without any ambiguity. The main reason for bug being marked as invalid is Insufficient troubleshooting by tester before reporting the bug. In this post I will focus only on troubleshooting to find main cause of the bug. Troubleshooting will help you to decide whether the ambiguity you found in your application under test is really a bug or any test setup mistake. Yes, 50% bugs get marked as Invalid bugs only due to testers incomplete testing setup. Lets say you found an ambiguity in application under test. You are now preparing the steps to report this ambiguity as a bug. But wait! Have you done enough troubleshooting before reporting this bug? Or have you confirmed if it is really a bug?

107

TESTING CONCEPTS

What troubleshooting you need to perform before reporting any bug? Troubleshooting of: Whats not working? Why its not working? How can you make it work? What are the possible reasons for the failure?

Answer for the first question whats not working? is sufficient for you to report the bug steps in bug tracking system. Then why to answer remaining three questions? Think beyond your responsibilities. Act smarter, dont be a dumb person who only follow his routine steps and dont even think outside of that. You should be able to suggest all possible solutions to resolve the bug and efficiency as well as drawbacks of each solution. This will increase your respect in your team and will also reduce the possibility of getting your bugs rejected, not due to this respect but due to your troubleshooting skill. Before reporting any bug, make sure it isnt your mistake while testing, you have missed any important flag to set or you might have not configured your test setup properly. Troubleshoot the reasons for the failure in application. On proper troubleshooting report the bug. I have complied a troubleshooting list. Check it out - what can be different reasons for failure. Reasons of failure: 1) If you are using any configuration file for testing your application then make sure this file is up to date as per the application requirements: Many times some global configuration file is used to pick or set some application flags. Failure to maintain this file as per your software requirements will lead to malfunctioning of your application under test. You cant report it as bug. 2) Check if your database is proper: Missing table is main reason that your application will not work properly.

108

TESTING CONCEPTS I have a classic example for this: One of my projects was querying many monthly user database tables for showing the user reports. First table existence was checked in master table (This table was maintaining only monthly table names) and then data was queried from different individual monthly tables. Many testers were selecting big date range to see the user reports. But many times it was crashing the application as those tables were not present in database of test machine server, giving SQL query error and they were reporting it as bug which subsequently was getting marked as invalid by developers. 3) If you are working on automation testing project then debug your script twice before coming to conclusion that the application failure is a bug. 4) Check if you are not using invalid access credentials for authentication. 5) Check if software versions are compatible. 6) Check if there is any other hardware issue that is not related to your application. 7) Make sure your application hardware and software prerequisites are correct. 8 ) Check if all software components are installed properly on your test machine. Check whether registry entries are valid. 9) For any failure look into system event viewer for details. You can trace out many failure reasons from system event log file. 10) Before starting to test make sure you have uploaded all latest version files to your test environment. How to write a good bug report? Tips and Tricks Why good Bug report? If your bug report is effective, chances are higher that it will get fixed. So fixing a bug depends on how effectively you report it. Reporting a bug is nothing but a skill and I will tell you how to achieve this skill. The point of writing problem report(bug report) is to get bugs fixed - By Cem Kaner. If tester is not reporting bug correctly, programmer will most likely

109

TESTING CONCEPTS reject this bug stating as irreproducible. This can hurt testers moral and some time ego also. (I suggest do not keep any type of ego. Egos like I have reported bug correctly, I can reproduce it, Why he/she has rejected the bug?, Its not my fault etc etc..) What are the qualities of a good software bug report? Anyone can write a bug report. But not everyone can write a effective bug report. You should be able to distinguish between average bug report and a good bug report. How to distinguish a good or bad bug report? Its simple, apply following characteristics and techniques to report a bug. 1) Having clearly specified bug number: Always assign a unique number to each bug report. This will help to identify the bug record. If you are using any automated bug-reporting tool then this unique number will be generated automatically each time you report the bug. Note the number and brief description of each bug you reported. 2) Reproducible: If your bug is not reproducible it will never get fixed. You should clearly mention the steps to reproduce the bug. Do not assume or skip any reproducing step. Step by step described bug problem is easy to reproduce and fix. 3) Be Specific: Do not write a essay about the problem. Be Specific and to the point. Try to summarize the problem in minimum words yet in effective way. Do not combine multiple problems even they seem to be similar. Write different reports for each problem. How to Report a Bug? Use following simple Bug report template: This is a simple bug report format. It may vary on the bug report tool you are using. If you are writing bug report manually then some fields need to specifically mention like Bug number which should be assigned manually. Reporter: Your name and email address.

110

TESTING CONCEPTS Product: In which product you found this bug. Version: The product version if any. Component: These are the major sub modules of the product. Platform: Mention the hardware platform where you found this bug. The various platforms like PC, MAC, HP, Sun etc. Operating system: Mention all operating systems where you found the bug. Operating systems like Windows, Linux, Unix, SunOS, Mac OS. Mention the different OS versions also if applicable like Windows NT, Windows 2000, Windows XP etc. Priority: When bug should be fixed? Priority is generally set from P1 to P5. P1 as fix the bug with highest priority and P5 as Fix when time permits. Severity: This describes the impact of the bug. Types of Severity: Blocker: No further testing work can be done. Critical: Application crash, Loss of data. Major: Major loss of function. Minor: minor loss of function. Trivial: Some UI enhancements. Enhancement: Request for new feature or some enhancement in existing one.

Status: When you are logging the bug in any bug tracking system then by default the bug status is New. Later on bug goes through various stages like Fixed, Verified, Reopen, Wont Fix etc. Click here to read more about detail bug life cycle. Assign To: If you know which developer is responsible for that particular module in which bug occurred, then you can specify email address of that developer. Else keep it blank

111

TESTING CONCEPTS this will assign bug to module owner or Manger will assign bug to developer. Possibly add the manager email address in CC list. URL: The page url on which bug occurred. Summary: A brief summary of the bug mostly in 60 or below words. Make sure your summary is reflecting what the problem is and where it is. Description: A detailed description of bug. Use following fields for description field: Reproduce steps: Clearly mention the steps to reproduce the bug. Expected result: How application should behave on above mentioned steps. Actual result: What is the actual result on running above steps i.e. the bug behavior. These are the important steps in bug report. You can also add the Report type as one more field which will describe the bug type. The report types are typically: 1) Coding error 2) Design error 3) New suggestion 4) Documentation issue 5) Hardware problem Some Bonus tips to write a good bug report: 1) Report the problem immediately: If you found any bug while testing, do not wait to write detail bug report later. Instead write the bug report immediately. This will ensure a good and reproducible bug report. If you decide to write the bug report later on then chances are high to miss the important steps in your report. 2) Reproduce the bug three times before writing bug report:Your bug should be reproducible. Make sure your steps are robust enough to reproduce the bug

112

TESTING CONCEPTS without any ambiguity. If your bug is not reproducible every time you can still file a bug mentioning the periodic nature of the bug. 3) Test the same bug occurrence on other similar module: Sometimes developer use same code for different similar modules. So chances are high that bug in one module can occur in other similar modules as well. Even you can try to find more severe version of the bug you found. 4) Write a good bug summary: Bug summary will help developers to quickly analyze the bug nature. Poor quality report will unnecessarily increase the development and testing time. Communicate well through your bug report summary. Keep in mind bug summary is used as a reference to search the bug in bug inventory. 5) Read bug report before hitting Submit button: Read all sentences, wording, steps used in bug report. See if any sentence is creating ambiguity that can lead to misinterpretation. Misleading words or sentences should be avoided in order to have a clear bug report. 6) Do not use Abusive language: Its nice that you did a good work and found a bug but do not use this credit for criticizing developer or to attack any individual. Conclusion: No doubt that your bug report should be a high quality document. Focus on writing good bug reports, spend some time on this task because this is main communication point between tester, developer and manager. Mangers should make aware to their team that writing a good bug report is primary responsibility of any tester. Your efforts towards writing good bug report will not only save company resources but also create a good relationship between you and developers.

113

TESTING CONCEPTS How to write software Testing Weekly Status Report Writing effective status report is as important as the actual work you did! How to write a effective status report of your weekly work at the end of each week? Here I am going to give some tips. Weekly report is important to track the important project issues, accomplishments of the projects, pending work and milestone analysis. Even using these reports you can track the team performance to some extent. From this report prepare future actionables items according to the priorities and make the list of next weeks actionable. So how to write weekly status report? Follow the below template: Prepared By: Project: Date of preparation: Status: A) Issues: Issues holding the QA team from delivering on schedule: Project: Issue description: Possible solution: Issue resolution date: You can mark these issues in red colour. These are the issues that requires managements help in resolving them. Issues that management should be aware: These are the issues that not hold the QA team from delivering on time but management should be aware of them. Mark these issues in Yellow colour. You can use above same template to report them. Project accomplishments: Mark them in Green colour. Use below template.

114

TESTING CONCEPTS Project: Accomplishment: Accomplishment date: B) Next week Priorities: Actionable items next week list them in two categories: 1) Pending deliverables: Mark them in blue color: These are previous weeks deliverables which should get released as soon as possible in this week. Project: Work update: Scheduled date: Reason for extending: 2) New tasks: List all next weeks new task here. You can use black color for this. Project: Scheduled Task: Date of release: C) Defect status: Active defects: List all active defects here with Reporter, Module, Severity, priority, assigned to. Closed Defects: List all closed defects with Reporter, Module, Severity, priority, assigned to. Test cases: List total number of test cases wrote, test cases passed, test cases failed, test cases to be executed. This template should give you the overall idea of the status report. Dont ignore the status report. Even if your managers are not forcing you to write these reports they are most important for your work assessment in future.

115

TESTING CONCEPTS How to hire the right candidates for software testing positions? Do companies really judge candidates testing ability in interviews? Do they ask the questions that really judge the candidates skill? What questions should be asked to judge the candidate for software testing field? What is the key process to hire good candidates for software testing positions? Ok, I am asking to many questions without giving answer to any of it. Well, each question mentioned above will require a separate post to address the problem fairly. Here we will address in short about - How to hire the right candidates for software testing positions? Companies or interviewers, who are not serious about hiring right candidates, often end with hiring poor performers. What I mean by Not serious here? - They dont know why and for what post they are hiring a candidate. - They either fake or fail to post the exact job opening details. - Or they dont want to hire skilled performers at all. Hmm, jealousy might be the key here! Whichever is the reason, there is definitely loss of organization. Loss in terms of both revenue and growth. If you need answer to these questions, here is an informative video from Pradeep Soundararajan - Consulting tester of Satisfice Inc in India. He explained what is the current situation of software testing interview process in India and how interviewers are wrong in selecting questions to be asked to candidates. A nice start to spread the awareness and importance of software testing interviews. Website Cookie Testing, Test cases for testing web application cookies? We will first focus on what exactly cookies are and how they work. It would be easy for you to understand the test cases for testing cookies when you have clear understanding of how cookies work? How cookies stored on hard drive? And how can we edit cookie settings?

116

TESTING CONCEPTS What is Cookie? Cookie is small information stored in text file on users hard drive by web server. This information is later used by web browser to retrieve information from that machine. Generally cookie contains personalized user data or information that is used to communicate between different web pages. Why Cookies are used? Cookies are nothing but the users identity and used to track where the user navigated throughout the web site pages. The communication between web browser and web server is stateless. For example if you are accessing domain http://www.example.com/1.html then web browser will simply query to example.com web server for the page 1.html. Next time if you type page as http://www.example.com/2.html then new request is send to example.com web server for sending 2.html page and web server dont know anything about to whom the previous page 1.html served. What if you want the previous history of this user communication with the web server? You need to maintain the user state and interaction between web browser and web server somewhere. This is where cookie comes into picture. Cookies serve the purpose of maintaining the user interactions with web server. How cookies work? The HTTP protocol used to exchange information files on the web is used to maintain the cookies. There are two types of HTTP protocol. Stateless HTTP and State ful HTTP protocol. Stateless HTTP protocol does not keep any record of previously accessed web page history. While State ful HTTP protocol do keep some history of previous web browser and web server interactions and this protocol is used by cookies to maintain the user interactions. Whenever user visits the site or page that is using cookie, small code inside that HTML page (Generally a call to some language script to write the cookie like cookies in JAVA Script, PHP, Perl) writes a text file on users machine called cookie. Here is one example of the code that is used to write cookie and can be placed inside any HTML page: Set-Cookie: NAME=VALUE; expires=DATE; path=PATH; domain=DOMAIN_NAME;

117

TESTING CONCEPTS When user visits the same page or domain later time this cookie is read from disk and used to identify the second visit of the same user on that domain. Expiration time is set while writing the cookie. This time is decided by the application that is going to use the cookie. Generally two types of cookies are written on user machine. 1) Session cookies: This cookie is active till the browser that invoked the cookie is open. When we close the browser this session cookie gets deleted. Some time session of say 20 minutes can be set to expire the cookie. 2) Persistent cookies: The cookies that are written permanently on user machine and lasts for months or years. Where cookies are stored? When any web page application writes cookie it get saved in a text file on user hard disk drive. The path where the cookies get stored depends on the browser. Different browsers store cookie in different paths. E.g. Internet explorer store cookies on path C:\Documents and Settings\Default User\Cookies Here the Default User can be replaced by the current user you logged in as. Like Administrator, or user name like Vijay etc. The cookie path can be easily found by navigating through the browser options. In Mozilla Fire fox browser you can even see the cookies in browser options itself. Open the Mozilla browser, click on Tools->Options->Privacy and then Show cookies button. How cookies are stored? Lets take example of cookie written by rediff.com on Mozilla Firefox browser: On Mozilla Fire fox browser when you open the page rediff.com or login to your rediffmail account, a cookie will get written on your Hard disk. To view this cookie simply click on Show cookies button mentioned on above path. Click on Rediff.com site under this cookie list. You can see different cookies written by rediff domain with different names. Site: Rediff.com Cookie name: RMID Name: RMID (Name of the cookie) Content: 1d11c8ec44bf49e0 (Encrypted content) Domain: .rediff.com

118

TESTING CONCEPTS Path: / (Any path after the domain name) Send For: Any type of connection Expires: Thursday, December 31, 2020 11:59:59 PM Applications where cookies can be used: 1) To implement shopping cart: Cookies are used for maintaining online ordering system. Cookies remember what user wants to buy. What if user adds some products in their shopping cart and if due to some reason user dont want to buy those products this time and closes the browser window? When next time same user visits the purchase page he can see all the products he added in shopping cart in his last visit. 2) Personalized sites: When user visits certain pages they are asked which pages they dont want to visit or display. User options are get stored in cookie and till the user is online, those pages are not shown to him. 3) User tracking: To track number of unique visitors online at particular time. 4) Marketing: Some companies use cookies to display advertisements on user machines. Cookies control these advertisements. When and which advertisement should be shown? What is the interest of the user? Which keywords he searches on the site? All these things can be maintained using cookies. 5) User sessions: Cookies can track user sessions to particular domain using user ID and password. Drawbacks of cookies: 1) Even writing Cookie is a great way to maintain user interaction, if user has set browser options to warn before writing any cookie or disabled the cookies completely then site containing cookie will be completely disabled and can not perform any operation resulting in loss of site traffic.

119

TESTING CONCEPTS 2) Too many Cookies: If you are writing too many cookies on every page navigation and if user has turned on option to warn before writing cookie, this could turn away user from your site. 3) Security issues: Some times users personal information is stored in cookies and if someone hack the cookie then hacker can get access to your personal information. Even corrupted cookies can be read by different domains and lead to security issues. 4) Sensitive information: Some sites may write and store your sensitive information in cookies, which should not be allowed due to privacy concerns. This should be enough to know what cookies are. If you want more cookie info see Cookie Central page. Some Major Test cases for web application cookie testing: The first obvious test case is to test if your application is writing cookies properly on disk. You can use the Cookie Tester application also if you dont have any web application to test but you want to understand the cookie concept for testing. Test cases: 1) As a Cookie privacy policy make sure from your design documents that no personal or sensitive data is stored in the cookie. 2) If you have no option than saving sensitive data in cookie make sure data stored in cookie is stored in encrypted format. 3) Make sure that there is no overuse of cookies on your site under test. Overuse of cookies will annoy users if browser is prompting for cookies more often and this could result in loss of site traffic and eventually loss of business. 4) Disable the cookies from your browser settings: If you are using cookies on your site, your sites major functionality will not work by disabling the cookies. Then try to access the web site under test. Navigate through the site. See if appropriate messages are displayed to user like For smooth functioning of this site make sure

120

TESTING CONCEPTS that cookies are enabled on your browser. There should not be any page crash due to disabling the cookies. (Please make sure that you close all browsers, delete all previously written cookies before performing this test) 5) Accepts/Reject some cookies: The best way to check web site functionality is, not to accept all cookies. If you are writing 10 cookies in your web application then randomly accept some cookies say accept 5 and reject 5 cookies. For executing this test case you can set browser options to prompt whenever cookie is being written to disk. On this prompt window you can either accept or reject cookie. Try to access major functionality of web site. See if pages are getting crashed or data is getting corrupted. 6) Delete cookie: Allow site to write the cookies and then close all browsers and manually delete all cookies for web site under test. Access the web pages and check the behavior of the pages. 7) Corrupt the cookies: Corrupting cookie is easy. You know where cookies are stored. Manually edit the cookie in notepad and change the parameters to some vague values. Like alter the cookie content, Name of the cookie or expiry date of the cookie and see the site functionality. In some cases corrupted cookies allow to read the data inside it for any other domain. This should not happen in case of your web site cookies. Note that the cookies written by one domain say rediff.com cant be accessed by other domain say yahoo.com unless and until the cookies are corrupted and someone trying to hack the cookie data. 8 ) Checking the deletion of cookies from your web application page: Some times cookie written by domain say rediff.com may be deleted by same domain but by different page under that domain. This is the general case if you are testing some action tracking web portal. Action tracking or purchase tracking pixel is placed on the action web page and when any action or purchase occurs by user the cookie written on disk get deleted to avoid multiple action logging from same cookie. Check if reaching to your action or purchase page deletes the cookie properly and no more invalid actions or purchase get logged from same user. 9) Cookie Testing on Multiple browsers: This is the important case to check if your web application page is writing the cookies properly on different browsers as intended and site works properly using these cookies. You can test your web

121

TESTING CONCEPTS application on Major used browsers like Internet explorer (Various versions), Mozilla Fire fox, Netscape, Opera etc. 10) If your web application is using cookies to maintain the logging state of any user then log in to your web application using some username and password. In many cases you can see the logged in user ID parameter directly in browser address bar. Change this parameter to different value say if previous user ID is 100 then make it 101 and press enter. The proper access message should be displayed to user and user should not be able to see other users account. Software Installation/Uninstallation Testing Have you performed software installation testing? How was the experience? Well, Installation testing (Implementation Testing) is quite interesting part of software testing life cycle. Installation testing is like introducing a guest in your home. The new guest should be properly introduced to all the family members in order to feel him comfortable. Installation of new software is also quite like above example. If your installation is successful on the new system then customer will be definitely happy but what if things are completely opposite. If installation fails then our program will not work on that system not only this but can leave users system badly damaged. User might require to reinstall the full operating system. In above case will you make any impression on user? Definitely not! Your first impression to make a loyal customer is ruined due to incomplete installation testing. What you need to do for a good first impression? Test the installer appropriately with combination of both manual and automated processes on different machines with different configuration. Major concerned of installation testing is Time! It requires lot of time to even execute a single test case. If you are going to test a big application installer then think about time required to perform such a many test cases on different configurations.

122

TESTING CONCEPTS We will see different methods to perform manual installer testing and some basic guideline for automating the installation process. To start installation testing first decide on how many different system configurations you want to test the installation. Prepare one basic hard disk drive. Format this HDD with most common or default file system, install most common operating system (Windows) on this HDD. Install some basic required components on this HDD. Each time create images of this base HDD and you can create other configurations on this base drive. Make one set of each configuration like Operating system and file format to be used for further testing. How we can use automation in this process? Well make some systems dedicated for creating basic images (use softwares like Norton Ghost for creating exact images of operating system quickly) of base configuration. This will save your tremendous time in each test case. For example if time to install one OS with basic configuration is say 1 hour then for each test case on fresh OS you will require 1+ hour. But creating image of OS will hardly require 5 to 10 minutes and you will save approximately 40 to 50 minutes! You can use one operating system with multiple attempts of installation of installer. Each time uninstalling the application and preparing the base state for next test case. Be careful here that your uninstallation program should be tested before and should be working fine.

123

TESTING CONCEPTS Installation testing tips with some broad test cases: 1) Use flow diagrams to perform installation testing. Flow diagrams simplify our task. See example flow diagram for basic installation testing test case. Add some more test cases on this basic flow chart Such as if our application is not the first release then try to add different logical installation paths. 2) If you have previously installed compact basic version of application then in next test case install the full application version on the same path as used for compact version. 3) If you are using flow diagram to test different files to be written on disk while installation then use the same flow diagram in reverse order to test uninstallation of all the installed files on disk. 4) Use flow diagrams to automate the testing efforts. It will be very easy to convert diagrams into automated scripts. 5) Test the installer scripts used for checking the required disk space. If installer is prompting required disk space 1MB, then make sure exactly 1MB is used or whether more disk space utilized during installation. If yes flag this as error. 6) Test disk space requirement on different file system format. Like FAT16 will require more space than efficient NTFS or FAT32 file systems. 7) If possible set a dedicated system for only creating disk images. As said above this will save your testing time.

124

TESTING CONCEPTS 8 ) Use distributed testing environment in order to carry out installation testing. Distributed environment simply save your time and you can effectively manage all the different test cases from a single machine. The good approach for this is to create a master machine, which will drive different slave machines on network. You can start installation simultaneously on different machine from the master system. 9) Try to automate the routine to test the number of files to be written on disk. You can maintain this file list to be written on disk in and excel sheet and can give this list as a input to automated script that will check each and every path to verify the correct installation. 10) Use softwares available freely in market to verify registry changes on successful installation. Verify the registry changes with your expected change list after installation. 11) Forcefully break the installation process in between. See the behavior of system and whether system recovers to its original state without any issues. You can test this break of installation on every installation step. 12) Disk space checking: This is the crucial checking in the installation-testing scenario. You can choose different manual and automated methods to do this checking. In manual methods you can check free disk space available on drive before installation and disk space reported by installer script to check whether installer is calculating and reporting disk space accurately. Check the disk space after the installation to verify accurate usage of installation disk space. Run various combination of disk space availability by using some tools to automatically making disk space full while installation. Check system behavior on low disk space conditions while installation. 13) As you check installation you can test for uninstallation also. Before each new iteration of installation make sure that all the files written to disk are removed after uninstallation. Some times uninstallation routine removes files from only last upgraded installation keeping the old version files untouched. Also check for rebooting option after uninstallation manually and forcefully not to reboot. I have addressed many areas of manual as well as automated installation testing procedure. Still there are many areas you need to focus on depending on

125

TESTING CONCEPTS the complexity of your software under installation. These not addressed important tasks includes installation over the network, online installation, patch installation, Database checking on Installation, Shared DLL installation and uninstallation etc. What are the Quality attributes? First in brief know what is Quality? Quality can be define in different manner. Quality definition may differ from person to person. But finally there should be some standards. So Quality can be defined as Degree of excellence - Oxford dictionary Fitness for purpose - Edward Deming Best for the customers use and selling price - Feigenbaum The totality of characteristics of an entity that bear on its ability to satisfy stated or implied needs - ISO How a Product developer will define quality - The product which meets the customer requirements. How Customer will define Quality - Required functionality is provided with user friendly manner. These are some quality definitions from different perspective. Now lets see how can one measure some quality attributes of product or application. Following factors are used to measure software development quality. Each attribute can be used to measure the product performance. These attributes can be used for Quality assurance as well as Quality control. Quality Assurance activities are oriented towards prevention of introduction of defects and Quality control activities are aimed at detecting defects in products and services. Reliability Measure if product is reliable enough to sustain in any condition. Should give consistently correct results. Product reliability is measured in terms of working of project under different working environment and different conditions.

126

TESTING CONCEPTS Maintainability Different versions of the product should be easy to maintain. For development its should be easy to add code to existing system, should be easy to upgrade for new features and new technologies time to time. Maintenance should be cost effective and easy. System be easy to maintain and correcting defects or making a change in the software. Usability This can be measured in terms of ease of use. Application should be user friendly. Should be easy to learn. Navigation should be simple. The system must be: Easy to use for input preparation, operation, and interpretation of output. Provide consistent user interface standards or conventions with our other frequently used systems. Easy for new or infrequent users to learn to use the system.

Portability This can be measured in terms of Costing issues related to porting, Technical issues related to porting, Behavioral issues related to porting. Correctness Application should be correct in terms of its functionality, calculations used internally and the navigation should be correct. This means application should adhere to functional requirements. Efficiency To Major system quality attribute. Measured in terms of time required to complete any task given to the system. For example system should utilize processor capacity, disk space and memory efficiently. If system is using all the available resources then user will get degraded performance failing the system for efficiency. If system is not efficient then it can not be used in real time applications. Integrity or security Integrity comes with security. System integrity or security should be sufficient to prevent unauthorized access to system functions, preventing information loss,

127

TESTING CONCEPTS ensure that the software is protected from virus infection, and protecting the privacy of data entered into the system. Testability System should be easy to test and find defects. If required should be easy to divide in different modules for testing. Flexibility Should be flexible enough to modify. Adaptable to other products with which it needs interaction. Should be easy to interface with other standard 3rd party components. Reusability Software reuse is a good cost efficient and time saving development way. Different code libraries classes should be generic enough to use easily in different application modules. Dividing application into different modules so that modules can be reused across the application. Interoperability Interoperability of one system to another should be easy for product to exchange data or services with other systems. Different system modules should work on different operating system platforms, different databases and protocols conditions. Applying above quality attributes standards we can determine whether system meets the requirements of quality or not. As specified above all these attributes are applied on QA and QC process so that tester or customer also can find quality of application or system. Developers are not good testers. What you say? This can be a big debate. Developers testing their own code - what will be the testing output? All happy endings! Yes, the person who develops the code generally sees only happy paths of the product and dont want to go in much details. The main concern of developer testing is - misunderstanding of requirements. If requirements are misunderstood by developer then no matter at what depth

128

TESTING CONCEPTS developer test the application, he will never find the error. The first place where the bug gets introduced will remain till end, as developer will see it as functionality. Optimistic developers - Yes, I wrote the code and I am confident its working properly. No need to test this path, no need to test that path, as I know its working properly. And right here developers skip the bugs. Developer vs. Tester: Developer always wants to see his code working properly. So he will test it to check if its working correctly. But you know why tester will test the application? To make it fail in any way, and tester surely will test how application is not working correctly. This is the main difference in developer testing and tester testing. Should developers test their own work? I personally dont mind developers testing their own code. After all its there baby They

know their code very well. They know what are the traps in their codes. Where it can fail, where to concentrate more, which is important path of the application. Developer can do unit testing very well and can effectively identify boundary cases. (Image credit) This is all applicable to a developer who is a good tester! But most of the developers consider testing as painful job, even they know the system well, due to their negligence they tend to skip many testing paths, as its a very painful experience for them. If developers find any errors in their code in unit testing then its comparatively easier to fix, as the code is fresh to them, rather than getting the bug from testers after two-three days. But this only possible if the developer is interested in doing that much testing. Its testers responsibility to make sure each and every path is tested or not. Testers should ideally give importance to all small possible details to verify application is not breaking anywhere.

129

TESTING CONCEPTS Developers, please dont review your own code. Generally you will overlook the issues in your code. So give it to others for review. Everyone is having specialization in particular subject. Developers generally think how to develop the application on the other hand testers think how the end user is going to use the application. Conclusion So in short there is no problem if developers are doing the basic unit testing and basic verification testing. Developers can test few exceptional conditions they know are critical and should not be missed. But there are some great testers out there. Through the build to test team. Dont waste your time as well. For success of any project there should be independent testing team validating your applications. After all its our (testers) responsibility to make the baby smarter!! Living life as a Software Tester! I will extract only points related to software testing. As a software tester keep in mind these simple points: Share everything: If you are a experienced tester on any project then help the new developers on your project. Some testers have habit to keep the known bugs hidden till they get implement in code and then they write a big defect report on that. Dont try to only pump your bug count, share everything with developers. Build trust: Let the developers know any bug you found in design phase. Do not log the bug repeatedly with small variations just to pump the bug count. Build trust in developer and tester relation. Dont blame others: As a tester you should not always blame developers for the bugs. Concentrate on bug, not always on pointing that bug in front of all people. Hit the bug and its cause not the developer!

130

TESTING CONCEPTS Clean up your own mess: When you finish doing any test scenario then reconfigure that machine to its original configuration. The same case applies for bug report. Write a clean effective bug report. Let the developer find it easy to repro and fix it. Give credit to others for their work: Do not take others credit. If you have referred any others work, immediately give credit to that person. Do not get frustrated if you not found any bug that later has been reported by client. Do work hard, use your skill. Remember to flush Like the toilets flush all the softwares at some point. While doing performance testing remember to flush the system cache. Take a nap everyday: We need time to think, get refresh or to regenerate our energy. Some times its important to take one step back in order to get fresh insight and to find different working approach. How to be a good tester? Its a every testers question. How to be a good tester? Apart from the technical knowledge, testing skills, tester should have some personal level skills which will help them to build a good rapport in the testing team. What are these abilities , skills which make a tester as a good tester? Well, I was reading Dave Whalens article Ugly Baby Syndrome! and found it very interesting. Dave compared software developers with the parents who deliver a baby (software) with countless efforts. Naturally the product managers, architectures, developers spent their countless time on developing application for the customer. Then they show it to us (testers) and asks: How is the baby (Application)? And testers tell them often that they have and ugly baby. (Application with Bugs!) Testers dont want to tell them that they have ugly baby, but unfortunately its our job. So effectively tester can convey the message to the developers without hurting them. How can be this done? Ya that is the skill of a good tester!

131

TESTING CONCEPTS Here are the tips sated by Dave to handle such a delicate situation: Be honest and Responsive: Tell developers what are your plans to attack their application. Be open and available: If any dev ask you to have a look at the application developed by him before the release, then politely give feedback on it and report any extra efforts needed. Dont log the bugs for these notes. Let them review your tests: If you have designed or wrote some test cases from the requirement specifications then just show them those test cases. Let them know your stuff as you are going to critic on developers work! Use of Bug tracker: Some testers have habit to report each and everything publicly. This attitude hurts the developers. So if you have logged any bug then let the bug tracking system report it to respective developers and managers. Also dont each time rely on bug tracker, talk personally to developers what you logged and why you logged? Finally some good personal points: Dont take it personally: Do the job of messenger. You could be a close target always. So build a thick skin! Be prepared: A good message in the end, Be prepared for everything! If worst things might not happened till now but they can happen at any moment in your career. So be ready to face them. Need of Skilled Testers Some years ago many companies preferred not to have separate test engineers in the project team. But this I have not seen in past 2-3 years in my career. As now all the companies have a clear idea of the need of the QA and test

132

TESTING CONCEPTS engineers. Also the QA and testers rolls are now concrete and there is no confusion. Unfortunately, still I find the perception of testing as a inferior role in some developers mind. This anyone can do attitude should be removed from those peoples mind. Lots of companies hiring any skilled personals to do this job and eventually suffering from the lost of money and time. Instead of hiring the junk of testers they should hire some gifted testers who can do there job beyond the developers limitations. If Managers and management remove this inferiority thinking from their mind then they can hire these gifted testers in their organization. Such testers can do complex job well, can find complex bugs and further more can add some procedures to the way of doing the routine jobs in order to make it more structured. Effective Software Testing In this tutorial you will learn about Effective Software Testing? How do we measure Effectiveness of Software Testing? Steps to Effective Software Testing, Coverage and Test Planning and Process. A 1994 study in US revealed that only about 9% of software projects were successful

A large number of projects upon completion do not have all the promised features or they do not meet all the requirements that were defined when the project was kicked off. It is an understatement to say that An increasing number of businesses depend on the software for their day-to-day businesses. Billions of Dollars change hands every day with the help of commercial software. Lots of lives depend on the reliability of the software for example running critical medical systems, controlling power plants, flying air planes and so on.

133

TESTING CONCEPTS Whether you are part of a team that is building a book keeping application or a software that runs a power plant you cannot afford to have less than reliable software. Unreliable software can severely hurt businesses and endanger lives depending on the criticality of the application. The simplest application poorly written can deteriorate the performance of your environment such as the servers, the network and thereby causing an unwanted mess. To ensure software application reliability and project success Software Testing plays a very crucial role. Everything can and should be tested Test if all defined requirements are met Test the performance of the application Test each component Test the components integrated with each other Test the application end to end Test the application in various environments Test all the application paths Test all the scenarios and then test some more

What is Effective Software Testing? How do we measure Effectiveness of Software Testing? The effectiveness of Testing can be measured if the goal and purpose of the testing effort is clearly defined. Some of the typical Testing goals are: Testing in each phase of the Development cycle to ensure that the bugs(defects) are eliminated at the earliest Testing to ensure no bugs creep through in the final product Testing to ensure the reliability of the software Above all testing to ensure that the user expectations are met

The effectiveness of testing can be measured with the degree of success in achieving the above goals.

134

TESTING CONCEPTS Steps to Effective Software Testing: Several factors influence the effectiveness of Software Testing Effort, which ultimately determine the success of the Project. A) Coverage: The testing process and the test cases should cover All the scenarios that can occur when using the software application Each business requirement that was defined for the project Specific levels of testing should cover every line of code written for the application There are various levels of testing which focus on different aspects of the software application. The often-quoted V model best explains this:

The various levels of testing illustrated above are: Unit Testing

135

TESTING CONCEPTS Integration Testing System Testing User Acceptance Testing

The goal of each testing level is slightly different thereby ensuring the overall project reliability. Each Level of testing should provide adequate test coverage.

Unit testing should ensure each and every line of code is tested.

Integration Testing should ensure the components can be integrated and all the interfaces of each component are working correctly.

System Testing should cover all the paths/scenarios possible when using the system.

The system testing is done in an environment that is similar to the production environment i.e. the environment where the product will be finally deployed. There are various types of System Testing possible which test the various aspects of the software application. B) Test Planning and Process: To ensure effective Testing Proper Test Planning is important An Effective Testing Process will comprise of the following steps: Test Strategy and Planning Review Test Strategy to ensure its aligned with the Project Goals Design/Write Test Cases Review Test Cases to ensure proper Test Coverage Execute Test Cases Capture Test Results Track Defects

136

TESTING CONCEPTS Capture Relevant Metrics Analyze

Having followed the above steps for various levels of testing the product is rolled. It is not uncommon to see various bugs/Defects even after the product is released to production. An effective Testing Strategy and Process helps to minimize or eliminate these defects. The extent to which it eliminates these post-production defects (Design Defects/Coding Defects/etc) is a good measure of the effectiveness of the Testing Strategy and Process. As the saying goes - 'the proof of the pudding is in the eating' Summary: The success of the project and the reliability of the software application depend a lot on the effectiveness of the testing effort. This article discusses What is effective Software Testing? A link to the 1994 study called The Chaos Report http://www.standishgroup.com/sample_research/chaos_1994_1.php Unit Testing: Why? What? & How? In this tutorial you will learn about unit testing, various levels of testing, various types of testing based upon the intent of testing, How does Unit Testing fit into the Software Development Life Cycle? Unit Testing Tasks and Steps, What is a Unit Test Plan? What is a Test Case? and Test Case Sample, Steps to Effective Unit Testing. There are various levels of testing: Unit Testing Integration Testing System Testing

There are various types of testing based upon the intent of testing such as: Acceptance Testing

137

TESTING CONCEPTS Performance Testing Load Testing Regression Testing

Based on the testing Techniques testing can be classified as: Black box Testing White box Testing

How does Unit Testing fit into the Software Development Life Cycle? This is the first and the most important level of testing. As soon as the programmer develops a unit of code the unit is tested for various scenarios. As the application is built it is much more economical to find and eliminate the bugs early on. Hence Unit Testing is the most important of all the testing levels. As the software project progresses ahead it becomes more and more costly to find and fix the bugs. In most cases it is the developers responsibility to deliver Unit Tested Code. Unit Testing Tasks and Steps: Step 1: Create a Test Plan Step 2: Create Test Cases and Test Data Step 3: If applicable create scripts to run test cases Step 4: Once the code is ready execute the test cases Step 5: Fix the bugs if any and re test the code Step 6: Repeat the test cycle until the unit is free of all bugs What is a Unit Test Plan? This document describes the Test Plan in other words how the tests will be carried out. This will typically include the list of things to be Tested, Roles and Responsibilities, prerequisites to begin Testing, Test Environment, Assumptions, what to do after a test is successfully carried out, what to do if test fails, Glossary and so on

138

TESTING CONCEPTS What is a Test Case? Simply put, a Test Case describes exactly how the test should be carried out. For example the test case may describe a test as follows: Step 1: Type 10 characters in the Name Field Step 2: Click on Submit Test Cases clubbed together form a Test Suite

Test Case Sample Test Case ID Test Case Description Input Data Expected Result Actual Result

Pass/Fail

Remarks

Additionally the following information may also be captured: a) Unit Name and Version Being tested b) Tested By c) Date d) Test Iteration (One or more iterations of unit testing may be performed) Steps to Effective Unit Testing: 1) Documentation: Early on document all the Test Cases needed to test your code. A lot of times this task is not given due importance. Document the Test Cases, actual Results when executing the Test Cases, Response Time of the code for each test case. There are several important advantages if the test cases and the actual execution of test cases are well documented. a. Documenting Test Cases prevents oversight. b. Documentation clearly indicates the quality of test cases c. If the code needs to be retested we can be sure that we did not miss anything

139

TESTING CONCEPTS d. It provides a level of transparency of what was really tested during unit testing. This is one of the most important aspects. e. It helps in knowledge transfer in case of employee attrition f. Sometimes Unit Test Cases can be used to develop test cases for other levels of testing 2) What should be tested when Unit Testing: A lot depends on the type of program or unit that is being created. It could be a screen or a component or a web service. Broadly the following aspects should be considered: a. For a UI screen include test cases to verify all the screen elements that need to appear on the screens b. For a UI screen include Test cases to verify the spelling/font/size of all the labels or text that appears on the screen c. Create Test Cases such that every line of code in the unit is tested at least once in a test cycle d. Create Test Cases such that every condition in case of conditional statements is tested once e. Create Test Cases to test the minimum/maximum range of data that can be entered. For example what is the maximum amount that can be entered or the max length of string that can be entered or passed in as a parameter f. Create Test Cases to verify how various errors are handled g. Create Test Cases to verify if all the validations are being performed 3) Automate where Necessary: Time pressures/Pressure to get the job done may result in developers cutting corners in unit testing. Sometimes it helps to write scripts, which automate a part of unit testing. This may help ensure that the necessary tests were done and may result in saving time required to perform the tests. Summary: Unit Testing is the first level of testing and the most important one. Detecting and fixing bugs early on in the Software Lifecycle helps reduce costly fixes later on. An Effective Unit Testing Process can and should be developed to increase the Software Reliability and credibility of the developer. The Above article explains how Unit

140

TESTING CONCEPTS Testing should be done and the important points that should be considered when doing Unit Testing. Many new developers take the unit testing tasks lightly and realize the importance of Unit Testing further down the road if they are still part of the project. This article serves as a starting point for laying out an effective (Unit) Testing Strategy.

Integration Testing: Why? What? & How? Introduction: As we covered in various articles in the Testing series there are various levels of testing: Unit Testing, Integration Testing, System Testing Each level of testing builds on the previous level. Unit testing focuses on testing a unit of the code. Integration testing is the next level of testing. This level of testing focuses on testing the integration of units of code or components. How does Integration Testing fit into the Software Development Life Cycle? Even if a software component is successfully unit tested, in an enterprise n-tier distributed application it is of little or no value if the component cannot be successfully integrated with the rest of the application. Once unit tested components are delivered we then integrate them together. These integrated components are tested to weed out errors and bugs caused due to the integration. This is a very important step in the Software Development Life Cycle. It is possible that different programmers developed different components. A lot of bugs emerge during the integration step.

141

TESTING CONCEPTS In most cases a dedicated testing team focuses on Integration Testing. Prerequisites for Integration Testing: Before we begin Integration Testing it is important that all the components have been successfully unit tested. Integration Testing Steps: Integration Testing typically involves the following Steps: Step 1: Create a Test Plan Step 2: Create Test Cases and Test Data Step 3: If applicable create scripts to run test cases Step 4: Once the components have been integrated execute the test cases Step 5: Fix the bugs if any and re test the code Step 6: Repeat the test cycle until the components have been successfully integrated What is an Integration Test Plan? As you may have read in the other articles in the series, this document typically describes one or more of the following: - How the tests will be carried out - The list of things to be Tested - Roles and Responsibilities - Prerequisites to begin Testing - Test Environment - Assumptions - What to do after a test is successfully carried out - What to do if test fails - Glossary How to write an Integration Test Case? Simply put, a Test Case describes exactly how the test should be carried out. The Integration test cases specifically focus on the flow of data/information/control from one component to the other.

142

TESTING CONCEPTS So the Integration Test cases should typically focus on scenarios where one component is being called from another. Also the overall application functionality should be tested to make sure the app works when the different components are brought together. The various Integration Test Cases clubbed together form an Integration Test Suite Each suite may have a particular focus. In other words different Test Suites may be created to focus on different areas of the application. As mentioned before a dedicated Testing Team may be created to execute the Integration test cases. Therefore the Integration Test Cases should be as detailed as possible. Sample Test Case Table: Test Case ID

Test Case Description

Input Data

Expected Result

Actual Result

Pass/Fail

Remarks

Additionally the following information may also be captured: a) Test Suite Name b) Tested By c) Date d) Test Iteration (One or more iterations of Integration testing may be performed) Working towards Effective Integration Testing: There are various factors that affect Software Integration and hence Integration Testing: 1) Software Configuration Management: Since Integration Testing focuses on Integration of components and components can be built by different developers and even different development teams, it is important the right version of components are tested. This may sound very basic, but the biggest problem faced in n-tier development is integrating the right version of components. Integration testing may run through several iterations and to fix bugs components may undergo changes. Hence it is important that a good Software Configuration Management (SCM) policy

143

TESTING CONCEPTS is in place. We should be able to track the components and their versions. So each time we integrate the application components we know exactly what versions go into the build process. 2) Automate Build Process where Necessary: A Lot of errors occur because the wrong version of components were sent for the build or there are missing components. If possible write a script to integrate and deploy the components this helps reduce manual errors. 3) Document: Document the Integration process/build process to help eliminate the errors of omission or oversight. It is possible that the person responsible for integrating the components forgets to run a required script and the Integration Testing will not yield correct results. 4) Defect Tracking: Integration Testing will lose its edge if the defects are not tracked correctly. Each defect should be documented and tracked. Information should be captured as to how the defect was fixed. This is valuable information. It can help in future integration and deployment processes. Metrics Used In Testing In this tutorial you will learn about metrics used in testing, The Product Quality Measures - 1. Customer satisfaction index, 2. Delivered defect quantities, 3. Responsiveness (turnaround time) to users, 4. Product volatility, 5. Defect ratios, 6. Defect removal efficiency, 7. Complexity of delivered product, 8. Test coverage, 9. Cost of defects, 10. Costs of quality activities, 11. Re-work, 12. Reliability and Metrics for Evaluating Application System Testing. The Product Quality Measures: 1. Customer satisfaction index This index is surveyed before product delivery and after product delivery (and on-going on a periodic basis, using standard questionnaires).The following are analyzed: Number of system enhancement requests per year

144

TESTING CONCEPTS Number of maintenance fix requests per year User friendliness: call volume to customer service hotline User friendliness: training time per new user Number of product recalls or fix releases (software vendors) Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc. 3. Responsiveness (turnaround time) to users Turnaround time for defect fixes, by level of severity Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality) 5. Defect ratios Defects found after product delivery per function point. Defects found after product delivery per LOC Pre-delivery defects: annual post-delivery defects Defects per function point of the system modifications

6. Defect removal efficiency Number of post-release defects (found by clients in field operation), categorized by level of severity Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects

145

TESTING CONCEPTS All defects include defects found internally plus externally (by customers) in the first year after product delivery

7. Complexity of delivered product McCabe's cyclomatic complexity counts across the system Halsteads measure Card's design complexity measures Predicted defects and maintenance costs, based on complexity measures

8. Test coverage Breadth of functional coverage Percentage of paths, branches or conditions that were actually tested Percentage by criticality level: perceived level of risk of paths The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects Business losses per defect that occurs during operation Business interruption costs; costs of work-arounds Lost sales and lost goodwill Litigation costs resulting from defects Annual maintenance cost (per function point) Annual operating cost (per function point) Measurable damage to your boss's career

10. Costs of quality activities Costs of reviews, inspections and preventive measures Costs of test planning and preparation Costs of test execution, defect tracking, version and change control

146

TESTING CONCEPTS Costs of diagnostics, debugging and fixing Costs of tools and tool support Costs of test case library maintenance Costs of testing & QA education associated with the product Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations) 11. Re-work Re-work effort (hours, as a percentage of the original coding hours) Re-worked LOC (source lines of code, as a percentage of the total delivered LOC) Re-worked software components (as a percentage of the total delivered components) 12. Reliability Availability (percentage of time a system is available, versus the time the system is needed to be available) Mean time between failure (MTBF). Man time to repair (MTTR) Reliability ratio (MTBF / MTTR) Number of product recalls or fix releases Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing: Metric = Formula Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code) Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code). Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria Defects per size = Defects detected / system size

147

TESTING CONCEPTS Test cost (in %) = Cost of testing / total cost *100 Cost to locate defect = Cost of testing / the number of defects located Achieving Budget = Actual cost of testing / Budgeted cost of testing Defects detected in testing = Defects detected in testing / total system defects Defects detected in production = Defects detected in production/system size Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100 Effectiveness of testing to business = Loss due to problems / total resources processed by the system. System complaints = Number of third party complaints / number of transactions processed Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10 Source Code Analysis = Number of source code statements changed / total number of tests. Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation Test Execution Productivity = No of Test cycles executed / Actual Effort for testing Life Cycle of Testing Process This article explains about Different steps in Life Cycle of Testing Process. In Each phase of the development process will have a specific input and a specific output. In the whole development process, testing consumes highest amount of time. But most of the developers oversee that and testing phase is generally neglected. As a consequence, erroneous software is released. The testing team should be involved right from the requirements stage itself.

148

TESTING CONCEPTS The various phases involved in testing, with regard to the software development life cycle are:

1. Requirements stage 2. Test Plan 3. Test Design. 4. Design Reviews 5. Code Reviews 6. Test Cases preparation. 7. Test Execution 8. Test Reports. 9. Bugs Reporting 10. Reworking on patches. 11. Release to production. Requirements Stage Normally in many companies, developers itself take part in the requirements stage. Especially for product-based companies, a tester should also be involved in this stage. Since a tester thinks from the user side whereas a developer cant. A separate panel should be formed for each module comprising a developer, a tester and a user. Panel meetings should be scheduled in order to gather everyones view. All the requirements should be documented properly for further use and this document is called Software Requirements Specifications. Test Plan Without a good plan, no work is a success. A successful work always contains a good plan. The testing process of software should also require good plan. Test plan document is the most important document that brings in a process oriented approach. A test plan document should be prepared after the requirements of the

149

TESTING CONCEPTS project are confirmed. The test plan document must consist of the following information: Total number of features to be tested. Testing approaches to be followed. The testing methodologies Number of man-hours required. Resources required for the whole testing process. The testing tools that are to be used. The test cases, etc Test Design Test Design is done based on the requirements of the project. Test has to be designed based on whether manual or automated testing is done. For automation testing, the different paths for testing are to be identified first. An end to end checklist has to be prepared covering all the features of the project. The test design is represented pictographically. The test design involves various stages. These stages can be summarized as follows: The different modules of the software are identified first. Next, the paths connecting all the modules are identified. Then the design is drawn. The test design is the most critical one, which decides the test case preparation. So the test design assesses the quality of testing process. Test Cases Preparation Test cases should be prepared based on the following scenarios: Positive scenarios Negative scenarios Boundary conditions and Real World scenarios

150

TESTING CONCEPTS The software design is done in systematical manner or using the UML language. The tester can do the reviews over the design and can suggest the ideas and the modifications needed. Code Reviews Code reviews are similar to unit testing. Once the code is ready for release, the tester should be ready to do unit testing for the code. He must be ready with his own unit test cases. Though a developer does the unit testing, a tester must also do it. The developers may oversee some of the minute mistakes in the code, which a tester may find out. Test Execution and Bugs Reporting Once the unit testing is completed and the code is released to QA, the functional testing is done. A top-level testing is done at the beginning of the testing to find out the top-level failures. If any top-level failures occur, the bugs should be reported to the developer immediately to get the required workaround. The test reports should be documented properly and the bugs have to be reported to the developer after the testing is completed. Release to Production Once the bugs are fixed, another release is given to the QA with the modified changes. Regression testing is executed. Once the QA assures the software, the software is released to production. Before releasing to production, another round of top-level testing is done. The testing process is an iterative process. Once the bugs are fixed, the testing has to be done repeatedly. Thus the testing process is an unending process. Technical Terms Used in Testing World In this tutorial you will learn about technical terms used in testing world, from Audit, Acceptance Testng to Validation, Verification and Testing.

151

TESTING CONCEPTS Audit: An independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria. Acceptance testing: Testing conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system. Alpha Testing: Acceptance testing performed by the customer in a controlled environment at the developer's site. The software is used by the customer in a setting approximating the target environment with the developer observing and recording errors and usage problems. Assertion Testing: A dynamic analysis technique which inserts assertions about the relationship between program variables into the program code. The truth of the assertions is determined as the program executes. Boundary Value: (1) A data value that corresponds to a minimum or maximum input, internal, or output value specified for a system or component. (2) A value which lies at, or just inside or just outside a specified range of valid input and output values. Boundary Value Analysis: A selection technique in which test data are chosen to lie along "boundaries" of the input domain [or output range] classes, data structures, procedure parameters, etc. Choices often include maximum, minimum, and trivial values or parameters. Branch Coverage: A test coverage criteria which requires that for each decision point each possible branch be executed at least once. Bug: A fault in a program which causes the program to perform in an unintended or unanticipated manner. Beta Testing: Acceptance testing performed by the customer in a live application of the software, at one or more end user sites, in an environment not controlled by the developer.

152

TESTING CONCEPTS Boundary Value Testing: A testing technique using input values at, just below, and just above, the defined limits of an input domain; and with input values causing outputs to be at, just below, and just above, the defined limits of an output domain. Branch Testing: Testing technique to satisfy coverage criteria which require that for each decision point, each possible branch [outcome] be executed at least once. Contrast with testing, path; testing, statement. See: branch coverage. Compatibility Testing: The process of determining the ability of two or more systems to exchange information. In a situation where the developed software replaces an already working program, an investigation should be conducted to assess possible comparability problems between the new software and other programs or systems. Cause Effect Graph: A Boolean graph linking causes and effects. The graph is actually a digital-logic circuit (a combinatorial logic network) using a simpler notation than standard electronics notation. Cause Effect Graphing: This is a Test data selection technique. The input and output domains are partitioned into classes and analysis is performed to determine which input classes cause which effect. A minimal set of inputs is chosen which will cover the entire effect set. It is a systematic method of generating test cases representing combinations of conditions. Code Inspection: A manual [formal] testing [error detection] technique where the programmer reads source code, statement by statement, to a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards. Code Review: A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Code Walkthrough: A manual testing [error detection] technique where program [source code] logic [structure] is traced manually [mentally] by a group with a small set of test cases, while the state of program variables is manually monitored, to analyze the programmer's logic and assumptions.

153

TESTING CONCEPTS Coverage Analysis: Determining and assessing measures associated with the invocation of program structural elements to determine the adequacy of a test run. Coverage analysis is useful when attempting to execute each statement, branch, path, or iterative structure in a program. Crash: The sudden and complete failure of a computer system or component. Criticality: The degree of impact that a requirement, module, error, fault, failure, or other item has on the development or operation of a system. Cyclomatic Complexity: The number of independent paths through a program. The cyclomatic complexity of a program is equivalent to the number of decision statements plus 1. Error: A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. Error Guessing: This is a Test data selection technique. The selection criterion is to pick values that seem likely to cause errors. Error Seeding: The process of intentionally adding known faults to those already in a computer program for the purpose of monitoring the rate of detection and removal, and estimating the number of faults remaining in the program. Contrast with mutation analysis. Exception: An event that causes suspension of normal program execution. Types include addressing exception, data exception, operation exception, overflow exception, protection exception, and underflow exception. Exhaustive Testing: Executing the program with all possible combinations of values for program variables. This type of testing is feasible only for small, simple programs. Failure: The inability of a system or component to perform its required functions within specified performance requirements. Fault: An incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner.

154

TESTING CONCEPTS Functional Testing: Testing that ignores the internal mechanism or structure of a system or component and focuses on the outputs generated in response to selected inputs and execution conditions. (2) Testing conducted to evaluate the compliance of a system or component with specified functional requirements and corresponding predicted results. Integration Testing: An orderly progression of testing in which software elements, hardware elements, or both are combined and tested, to evaluate their interactions, until the entire system has been integrated. Interface Testing: Testing conducted to evaluate whether systems or components pass data and control correctly to one another. Mutation Testing: A testing methodology in which two or more program mutations are executed using the same test cases to evaluate the ability of the test cases to detect differences in the mutations. Operational Testing: Testing conducted to evaluate a system or component in its operational environment. Parallel Testing: Testing a new or an altered data processing system with the same source data that is used in another system. The other system is considered as the standard of comparison. Path Testing: Testing to satisfy coverage criteria that each logical path through the program be tested. Often paths through the program are grouped into a finite set of classes. One path from each class is then tested. Performance Testing: Functional testing conducted to evaluate the compliance of a system or component with specified performance requirements. Qualification Testing: Formal testing, usually conducted by the developer for the consumer, to demonstrate that the software meets its specified requirements. Quality Assurance: (1) The planned systematic activities necessary to ensure that a component, module, or system conforms to established technical requirements. (2) All actions that are taken to ensure that a development organization delivers products that meet performance requirements and adhere to standards and

155

TESTING CONCEPTS procedures. (3) The policy, procedures, and systematic actions established in an enterprise for the purpose of providing and maintaining some degree of confidence in data integrity and accuracy throughout the life cycle of the data, which includes input, update, manipulation, and output. (4) The actions, planned and performed, to provide confidence that all systems and components that influence the quality of the product are working as expected individually and collectively. Quality Control: The operational techniques and procedures used to achieve quality requirements. Regression Testing: Rerunning test cases which a program has previously executed correctly in order to detect errors spawned by changes or corrections made during software development and maintenance. Review: A process or meeting during which a work product or set of work products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include code review, design review, formal qualification review, requirements review, test readiness review. Risk: A measure of the probability and severity of undesired effects. Risk Assessment: A comprehensive evaluation of the risk and its associated impact. Software Review: An evaluation of software elements to ascertain discrepancies from planned results and to recommend improvement. This evaluation follows a formal process. Syn: software audit. See: code audit, code inspection, code review, code walkthrough, design review, specification analysis, static analysis Static Analysis: Analysis of a program that is performed without executing the program. The process of evaluating a system or component based on its form, structure, content, documentation is also called as Static Analysis. Statement Testing: Testing to satisfy the criterion that each statement in a program be executed at least once during program testing. Storage Testing: This is a determination of whether or not certain processing conditions use more storage [memory] than estimated.

156

TESTING CONCEPTS Stress Testing: Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements. Structural Testing: Testing that takes into account the internal mechanism [structure] of a system or component. Types include branch testing, path testing, statement testing. (2) Testing to insure each program statement is made to execute during testing and that each program statement performs its intended function. System Testing: The process of testing an integrated hardware and software system to verify that the system meets its specified requirements. Such testing may be conducted in both the development environment and the target environment. Test: An activity in which a system or component is executed under specified conditions, the results are observed or recorded and an evaluation is made of some aspect of the system or component. Testability: The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. Test case: Documentation specifying inputs, predicted results, and a set of execution conditions for a test item. Test case Generator: A software tool that accepts as input source code, test criteria, specifications, or data structure definitions; uses these inputs to generate test input data; and, sometimes, determines expected results. Test Design: Documentation specifying the details of the test approach for a software feature or combination of software features and identifying the associated tests. Test Documentation: Documentation describing plans for, or results of, the testing of a system or component, Types include test case specification, test incident report, test log, test plan, test procedure, test report. Test Driver: A software module used to invoke a module under test and, often, provide test inputs, control and monitor execution, and report test results.

157

TESTING CONCEPTS Test Incident Report: A document reporting on any event that occurs during testing that requires further investigation. Test Item: A software item which is the object of testing. Test Log: A chronological record of all relevant details about the execution of a test. Test Phase: The period of time in the software life cycle in which the components of a software product are evaluated and integrated, and the software product is evaluated to determine whether or not requirements have been satisfied. Test Plan: Documentation specifying the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, responsibilities, required, resources, and any risks requiring contingency planning. See: test design, validation protocol. Test Procedure: A formal document developed from a test plan that presents detailed instructions for the setup, operation, and evaluation of the results for each defined test. Test Report: A document describing the conduct and results of the testing carried out for a system or system component. Test Result Analyzer: A software tool used to test output data reduction, formatting, and printing. Testing: (1) The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component. (2) The process of analyzing a software item to detect the differences between existing and required conditions, i.e. bugs, and to evaluate the features of the software items. Traceability Matrix: A matrix that records the relationship between two or more products; e.g., a matrix that records the relationship between the requirements and the design of a given software component. See: traceability, traceability analysis. Unit Testing: Testing of a module for typographic, syntactic, and logical errors, for correct implementation of its design, and for satisfaction of its requirements (or)

158

TESTING CONCEPTS Testing conducted to verify the implementation of the design for one software element; e.g., a unit or module; or a collection of software elements. Usability: The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component. Usability Testing: Tests designed to evaluate the machine/user interface. Validation: Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes. Validation, Verification and Testing: Used as an entity to define a procedure of review, analysis, and testing throughout the software life cycle to discover errors, determine functionality, and ensure the production of quality software. Volume Testing: Testing designed to challenge a system's ability to manage the maximum amount of data over a period of time. This type of testing also evaluates a system's ability to handle overload situations in an orderly fashion. Positive and Negative Testing The notion of something like "Integration Testing" or "System Testing" can (and should) be defined so that everyone knows what is meant by that activity within the same organization, but terms like "negative test" and "positive test" are more of a concept than a strict activity. In both instances you are dealing with an input, an action, and an output. The action acts upon the input to derive a certain output. So a test case (and thus a good test) is just one that deals with those three things. Both test cases can produce errors and, in fact, some say that the success of a test case is based upon the probability of it finding new errors in an application. What I want to do here, however, is state clearly one viewpoint of what the distinction between positive and negative testing is. Then I want to play Devil's Advocate and try to undermine that viewpoint by presenting an argument that others have put forth - an alternative viewpoint. The real point of this will be to show that sometimes trying to adhere too rigidly to conceptual terms like this can lead to a lot of stagnating action. Read this section as a sort of extended argument that I am

159

TESTING CONCEPTS having with myself as I come to grips with these terms. So let us first state a simple hypothetical definition: positive testing is that testing which attempts to show that a given module of an application does what it is supposed to do. Negative testing is that testing which attempts to show that the module does not do anything that it is not supposed to do. So, by that logic, and to make a concrete example, an application delivering an error when it should is actually an example of a positive test. A negative test would be the program not delivering an error when it should or delivering an error when it should not. But this sounds like it is more based on what the application does during testing rather than how the tester is actually going about testing it. Well, sort of. The idea here is that neither test necessarily has to force an error condition, per se, at least by strict definition. But both concepts (negative and positive) are looking for different types of error conditions. Consider that one part of negative testing is often considered to be boundary analysis. In this case, you are not so much "forcing an error" because, of course, the application should handle boundary problems. But what you are doing is seeing if the boundary problem is not, in fact, handled. So if the program is supposed to give an error when the person types in "101" on a field that should be between "1" and "100", then that is valid if an error shows up. If, however, the application does not give an error when the user typed "101" then you have a problem. So really negative testing and positive testing are the same kinds of things when you really boil it right down. Now, some make a distinguishing remark from what I said. I said the following: Positive testing is that testing which attempts to show that a given module of an application does what it is supposed to do. Negative testing is that testing which attempts to show that the module does not do anything that it is not supposed to do. Playing the Devil's Advocate, others would change this around and say the following is a better distinction: Positive testing is that testing which attempts to show that a given module of an application does not do what it is supposed to do. Negative testing is that testing which attempts to show that the module does

160

TESTING CONCEPTS something that it is not supposed to do. Let us look at this slightly shifted point of view. By this logic, we would say that most syntax/input validation tests are positive tests. Even if you give an invalid input, you are expecting a positive result (e.g., an error message) in the hope of finding a situation where the module either gives the wrong error message or actually allows the invalid input. A negative test is, by this logic, more trying to get the module to do something differently than it was designed to do. For example, if you are testing a state transition machine and the state transition sequence is: State 1 -> State 2 -> State 3 -> State 4, then trying to get the module to go from State 2 to State 4, skipping State 3, is a negative test. So, negative testing, in this case, is about thinking of how to disrupt the module and, by extension, positive testing is examining how well/badly the module does its task. Now, in response to this, I would agree that most can see looking at it this way from what the tester hopes to find. Testing pundits often tell testers to look for error because if you look for success, you will often find success - even when there is error. By proxy, if you do not find an error and you have reliable test cases (that latter point is crucial), then a positive test case will show that the application did not, in fact, manifest that error. However, showing an error when it should have done so is an example of a "positive test" by the strict definition of that term. So in other words: Positive Testing = (Not showing error when not supposed to) + (Showing error when supposed to) So if either of the situations in parentheses happens you have a positive test in terms of its result - not what the test was hoping to find. The application did what it was supposed to do. By that logic: Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to) (Usually these situations crop up during boundary testing or cause-effect testing.) Here if either of the situations in parentheses happens you have a negative test in terms of its result - again, not what the test was hoping to find. The application did what it was not supposed to do.

161

TESTING CONCEPTS However, in both cases, these were good results because they showed you what the application was doing and you were able to determine if it was working correctly or not. So, by my original definitions, the testing is all about errors and finding them. It is just how you are looking for those errors that make the distinction. (Granted, how you are looking will often dictate what you are hoping to find but since that is the case, it hardly makes sense to make a grand distinction between them.) Now, regarding the point I made above, as a Devil's Advocate: "A negative test is more trying to get the module to do something differently than it was designed to do." We have to realize, I think, that what we call "negative testing" is often about exercising boundary conditions - and those boundaries exist within the context of design. Granted, that can be trying to get a value in to a field that it should not accept. However, a good application should have, during the requirements stage, had provisions for invalid input. Thus really what you are testing here is (a) whether the provisions for invalid input exist and (b) whether they are working correctly. And, again, that is why this distinction, for me (between positive and negative), is somewhat banal. Your negative test can turn into a positive test just be shifting the emphasis of what you are looking for. To get the application to do something it is not designed to do could be looked at as accepting invalid input. However, if you find that the application does accept invalid input and does not, in fact, give a warning, I would agree that is a negative test if it was specified in requirements that the application should respond to invalid input. In this case the application did not, but it was not also specified that it should. So, here, by strict requirements did the application do what it was supposed to do? Technically, yes. If requirements did not specify differently, design was not put in place to handle the issue. Thus you are not testing something outside the scope of design. Rather, you are testing something that was not designed in the first place. So, going back to one of the previous points, one thing we can probably all agree on: it entirely depends on how you view a test. But are we saying the result of the test determines whether it was a positive or negative test? If so, many would disagree with that, indicating that it is the thinking behind the test that should be positive or negative. In actuality, most experienced testers do not think in terms of positive or negative, they think in terms of "what can I do to establish the level of

162

TESTING CONCEPTS risk?" However, to this point, I would argue that if that is truly how the tester thinks of things then all concepts of positive/negative go right out of the window (as I think they mostly should anyway). Obviously you could classify the test design in terms of negative or positive, but to some extent that is irrelevant. However, without getting into that, I am not sure we are saying that the result of the test determines positivity or negativity. What I said earlier, relative to my example, was that "in both cases, these were good results because they showed you what the application was doing and you were able to determine if it was working correctly or not." If the application was behaving correctly or incorrectly, you still determined what the application was actually doing and, as such, those are good results. Thus the result tells you about the application and that is good (without recourse to terms like positive and negative). If the result tells you nothing about how the application is functioning that is, obviously, bad (and, again, this is without recourse to positive or negative). We can apply the term "effective" to these types of test cases and we can say that all test cases, positive or negative, should be effective. But what about the idea of relying on the thinking behind the test? This kind of concept is just a little too vague for me because people's thinking can be more or less different, even on this issue, which can often depend on what people have been taught regarding these concepts. As I showed, you can transform a postive test mentality into a negative test mentality just by thinking about the results of the test differently. And if negative testing is just about "disrupting a module" (the Devil's Advocate position), even a positive test can do that if there is a fault. However I am being a little flip because with the notion of the thinking behind the test, obviously someone here would be talking about intent. The intent is to disrupt the module so as to cause a fault fault and that would constitute a negative test (by the Devil's Advocate position) while a positive test would not be trying to disrupt the module - even though disruption might occur (again, by the Devil's Advocate position). The key differentiator is the intent. I could sort of buy that but, then again, boundary testing is an attempt to disrupt modules because you are seeing if the system can handle the boundary violation. This can also happen with results. As I said: "Your negative test can turn into a positive test just be shifting the emphasis of what you are looking for." That sort of speaks to the intention of what you are hoping to find but also how you view the problem. If the disruption you tried to cause in the module is, in fact, handled by the code then you will get a positive test result - an error

163

TESTING CONCEPTS message of some sort. Now I want to keep on this point because, again, some people state that negative testing is about exercising boundary conditions. Some were taught that this is not negative testing; rather that this is testing invalid inputs, which are positive tests - so it depends how you were taught. And figure that a boundary condition, if not handled by the code logic, will potentially severely disrupt the module - which is the point of negative testing according to some views of it. However, that is not the intent here according to some. And yet while that was not the intent, that might be the result. That is why the distinction, for me, blurs. But here is where the crux of the point is for me: you can generlaly forget all about intent of test case design for the moment and look at the distinction of what the result is in terms of a "positive result" (the application showed me an error when it should have) and a "negative result" (the application did not show me an error when it should have). The latter is definitely a more negative connotation than the former, regardless of the intent of the tester during design of the test case and that is important to realize because sometimes our intentions for tests are changed by the reality of what exists and what happens as a result of running the tests. So, in the case of intent for the situation of the application not showing an error when it was supposed to, this is simply a matter of writing "negative test cases" (if we stick with the term for a moment) that will generate conditions that should, in turn, generate error messages. But the point is that the intent of the test case is to see if the application does not, in fact, generate that error message. In other words, you are looking for a negative result. But, then again, we can say: "Okay, now I will look that the application does generate the error message that it should." Well, in that case, we are really just running the negative test case! Either way the result is that the error either will or will not show up and thus the result is, at least to some extent, determining the nature of the test case (in terms of negative or positive connotation). If the error does not show up, the invalid input might break the module. So is the breakdown this:

164

TESTING CONCEPTS P: Not showing error when not supposed to N: Not showing error when supposed to P: Showing error when supposed to N: Showing error when not supposed to I think the one thing we have to consider is the viewpoint: hinges on the idea of "negative testing" being looked at as forcing the module to do something it was not designed to do. However, if the module was never designed to do the thing you are trying, then your testing is of an interesting sort because, after all, you know nothing exists to handle it. So the real question should not be: "What happens when I do this?" but rather "Why have we not designed this to handle this situation?" Let us say that something is designed to handle the "module disruption" you are proposed to test. In that case, you are actually positively testing the code that handles that situation. To a strict degree, forcing a module to do something it was not designed to do suggests that this is something your average user can do. In other words, your average user could potentially use the application in such a fashion that the negative test case you are putting forth could be emulated by the user. However, if that is the case, design should be in place to mitigate that problem. And, again, you are then positively testing. Now, one can argue, "Well, it is possible that the user can try something that there simply is no way to design around." Okay. But then I ask: "Like what?" If there is no way you can design around it or even design something to watch for the event, or have the system account for it, how do you write a valid test case for that? I mean, you can write a test case that breaks the application by disrupting the module but -you already knew that was going to happen. However, this is not as cut and dry as I am sure anyone reading this could point out. After all, in some cases maybe you are not sure that what you are writing as a test case will be disruptive. Ah, but that is the rub. We just defined "negative testing" as trying to disrupt the module. Whether we succeed or not is a different issue (and speaks to the result), but that was the intent. We are trying to do something that is outside the bounds of design and thus it is not so much a matter of testing for disruption as it is testing for the effects of that disruption. If the effects could be mitigated, that must be some sort of design that is mitigating them and then you are positively testing that mitigating influence. As an example, a good test case for a word processer might be: "Turn off the

165

TESTING CONCEPTS computer to simulate a power failure when an unsaved document is present in the application." Now, the idea here is that you might have some document saving feature that automatically kicks in when the application suddenly terminates, say via a General Protection Fault (GPF). However, strictly speaking, powering down the computer is different than a GPF. So here you are testing to see what happens if the application is shut down via a power-off of the PC, which, let us say, the application was not strictly designed to really handle. So my intent is to disrupt the module. However, in this case, since I can state the negative condition, I can state a possible design that could account for it. After all: we already know that the document will not be saved because nothing was designed to account for that. But the crucial point is that if nothing was designed into the system to account for the power-off of the PC, then what are you really testing? You are testing that the application does what the application does when a power-off occurs. But if nothing is designed to happen one way or the other, then testing for disruption really does you no good. After all, you know it is going to be disrupted. That is not in question. What is (or should be) in question is how you can handle that disruption and then test how that handling works. So let us take those alternate (Devil's Advocate) definitions: Positive testing is that testing which attempts to show that a given module of an application does not do what it is supposed to do. In this case of the power-down test case, we are not positive testing because we did not test that the application did not do what it was supposed to do. The application was not "supposed" to do anything because nothing was designed to handle the power-down. Negative testing is that testing which attempts to show that the module does something that it is not supposed to do. In the case of the power-down test case, we are also not negative testing by this definition because the application, in not saving the document or doing anything (since it was not designed to do anything in the first place), is not doing something that it is not supposed to do. Again, the application is not "supposed" to do anything since it was not designed to do anything in this situation.

166

TESTING CONCEPTS Now consider my quasi-definition/equation for positive testing that I gave earlier: Positive Testing = (Not showing error when not supposed to) + (Showing error when supposed to) I would have to loosen my language a little but, basically, the application was not supposed to show an error and, in fact, did not do so in this case. But what if the application was, in fact, supposed to handle that situation of a power-down? Let us say the developers hooked into the API so that if a shut-down event was fired off, the application automatically issues an error/warning and then saves the document in a recovery mode format. Now let us say I test that and find that the application did not, in fact, save the file. Consider again my quasi-definition/equation for negative testing: Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to) In this case I have done negative testing because the application was supposed to issue an error/warning but did not. However, notice, that the test case is the same exact test case. The intent of my testing was simply to test this aspect of the application. The result of the test relative to the stated design is what determines if the test was negative or positive by my definitions. Now, because I want to be challenged on this stuff, you could also say: "Yes, but forget the document in the word processor. What if the application gets corrupted because of the power-off?" Let us say that the corruption is just part of the Windows environment and there is nothing that can be done about it. Is this negative testing? By the Devil's Advocate definition, strictly it is not, because remember by that definition: "Negative testing is that testing which attempts to show that the module does something that it is not supposed to do." But, in this case, the module did not do something (become corrupted) that it was not supposed to do. This simply happened as a by-product of a Windows event that cannot be handled. But we did, after all, try to disrupt the module, right? So is it a negative test or not by the definition of disruption? Incidentally, by my definition, it is not a negative test either. However, what is common in all of what I have said is that the power-down test case is an effective test case and this is the case regardless of whether you choose to connote it with a "positive" or "negative" qualifier. Since that can be the case, then, for me, the use of the qualifier is irrelevant.

167

TESTING CONCEPTS But now let us consider another viewpoint from the Devil's Advocate and one that I think is pretty good. Consider this example: An application takes mouse clicks as input. The requirement is for one mouse click to be processed at a time, the user hitting multiple mouse clicks will cause the application to discard anything but the first. Any tester will do the obvious and design a test to hit multiple mouse clicks. Now the application is designed to discard anything but the first, so the test could be classified (by my definition) a negative one as the application is designed not to process multiple mouse clicks. The negative test is to try to force the application to process more than the first. BUT, I hear you say, this is an input validation test that tests that the application does discard mutiple mouse clicks, therefore it is a positive test (again, by my definition), and I would then agree, it is a positive test. However, the tester might also design a test that overflows the input buffer with mouse clicks is that a negative test?. Note, this situation is not covered explicitly in the requirements - and that is crucial to what I would call negative testing, that very often it is the tester's "what if" analysis that designs negative tests - so, yes, it is a negative test as you are forcing the application into a situation it may not have been designed and/or coded for - you may not know whether it had or not. The actual result of the test may be that the application stops accepting any more clicks on its input buffer and causes an error message or it may crash. Now, having said all this, it makes me realize how my point starts to coalesce with the Devil's Advocate. One way that might happen is via the use of the term "error" that has gotten tossed around a lot. My language seemed too restrictive in the sense that when I used the word "error" (as in "showing error when not supposed to") I did not make it clear enough that I was not necessarily talking about an error screen of some sort, but rather an error condition or a failure. With this, my negative testing definition really starts to coalesce with the Devil's Advocate's definition ("does something that it is not supposed to do"). I had said: "Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to)" and broadening my language more, really what I am saying is that the application is either showing (doing) something it is not supposed to (which matches the Devil's Advocate thought) but I was also saying that the application is not showing (doing) something that it was supposed to. And to the Devil's Advocate that latter is positive testing. Let me restate the two viewpoints somewhat:

168

TESTING CONCEPTS Positive Testing (Jeff): Not doing something it was not supposed to do. Doing something it was supposed to do. Positive Testing (Devil's Advocate): Not doing what it is supposed to do. Negative Testing (Jeff): Doing something it was not supposed to do. Not doing something it was supposed to do. Negative Testing (Devil's Advocate): Doing something that it is not supposed to do. I think I was essentially saying the same thing in terms of negative testing as my hypothetical opponent, just not in terms of positive testing. If you notice, both of our "negative testings" really contain the same point. On the other hand, relative to the common Devil's Advocate position, I am having a hard time seeing the major distinction between positive and negative. The Devil's Advocate's original conception: "Positive testing is that testing which attempts to show that a given module of an application does NOT do what it is supposed to do. Negative testing is that testing which attempts to show that the module does something that it is not supposed to do." To me, "doing something you are not supposed to" (the Devil's Advocate negative test) and "not doing something you are supposed to" (the Devil's Advocate positive test) are really two sides of the same coin or maybe just two ways of saying the same thing. So let us say that our requirement is "do not process multiple mouse clicks". In that case, "not doing something you are supposed to" (Devil's Advocate positive test) means, in this case, "processing multiple mouse clicks". In other words, the application should not process multiple mouse clicks. If it does, it is doing something it is not supposed to. Likewise, "doing something you are not supposed to do" (Devil's Advocate negative test) means, in this case, "processing multiple mouse clicks". In other words, the application should not process multiple mouse clicks. Either way, it is saying the same thing. So what we are testing for is "application not

169

TESTING CONCEPTS processing multiple mouse clicks". It would seem that if the application does process multiple mouse clicks it is both not doing what it is supposed to do (not processing them) and doing something it is not supposed to do (processing them). The same statement, just made different ways. Now, let me see if that works with my definitions. Again, the lemma is "do not process multiple mouse clicks". If the application does this then it falls under lemma 1 of my negative test ("Doing something it was not supposed to do.") If the application does not do this, it falls under lemma 1 of my positive test ("not doing something it was not supposed to do"). Even with the mouse click example we have two aspects: Application designed not to process multiple mouse clicks Application designed to process only one mouse click Saying the same thing, and yet a subtle shift in emphasis if you want to go by the positive and negative distinctions. The difference, however, is also whether you are dealing with active design or passive design. In other words, does the application actively make sure that only one mouse click is handled (by closing the buffer) or does it simply only process one click, but allow the buffer to fill up anyway. I like the idea of tying this whole thing in with "mitgating design factors". I think that we can encapsulate "intent" and "result" (both of which are important to test casing) by looking more at the efficient and effective demarcations. We have to consider result that is part of how you do test case effectiveness metrics as well as proactive defect detection metrics. If a test case is a tautology test then it is not really efficient or effective - but that is based solely on the result, not the intent or anything else Conclusion: BVT is nothing but a set of regression test cases that are executed each time for new build. This is also called as smoke test. Build is not assigned to test team unless and until the BVT passes. BVT can be run by developer or tester and BVT result is communicated throughout the team and immediate action is taken to fix the bug if BVT fails. BVT process is typically automated by writing scripts for test cases. Only critical test cases are included in BVT. These test cases should ensure application test coverage. BVT is very effective for daily as well as long term builds. This saves significant time, cost, resources and after all no frustration of test team for incomplete build.

170

TESTING CONCEPTS

Definitions
A abstract test case: See high level test case. acceptance: See acceptance testing. acceptance criteria: The exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. [IEEE 610] acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610] accessibility testing: Testing to determine the ease by which users with disabilities can use a component or system. [Gerrard] accuracy: The capability of the software product to provide the right or agreed results or effects with the needed degree of precision. [ISO 9126] See also functionality testing. actual outcome: See actual result. actual result: The behavior produced/observed when a component or system is tested. ad hoc review: See informal review. ad hoc testing: Testing carried out informally; no formal test preparation takes place, no recognized test design technique is used, there are no expectations for results and arbitrariness guides the test execution activity. adaptability: The capability of the software product to be adapted for different specified environments without applying actions or means other than those provided for this purpose for the software considered. [ISO 9126] See also portability. agile testing: Testing practice for a project using agile methodologies, such as extreme programming (XP), treating development as the customer of testing and emphasizing the test-first design paradigm. See also test driven development. algorithm test [TMap]: See branch testing. alpha testing: Simulated or actual operational testing by potential users/customers or an independent test team at the developers site, but outside the development organization. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing.

171

TESTING CONCEPTS analyzability: The capability of the software product to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified to be identified. analyzer: See static analyzer. anomaly: Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc. or from someones perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation. [IEEE 1044] See also defect, deviation, error, fault, failure, incident, problem. arc testing: See branch testing. attractiveness: The capability of the software product to be attractive to the user. [ISO 9126] See also usability. audit: An independent evaluation of software products or processes to ascertain compliance to standards, guidelines, specifications, and/or procedures based on objective criteria, including documents that specify: (1) the form or content of the products to be produced (2) the process by which the products shall be produced (3) how compliance to standards or guidelines shall be measured. [IEEE 1028] audit trail: A path by which the original input to a process (e.g. data) can be traced back through the process, taking the process output as a starting point. This facilitates defect analysis and allows a process audit to be carried out. [After TMap] automated testware: Testware used in automated testing, such as tool scripts. availability: The degree to which a component or system is operational and accessible when required for use. Often expressed as a percentage. [IEEE 610] B back-to-back testing: Testing in which two or more variants of a component or system are executed with the same inputs, the outputs compared, and analyzed in cases of discrepancies. [IEEE 610] baseline: A specification or software product that has been formally reviewed or agreed upon, that thereafter serves as the basis for further development, and that can be changed only through a formal change control process. [After IEEE 610] basic block: A sequence of one or more consecutive executable statements containing no branches.

172

TESTING CONCEPTS basis test set: A set of test cases derived from the internal structure of a component or specification to ensure that 100% of a specified coverage criterion will be achieved. bebugging: See error seeding. [Abbott] behavior: The response of a component or system to a set of input values and preconditions. benchmark test: (1) A standard against which measurements or comparisons can be made. (2) A test that is be used to compare components or systems to each other or to a standard as in (1). [After IEEE 610] bespoke software: Software developed specifically for a set of users or customers. The opposite is off-the-shelf software. best practice: A superior method or innovative practice that contributes to the improved performance of an organization under given context, usually recognized as best by other peer organizations. beta testing: Operational testing by potential and/or existing users/customers at an external site not otherwise involved with the developers, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes. Beta testing is often employed as a form of external acceptance testing for off-the-shelf software in order to acquire feedback from the market. big-bang testing: A type of integration testing in which software elements, hardware elements, or both are combined all at once into a component or an overall system, rather than in stages. [After IEEE 610] See also integration testing. black-box technique: See black box test design technique. black-box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system. black-box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure. blocked test case: A test case that cannot be executed because the preconditions for its execution are not fulfilled. bottom-up testing: An incremental approach to integration testing where the lowest level components are tested first, and then used to facilitate the testing of higher level components. This process is repeated until the component at the top of the hierarchy is tested. See also integration testing.

173

TESTING CONCEPTS boundary value: An input value or output value which is on the edge of an equivalence partition or at the smallest incremental distance on either side of an edge, for example the minimum or maximum value of a range. boundary value analysis: A black box test design technique in which test cases are designed based on boundary values. boundary value coverage: The percentage of boundary values that have been exercised by a test suite. boundary value testing: See boundary value analysis. branch: A basic block that can be selected for execution based on a program construct in which one of two or more alternative program paths are available, e.g. case, jump, go to, ifthen- else. branch condition: See condition. branch condition combination coverage: See multiple condition coverage. branch condition combination testing: See multiple condition testing. branch condition coverage: See condition coverage. branch coverage: The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage. branch testing: A white box test design technique in which test cases are designed to execute branches. bug: See defect. bug: See defect report. business process-based testing: An approach to testing in which test cases are designed based on descriptions and/or knowledge of business processes. C Capability Maturity Model (CMM): A five level staged framework that describes the key elements of an effective software process. The Capability Maturity Model covers bestpractices for planning, engineering and managing software development and maintenance. [CMM] Capability Maturity Model Integration (CMMI): A framework that describes the key elements of an effective product development and maintenance process. The Capability Maturity Model Integration covers best-practices for planning, engineering and managing product development and maintenance. CMMI is the designated successor of the CMM. [CMMI]

174

TESTING CONCEPTS capture/playback tool: A type of test execution tool where inputs are recorded during manual testing in order to generate automated test scripts that can be executed later (i.e. replayed). These tools are often used to support automated regression testing. capture/replay tool: See capture/playback tool. CASE: Acronym for Computer Aided Software Engineering. CAST: Acronym for Computer Aided Software Testing. See also test automation. cause-effect graph: A graphical representation of inputs and/or stimuli (causes) with their associated outputs (effects), which can be used to design test cases. cause-effect graphing: A black box test design technique in which test cases are designed from cause-effect graphs. [BS 7925/2] cause-effect analysis: See cause-effect graphing. cause-effect decision table: See decision table. certification: The process of confirming that a component, system or person complies with its specified requirements, e.g. by passing an exam. changeability: The capability of the software product to enable specified modifications to be implemented. [ISO 9126] See also maintainability. change control: See configuration control. change control board: See configuration control board. checker: See reviewer. Chow's coverage metrics: See N-switch coverage. [Chow] classification tree method: A black box test design technique in which test cases, described by means of a classification tree, are designed to execute combinations of representatives of input and/or output domains. [Grochtmann] code: Computer instructions and data definitions expressed in a programming language or in a form output by an assembler, compiler or other translator. [IEEE 610] code analyzer: See static code analyzer. code coverage: An analysis method that determines which parts of the software have been executed (covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage. code-based testing: See white box testing. co-existence: The capability of the software product to co-exist with other independent software in a common environment sharing common resources. [ISO 9126] See also portability.

175

TESTING CONCEPTS commercial off-the-shelf software: See off-the-shelf software. comparator: See test comparator. compatibility testing: See interoperability testing. compiler: A software tool that translates programs expressed in a high order language into their machine language equivalents. [IEEE 610] complete testing: See exhaustive testing. completion criteria: See exit criteria. complexity: The degree to which a component or system has a design and/or internal structure that is difficult to understand, maintain and verify. See also cyclomatic complexity. compliance: The capability of the software product to adhere to standards, conventions or regulations in laws and similar prescriptions. [ISO 9126] compliance testing: The process of testing to determine the compliance of the component or system. component: A minimal software item that can be tested in isolation. component integration testing: Testing performed to expose defects in the interfaces and interaction between integrated components. component specification: A description of a components function in terms of its output values for specified input values under specified conditions, and required nonfunctional behavior (e.g. resource-utilization). component testing: The testing of individual software components. [After IEEE 610] compound condition: Two or more single conditions joined by means of a logical operator (AND, OR or XOR), e.g. A>B AND C>1000. concrete test case: See low level test case. concurrency testing: Testing to determine how the occurrence of two or more activities within the same interval of time, achieved either by interleaving the activities or by simultaneous execution, is handled by the component or system. [After IEEE 610] condition: A logical expression that can be evaluated as True or False, e.g. A>B. See also test condition. condition combination coverage: See multiple condition coverage. condition combination testing: See multiple condition testing.

176

TESTING CONCEPTS condition coverage: The percentage of condition outcomes that have been exercised by a test suite. 100% condition coverage requires each single condition in every decision statement to be tested as True and False. condition determination coverage: The percentage of all single condition outcomes that independently affect a decision outcome that have been exercised by a test case suite. 100% condition determination coverage implies 100% decision condition coverage. condition determination testing: A white box test design technique in which test cases are designed to execute single condition outcomes that independently affect a decision outcome. condition testing: A white box test design technique in which test cases are designed to execute condition outcomes. condition outcome: The evaluation of a condition to True or False. confidence test: See smoke test. configuration: The composition of a component or system as defined by the number, nature, and interconnections of its constituent parts. configuration auditing: The function to check on the contents of libraries of configuration items, e.g. for standards compliance. [IEEE 610] configuration control: An element of configuration management, consisting of the evaluation, co-ordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. [IEEE 610] configuration control board (CCB): A group of people responsible for evaluating and approving or disapproving proposed changes to configuration items, and for ensuring implementation of approved changes. [IEEE 610] configuration identification: An element of configuration management, consisting of selecting the configuration items for a system and recording their functional and physical characteristics in technical documentation. [IEEE 610] 11 configuration item: An aggregation of hardware, software or both, that is designated for configuration management and treated as a single entity in the configuration management process. [IEEE 610] configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics,

177

TESTING CONCEPTS record and report change processing and implementation status, and verify compliance with specified requirements. [IEEE 610] configuration management tool: A tool that provides support for the identification and control of configuration items, their status over changes and versions, and the release of baselines consisting of configuration items. configuration testing: See portability testing. confirmation testing: See re-testing. conformance testing: See compliance testing. consistency: The degree of uniformity, standardization, and freedom from contradiction among the documents or parts of a component or system. [IEEE 610] control flow: A sequence of events (paths) in the execution through a component or system. control flow graph: A sequence of events (paths) in the execution through a component or system. control flow path: See path. conversion testing: Testing of software used to convert data from existing systems for use in replacement systems. COTS: Acronym for Commercial Off-The-Shelf software. See off-the-shelf software. coverage: The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite. coverage analysis: Measurement of achieved coverage to a specified coverage item during test execution referring to predetermined criteria to determine whether additional testing is required and if so, which test cases are needed. coverage item: An entity or property used as a basis for test coverage, e.g. equivalence partitions or code statements. coverage tool: A tool that provides objective measures of what structural elements, e.g. statements, branches have been exercised by a test suite. custom software: See bespoke software. cyclomatic complexity: The number of independent paths through a program. Cyclomatic complexity is defined as: L N + 2P, where - L = the number of edges/links in a graph - N = the number of nodes in a graph - P = the number of disconnected parts of the graph (e.g. a called graph and a subroutine) cyclomatic number: See cyclomatic complexity.

178

TESTING CONCEPTS D daily build: a development activity where a complete system is compiled and linked every day (usually overnight), so that a consistent system is available at any time including all latest changes. data definition: An executable statement where a variable is assigned a value. data driven testing: A scripting technique that stores test input and expected results in a table or spreadsheet, so that a single control script can execute all of the tests in the table. Data driven testing is often used to support the application of test execution tools such as capture/playback tools. [Fewster and Graham] See also keyword driven testing. data flow: An abstract representation of the sequence and possible changes of the state of data objects, where the state of an object is any of: creation, usage, or destruction. [Beizer] data flow analysis: A form of static analysis based on the definition and usage of variables. data flow coverage: The percentage of definition-use pairs that have been exercised by a test suite. data flow testing: A white box test design technique in which test cases are designed to execute definition and use pairs of variables. data integrity testing: See database integrity testing. database integrity testing: Testing the methods and processes used to access and manage the data(base), to ensure access methods, processes and data rules function as expected and that during access to the database, data is not corrupted or unexpectedly deleted, updated or created. dead code: See unreachable code. debugger: See debugging tool. debugging: The process of finding, analyzing and removing the causes of failures in software. debugging tool: A tool used by programmers to reproduce failures, investigate the state of programs and find the corresponding defect. Debuggers enable programmers to execute programs step by step, to halt a program at any program statement and to set and examine program variables. decision: A program point at which the control flow has two or more alternative routes. A node with two or more links to separate branches.

179

TESTING CONCEPTS decision condition coverage: The percentage of all condition outcomes and decision outcomes that have been exercised by a test suite. 100% decision condition coverage implies both 100% condition coverage and 100% decision coverage. decision condition testing: A white box test design technique in which test cases are designed to execute condition outcomes and decision outcomes. decision coverage: The percentage of decision outcomes that have been exercised by a test suite. 100% decision coverage implies both 100% branch coverage and 100% statement coverage. decision table: A table showing combinations of inputs and/or stimuli (causes) with their associated outputs and/or actions (effects), which can be used to design test cases. 13 decision table testing: A black box test design techniques in which test cases are designed to execute the combinations of inputs and/or stimuli (causes) shown in a decision table. [Veenendaal] decision testing: A white box test design technique in which test cases are designed toexecute decision outcomes. decision outcome: The result of a decision (which therefore determines the branches to be taken). defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system. defect density: The number of defects identified in a component or system divided by the size of the component or system (expressed in standard measurement terms, e.g. lines-ofcode, number of classes or function points). Defect Detection Percentage (DDP): the number of defects found by a test phase, divided by the number found by that test phase and any other means afterwards. defect management: The process of recognizing, investigating, taking action and disposing of defects. It involves recording defects, classifying them and identifying the impact. [After IEEE 1044] defect management tool: A tool that facilitates the recording and status tracking of defects. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of defects and provide reporting facilities. See also incident management

180

TESTING CONCEPTS tool. defect masking: An occurrence in which one defect prevents the detection of another. [After IEEE 610] defect report: A document reporting on any flaw in a component or system that can cause the component or system to fail to perform its required function. [After IEEE 829] defect tracking tool: See defect management tool. definition-use pair: The association of the definition of a variable with the use of that variable. Variable uses include computational (e.g. multiplication) or to direct the execution of a path (predicate use). deliverable: Any (work) product that must be delivered to someone other than the (work)products author. design-based testing: An approach to testing in which test cases are designed based on the architecture and/or detailed design of a component or system (e.g. tests of interfaces between components or systems). desk checking: Testing of software or specification by manual simulation of its execution. See also static analysis. development testing: Formal or informal testing conducted during the implementation of a component or system, usually in the development environment by developers. [After IEEE 610] deviation: See incident. deviation report: See incident report. dirty testing: See negative testing. documentation testing: Testing the quality of the documentation, e.g. user guide or installation guide. domain: The set from which valid input and/or output values can be selected. driver: A software component or test tool that replaces a component that takes care of the control and/or the calling of a component or system. [After TMap] dynamic analysis: The process of evaluating behavior, e.g. memory performance, CPU usage, of a system or component during execution. [After IEEE 610] dynamic analysis tool: A tool that provides run-time information on the state of the software code. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic and to monitor the allocation, use and deallocation of memory and to flag memory leaks.

181

TESTING CONCEPTS dynamic comparison: Comparison of actual and expected results, performed while the software is being executed, for example by a test execution tool. dynamic testing: Testing that involves the execution of the software of a component or system. E efficiency: The capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions. [ISO 9126] efficiency testing: The process of testing to determine the efficiency of a software product. elementary comparison testing: A black box test design techniques in which test cases are designed to execute combinations of inputs using the concept of condition determination coverage. [TMap] emulator: A device, computer program, or system that accepts the same inputs and produces the same outputs as a given system. [IEEE 610] See also simulator. entry criteria: the set of generic and specific conditions for permitting a process to go forward with a defined task, e.g. test phase. The purpose of entry criteria is to prevent a task from starting which would entail more (wasted) effort compared to the effort needed to remove the failed entry criteria. [Gilb and Graham] entry point: The first executable statement within a component. equivalence class: See equivalence partition. equivalence partition: A portion of an input or output domain for which the behavior of a component or system is assumed to be the same, based on the specification. equivalence partition coverage: The percentage of equivalence partitions that have been exercised by a test suite. equivalence partitioning: A black box test design technique in which test cases are designed to execute representatives from equivalence partitions. In principle test cases are designed to cover each partition at least once. error: A human action that produces an incorrect result. [After IEEE 610] 15 error guessing: A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them.

182

TESTING CONCEPTS error seeding: The process of intentionally adding known defects to those already in the component or system for the purpose of monitoring the rate of detection and removal, and estimating the number of remaining defects. [IEEE 610] error tolerance: The ability of a system or component to continue normal operation despite the presence of erroneous inputs. [After IEEE 610]. evaluation: See testing. exception handling: Behavior of a component or system in response to erroneous input, from either a human user or from another component or system, or to an internal failure. executable statement: A statement which, when compiled, is translated into object code, and which will be executed procedurally when the program is running and may perform an action on data. exercised: A program element is said to be exercised by a test case when the input value causes the execution of that element, such as a statement, decision, or other structural element. exhaustive testing: A test approach in which the test suite comprises all combinations of input values and preconditions. exit criteria: The set of generic and specific conditions, agreed upon with the stakeholders, for permitting a process to be officially completed. The purpose of exit criteria is to prevent a task from being considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham] exit point: The last executable statement within a component. expected outcome: See expected result. expected result: The behavior predicted by the specification, or another source, of the component or system under specified conditions. experienced-based test design technique: Procedure to derive and/or select test cases based on the testers experience, knowledge and intuition. exploratory testing: An informal test design technique where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. [After Bach] F fail: A test is deemed to fail if its actual result does not match its expected result. failure: Deviation of the component or system from its expected delivery, service or result. [After Fenton]

183

TESTING CONCEPTS failure mode: The physical or functional manifestation of a failure. For example, a system in failure mode may be characterized by slow operation, incorrect outputs, or complete termination of execution. [IEEE 610] 16 Failure Mode and Effect Analysis (FMEA): A systematic approach to risk identification and analysis of identifying possible modes of failure and attempting to prevent their occurrence. failure rate: The ratio of the number of failures of a given category to a given unit of measure, e.g. failures per unit of time, failures per number of transactions, failures per number of computer runs. [IEEE 610] fault: See defect. fault density: See defect density. Fault Detection Percentage (FDP): See Defect Detection Percentage (DDP). fault masking: See defect masking. fault tolerance: The capability of the software product to maintain a specified level of performance in cases of software faults (defects) or of infringement of its specified interface. [ISO 9126] See also reliability. fault tree analysis: A method used to analyze the causes of faults (defects). feasible path: A path for which a set of input values and preconditions exists which causes it to be executed. feature: An attribute of a component or system specified or implied by requirements documentation (for example reliability, usability or design constraints). [After IEEE 1008] field testing: See beta testing. finite state machine: A computational model consisting of a finite number of states and transitions between those states, possibly with accompanying actions. [IEEE 610] finite state testing: See state transition testing. formal review: A review characterized by documented procedures and requirements, e.g. inspection. frozen test basis: A test basis document that can only be amended by a formal change control process. See also baseline. Function Point Analysis (FPA): Method aiming to measure the size of the functionality of an information system. The measurement is independent of the technology. This measurement may be used as a basis for the measurement of productivity, the estimation of the needed resources, and project control.

184

TESTING CONCEPTS functional integration: An integration approach that combines the components or systems for the purpose of getting a basic functionality working early. See also integration testing. functional requirement: A requirement that specifies a function that a component or system must perform. [IEEE 610] functional test design technique: Procedure to derive and/or select test cases based on an analysis of the specification of the functionality of a component or system without reference to its internal structure. See also black box test design technique. functional testing: Testing based on an analysis of the specification of the functionality of a component or system. See also black box testing. functionality: The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions. [ISO 9126] 17 functionality testing: The process of testing to determine the functionality of a software product. G glass box testing: See white box testing. H heuristic evaluation: A static usability test technique to determine the compliance of a user interface with recognized usability principles (the so-called heuristics). high level test case: A test case without concrete (implementation level) values for input data and expected results. Logical operators are used; instances of the actual values are not yet defined and/or available. See also low level test case. horizontal traceability: The tracing of requirements for a test level through the layers of test documentation (e.g. test plan, test design specification, test case specification and test procedure specification or test script). I impact analysis: The assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements. incident: Any event occurring that requires investigation. [After IEEE 1008] incident logging: Recording the details of any incident that occurred, e.g. during testing.

185

TESTING CONCEPTS incident management: The process of recognizing, investigating, taking action and disposing of incidents. It involves logging incidents, classifying them and identifying the impact. [After IEEE 1044] incident management tool: A tool that facilitates the recording and status tracking of incidents. They often have workflow-oriented facilities to track and control the allocation, correction and re-testing of incidents and provide reporting facilities. See also defect management tool. incident report: A document reporting on any event that occurred, e.g. during the testing, which requires investigation. [After IEEE 829] incremental development model: A development life cycle where a project is broken into a series of increments, each of which delivers a portion of the functionality in the overall project requirements. The requirements are prioritized and delivered in priority order in the appropriate increment. In some (but not all) versions of this life cycle model, each subproject follows a mini V-model with its own design, coding and testing phases. incremental testing: Testing where components or systems are integrated and tested one or some at a time, until all the components or systems are integrated and tested. independence: Separation of responsibilities, which encourages the ccomplishment of objective testing. [After DO-178b] infeasible path: A path that cannot be exercised by any set of possible input values. informal review: A review not based on a formal (documented) procedure. input: A variable (whether stored within a component or outside) that is read by a component. 18 input domain: The set from which valid input values can be selected. See also domain. input value: An instance of an input. See also input. inspection: A type of peer review that relies on visual examination of documents to detect defects, e.g. violations of development standards and non-conformance to higher level documentation. The most formal review technique and therefore always based on a documented procedure. [After IEEE 610, IEEE 1028] See also peer review. inspection leader: See moderator. inspector: See reviewer.

186

TESTING CONCEPTS installability: The capability of the software product to be installed in a specified environment [ISO 9126]. See also portability. installability testing: The process of testing the installability of a software product. See also portability testing. installation guide: Supplied instructions on any suitable media, which guides the installer through the installation process. This may be a manual guide, step-by-step procedure, installation wizard, or any other similar process description. installation wizard: Supplied software on any suitable media, which leads the installer through the installation process. It normally runs the installation process, provides feedback on installation results, and prompts for options. instrumentation: The insertion of additional code into the program in order to collect information about program behavior during execution, e.g. for measuring code coverage. instrumenter: A software tool used to carry out instrumentation. intake test: A special instance of a smoke test to decide if the component or system is ready for detailed and further testing. An intake test is typically carried out at the start of the test execution phase. See also smoke test. integration: The process of combining components or systems into larger assemblies. integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing. integration testing in the large: See system integration testing. integration testing in the small: See component integration testing. interface testing: An integration test type that is concerned with testing the interfaces between components or systems. interoperability: The capability of the software product to interact with one or more specified components or systems. [After ISO 9126] See also functionality. interoperability testing: The process of testing to determine the interoperability of a software product. See also functionality testing. invalid testing: Testing using input values that should be rejected by the component or system. See also error tolerance. isolation testing: Testing of individual components in isolation from surrounding components, with surrounding components being simulated by stubs and drivers, if needed.

187

TESTING CONCEPTS item transmittal report: See release note. 19 iterative development model: A development life cycle where a project is broken into a usually large number of iterations. An iteration is a complete development loop resulting in a release (internal or external) of an executable product, a subset of the final product under development, which grows from iteration to iteration to become the final product. K key performance indicator: See performance indicator. keyword driven testing: A scripting technique that uses data files to contain not only test data and expected results, but also keywords related to the application being tested. The keywords are interpreted by special supporting scripts that are called by the control script for the test. See also data driven testing. L LCSAJ: A Linear Code Sequence And Jump, consisting of the following three items (conventionally identified by line numbers in a source code listing): the start of the linear sequence of executable statements, the end of the linear sequence, and the target line to which control flow is transferred at the end of the linear sequence. LCSAJ coverage: The percentage of LCSAJs of a component that have been exercised by a test suite. 100% LCSAJ coverage implies 100% decision coverage. LCSAJ testing: A white box test design technique in which test cases are designed to execute LCSAJs. learnability: The capability of the software product to enable the user to learn its application. [ISO 9126] See also usability. level test plan: A test plan that typically addresses one test level. See also test plan. link testing: See component integration testing. load testing: A test type concerned with measuring the behavior of a component or system with increasing load, e.g. number of parallel users and/or numbers of transactions to determine what load can be handled by the component or system. See also stress testing. logic-coverage testing: See white box testing. [Myers] logic-driven testing: See white box testing. logical test case: See high level test case. low level test case: A test case with concrete (implementation level) values for input data and expected results. Logical operators from high level test cases are

188

TESTING CONCEPTS replaced by actual values that correspond to the objectives of the logical operators. See also high level test case. M maintenance: Modification of a software product after delivery to correct defects, to improve performance or other attributes, or to adapt the product to a modified environment. [IEEE 1219] maintenance testing: Testing the changes to an operational system or the impact of a changed environment to an operational system. maintainability: The ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment. [ISO 9126] 20 maintainability testing: The process of testing to determine the maintainability of a software product. management review: A systematic evaluation of software acquisition, supply, development, operation, or maintenance process, performed by or on behalf of management that monitors progress, determines the status of plans and schedules, confirms requirements and their system allocation, or evaluates the effectiveness of management approaches to achieve fitness for purpose. [After IEEE 610, IEEE 1028] master test plan: A test plan that typically addresses multiple test levels. See also test plan. maturity: (1) The capability of an organization with respect to the effectiveness and efficiency of its processes and work practices. See also Capability Maturity Model, Test Maturity Model. (2) The capability of the software product to avoid failure as a result of defects in the software. [ISO 9126] See also reliability. measure: The number or category assigned to an attribute of an entity by making a measurement. [ISO 14598] measurement: The process of assigning a number or category to an entity to describe an attribute of that entity. [ISO 14598] measurement scale: A scale that constrains the type of data analysis that can be performed on it. [ISO 14598] memory leak: A defect in a program's dynamic store allocation logic that causes it to fail to reclaim memory after it has finished using it, eventually causing the program to fail due to lack of memory. metric: A measurement scale and the method used for measurement. [ISO 14598] migration testing: See conversion testing.

189

TESTING CONCEPTS milestone: A point in time in a project at which defined (intermediate) deliverables and results should be ready. mistake: See error. moderator: The leader and main person responsible for an inspection or other review process. modified condition decision coverage: See condition determination coverage. modified condition decision testing: See condition determination coverage testing. modified multiple condition coverage: See condition determination coverage. modified multiple condition testing: See condition determination coverage testing. module: See component. module testing: See component testing. monitor: A software tool or hardware device that runs concurrently with the component or system under test and supervises, records and/or analyses the behavior of the component or system. [After IEEE 610] monitoring tool: See monitor. multiple condition: See compound condition. 21 multiple condition coverage: The percentage of combinations of all single condition outcomes within one statement that have been exercised by a test suite. 100% multiple condition coverage implies 100% condition determination coverage. multiple condition testing: A white box test design technique in which test cases are designed to execute combinations of single condition outcomes (within one statement). mutation analysis: A method to determine test suite thoroughness by measuring the extent to which a test suite can discriminate the program from slight variants (mutants) of the program. mutation testing: See back-to-back testing. N N-switch coverage: The percentage of sequences of N+1 transitions that have been exercised by a test suite. [Chow] N-switch testing: A form of state transition testing in which test cases are designed to execute all valid sequences of N+1 transitions. [Chow] See also state transition testing.

190

TESTING CONCEPTS negative testing: Tests aimed at showing that a component or system does not work. Negative testing is related to the testers attitude rather than a specific test approach or test design technique, e.g. testing with invalid input values or exceptions. [After Beizer]. non-conformity: Non fulfillment of a specified requirement. [ISO 9000] non-functional requirement: A requirement that does not relate to functionality, but to attributes such as reliability, efficiency, usability, maintainability and portability. non-functional testing: Testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability. non-functional test design techniques: Procedure to derive and/or select test cases for nonfunctional testing based on an analys is of the specification of a component or system without reference to its internal structure. See also black box test design technique. O off-the-shelf software: A software product that is developed for the general market, i.e. for a large number of customers, and that is delivered to many customers in identical format. operability: The capability of the software product to enable the user to operate and control it. [ISO 9126] See also usability. operational environment: Hardware and software products installed at users or customers sites where the component or system under test will be used. The software may include operating systems, database management systems, and other applications. operational profile testing: Statistical testing using a model of system operations (short duration tasks) and their probability of typical use. [Musa] operational testing: Testing conducted to evaluate a component or system in its operational environment. [IEEE 610] oracle: See test oracle. outcome: See result. output: A variable (whether stored within a component or outside) that is written by a component. 22 output domain: The set from which valid output values can be selected. See also domain.

191

TESTING CONCEPTS output value: An instance of an output. See also output. P pair programming: A software development approach whereby lines of code (production and/or test) of a component are written by two programmers sitting at a single computer. This implicitly means ongoing real-time code reviews are performed. pair testing: Two persons, e.g. two testers, a developer and a tester, or an end-user and a tester, working together to find defects. Typically, they share one computer and trade control of it while testing. partition testing: See equivalence partitioning. [Beizer] pass: A test is deemed to pass if its actual result matches its expected result. pass/fail criteria: Decision rules used to determine whether a test item (function) or feature has passed or failed a test. [IEEE 829] path: A sequence of events, e.g. executable statements, of a component or system from an entry point to an exit point. path coverage: The percentage of paths that have been exercised by a test suite. 100% path coverage implies 100% LCSAJ coverage. path sensitizing: Choosing a set of input values to force the execution of a given path. path testing: A white box test design technique in which test cases are designed to execute paths. peer review: A review of a software work product by colleagues of the producer of the product for the purpose of identifying defects and improvements. Examples are inspection, technical review and walkthrough. performance: The degree to which a system or component accomplishes its designated functions within given constraints regarding processing time and throughput rate. [After IEEE 610] See also efficiency. performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive development, e.g. lead-time slip for software development. [CMMI] performance testing: The process of testing to determine the performance of a software product. See also efficiency testing. performance testing tool: A tool to support performance testing and that usually has two main facilities: load generation and test transaction measurement. Load generation can simulate either multiple users or high volumes of input data. During

192

TESTING CONCEPTS execution, response time measurements are taken from selected transactions and these are logged. Performance testing tools normally provide reports based on test logs and graphs of load against response times. phase test plan: A test plan that typically addresses one test phase. See also test plan. portability: The ease with which the software product can be transferred from one hardware or software environment to another. [ISO 9126] portability testing: The process of testing to determine the portability of a software product. 23 postcondition: Environmental and state conditions that must be fulfilled after the execution of a test or test procedure. post-execution comparison: Comparison of actual and expected results, performed after the software has finished running. precondition: Environmental and state conditions that must be fulfilled before the component or system can be executed with a particular test or test procedure. predicted outcome: See expected result. pretest: See intake test. priority: The level of (business) importance assigned to an item, e.g. defect. probe effect: The effect on the component or system by the measurement instrument when the component or system is being measured, e.g. by a performance testing tool or monitor. For example performance may be slightly worse when performance testing tools are being used. problem: See defect. problem management: See defect management. problem report: See defect report. process: A set of interrelated activities, which transform inputs into outputs. [ISO 12207] process cycle test: A black box test design technique in which test cases are designed to execute business procedures and processes. [TMap] product risk: A risk directly related to the test object. See also risk. project: A project is a unique set of coordinated and controlled activities with start and finish dates undertaken to achieve an objective conforming to specific requirements, including the constraints of time, cost and resources. [ISO 9000]

193

TESTING CONCEPTS project risk: A risk related to management and control of the (test) project. See also risk. program instrumenter: See instrumenter. program testing: See component testing. project test plan: See master test plan. pseudo-random: A series which appears to be random but is in fact generated according to some prearranged sequence. Q quality: The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations. [After IEEE 610] quality assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled. [ISO 9000] quality attribute: A feature or characteristic that affects an items quality. [IEEE 610] quality characteristic: See quality attribute. quality management: Coordinated activities to direct and control an organization with regard to quality. Direction and control with regard to quality generally includes the establishment 24 of the quality policy and quality objectives, quality planning, quality control, quality assurance and quality improvement. [ISO 9000] R random testing: A black box test design technique where test cases are selected, possibly using a pseudo-random generation algorithm, to match an operational profile. This technique can be used for testing non-functional attributes such as reliability and performance. recorder: See scribe. record/playback tool: See capture/playback tool. recoverability: The capability of the software product to re-establish a specified level of performance and recover the data directly affected in case of failure. [ISO 9126] See alsoreliability. recoverability testing: The process of testing to determine the recoverability of a software product. See also reliability testing. recovery testing: See recoverability testing. regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of

194

TESTING CONCEPTS the software, as a result of the changes made. It is performed when the software or its environment is changed. regulation testing: See compliance testing. release note: A document identifying test items, their configuration, current status and other delivery information delivered by development to testing, and possibly other stakeholders, at the start of a test execution phase. [After IEEE 829] reliability: The ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations. [ISO 9126] reliability testing: The process of testing to determine the reliability of a software product. replaceability: The capability of the software product to be used in place of another specified software product for the same purpose in the same environment. [ISO 9126] See also portability. requirement: A condition or capability needed by a user to solve a problem or achieve an objective that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. [After IEEE 610] requirements-based testing: An approach to testing in which test cases are designed based on test objectives and test conditions derived from requirements, e.g. tests that exercise specific functions or probe non-functional attributes such as reliability or usability. requirements management tool: A tool that supports the recording of requirements, requirements attributes (e.g. priority, knowledge responsible) and annotation, and facilitates traceability through layers of requirements and requirements change management. Some requirements management tools also provide facilities for static analysis, such as consistency checking and violations to pre-defined requirements rules. requirements phase: The period of time in the software life cycle during which the requirements for a software product are defined and documented. [IEEE 610] 25 resource utilization: The capability of the software product to use appropriate amounts and types of resources, for example the amounts of main and secondary memory used by the program and the sizes of required temporary or overflow files, when the software performs its function under stated conditions. [After ISO 9126] See also efficiency.

195

TESTING CONCEPTS resource utilization testing: The process of testing to determine the resourceutilization of a software product. See also efficiency testing. result: The consequence/outcome of the execution of a test. It includes outputs to screens, changes to data, reports, and communication messages sent out. See also actual result, expected result. resumption criteria: The testing activities that must be repeated when testing is re-started after a suspension. [After IEEE 829] re-testing: Testing that runs test cases that failed the last time they were run, in order to verify the success of corrective actions. review: An evaluation of a product or project status to ascertain discrepancies from planned results and to recommend improvements. Examples include management review, informal review, technical review, inspection, and walkthrough. [After IEEE 1028] reviewer: The person involved in the review that identifies and describes anomalies in the product or project under review. Reviewers can be chosen to represent different viewpoints and roles in the review process. review tool: A tool that provides support to the review process. Typical features include review planning and tracking support, communication support, collaborative reviews and a repository for collecting and reporting of metrics. risk: A factor that could result in future negative consequences; usually expressed as impact and likelihood. risk analysis: The process of assessing identified risks to estimate their impact and probability of occurrence (likelihood). risk-based testing: Testing oriented towards exploring and providing information about product risks. [After Gerrard] risk control: The process through which decisions are reached and protective measures are implemented for reducing risks to, or maintaining risks within, specified levels. risk identification: The process of identifying risks using techniques such as brainstorming, checklists and failure history. risk management: Systematic application of procedures and practices to the tasks of identifying, analyzing, prioritizing, and controlling risk. risk mitigation: See risk control.

196

TESTING CONCEPTS robustness: The degree to which a component or system can function correctly in the presence of invalid inputs or stressful environmental conditions. [IEEE 610] See also error-tolerance, fault-tolerance. robustness testing: Testing to determine the robustness of the software product. root cause: An underlying factor that caused a non-conformance and possibly should be permanently eliminated through process improvement. 26 S safety: The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use. [ISO 9126] safety testing: Testing to determine the safety of a software product. sanity test: See smoke test. scalability: The capability of the software product to be upgraded to accommodate increased loads. [After Gerrard] scalability testing: Testing to determine the scalability of the software product. scenario testing: See use case testing. scribe: The person who records each defect mentioned and any suggestions for process improvement during a review meeting, on a logging form. The scribe has to ensure that the logging form is readable and understandable. scripting language: A programming language in which executable test scripts are written, used by a test execution tool (e.g. a capture/playback tool). security: Attributes of software products that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs and data. [ISO 9126] See also functionality. security testing: Testing to determine the security of the software product. See also functionality testing. security testing tool: A tool that provides support for testing security characteristics and vulnerabilities. security tool: A tool that supports operational security. serviceability testing: See maintainability testing. severity: The degree of impact that a defect has on the development or operation of a component or system. [After IEEE 610] simulation: The representation of selected behavioral characteristics of one physical or abstract system by another system. [ISO 2382/1]

197

TESTING CONCEPTS simulator: A device, computer program or system used during testing, which behaves or operates like a given system when provided with a set of controlled inputs. [After IEEE 610, DO178b] See also emulator. site acceptance testing: Acceptance testing by users/customers at their site, to determine whether or not a component or system satisfies the user/customer needs and fits within the business processes, normally including hardware as well as software. smoke test: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertaining that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. See also intake test. software: Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system [IEEE 610] software feature: See feature. 27 software quality: The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs. [After ISO 9126] software quality characteristic: See quality attribute. software test incident: See incident. software test incident report: See incident report. Software Usability Measurement Inventory (SUMI): A questionnaire based usability test technique to evaluate the usability, e.g. user-satisfaction, of a component or system. [Veenendaal] source statement: See statement. specification: A document that specifies, ideally in a complete, precise and verifiable manner, the requirements, design, behavior, or other characteristics of a component or system, and, often, the procedures for determining whether these provisions have been satisfied. [After IEEE 610] specification-based testing: See black box testing. specification-based test design technique: See black box test design technique. specified input: An input for which the specification predicts a result. stability: The capability of the software product to avoid unexpected effects from modifications in the software. [ISO 9126] See also maintainability. standard software: See off-the-shelf software. standards testing: See compliance testing.

198

TESTING CONCEPTS state diagram: A diagram that depicts the states that a component or system can assume, and shows the events or circumstances that cause and/or result from a change from one state to another. [IEEE 610] state table: A grid showing the resulting transitions for each state combined with each possible event, showing both valid and invalid transitions. state transition: A transition between two states of a component or system. state transition testing: A black box test design technique in which test cases are designed to execute valid and invalid state transitions. See also N-switch testing. statement: An entity in a programming language, which is typically the smallest indivisible unit of execution. statement coverage: The percentage of executable statements that have been exercised by a test suite. statement testing: A white box test design technique in which test cases are designed to execute statements. static analysis: Analysis of software artifacts, e.g. requirements or code, carried out without execution of these software artifacts. static analysis tool: See static analyzer. static analyzer: A tool that carries out static analysis. static code analysis: Analysis of source code carried out without execution of that software. 28 static code analyzer: A tool that carries out static code analysis. The tool checks source code, for certain properties such as conformance to coding standards, quality metrics or data flow anomalies. static testing: Testing of a component or system at specification or implementation level without execution of that software, e.g. reviews or static code analysis. statistical testing: A test design technique in which a model of the statistical distribution of the input is used to construct representative test cases. See also operational profile testing. status accounting: An element of configuration management, consisting of the recording and reporting of information needed to manage a configuration effectively. This information includes a listing of the approved configuration identification, the status of proposed changes to the configuration, and the implementation status of the approved changes. [IEEE 610] storage: See resource utilization. storage testing: See resource utilization testing.

199

TESTING CONCEPTS stress testing: Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements. [IEEE 610] See also load testing. structure-based techniques: See white box test design technique. structural coverage: Coverage measures based on the internal structure of a component or system. structural test design technique: See white box test design technique. structural testing: See white box testing. structured walkthrough: See walkthrough. stub: A skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component. [After IEEE 610] subpath: A sequence of executable statements within a component. suitability: The capability of the software product to provide an appropriate set of functions for specified tasks and user objectives. [ISO 9126] See also functionality. suspension criteria: The criteria used to (temporarily) stop all or a portion of the testing activities on the test items. [After IEEE 829] syntax testing: A black box test design technique in which test cases are designed based upon the definition of the input domain and/or output domain. system: A collection of components organized to accomplish a specific function or set of functions. [IEEE 610] system integration testing: Testing the integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet). system testing: The process of testing an integrated system to verify that it meets specified requirements. [Hetzel] T technical review: A peer group discussion activity that focuses on achieving consensus on the technical approach to be taken. [Gilb and Graham, IEEE 1028] See also peer review. 29 test: A set of one or more test cases [IEEE 829] test approach: The implementation of the test strategy for a specific project. It typically includes the decisions made that follow based on the (test) projects goal and the risk assessment carried out, starting points regarding the test process, the test design techniques to be applied, exit criteria and test types to be performed. test automation: The use of software to perform or support test activities, e.g. test

200

TESTING CONCEPTS management, test design, test execution and results checking. test basis: All documents from which the requirements of a component or system can be inferred. The documentation on which the test cases are based. If a document can be amended only by way of formal amendment procedure, then the test basis is called a frozen test basis. [After TMap] test bed: See test environment. test case: A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610] test case design technique: See test design technique. test case specification: A document specifying a set of test cases (objective, inputs, test actions, expected results, and execution preconditions) for a test item. [After IEEE 829] test case suite: See test suite. test charter: A statement of test objectives, and possibly test ideas on how to test. Test charters are for example often used in exploratory testing. See also exploratory testing. test closure: During the test closure phase of a test process data is collected from completed activities to consolidate experience, testware, facts and numbers. The test closure phase consists of finalizing and archiving the testware and evaluating the test process, including preparation of a test evaluation report. See also test process. test comparator: A test tool to perform automated test comparison. test comparison: The process of identifying differences between the actual results produced by the component or system under test and the expected results for a test. Test comparison can be performed during test execution (dynamic comparison) or after test execution. test completion criteria: See exit criteria. test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element. test control: A test management task that deals with developing and applying a set of corrective actions to get a test project on track when monitoring shows a deviation from what was planned. See also test management. test coverage: See coverage.

201

TESTING CONCEPTS test cycle: Execution of the test process against a single identifiable release of the test object. test data: Data that exists (for example, in a database) before a test is executed, and that affects or is affected by the component or system under test. test data preparation tool: A type of test tool that enables data to be selected from existing databases or created, generated, manipulated and edited for use in testing. 30 test design: See test design specification. test design specification: A document specifying the test conditions (coverage items) for a test item, the detailed test approach and identifying the associated high level test cases. [After IEEE 829] test design technique: Procedure used to derive and/or select test cases. test design tool: A tool that supports the test design activity by generating test inputs from a specification that may be held in a CASE tool repository, e.g. requirements management tool, from specified test conditions held in the tool itself, or from code. test driver: See driver. test driven development: A way of developing software where the test cases are developed, and often automated, before the software is developed to run those test cases. test environment: An environment containing hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test. [After IEEE 610] test evaluation report: A document produced at the end of the test process summarizing all testing activities and results. It also contains an evaluation of the test process and lessons learned. test execution: The process of running a test on the component or system under test, producing actual result(s). test execution automation: The use of software, e.g. capture/playback tools, to control the execution of tests, the comparison of actual results to expected results, the setting up of test preconditions, and other test control and reporting functions. test execution phase: The period of time in a software development life cycle during which the components of a software product are executed, and the software product is evaluated to determine whether or not requirements have been satisfied. [IEEE 610]

202

TESTING CONCEPTS test execution schedule: A scheme for the execution of test procedures. The test procedures are included in the test execution schedule in their context and in the order in which they are to be executed. test execution technique: The method used to perform the actual test execution, either manually or automated. test execution tool: A type of test tool that is able to execute other software using an automated test script, e.g. capture/playback. [Fewster and Graham] test fail: See fail. test generator: See test data preparation tool. test leader: See test manager. test harness: A test environment comprised of stubs and drivers needed to execute a test. test incident: See incident. test incident report: See incident report. test infrastructure: The organizational artifacts needed to perform testing, consisting of test environments, test tools, office environment and procedures. test input: The data received from an external source by the test object during test execution. The external source can be hardware, software or human. 31 test item: The individual element to be tested. There usually is one test object and many test items. See also test object. test item transmittal report: See release note. test leader: See test manager. test level: A group of test activities that are organized and managed together. A test level is linked to the responsibilities in a project. Examples of test levels are component test, integration test, system test and acceptance test. [After TMap] test log: A chronological record of relevant details about the execution of tests. [IEEE 829] test logging: The process of recording information about tests executed into a test log. test manager: The person responsible for project management of testing activities and resources, and evaluation of a test object. The individual who directs, controls, administers, plans and regulates the evaluation of a test object.

203

TESTING CONCEPTS test management: The planning, estimating, monitoring and control of test activities, typically carried out by a test manager. test management tool: A tool that provides support to the test management and control part of a test process. It often has several capabilities, such as testware management, scheduling of tests, the logging of results, progress tracking, incident management and test reporting. Test Maturity Model (TMM): A five level staged framework for test process improvement, related to the Capability Maturity Model (CMM) that describes the key elements of an effective test process. test monitoring: A test management task that deals with the activities related to periodically checking the status of a test project. Reports are prepared that compare the actuals to that which was planned. See also test management. test object: The component or system to be tested. See also test item. test objective: A reason or purpose for designing and executing a test. test oracle: A source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual, or an individuals specialized knowledge, but should not be the code. [After Adrion] test outcome: See result. test pass: See pass. test performance indicator: A high level metric of effectiveness and/or efficiency used to guide and control progressive test development, e.g. Defect Detection Percentage (DDP). test phase: A distinct set of test activities collected into a manageable phase of a project, e.g. the execution activities of a test level. [After Gerrard] test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process. [After IEEE 829] test planning: The activity of establishing or updating a test plan. 32 test policy: A high level document describing the principles, approach and major objectives of the organization regarding testing.

204

TESTING CONCEPTS Test Point Analysis (TPA): A formula based test estimation method based on function point analysis. [TMap] test procedure: See test procedure specification. test procedure specification: A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829] test process: The fundamental test process comprises planning, specification, execution, recording, checking for completion and test closure activities. [After BS 7925/2] Test Process Improvement (TPI): A continuous framework for test process improvement that describes the key elements of an effective test process, especially targeted at system testing and acceptance testing. test record: See test log. test recording: See test logging. test reproduceability: An attribute of a test indicating whether the same results are produced each time the test is executed. test report: See test summary report. test requirement: See test condition. test run: Execution of a test on a specific version of the test object. test run log: See test log. test result: See result. test scenario: See test procedure specification. test script: Commonly used to refer to a test procedure specification, especially an automated one. test set: See test suite. test situation: See test condition. test specification: A document that consists of a test design specification, test case specification and/or test procedure specification. test specification technique: See test design technique. test stage: See test level. test strategy: A high-level description of the test levels to be performed and the testing within those levels for an organization or programme (one or more projects). test suite: A set of several test cases for a component or system under test, where the post condition of one test is often used as the precondition for the next one.

205

TESTING CONCEPTS test summary report: A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items against exit criteria. [After IEEE 829] test target: A set of exit criteria. test technique: See test design technique. test tool: A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis. [TMap] See also CAST. test type: A group of test activities aimed at testing a component or system focused on a specific test objective, i.e. functional test, usability test, regression test etc. A test type may take place on one or more test levels or test phases. [After TMap] testability: The capability of the software product to enable modified software to be tested. [ISO 9126] See also maintainability. testability review: A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act as an input document for the test process. [After TMap] testable requirements: The degree to which a requirement is stated in terms that permit establishment of test designs (and subsequently test cases) and execution of tests to determine whether the requirements have been met. [After IEEE 610] tester: A skilled professional who is involved in the testing of a component or system. testing: The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. testware: Artifacts produced during the test process required to plan, design, and execute tests, such as documentation, scripts, inputs, expected results, set-up and clear-up procedures, files, databases, environment, and any additional software or utilities used in testing. [After Fewster and Graham] thread testing: A version of component integration testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by levels of a hierarchy. time behavior: See performance. top-down testing: An incremental approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level

206

TESTING CONCEPTS components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested. See also integration testing. traceability: The ability to identify related items in documentation and software, such as requirements with associated tests. See also horizontal traceability, vertical traceability. U understandability: The capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use. [ISO 9126] See also usability. unit: See component. unit testing: See component testing. unreachable code: Code that cannot be reached and therefore is impossible to execute. usability: The capability of the software to be understood, learned, used and attractive to the user when used under specified conditions. [ISO 9126] 34 usability testing: Testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions. [After ISO 9126] use case: A sequence of transactions in a dialogue between a user and the system with a tangible result. use case testing: A black box test design technique in which test cases are designed to execute user scenarios. user acceptance testing: See acceptance testing. user scenario testing: See use case testing. user test: A test whereby real-life users are involved to evaluate the usability of a component or system. V V-model: A framework to describe the software development life cycle activities from requirements specification to maintenance. The V-model illustrates how testing activities can be integrated into each phase of the software development life cycle. validation: Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]

207

TESTING CONCEPTS variable: An element of storage in a computer that is accessible by a software program by referring to it by a name. verification: Confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled. [ISO 9000] vertical traceability: The tracing of requirements through the layers of development documentation to components. version control: See configuration control. volume testing: Testing where the system is subjected to large volumes of data. See also resource-utilization testing. W walkthrough: A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content. [Freedman and Weinberg, IEEE 1028] See also peer review. white-box test design technique: Procedure to derive and/or select test cases based on an analysis of the internal structure of a component or system. white-box testing: Testing based on an analysis of the internal structure of the component or system. Wide Band Delphi: An expert based test estimation technique that aims at making an accurate estimation using the collective wisdom of the team members. 35

208

TESTING CONCEPTS

209

TESTING CONCEPTS

210

TESTING CONCEPTS

211