White Box test~ng

61

Path coverage provides a stronger condition of coverage than statement coverage as it relates to the various logical paths in the program rather than just program statements. 3.3.2.3 Conditioncoverage In the aboveexample,even ifwe have covered all the paths possible, it would not mean that the program is fully tested. For example, we can make the program take the path A by giving a value less than 1 (for example, 0) to mm and find that we have covered the path A and the program has detected that the month is invalid. But, the program may still not be correctly testing for the other condition namely mm > 12.Furthermore, most compliers perform optimizations to minimize the number of Boolean operations and all the conditions may not get evaluated, even though the right path is chosen. For example, when there is an OR condition (as in the first IF statement above), once the first part of the IF (for example, mm < 1) is found to be true, the second part will not be evaluated at all as the overall value of the Boolean is TRUE. Similarly, when there is an AND condition in a Boolean expression, when the first condition evaluates to FALSE, the rest of the expression need not be evaluated at all. For all these reasons, path testing may not be sufficient. It is necessary to have test cases that exercise each Boolean expression and have test cases test produce the TRUE as well as FALSE paths. Obviously, this will mean more test cases and the number of test cases will rise exponentially with the number of conditions and Boolean expressions. However, in reality, the situation may not be very bad as these conditions usually have some dependencies on one another. The condition coverage, as defined by the formula alongside in the margin gives an indication of the percentage of conditions covered by a set of test cases. Condition coverage is a much stronger criteria than path coverage, which in turn is a much stronger criteria than statement coverage. 3.3.2.4 Function coverage This is a new addition to structural testing to identify how many program functions (similar to functions in "C" language) are covered by test cases. The requirements of a product are mapped into functions during the design phase and each of the functions form a logical unit. For example, in a database software, "inserting a row into the database" could be a function. Or, ina payroll application, "calculate tax" could be a function. Each function could, in turn, be implemented using other functions. While providing function coverage, test cases can be written so as to exercise each of the different functions in the code. The advantages that function coverage provides over the other types of coverage are as follows. 1. 2. Functions are easier to identify in a program and hence it is easier to write test cases to provide function coverage. Since functions are at a much higher level of abstraction than code, it is easier to achieve 100percent function coverage than 100percent coverage in any of the earlier methods.

62

Software Testing

3.

4.

Functions have a more logical mapping to requirements and hence can provide a more direct correlation to the test coverage of the product. In the next chapter, we will be discussing the requirements traceability matrix, which track a requirement through design, coding, and testing phases. Functions provide one means to achieve this traceability. Function coverage provides a way of testing this traceabili ty. Since functions are a means of realizing requirements, the importance of functions can be prioritized based on the importance of the requirements they realize. Thus, it would be easier to prioritize the functions for testing. This is not necessarily the case with the earlier methods of coverage. Function coverage provides a natural transition to black box testing.

5.

We can also measure how many times a given function is called. This will indicate which functions are used most often and hence these functions become the target of any performance testing and optimization. As an example, if in a networking software, we find that the function that assembles and disassembles the data packets is being used most often, it is appropriate to spend extra effort in improving the quality and performance of that function. Thus, function coverage can help in improving the performance as well as quality of the product. 3.3.2.5 Summary Code coverage testing involves "dynamic testing" methods of executing the product with pre-written test cases, and finding out how much of code has been covered. If a better coverage of a code is desired, several iterations of testing may be required. For each iteration, one has to go through the statistics and write a new set of test cases for covering portions of the code not covered by earlier test cases. To do this type of testing not only does one need to understand the code, logic but also need to understand how to write effective test cases that can cover good portions of the code. This type of testing can also be referred to as "gray box testing" as this uses the combination of "white box and black box methodologies" (white + black = gray) for effectiveness. Thus, better code coverage is the result of better code flow understanding and writing effective test cases. Code coverage up to 40-50 percent is usually achievable. Code coverage of more than 80 percent requires enormous amount of effort and understanding of the code. The multiple code coverage techniques we have discussed so far are not mutually exclusive. They supplement and augment one another. While statement coverage can provide a basic comfort factor, path, decision, and function coverage provide more confidence by exercising various logical paths and functions. In the above discussion, we have looked at the use of code coverage testing for various functional requirements. There are also a few other uses for these methods.

White Box Testing

63

Performance analysis and optimization Code coverage tests can identify the areas of a code that are executed most frequently. Extra attention can then be paid to these sections of the code. If further performance improvement is no longer possible, then other strategies like caching can be considered. Code coverage testing provides information that is useful in making such performance-oriented decisions. Resource usage analysis White box testing, especially with instrumented code, is useful in identifying bottlenecks in resource usage. For example, if a particular resource like the RAM or network is perceived as a bottleneck, then instrumented code can help identify where the bottlenecks are and point towards possible solutions. Checking of critical sections or concurrency related parts of code Critical sections are those parts of a code that cannot have multiple processes executing at the same time. Coverage tests with instrumented code is one of the best means of identifying any violations of such concurrency constraints through critical sections. Identifying memory leaks Every piece of memory that is acquired or allocated by a process (for example, by malloe in C) should be explicitly released (for example, by free in C). If not, the acquired memory is "lost" and the amount of available memory decreases correspondingly. Over time, there would be no memory available for allocation to meet fresh memory requests and processes start failing for want of memory. The various white box testing methods can help identify memory leaks. Most debuggers or instrumented code can tally allocated and freed memory. Dynamically generated code White box testing can help identify security holes effectively, especially in a dynamically generated code. In instances where a piece of code is dynamically created and executed, the functionality of the generated code should be tested on the fly. For example, when using web services, there may be situations wherein certain parameters are accepted from the users and html/java code may be generated and passed on to a remote machine for execution. Since after the transaction or service is executed, the generated code ceases to exist, testing the generated code requires code knowledge. Hence, the various techniques of white box testing discussed in this chapter come in handy.

3.3.3

Code Complexity Testing

In previous sections, we saw the different types of coverage that can be provided to test a program. Two questions that come to mind while using these coverage are: 1. 2. Which of the paths are independent? If two paths are not independent, then we may be able to minimize the number of tests. Is there an upper bound on the number of tests that must be run to ensure that all the statements have been executed at least once?

-~
64 SojtwareTesting

Cyclomatic complexity is a metric that quantifies the complexity of a program and thus provides answers to the above questions. A program is represented in the form of aflow graph. A flow graph consists of nodes and edges. In order to convert a standard flow chart into a flow graph to compute cyclomatic complexity, the following steps can be taken. 1. Identify the predicates or decision points (typically the Boolean conditions or conditional statements) in the program. 2. Ensure that the predicates are simple (that is, no and/or, and so on in each predicate). Figure 3.3 shows how to break up a condition having or into simple predicates. Similarly, if there are loop constructs, break the loop termination checks into simple predicates. 3. Combine all sequential statements into a single node. The reasoning here is that these statements all get executed, once started. 4. When a set of sequential statements are followed by a simple predicate (as simplified in (2) above), combine all the sequential statements and the predicate check into one node and have two edges emanating from this one node. Such nodes with two edges emanating from them are called predicate nodes. 5. Make sure that all the edges terminate at some node; add a node to represent all the sets of sequential statements at the end of the program.

We have illustrated the above transformation rules of a conventional flow chart to a flow diagram in Figure 3.4.We have color coded the different boxes (coloured figure on page 459) so that the reader can see the transformations more clearly. The flow chart elements of a given color on the left-hand side get mapped to flow graph elements of the corresponding nodes on the righthand side. Intuitively, a flow graph and the cyclomatic complexity provide indicators to the complexity of the logic flow in a program and to the number of independent paths in a program. The primary contributors to both the

Figure 3.3 Flow graph translation of an OR to a simple predicate.

(a) A predicate with a Boolean OR

(b) An equivalent set of simple predicates

_if
Figure 3.4 Converting a conventional flow chart to a flow graph.

7' ,.,,;.'~L':~'E::::'"
Conventional flow chart

White

Box

Testing

65 .

Flow diagram for calculating complexity

Figure 3.5 A hypothetical program with no decision node.

# of independent paths

=1 # of nodes, N=2 # of edges, E=1 Cyclomatic complexity = E N + 2 = 1 # of predicate nodes, P = 0 Cyclomatic complexity = P+ 1 = 1

complexity and independent paths are the decision points in the program. Consider a hypothetical program with no decision points. The flow graph of such a program (shown in Figure 3.5 above) would have two nodes, one for the code and one for the termination node. Since all the sequential steps are combined into one node (the first node), there is only one edge, which connects the two nodes. This edge is the only independent path. Hence, for this flow graph, cyclomatic complexity is equal to one. This graph has no predicate nodes because there are no decision points. Hence, the cyclomatic complexity is also equal to the number of predicate nodes (0) + 1. Note that in this flow graph, the edges (E) = 1; nodes (N) cyclomatic complexity is also equal to 1 = 1 + 2 - 2 = E - N + 2.
=

2. The

66

Software Te~ting

Figure 3.6 Adding one decision node.

~# of independent paths ~# of nodes, N=4 ~# of edges, E=4

~Cyclomatic ~Cyclomatic complexity complexity

=2

~# of predicate nodes, P

= E- N+2 =2 =1 = P+1 = 2

When a predicate node is added to the flow graph (shown in Figure 3.6 above), there are obviously two independent paths, one following the path when the Boolean condition is TRUE and one when the Boolean condition is FALSE. Thus, the cyclomatic complexity of the graph is 2. Incidentally, this number of independent paths, 2, is again equal to the number of predicate nodes (1) + 1. When we add a predicate node (a node with two edges), complexity increases by 1, since the "E" in the E - N + 2 formula is increased by one while the "N" is unchanged. As a result, the complexity using the formula E - N + 2 also works out to 2. From the above reasoning, the reader would hopefully have got an idea about the two different ways to calculate cyclomatic complexity and the relevance of cyclomatic complexity in identifying independent paths through a program. We have summarized these formulae below. We are not going to formally prove these formulae. Suffice to say that these formulae are extremely useful. The above two formulae provide an easy means to calculate cyclomatic complexity, given a flow graph. In fact the first formula can be used even without drawing the flow graph, by simply counting the number of the basic predicates. There are other formulations of cyclomatic complexity derived from the foundations of Graph Theory, which we have not covered here. The references given at the end can provide pointers for the interested reader. Using the flow graph, an independent path can be defined as a path in the flow graph that has at least one edge that has not been traversed before in other paths. A set of independent paths that cover all the edges is a basis set. Once the basis set is formed, test cases should be written to execute all the paths in the basis set.

White Box Testing

Table 3.1

67

3.3.3.1 calculating and using c.ydomatic complexity Forsmallprograms cyclomaticcomplexity can be calculated manually, but automated tools are essential as several thousands of lines of code are possible in each program in a project. It will be very difficult to manually create flow graphs for large programs. There are several tools that are available in the market which can compute cyclomatic complexity. But, we would like to caution that calculating the complexity of a module after it has been built and tested may be too late-it may not be possible to redesign a complex module after it has been tested. Thus some basic complexity checks must be performed on the modules before embarking upon the testing (or even coding) phase. This can become one of the items to check for in a code review. Based on the complexity number that emerges from using the tool, one can conclude what actions need to be taken for complexity measure using Table 3.1.

3.4

CHALLENGESIN WHITE BOX TESTING

White box testing requires a sound knowledge of the program code and the programming language. This means that the developers should get intimately involved in white box testing. Developers, in general, do not like to perform testing functions. This applies to structural testing as well as static testing methods such as reviews. In addition, because of the timeline pressures, the programmers may not "find time" for reviews (an euphemism for wanting to do more coding). We will revisit this myth of dichotomy between testing and development functions in the chapter on people issues (Chapter 13). Human tendency of a developer being unable to find the defects in his or her code As we ~aw earlier, most of us have blind spots in detecting errors in our own products. Since white box testing involves programmers who write the code, it is quite possible that they may not be most effectivein detecting defects in their own work products. An independent perspective could certainly help. Fully tested code may not correspond to realistic scenarios Programmers generally do not have a full appreciation of the external (customer)

68

Software

perspective or the domain knowledge to visualize how a product will be deployed in realistic scenarios. This may mean that even after extensive testing, some of the common user scenarios may get left out and defects may creep in. These challenges do not mean that white box testing is ineffective. But when white box testing is carried out and these challenges are addressed by other means of testing, there is a higher likelihood of more effective testing. Black box testing, to be discussed in the following chapter addresses some of these challenges.

REFERENGES
[MCCA-76]defined the Cyclomatic Complexity. The web reference [NIST-I] discusses code coverage and code complexity. [FAGA-86]covers the details of inspection. [PRES-97]provides a comprehensive coverage of all white box testing issues.

I ~IPROBLEMS
1.

AND EXERGISE
In the book, we discussed how to convert a Boolean or to simple predicates that can be used to derive the flow graph. Following similar technique, convert an expression like if A and B to a flow graph. Given below is a simple C program to accept a set of inputs and calculate standard deviation. It has consciously been seeded with defects. If you were asked to perform white box testing on this program, identify some of the defects in the program. Also, list the methodology you used to identify these defects

2.

White Box Testing

69

3. 4.

For the above program, draw the flow graph and hence calculate the cyclomatic complexity of the program. Given below is a C program for deleting an element from a linked list. Suggest a set of test data to cover each and every statement of this program.

"".;.:

:.-c'i'o-'- -----;.;."':

'Software Testing

5.

In the example of the previous problem, even when the set of test

6.

data provides 100% statement coverage,show that there are still uncovered defects. Given below are parts of the code segment actually written by two students for date validation. The two students used two different approaches as given in the code segments 6a and 6b. Which of these two would you think is easier from a white box testing perspective? Which of the techniques of white box testing discussed in this chapter would be most relevant to apply for 6a and which for 6b? How would you improve the effectiveness of the code in terms of susceptibility to defects in either case and ability to detect defects more easily?

White130x Testing

7.

In Problem 6b, the test data presented in Table PI was used to test the leap year part of the validation. Calculate the various coverage factors for this test data for the leap year part of the code.

Table P1

Values for testing segment 38.

8.

Write a simple matrix multiplication program and try to take care of as many valid and invalid conditions as possible. Identify what test data would you use for testing program. Justify your answer.

9.

A reentrant code is a piece of program that does not modify itself. In contrast, a non-reentrant code is a piece of program that modifies itself. Discuss the problems you will face from the white box testing perspective and from the point of view of ongoing maintenance of such a program. 10. Discuss the negative effects of the following constructs from a white box testing perspective: a. GOTO statements b. Global variables 11. Discuss the pros and cons of function coverage vis a vis other forms of coverage.

In this chapter../ What is black box testing ../ Why black box testing ../ When to do black box testing ../ How to do black box testing ../ Conclusion

74

Software Testing

4.1

WHAT IS BLACK BOX TESTING?

Black box testing involves looking at the specifications and does not , require examining the code of a program. Black box testing is done from the customer's viewpoint. The test engineer engaged in black box testing only knows the set of inputs and expected outputs and is unaware of how those inputs are transformed into outputs by the software. Black box tests are convenient to administer because they use the complete finished product and do not require any knowledge of its construction. Independent test laboratories can administer black box tests to ensure functionality and compatibility.

Black Box Testing

7S

Black box testing thus requires a functional knowledge of the product to be tested. It does not mandate the knowledge of the internal logic of the system nor does it mandate the knowledge of the programming language used to build the product. Our tests in the above example were focused towards testing the features of the product (lock and key), the different states, we already knew the expected outcome. You may check if the lock works with some other key (other than its own). You may also want to check with a hairpin or any thin piece of wire if the lock works. We shall see in further sections, in detail, about the different kinds of tests that can be performed in a given product.

4.2

WHY BLACK BOX TESTING

Black box testing helps in the overall functionality verification of the system under test. Black box testing is done based on requirements It helps in identifying any incomplete, inconsistent requirement as well as any issues involved when the system is tested as a complete entity. Black box testing addresses the stated requirements as well as implied requirements Not all the requirements are stated explicitly,but are deemed implicit. For example, inclusion of dates, page header, and footer may not be explicitly stated in the report generation requirements specification. However, these would need to be included while providing the product to the customer to enable better readability and usability. Black box testing encompasses the end user perspectives Since we want to test the behavior of a product from an external perspective, end-user perspectives are an integral part of black box testing. Black box testing handles valid and invalid inputs It is natural for users to make errors while using a product. Hence,it is not sufficientfor black box testing to simply handle valid inputs. Testing from the end-user perspective includes testing for these error or invalid conditions.Thisensures that the product behaves as expected in a valid situation and does not hang or crash when provided with an invalid input. These are called positive and negative test cases. The tester mayor may not know the technology or the internal logic of the product. However, knowing the technology and the system internals helps in constructing test cases specific to the error-prone areas.

76

Software Testing

Test scenarios can be generated as soon as the specifications are ready. Since requirements specifications are the major inputs for black box testing, test design can be started early in the cycle.

4.3

WHEN TO DO BLACK BOX TESTING?

Black box testing activities require involvement of the testing team from the beginning of the software project life cycle, regardless of the software development life cycle model chosen for the project. Testers can get involved right from the requirements gathering and analysis phase for the system under test. Test scenarios and test data are prepared during the test construction phase of the test life cycle, when the software is in the design phase. Once the code is ready and delivered for testing, test execution can be done. All the test scenarios developed during the construction phase are executed. Usually, a subset of these test scenarios is selected for regression testing.

4.4

HOW TO DO BLACK BOX TESTING?

As we saw in Chapter I, it is not possible to exhaustively test a product, however simple the product is. Since we are testing external functionality in black box testing, we need to arrive at a judicious set of tests that test as much of the external functionality as possible, uncovering as many defects as possible, in as short a time as possible. While this may look like a utopian wish list, the techniques we will discuss in this section facilitates this goal. This section deals with the various techniques to be used to generate test scenarios for effective black box testing. The various techniques we will discuss are as follows. 1. 2. 3. 4. 5. 6. 7. 8. 9. Requirements based testing Positive and negative testing Boundary value analysis Decision tables Equivalence partitioning State based testing Compatibility testing User documentation testing Domain testing

4.4.1

Requirements Based Testing

Requirements testing deals with validating the requirements given in the Software Requirements Specification (SRS)of the software system.

Black Box

Testing

77

As mentioned in earlier chapters, not all requirements are explicitly stated; some of the requirements are implied or implicit. Explicit requirements are stated and documented as part of the requirements specification. Implied or implicit requirements are those that are nor documented but assumed to be incorporated in the system. The precondition for requirements testing is a detailed review of the requirem~s sper~cation. ReqvJEfments review ensures that they are (jconsistent;!torrect~omplete, and=testable. This process ensures that some implied requirements are converted and documented as explicit requirements, thereby bringing better clarity to requirements and making requirements based testing more effective. Some organizations follow a variant of this method to bring more details into requirements. All explicit requirements (from the Systems Requirements Specifications) and implied requirements (inferred by the test team) are collected and documented as "Test Requirements Specification" (TRS).Requirements based testing can also be conducted based on such a TRS, as it captures the testers' perspective as well. However, for simplicity, we will consider SRSand TRSto be one and the same. A requirements specification for the lock and key example explained earlier can be documented as given in Table 4.1. Requirements (like the ones given above) are tracked by a Requirements Traceability Matrix (RTM). An RTM traces all the requirements from their

Table 4.1

Sample requirements

specification for lock and key system.

78

Software Testing

genesis through design! development! and testing. This matrix evolves through the life cycle of the project. To start with, each requirement is given a unique id along with a brief description. The requirement identifier and description can be taken from the Requirements Specification (above table) or any other available document that lists the requirements to be tested for the product. In the above table, the naming convention uses a prefix UBRU followed by a two-digit number. BR indicates the type of testing-UBlack box-requirements testing." The two-digit numerals count the number of requirements. In systems that are more complex! an identifier representing a module and a running serial number within the module (for example! INV-OL AP-02! and so on) can identify a requirement. Each requirement is assigned a requirement priority, classified as high, medium or low. This not

Table 4.2

Sample Requirements Traceability Matrix.

only enables prioritizing the resources for development of features but is also used to sequence and run tests. Tests for higher priority requirements will get precedence over tests for lower priority requirements. This ensures that the functionality that has the highest risk is tested earlier in the cycle. Defects reported by such testing can then be fixed as early as possible. As we move further down in the life cycle of the product, and testing phases, the cross-reference between requirements and the subsequent phases is recorded in the RTM. In the example given here, we only list the mapping between requirements and testing; in a more complete RTM, there will be columns to reflect the mapping of requirements to design and code. The "test conditions" column lists the different ways of testing the requirement. Test conditions can be arrived at using the techniques given in this chapter. Identification of all the test conditions gives a comfort feeling that we have not missed any scenario that could produce a defect in the end-user environment. These conditions can be grouped together to form a single test case. Alternatively, each test condition can be mapped to one test case. The "test case IDs" column can be used to complete the mapping between test cases and the requirement. Test case IDs should follow naming conventions so as to enhance their usability. For example, in Table 4.2, test cases are serially numbered and prefixed with the name of the product. In a more complex product made up of multiple modules, a test case ID may be identified by a module code and a serial number. Once the test case creation is completed, the RTM helps in identifying the relationship between the requirements and test cases. The following combinations are possible. ~ ~ ~ ~ One to one-For example, BR-Ol) each requirement there is one test case (for

One to many-For each requirement there are many test cases (for example, BR-03) Many to one-A set of requirements can be tested by one test case (not represented in Table 4.2) Many to many - Many requirements can be tested by many test cases (these kind of test cases are normal with integration and system testing; however, an RTM is not meant for this purpose) One to none- The set of requirements can have no test cases. The test team can take a decision not to test a requirement due to nonimplementation or the requirement being low priority (for example, BR-08)

A requirement is subjected to multiple phases of testing-unit, component, integration, and system testing. This reference to the phase of testing can be provided in a column in the Requirements Traceability Matrix. This column indicates when a requirement will be tested and at what phase of testing it needs to be considered for testing.

80

Software Testing

An RTM plays a valuable role in requirements based testing. 1. Regardless of the number of requirements, ideally each of the requirements has to be tested. When there are a large numbers of requirements, it would not be possible for someone to manually keep a track of the testing status of each requirement. The RTM provides a tool to track the testing status of each requirement, without missing any (key) requirements. 2. By prioritizing the requirements, the RTM enables testers to prioritize the test cases execution to catch defects in the highpriority area as early as possible. It is also used to find out whether there are adequate test cases for high-priority requirements and to reduce the number of test cases for low-priority requirements. In addition, if there is a crunch for time for testing, the prioritization enables selecting the right features to test. 3. Test conditions can be grouped to create test cases or can be represented as unique test cases. The list of test case(s) that address a particular requirement can be viewed from the RTM. 4. Test conditions/cases can be used as inputs to arrive at a size / effort / schedule estimation of tests.

The Requirements Traceability Matrix provides a wealth of information on various test metrics. Some of the metrics that can be collected or inferred from this matrix are as follows. ~ Requirements addressed prioritywise- This metric helps in knowing the test coverage based on the requirements. Number of tests that is covered for high-priority requirement versus tests created for low-priority requirement. Number of test cases requirementwise-For each requirement, the total number of test cases created. Total number of test cases prepared - Total of all the test cases prepared for ail requirements.

~ ~

Once the test cases are executed, the test results can be used to collect metrics such as ~ Total number of test cases (or requirements) passed-Once test execution is completed, the total passed test cases and what percent of requirements they correspond. Total number of test cases (or requirements) failed - Once test execution is completed, the total number of failed test cases and what percent of requirements they correspond. Total number of defects in requirements-List of defects reported for each requirement (defect density for requirements). This helps in doing an impact analysis of what requirements have more defects and how they will impact customers. A comparatively high-defect density in low-priority requirements is acceptable for a release. A high-defect density in high-priority requirement is considered a high-risk area, and may prevent a product release.