BLACK BOX TESTING

 Black Box Testing is testing without knowledge of the internal workings

of the item being tested .It treats the system as a "black-box". For
example, when black box testing is applied, the tester would only know
the "legal" inputs and what the expected outputs should be, but not how
the program actually arrives at those outputs. It is because of this that
black box testing can be considered testing with respect to the
specifications, no other knowledge of the program is necessary.

 The opposite of this would be white box testing, where test data are
derived from direct examination of the code to be tested. For white box
testing, the test cases cannot be determined until the code has actually
been written.

 Black box testing, also called functional testing and behavioral testing,
focuses on determining whether or not a program does what it is
supposed to do based on its functional requirements. Black box testing
attempts to find errors in the external behavior of the code in the
following categories:

 Incorrect or missing functionality;

 Interface errors;
 Errors in data structures used by interfaces;
 Behavior or performance errors; and
 Initialization and termination errors.

 Through this testing, we can determine if the functions appear to work

according to specifications. However, it is important to note that no
amount of testing can unequivocally demonstrate the absence of errors
and defects in your code.

 It is best if the person who plans and executes black box tests is not the
programmer of the code and does not know anything about the structure
of the code. The programmers Testing Overview and Black-Box Testing
 Techniques of the code are innately biased and are likely to test that the
program does what they programmed it to do. What are needed are tests
to make sure that the program does what the customer wants it to do. As
a result, most organizations have independent testing groups to perform
black box testing. These testers are not the developers and are often
referred to as third-party testers. Testers should just be able to understand
and specify what the desired output should be for a given input into the
program, as shown in Figure 3.

Testing Strategies/Techniques

 Black box testing should make use of randomly generated inputs (only a
test range should be specified by the tester), to eliminate any guess work
by the tester as to the methods of the function.

 Data outside of the specified input range should be tested to check the
robustness of the program.

 Boundary cases should be tested (top and bottom of specified range) to

make sure the highest and lowest allowable inputs produce proper output.

 The number zero should be tested when numerical data is to be input.

 Stress testing should be performed (try to overload the program with

inputs to see where it reaches its maximum capacity), especially with real
time systems.
 Crash testing should be performed to see what it takes to bring the system
down.

 Test monitoring tools should be used whenever possible to track which

tests have already been performed and the outputs of these tests to avoid
repetition and to aid in the software maintenance.

 Other functional testing techniques include: transaction testing, syntax

testing, domain testing, logic testing, and state testing.

 Finite state machine models can be used as a guide to design functional

tests.

According to Beizer, the following is a general order by which tests should

be designed:

1. Clean tests against requirements.

2. Additional structural tests for branch coverage, as needed.

3. Additional tests for data-flow coverage as needed.

4. Domain tests not covered by the above.

5. Special techniques as appropriate--syntax, loop, state, etc.

6. Any dirty tests not covered by the above.

Though centered around the knowledge of user requirements, black box tests
do not necessarily involve the participation of users. Among the most
important black box tests that do not involve users are functionality testing,
volume tests, stress tests, recovery testing, and benchmarks (Thaller94, 138-
149). Additionally, there are two types of black box test that involve users,
i.e. field and laboratory tests (Karat90, 352) and (Crellin90, 330). In the
following the most important aspects of these black box tests will be
described briefly.

Advantages of Black Box Testing

1. It is more effective on larger units of code than glass box testing.

2. The tester needs no knowledge of implementation, including specific

programming languages.

3. The tester and programmer are independent of each other.

4. The tests are done from a user's point of view.

5. It helps to expose any ambiguities or inconsistencies in the

specifications.

6. The test cases can be designed as soon as the specifications are

complete.

Disadvantages of Black Box Testing

1. Only a small number of possible inputs can actually be tested, to test

every possible input stream would take nearly forever.

2. Without clear and concise specifications, test cases are hard to design.

3. There may be unnecessary repetition of test inputs if the tester is not

informed of test cases the programmer has already tried.

4. It may leave many program paths untested.

5. It cannot be directed toward specific segments of code which may be

very complex (and therefore more error prone).
 6. Most testing related research has been directed toward glass box
testing.

Black box testing - without user involvement

The so-called ``functionality testing'' is central to most testing exercises. Its
primary objective is to assess whether the program does what it is supposed
to do, i.e. what is specified in the requirements. There are different
approaches to functionality testing. One is the testing of each program
feature or function in sequence (Musa87, 521). The other is to test module
by module, i.e. each function where it is called first (Thaller94, 138).

The objective of volume tests is to find the limitations of the software by

processing a huge amount of data (Thaller94, 139). A volume test can
uncover problems that are related to the efficiency of a system, e.g. incorrect
buffer sizes, a consumption of too much memory space, or only show that an
error message would be needed telling the user that the system cannot
process the given amount of data.

During a stress test, the system has to process a huge amount of data or
perform many function calls within a short period of time. A typical example
could be to perform the same function from all workstations connected in a
LAN within a short period of time (e.g. sending e-mails, or, in the NLP area,
to modify a term bank via different terminals simultaneously).

The aim of recovery testing is to make sure to which extent data can be
recovered after a system breakdown. Does the system provide possibilities
to recover all of the data or part of it? How much can be recovered and how?
Is the recovered data still correct and consistent? Particularly for software
that needs high reliability standards, recovery testing is very important.

The notion of benchmark tests involves the testing of program efficiency.

The efficiency of a piece of software strongly depends on the hardware
environment and therefore benchmark tests always consider the
soft/hardware combination (Thaller94, 147). Whereas for most software
engineers benchmark tests are concerned with the quantitative measurement
of specific operations (Thaller94, 146), some also consider user tests that
compare the efficiency of different software systems as benchmark tests
(Lewis90, 337-343). In the context of this document, however, benchmark
tests only denote operations that are independent of personal variables.

Black box testing - with user involvement

For tests involving users, methodological considerations are rare in SE
literature. Rather, one may find practical test reports that distinguish roughly
between field and laboratory tests (Karat90), (Crellin90) and (Moll88). In
the following only a rough description of field and laboratory tests will be
given. For details see Scenario Tests.

In field tests users are observed while using the software system at their
normal working place. Apart from general usability-related aspects, field
tests are particularly useful for assessing the interoperability of the software
system, i.e. how the technical integration of the system works. Moreover,
field tests are the only real means to elucidate problems of the organisational
integration of the software system into existing procedures. Particularly in
the NLP environment this problem has frequently been underestimated. A
typical example of the organisational problem of implementing a translation
memory is the language service of a big automobile manufacturer, where the
major implementation problem is not the technical environment, but the fact
that many clients still submit their orders as print-out, that neither source
texts nor target texts are properly organised and stored and, last but not least,
individual translators are not too motivated to change their working habits.

Laboratory tests are mostly performed to assess the general usability of the
system. Due to the high laboratory equipment costs laboratory tests are
mostly only performed at big software houses such as IBM or Microsoft.
Since laboratory tests provide testers with many technical possibilities, data
collection and analysis are easier than for field tests.