The document summarizes the purposes, activities, processes, and context of software testing. The key purposes of testing are to ensure software systems work as expected and to detect and fix problems. There are four main testing processes - test planning and preparation, test execution, analysis and follow up, and defect prevention techniques. Software testing can be functional or structural, with functional testing focusing on external behavior and structural testing examining internal implementations. Coverage-based and usage-based criteria are discussed for determining when to stop testing.
The document summarizes the purposes, activities, processes, and context of software testing. The key purposes of testing are to ensure software systems work as expected and to detect and fix problems. There are four main testing processes - test planning and preparation, test execution, analysis and follow up, and defect prevention techniques. Software testing can be functional or structural, with functional testing focusing on external behavior and structural testing examining internal implementations. Coverage-based and usage-based criteria are discussed for determining when to stop testing.
The document summarizes the purposes, activities, processes, and context of software testing. The key purposes of testing are to ensure software systems work as expected and to detect and fix problems. There are four main testing processes - test planning and preparation, test execution, analysis and follow up, and defect prevention techniques. Software testing can be functional or structural, with functional testing focusing on external behavior and structural testing examining internal implementations. Coverage-based and usage-based criteria are discussed for determining when to stop testing.
Purposes, activities , processes , and context of Software Testing.
Purpose: to ensure that the software systems would work as expected when they are used by their target customers and users. o Demonstrations of proper behavior /providing evidence of quality (terms of s/w QA) /meeting certain quality goals is primary purpose of testing. o to detect and fix problems Activities: to show that s/w fulfills the expectations we demonstrate their operations through some dry-runs or controlled experimentations through program executions before the products are released/delivered TESTING. Processes: focus testing in development process to achieve defect free manufacturing process. Major activities and the generic testing process: Test Planning and preparation: which set the goals for testing, select an overall testing strategy and prepare specific test cases and the general test procedure. Test execution: and related activities. Which also include related observation and measurement of product behavior. Analysis and follow up: includes result checking and analysis to determine if a failure has been observed. If so follow up activities are initiated and monitored to ensure the removal of the faults / causes that led to the failures. Refer to fig 6.1 text book. Test run the execution of a specific test case, or a sub-division of the overall test execution sequence for some systems that require continuous operation. Effective test execution is handling the problems of the failed test runs so that it wont block the executions of the other test runs especially in systems which require continuous operations. If the test run fails , the data collected during execution or other measurements can be used to locate and fix the underlying faults that led to the failure. Analysis reports provide assessments of current status wrt progress,effort,defect, and product quality so decisions of when to stop testing can be made based on the facts . Sub-activities in test planning and preparation: Goal setting: quality views and attributes have been decided by the overall quality engineering process. Specific testing goals ,such as reliability or coverage goals, to be used as exit criteria. Test case preparation: includes constructing new test cases or generating them automatically, selecting from existing ones for legacy products and organizing them in some systematic ways for easy execution and management. Test procedure preparation: for systematic testing on large scale for today s/w products and systems, a formal procedure is more of a necessity. It can be defined and followed to ensure effective test execution, problem handling and resolution and overall test process management. Other QA alternatives to complement testing: Defect prevention reduces defect injections during s/w dev, resulting in fewer faults to be detected and removed during testing, thus reducing testing efforts and expenditure. Formal verification to verify correctness of some core functions in a product instead of applying exhaustive testing to them. Fault tolerance and failure containment strategies might be appropriate for critical systems. Test result analysis and follow up activities have to be carried out in parallel to testing and should not stop to ensure discovered problems are all resolved and long term improvement initiatives are planned and carried out. Functional VS Structural testing : There are different levels of testing corresponding to different views of the code and different levels of abstraction, as follows: Detailed level: o Individual program elements- individual statements, decisions, data items can be tested. o Depending on the programming language used, this unit may correspond to a function, a procedure, a subroutine or a method. o Components- collection of smaller units that together accomplish something or form an object Intermediate level: o Various program elements or program components may be treated as interconnected group and tested. o Done at component, sub-system, or system levels with the help of some models to capture the interconnection and other relations among different elements or components. Abstract level: o Whole s/w systems can be treated as black box. o Focus on the functions or input-output relations Tendency is that at higher levels of abstraction, functional testing is used and at lower levels of abstraction, structural testing is used. Functional / black box testing(BBT): Verifies the correct handling of the external functions provided by the s/w, through the observation of the program external behavior during execution. Ad-hoc testing- simplest form of BBT start running the s/w and make observations in the hope that it is easy to distinguish between expected and unexpected behavior. o Unexpected behavior like, crashes are easy to detect and after multiple executions to rule out hardware problems, the info is passed to responsible party to get it fixed. Specification Checklists- common form of BBT- lists the external functions that are supposed to be present, as well as some info about the expected behavior or input- output pairing. o Input/output-any action, artifact, or result produced by running program either at beginning /end or at any time during program execution. Formalized and systematic BBT can be based on some formal models which are derived from system requirement or functional specifications. BBT can follow generic testing process described above. o Planning stage: identify external functions to test, and derive input conditions to test these functions. o Ex: compiler where the input is the source code to be complied and output is executable code. o Part of expected behavior is system termination- compiler must produce output within given time. o Another expected behavior is when illegal program is inputted then the output is not generated. o Collection of programs to be complied constitutes the test suite or collection of test cases which consists or legal and illegal programs to cover the expected spectrum of input. Focus is to observe the external behavior, to ensure orderly execution of all the test cases, and to record execution information for analysis and follow up activities. If observed behavior patterns cannot be identified as failures, info needs to be recorded for further analysis. Oracle testing problem- once execution result is obtained, analysis is carried out to compare the specific behavior and the output with expected ones to determine if expected behavior or failure occurred. Once failures are detected follow up activities are initiated to detect faults and remove them. Info recorded during test execution is used to recreate the failure scenarios, diagnose problems and causes and fix them. Emphasis is on reducing chances of encountering functional problems by target customers. Structural/ white box testing (WBT): Verifies the correct implementation of internal units, such as program statements, data structures, blocks, etc. and relations among them. This is done through test execution by observing the program behavior related to these specific units. Connection between execution behavior and internal units needs to be done and its done with various s/w tools. Statement coverage testing- simplest form of WBT- helps trace the program executions with help of various debugging tools and debuggers. o Tester can see if a specific statement has been executed and if the result or behavior is as expected. Adv: problem once detected is also located. Dis adv: omission or design problems are not easily detected as only what is present in code is tested. Tester should be familiar with code under testing to trace through the executions and also to perform related activities. This dual role makes defect fixing easy. Formalized and systematic WBT can be based on formal models which are derived from system implementation details. WBT can also follow generic testing process but due to extensive amount of implementation knowledge required, WBT is limited to small scale. For small products not much formal testing process is needed to plan and execute the test cases and follow up on execution results. Stopping criterion: internal quality measures, defect levels, once planned coverage has been achieved by exercising all the statements and all paths. Comparing BBT and WBT see text book: pg . 77 Coverage based VS Usage based testing : when to stop testing? Without formal assessment the decision to stop testing is made in 2 general forms: o Resource based criteria: where decision is made on resource consumptions. Stop when you run out of time Stop when you run out of money. This is used when product schedule or cost is dominant concerns but it is not good for quality of product. o Activity based criteria: stop when you complete the planned test activities. Based on formal analyses and assessments: o On global level , exit criteria determined by product release which is determined by the reliability assessment-involves achieving quality goals and project goals in over s/w dev process to meet the customers expectations. o In reliability criterion, u set the reliability goal in the quality planning activity during the product planning and requirement analysis and later on compares the reliability assessment based on testing data to see if this pre-set goal has been reached. If yes, product can be released else continue testing. o Implications of using this criteria For stopping testing is that Reliability testing should be as close to what actual users would expect which requires testing right before product release. Resembles what actual usages by customers. This resulted in usage based testing o For testing internal components the use of reliability criteria is not relevant as users are not in direct contact with it. So alternative exit criteria is needed. o Rule of thumb: products should not be released unless every component has been tested. o Implication for using coverage as stopping criterion is that everything covered is defect free with respect to this specific coverage aspect as all defects found would have been fixed before product release. o Specific quality goal can be translated into specific coverage goal.
Usage based testing (UBST) and operational profiles(Ops) Actual customer usage of software products can be viewed as a form of usage based testing and problems if encountered can be reported to software vendors and integrated fixes can be constructed and delivered. o This is very expensive due to large no of s/w installations o Frequent fixes can damage the s/w vendors reputation n business Beta test helps s/w vendors through controlled release to beta customers who help improve their s/w quality. Testing environment resembles the actual operational environment for the s/w product in the field and the overall testing sequence as represented by the orderly execution of specific test cases in a test suite, resembles the usage scenarios, sequences, and patterns of actual software usage by the customers. For practical implementation of such a testing strategy, actual usage info needs to be captured in various models, commonly referred to as operational profiles or Ops. Different OPs are associated with different testing techniques for UBST.2 models are: o Flat Ops or Musa OPs: which is present commonly used operations in a list, a histogram, or a tree structure, together with the associated occurrence probabilities. Simple in model construction and usage. o Markov chain based usage model/Markov OPs: present the commonly used operational units in markovs chain, where the state transition probabilities are history independent. Complete operations can be constructed by linking various states together following the state transitions, and the probability for whole path is the product of its individual transition probabilities. Markov models based on state transitions can capture navigation patterns than flat OPs, but are more expensive to maintain and use. UBST is generally applicable to final stage of testing, typically referred to as acceptance testing right before product release, so stopping testing is = to releasing the product. Coverage and Coverage based Testing(CBT) There are various forms of test coverage as stopping criteria: o Completing various checklists is the simplest form. For BBT- checklist of major functions based on product specifications WBT checklist of all product components or all statements o Testing can be performed until all the items on the checklist have been checked off. For systematic testing, formal models are used: o Formally defined partitions can be used as the basis for various testing techniques which are similar to checklists but ensure: Mutual exclusion of checklist items to avoid unnecessary repetition. Complete coverage defined accordingly o A specialized type of partitions, input domain partitions into sub-domains, can also be used to test these sub-domains and related boundary conditions. o Various programming or functional states can be defined and linked together to form finite state machines(FSMs) to model the system as the basis for various testing techniques to ensure state coverage and coverage of related state transitions and execution sequences. o This FSM can also be extended to analyse and cover the execution paths and data dependencies through various testing techniques.