Testing Basics

Test case
From Wikipedia, the free encyclopedia
Jump to: navigation, search This article is about the term in software engineering. For the use of the term in law, see Test case (law). This article does not cite any references or sources.
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (July 2009) Software Testing portal
A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test oracle. In some settings, an oracle could be a requirement or use case, while in others it could be a heuristic. It may take many test cases to determine that a software program or system is functioning correctly. Test cases are often referred to as test scripts, particularly when written. Written test cases are usually collected into test suites.
Contents
[hide]

1 Formal test cases 2 Informal test cases 3 Typical written test case format 4 References 5 External links
[edit] Formal test cases

In order to fully test that all the requirements of an application are met, there must be at least two test cases for each requirement: one positive test and one negative test; unless a requirement has sub-requirements. In that situation, each sub-requirement must have at least two test cases. Keeping track of the link between the requirement and the test is frequently done using a traceability matrix. Written test cases should include a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted. A formal written test-case is characterized by a known input and by an expected output, which is worked out before the test is executed. The known input should test a precondition and the expected output should test a postcondition.
[edit] Informal test cases
For applications or systems without formal requirements, test cases can be written based on the accepted normal operation of programs of a similar class. In some schools of testing, test cases are not written at all but the activities and results are reported after the tests have been run. In scenario testing, hypothetical stories are used to help the tester think through a complex problem or system. These scenarios are usually not written down in any detail. They can be as simple as a diagram for a testing environment or they could be a description written in prose. The ideal scenario test is a story that is motivating, credible, complex, and easy to evaluate. They are usually different from test cases in that test cases are single steps while scenarios cover a number of steps.
[edit] Typical written test case format

A test case is usually a single step, or occasionally a sequence of steps, to test the correct behaviour/functionalities, features of an application. An expected result or expected outcome is usually given. Additional information that may be included:

test case ID test case description test step or order of execution number related requirement(s) depth test category author check boxes for whether the test is automatable and has been automated.
Additional fields that may be included and completed when the tests are executed:

pass/fail remarks
Larger test cases may also contain prerequisite states or steps, and descriptions. A written test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table. Test suites often also contain

Test summary Configuration
Besides a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted, the most time consuming part in the test case is creating the tests and modifying them when the system changes. Under special circumstances, there could be a need to run the test, produce results, and then a team of experts would evaluate if the results can be considered as a pass. This happens often on new products' performance number determination. The first test is taken as the base line for subsequent test / product release cycles. Acceptance tests, which use a variation of a written test case, are commonly performed by a group of end-users or clients of the system to ensure the developed system meets the requirements specified or the contract. User acceptance tests are differentiated by the inclusion of happy path or positive test cases to the almost complete exclusion of negative test cases.
Test automation
Jump to: navigation, search It has been suggested that this article or section be merged with Test execution engine. (Discuss) It has been suggested that this article or section be merged with Test automation framework. (Discuss) Compare with Manual testing. Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions[1]. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
Contents
[hide]

1 Overview 2 Code-driven testing 3 Graphical User Interface (GUI) testing 4 What to test 5 Framework approach in automation 6 Popular Test Automation Tools 7 See also 8 References 9 External links
[edit] Overview
Although manual tests may find many defects in a software application, it is a laborious and time consuming process. In addition, it may not be effective in finding certain classes of defects. Test automation is a process of writing a computer program to do testing that would otherwise need to be done manually. Once tests have been automated, they can be run quickly. This is often the most cost effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features to break which were working at an earlier point in time. There are two general approaches to test automation:
Code-driven testing. The public (usually) interfaces to classes, modules, or libraries are tested with a variety of input arguments to validate that the results that are returned are correct.
Graphical user interface testing. A testing framework generates user interface events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.
Test automation tools can be expensive, and it is usually employed in combination with manual testing. It can be made cost-effective in the longer term, especially when used repeatedly in regression testing. One way to generate test cases automatically is model-based testing through use of a model of the system for test case generation but research continues into a variety of alternative methodologies for doing so.[citation needed] What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make. Selecting the correct features of the product for automation largely determines the success of the automation. Automating unstable features or features that are undergoing changes should be avoided.[2]
[edit] Code-driven testing

A growing trend in software development is the use of testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) that allow the execution of unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. Code driven test automation is a key feature of Agile software development, where it is known as Test-driven development (TDD). Unit tests are written to define the functionality before the code is written. Only when all tests pass is the code considered complete. Proponents argue that it produces software that is both more reliable and less costly than code that is tested by manual exploration. It is considered more reliable because the code coverage is better, and because it is run constantly during development rather than once at the end of a waterfall development cycle. The developer discovers defects immediately upon making a change, when it is least expensive to fix. Finally, code refactoring is safer; transforming the code into a simpler form with less code duplication, but equivalent behavior, is much less likely to introduce new defects.
[edit] Graphical User Interface (GUI) testing

Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no software development. This approach can be applied to any application that has a graphical user interface. However, reliance on these features poses major reliability and maintainability problems. Relabelling a button or moving it to another part of the window may require the test to be re-recorded. Record and playback also often adds irrelevant activities or incorrectly records some activities. A variation on this type of tool is for testing of web sites. Here, the "interface" is the web page. This type of tool also requires little or no software development. However, such a
framework utilizes entirely different techniques because it is reading html instead of observing window events.[citation needed] Another variation is scriptless test automation that does not use record and playback, but instead builds a model of the application under test and then enables the tester to create test cases by simply editing in test parameters and conditions. This requires no scripting skills, but has all the power and flexibility of a scripted approach. Test-case maintenance is easy, as there is no code to maintain and as the application under test changes the software objects can simply be re-learned or added. It can be applied to any GUI-based software application.
[edit] What to test

Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion. One must keep satisfying popular requirements when thinking of test automation:

Platform and OS independence Data driven capability (Input Data, Output Data, Meta Data) Customizable Reporting (DB Access, crystal reports) Easy debugging and logging Version control friendly minimal binary files Extensible & Customizable (Open APIs to be able to integrate with other tools) Common Driver (For example, in the Java development ecosystem, that means Ant or Maven and the popular IDEs). This enables tests to integrate with the developers' workflows. Support unattended test runs for integration with build processes and batch runs. Continuous Integration servers require this. Email Notifications (automated notification on failure or threshold levels). This may be the test runner or tooling[clarification needed] that executes it. Support distributed execution environment (distributed test bed) Distributed application support (distributed SUT)
[edit] Framework approach in automation

A framework is an integrated system that sets the rules of Automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort. There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are: 1. 2. 3. 4. Data-driven testing Modularity-driven testing Keyword-driven testing Hybrid testing
5. Model-based testing
[edit] Popular Test Automation Tools

Tool Name Company Name Latest Version HP QuickTest Professional HP 11.0 IBM Rational Functional Tester IBM Rational 8.1.0.3 Parasoft SOAtest Parasoft 9.0 Rational robot IBM Rational 2003 Selenium OpenSource Tool 1.0.6 SilkTest Micro Focus 2010 TestComplete SmartBear Software 8.0 TestPartner Micro Focus 6.3 WATIR OpenSource Tool 1.6.5
System testing
(Redirected from Functional test) Jump to: navigation, search System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. [1] As a rule, system testing takes, as its input, all of the "integrated" software components that have successfully passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limited type of testing; it seeks to detect defects both within the "inter-assemblages" and also within the system as a whole.
Software Testing portal
Contents
[hide]

1 Testing the whole system 2 Types of tests to include in system testing 3 See also 4 References
[edit] Testing the whole system

System testing is performed on the entire system in the context of a Functional Requirement Specification(s) (FRS) and/or a System Requirement Specification (SRS). System testing tests not only the design, but also the behaviour and even the believed expectations of the customer. It is also intended to test up to and beyond the bounds defined in the software/hardware requirements specification(s).[citation needed]
[edit] Types of tests to include in system testing

The following examples are different types of testing that should be considered during System testing:

GUI software testing Usability testing Performance testing
Compatibility testing Error handling testing Load testing Volume testing Stress testing Security testing Scalability testing Sanity testing Smoke testing Exploratory testing Ad hoc testing Regression testing Reliability testing Installation testing Maintenance testing Recovery testing and failover testing. Accessibility testing, including compliance with: o Americans with Disabilities Act of 1990 o Section 508 Amendment to the Rehabilitation Act of 1973 o Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C)
Although different testing organizations may prescribe different tests as part of System testing, this list serves as a general framework or foundation to begin with
Software development process

From Wikipedia, the free encyclopedia Jump to: navigation, search

Activities and steps Requirements Specification Architecture Design Implementation Testing Deployment Maintenance Methodologies Agile Cleanroom Iterative RAD RUP Spiral Waterfall XP Lean Scrum V-Model TDD Supporting disciplines Configuration management Documentation Quality assurance (SQA) Project management User experience design Tools Compiler Debugger Profiler GUI designer IDE
vde
A software development process is a structure imposed on the development of a software product. Similar terms include software life cycle and software process. There are several models for such processes, each describing approaches to a variety of tasks or activities that take place during the process. Some people consider a lifecycle model a more general term and a software development process a more specific term. For example, there are many specific software development processes that 'fit' the spiral lifecycle model.
Contents
[hide]

1 Overview 2 Software development activities o 2.1 Planning o 2.2 Implementation, testing and documenting o 2.3 Deployment and maintenance 3 Software Development Models o 3.1 Waterfall Model o 3.2 Spiral Model o 3.3 Iterative and Incremental development o 3.4 Agile development 4 Process Improvement Models 5 Formal methods 6 See also 7 References 8 External links
[edit] Overview
The large and growing body of software development organizations implement process methodologies. Many of them are in the defense industry, which in the U.S. requires a rating based on 'process models' to obtain contracts. The international standard for describing the method of selecting, implementing and monitoring the life cycle for software is ISO 12207. A decades-long goal has been to find repeatable, predictable processes that improve productivity and quality. Some try to systematize or formalize the seemingly unruly task of writing software. Others apply project management techniques to writing software. Without project management, software projects can easily be delivered late or over budget. With large numbers of software projects not meeting their expectations in terms of functionality, cost, or delivery schedule, effective project management appears to be lacking. Organizations may create a Software Engineering Process Group (SEPG), which is the focal point for process improvement. Composed of line practitioners who have varied skills, the group is at the center of the collaborative effort of everyone in the organization who is involved with software engineering process improvement.
[edit] Software development activities
The activities of the software development process represented in the waterfall model. There are several other models to represent this process. [edit] Planning
The important task in creating a software product is extracting the requirements or requirements analysis. Customers typically have an abstract idea of what they want as an end result, but not what software should do. Incomplete, ambiguous, or even contradictory requirements are recognized by skilled and experienced software engineers at this point. Frequently demonstrating live code may help reduce the risk that the requirements are incorrect. Once the general requirements are gathered from the client, an analysis of the scope of the development should be determined and clearly stated. This is often called a scope document. Certain functionality may be out of scope of the project as a function of cost or as a result of unclear requirements at the start of development. If the development is done externally, this document can be considered a legal document so that if there are ever disputes, any ambiguity of what was promised to the client can be clarified.
[edit] Implementation, testing and documenting
Implementation is the part of the process where software engineers actually program the code for the project. Software testing is an integral and important part of the software development process. This part of the process ensures that defects are recognized as early as possible. Documenting the internal design of software for the purpose of future maintenance and enhancement is done throughout development. This may also include the writing of an API, be it external or internal. It is very important to document everything in the project.
[edit] Deployment and maintenance
Deployment starts after the code is appropriately tested, is approved for release and sold or otherwise distributed into a production environment. Software Training and Support is important and a lot of developers fail to realize that. It would not matter how much time and planning a development team puts into creating software if nobody in an organization ends up using it. People are often resistant to change and avoid venturing into an unfamiliar area, so as a part of the deployment phase, it is very important to have training classes for new clients of your software. Maintaining and enhancing software to cope with newly discovered problems or new requirements can take far more time than the initial development of the software. It may be necessary to add code that does not fit the original design to correct an unforeseen problem or it may be that a customer is requesting more functionality and code can be added to accommodate their requests. If the labor cost of the maintenance phase exceeds 25% of the prior-phases' labor cost, then it is likely that the overall quality of at least one prior phase is poor.[citation needed] In that case, management should consider the option of rebuilding the system (or portions) before maintenance cost is out of control. Bug Tracking System tools are often deployed at this stage of the process to allow development teams to interface with customer/field teams testing the software to identify any real or perceived issues. These software tools, both open source and commercially licensed, provide a customizable process to acquire, review, acknowledge, and respond to reported issues. (software maintenance).
[edit] Software Development Models

Several models exist to streamline the development process. Each one has its pros and cons, and it's up to the development team to adopt the most appropriate one for the project. Sometimes a combination of the models may be more suitable.
[edit] Waterfall Model Main article: Waterfall model
The waterfall model shows a process, where developers are to follow these phases in order:
1. 2. 3. 4. 5. 6. Requirements specification (Requirements analysis) Software Design Integration Testing (or Validation) Deployment (or Installation) Maintenance
In a strict Waterfall model, after each phase is finished, it proceeds to the next one. Reviews may occur before moving to the next phase which allows for the possibility of changes (which may involve a formal change control process). Reviews may also be employed to ensure that the phase is indeed complete; the phase completion criteria are often referred to as a "gate" that the project must pass through to move to the next phase. Waterfall discourages
revisiting and revising any prior phase once it's complete. This "inflexibility" in a pure Waterfall model has been a source of criticism by other more "flexible" models.
[edit] Spiral Model Main article: Spiral model
The key characteristic of a Spiral model is risk management at regular stages in the development cycle. In 1988, Barry Boehm published a formal software system development "spiral model", which combines some key aspect of the waterfall model and rapid prototyping methodologies, but provided emphasis in a key area many felt had been neglected by other methodologies: deliberate iterative risk analysis, particularly suited to large-scale complex systems. The Spiral is visualized as a process passing through some number of iterations, with the four quadrant diagram representative of the following activities:
1. formulate plans to: identify software targets, selected to implement the program, clarify the project development restrictions; 2. Risk analysis: an analytical assessment of selected programs, to consider how to identify and eliminate risk; 3. the implementation of the project: the implementation of software development and verification;
Risk-driven spiral model, emphasizing the conditions of options and constraints in order to support software reuse, software quality can help as a special goal of integration into the product development. However, the spiral model has some restrictive conditions, as follows:
1. spiral model emphasize risk analysis, but require customers to accept and believe that much of this analysis, and make the relevant response is not easy, therefore, this model is often adapted to large-scale internal software development. 2. If the implementation of risk analysis will greatly affect the profits of the project, then risk analysis is meaningless, therefore, spiral model is only suitable for large-scale software projects. 3. Good software developers should look for possible risks, an accurate analysis of risk, otherwise it will lead to greater risk.
First stage is to determine the stage of the goal of accomplishing these objectives, options and constraints, and then from the perspective of risk analysis program, development strategy, and strive to remove all potential risks, and sometimes necessary to achieve through the construction of the prototype. If some risk can not be ruled out, the program to end immediately, or else start the development of the next steps. Finally, evaluation results of the stage, and the design of the next phase.
[edit] Iterative and Incremental development Main article: Iterative and incremental development
Iterative development[1] prescribes the construction of initially small but ever larger portions of a software project to help all those involved to uncover important issues early before problems or faulty assumptions can lead to disaster. Iterative processes are preferred[citation
needed]
by commercial developers because it allows a potential of reaching the design goals of a customer who does not know how to define what they want.
[edit] Agile development Main article: Agile software development
Agile software development uses iterative development as a basis but advocates a lighter and more people-centric viewpoint than traditional approaches. Agile processes use feedback, rather than planning, as their primary control mechanism. The feedback is driven by regular tests and releases of the evolving software. There are many variations of agile processes:
In Extreme Programming (XP), the phases are carried out in extremely small (or "continuous") steps compared to the older, "batch" processes. The (intentionally incomplete) first pass through the steps might take a day or a week, rather than the months or years of each complete step in the Waterfall model. First, one writes automated tests, to provide concrete goals for development. Next is coding (by a pair of programmers), which is complete when all the tests pass, and the programmers can't think of any more tests that are needed. Design and architecture emerge out of refactoring, and come after coding. Design is done by the same people who do the coding. (Only the last feature merging design and code is common to all the other agile processes.) The incomplete but functional system is deployed or demonstrated for (some subset of) the users (at least one of which is on the development team). At this point, the practitioners start again on writing tests for the next most important part of the system. Scrum
[edit] Process Improvement Models

Capability Maturity Model Integration The Capability Maturity Model Integration (CMMI) is one of the leading models and based on best practice. Independent assessments grade organizations on how well they follow their defined processes, not on the quality of those processes or the software produced. CMMI has replaced CMM. ISO 9000 ISO 9000 describes standards for a formally organized process to manufacture a product and the methods of managing and monitoring progress. Although the standard was originally created for the manufacturing sector, ISO 9000 standards have been applied to software development as well. Like CMMI, certification with ISO 9000 does not guarantee the quality of the end result, only that formalized business processes have been followed. ISO 15504 ISO 15504, also known as Software Process Improvement Capability Determination (SPICE), is a "framework for the assessment of software processes". This standard is aimed at setting
out a clear model for process comparison. SPICE is used much like CMMI. It models processes to manage, control, guide and monitor software development. This model is then used to measure what a development organization or project team actually does during software development. This information is analyzed to identify weaknesses and drive improvement. It also identifies strengths that can be continued or integrated into common practice for that organization or team.
[edit] Formal methods

Formal methods are mathematical approaches to solving software (and hardware) problems at the requirements, specification and design levels. Examples of formal methods include the BMethod, Petri nets, Automated theorem proving, RAISE and VDM. Various formal specification notations are available, such as the Z notation. More generally, automata theory can be used to build up and validate application behavior by designing a system of finite state machines. Finite state machine (FSM) based methodologies allow executable software specification and by-passing of conventional coding (see virtual finite state machine or event driven finite state machine). Formal methods are most likely to be applied in avionics software, particularly where the software is safety critical. Software safety assurance standards, such as DO178B demand formal methods at the highest level of categorization (Level A). Formalization of software development is creeping in, in other places, with the application of Object Constraint Language (and specializations such as Java Modeling Language) and especially with Model-driven architecture allowing execution of designs, if not specifications. Another emerging trend in software development is to write a specification in some form of logic (usually a variation of FOL), and then to directly execute the logic as though it were a program. The OWL language, based on Description Logic, is an example. There is also work on mapping some version of English (or another natural language) automatically to and from logic, and executing the logic directly. Examples are Attempto Controlled English, and Internet Business Logic, which does not seek to control the vocabulary or syntax. A feature of systems that support bidirectional English-logic mapping and direct execution of the logic is that they can be made to explain their results, in English, at the business or scientific level. The Government Accountability Office, in a 2003 report on one of the Federal Aviation Administrations air traffic control modernization programs,[2] recommends following the agencys guidance for managing major acquisition systems by
establishing, maintaining, and controlling an accurate, valid, and current performance measurement baseline, which would include negotiating all authorized, unpriced work within 3 months; conducting an integrated baseline review of any major contract modifications within 6 months; and preparing a rigorous life-cycle cost estimate, including a risk assessment, in accordance with the Acquisition System Toolsets guidance and identifying the level of uncertainty inherent in the estimate.
Traceability matrix
Jump to: navigation, search

Activities and steps Requirements Specification Architecture Design Implementation Testing Deployment Maintenance Methodologies Agile Cleanroom Iterative RAD RUP Spiral Waterfall XP Lean Scrum V-Model TDD Supporting disciplines Configuration management Documentation Quality assurance (SQA) Project management User experience design Tools Compiler Debugger Profiler GUI designer IDE
vde
A traceability matrix is a document, usually in the form of a table, that correlates any two baselined documents that require a many to many relationship to determine the completeness of the relationship. It is often used with high-level requirements (these often consist of
marketing requirements) and detailed requirements of the software product to the matching parts of high-level design, detailed design, test plan, and test cases. A requirements traceability matrix may be used to check to see if the current project requirements are being met, and to help in the creation of a Request for Proposal, various deliverable documents, and project plan tasks.[1] Common usage is to take the identifier for each of the items of one document and place them in the left column. The identifiers for the other document are placed across the top row. When an item in the left column is related to an item across the top, a mark is placed in the intersecting cell. The number of relationships are added up for each row and each column. This value indicates the mapping of the two items. Zero values indicate that no relationship exists. It must be determined if one must be made. Large values imply that the relationship is too complex and should be simplified. To ease the creation of traceability matrices, it is advisable to add the relationships to the source documents for both backward traceability and forward traceability. In other words, when an item is changed in one baselined document, it's easy to see what needs to be changed in the other.
Contents
[hide]

1 Sample traceability matrix 2 See also 3 References 4 External links
[edit] Sample traceability matrix

REQ REQ REQ REQ REQ REQ REQ REQ REQ REQ REQ REQ REQ REQ 1 1 1 Requirem Reqs 1 1 1 1 1 1 1 1 1 1 1 Teste ent TEC TEC TEC UC UC UC UC UC UC UC UC UC UC UC Identifiers d H H H 1.1 1.2 1.3 2.1 2.2 2.3.1 2.3.2 2.3.3 2.4 3.1 3.2 1.1 1.2 1.3
Test Cases 321 Tested 77 Implicitly 3 2 3 1 1 1 1 1 1 2 3 1 1 1
1.1.1 1.1.2 1.1.3 1.1.4 1.1.5 1.1.6
1 2 2 1 2 1
x x x x x x x x x
1.1.7 1.2.1 1.2.2 1.2.3 1.3.1 1.3.2 1.3.3 1.3.4 1.3.5 etc 5.6.2
1 2 2 2 1 1 1 1 1
x x x x x x x x x x x x
Baseline (configuration management)

From Wikipedia, the free encyclopedia Jump to: navigation, search This article is about configuration management. For other uses, see Baseline (disambiguation). This article may be confusing or unclear to readers. Please help clarify the article; suggestions may be found on the talk page. (August 2010) This article needs attention from an expert on the subject. See the talk page for details. Consider associating this request with a WikiProject. (August 2010)
Configuration management is the process of managing change in hardware, software, firmware, documentation, measurements, etc. As change requires an initial state and next state, the marking of significant states within a series of several changes becomes important. The identification of significant states within the revision history of a configuration item is the central purpose of baseline identification.[1] Typically, significant states are those that receive a formal approval status, either explicitly or implicitly (approval statuses may be marked individually, when such a marking has been defined, or signified merely by association to a certain baseline). Nevertheless, this approval status is usually recognized publicly. Thus, a baseline may also mark an approved configuration item, e.g. a project plan that has been signed off for execution. In a similar manner, associating multiple configuration items with such a baseline indicates those items as being approved.
Generally, a baseline may be a single work product, or set of work products that can be used as a logical basis for comparison. A baseline may also be established (whose work products meet certain criteria) as the basis for subsequent select activities. Such activities may be attributed with formal approval.
Conversely, the configuration of a project often includes one or more baselines, the status of the configuration, and any metrics collected. The current configuration refers to the current status, current audit, current metrics, and latest revision of all configuration items. Similarly, but less frequently, a baseline may refer to all items associated with a specific project. This may include all revisions of all items, or only the latest revision of all items in the project, depending upon context, e.g. "the baseline of the project is proceeding as planned." A baseline may be specialized as a specific type of baseline.[2] Some examples include:

Functional Baseline: initial specifications established; contract, etc. Allocated Baseline: state of work products once requirements are approved Developmental Baseline: state of work products amid development Product Baseline: contains the releasable contents of the project others, based upon proprietary business practices
Contents
[hide]

1 Capabilities of Baselines o 1.1 Baselining Configuration Items 2 Baseline Control 3 Application 4 See also 5 References
[edit] Capabilities of Baselines

While marking approval status covers the majority of uses for a baseline, baselines may also be established merely to signify the progress of work through the passage of time. In this case, a baseline is a visible stake through an endured collective effort, e.g. a developmental baseline. Baselines may also mark milestones; albeit some milestones also signify approval. Baselines themselves are valued not only for their ability to identify the notable state of work product(s) but also provide particular importance in their ability to be retrieved. Once retrieved, the state of the work product(s) in that subset share the same significance in their history of changes that this significance was observed. The baseline is then regarded with poignant qualities (either favorably or disfavorably). For this reason, baseline identification, monitoring, and retrieval are critical to the success of configuration management. However, the ease of retrieving any given baseline varies according to the system employed for performing configuration management which may use a manual, automated, or hybrid approach. Once retrieved, the baseline may be compared to a particular configuration or another baseline. Most baselines are established at a fixed point in time[2] and serve to continue to reference that point (identification of state). However, some baselines are established to carry forward as a reference to the item itself regardless of any changes to the item. These latter baselines evolve with the progression of the work effort but continue to identify notable work products in the project.
[edit] Baselining Configuration Items
In the process of performing configuration management, configuration items (or work products) may be baselined so as to establish them with a certain status to interested parties. In this sense, to baseline a work product may require certain change(s) to the work product to ensure its conformance to the characteristics associated with the baseline referenced. This varies upon context, but in many cases this has the implication that the work product is "reset" to an initial (possibly inherently approved) state from which work may proceed.
[edit] Baseline Control
In many environments, baselines are controlled such that certain subsequent activities against work products in that baseline are either prohibited or permitted. These activities are carefully selected and controlled, and again, depending upon the configuration management system, are also monitored. Consequently, baselines are ordinarily subjected to configuration management audits. Audits may include an examination of specific actions performed against the baseline, identification of individuals involved in any action, an evaluation of change within the baseline, (re-)certification for approval, accounting, metric collection, comparison to another baseline, or all of these.
[edit] Application
Though common in software revision control systems as labels or tags, the existence of baselines is found in several other technology-related domains. Baselines can be found in UML modeling systems and business rule management systems, among others. In addition to the field of Hardware and Software Engineering, baselines can be found in Medicine (e.g. monitoring health progress), Politics (e.g. statistics), Physics & Chemistry (e.g. observations and changes), Finance (e.g. budgeting), and others.
Integration testing
From Wikipedia, the free encyclopedia Jump to: navigation, search This article does not cite any references or sources.
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (August 2010)
Integration testing (sometimes called Integration and Testing, abbreviated "I&T") is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.
Contents
[hide]
1 Purpose o 1.1 Big Bang o 1.2 Top-down and Bottom-up 2 Limitations 3 See also
[edit] Purpose
The purpose of integration testing is to verify functional, performance, and reliability requirements placed on major design items. These "design items", i.e. assemblages (or groups of units), are exercised through their interfaces using Black box testing, success and error cases being simulated via appropriate parameter and data inputs. Simulated usage of shared data areas and inter-process communication is tested and individual subsystems are exercised through their input interface. Test cases are constructed to test that all components within assemblages interact correctly, for example across procedure calls or process activations, and this is done after testing individual modules, i.e. unit testing. The overall idea is a "building block" approach, in which verified assemblages are added to a verified base which is then used to support the integration testing of further assemblages. Some different types of integration testing are big bang, top-down, and bottom-up.
[edit] Big Bang
In this approach, all or most of the developed modules are coupled together to form a complete software system or major part of the system and then used for integration testing. The Big Bang method is very effective for saving time in the integration testing process. However, if the test cases and their results are not recorded properly, the entire integration process will be more complicated and may prevent the testing team from achieving the goal of integration testing. A type of Big Bang Integration testing is called Usage Model testing. Usage Model testing can be used in both software and hardware integration testing. The basis behind this type of integration testing is to run user-like workloads in integrated user-like environments. In doing the testing in this manner, the environment is proofed, while the individual components are proofed indirectly through their use. Usage Model testing takes an optimistic approach to testing, because it expects to have few problems with the individual components. The strategy relies heavily on the component developers to do the isolated unit testing for their product. The goal of the strategy is to avoid redoing the testing done by the developers, and instead flesh out problems caused by the interaction of the components in the environment. For integration testing, Usage Model testing can be more efficient and provides better test coverage than traditional focused functional integration testing. To be more efficient and accurate, care must be used in defining the user-like workloads for creating realistic scenarios in exercising the environment. This gives that the integrated environment will work as expected for the target customers.
[edit] Top-down and Bottom-up Main article: Top-down and bottom-up design
Bottom Up Testing is an approach to integrated testing where the lowest level components are tested first, then used to facilitate the testing of higher level components. The process is repeated until the component at the top of the hierarchy is tested. All the bottom or low-level modules, procedures or functions are integrated and then tested. After the integration testing of lower level integrated modules, the next level of modules will be formed and can be used for integration testing. This approach is helpful only when all or most of the modules of the same development level are ready. This method also helps to determine the levels of software developed and makes it easier to report testing progress in the form of a percentage. Top Down Testing is an approach to integrated testing where the top integrated modules are tested and the branch of the module is tested step by step until the end of the related module. Sandwich Testing is an approach to combine top down testing with bottom up testing. The main advantage of the Bottom-Up approach is that bugs are more easily found. With Top-Down, it is easier to find a missing branch link.
[edit] Limitations
Any conditions not stated in specified integration tests, outside of the confirmation of the execution of design items, will generally not be tested.
Requirements analysis
Requirements analysis is the first stage in the systems engineering process and software development process.[1]

Activities and steps Requirements Specification Architecture Design Implementation Testing Deployment Maintenance Methodologies
Agile Cleanroom Iterative RAD RUP Spiral Waterfall XP Lean Scrum V-Model TDD Supporting disciplines Configuration management Documentation Quality assurance (SQA) Project management User experience design Tools Compiler Debugger Profiler GUI designer IDE
vde
Requirements analysis in systems engineering and software engineering, encompasses those tasks that go into determining the needs or conditions to meet for a new or altered product, taking account of the possibly conflicting requirements of the various stakeholders, such as beneficiaries or users. Requirements analysis is critical to the success of a development project.[2] Requirements must be documented, actionable, measurable, testable, related to identified business needs or opportunities, and defined to a level of detail sufficient for system design. Requirements can be architectural, structural, behavioral, functional, and non-functional.
Contents
[hide]

1 Overview 2 Requirements engineering 3 Requirements analysis topics o 3.1 Stakeholder identification o 3.2 Stakeholder interviews o 3.3 Joint Requirements Development (JRD) Sessions o 3.4 Contract-style requirement lists 3.4.1 Strengths
3.4.2 Weaknesses 3.4.3 Alternative to requirement lists o 3.5 Measurable goals o 3.6 Prototypes o 3.7 Use cases o 3.8 Software requirements specification 4 Types of Requirements 5 Requirements analysis issues o 5.1 Stakeholder issues o 5.2 Engineer/developer issues o 5.3 Attempted solutions 6 See also 7 References 8 Further reading 9 External links
[edit] Overview
Conceptually, requirements analysis includes three types of activity:

Eliciting requirements: the task of communicating with customers and users to determine what their requirements are. This is sometimes also called requirements gathering. Analyzing requirements: determining whether the stated requirements are unclear, incomplete, ambiguous, or contradictory, and then resolving these issues. Recording requirements: Requirements might be documented in various forms, such as natural-language documents, use cases, user stories, or process specifications.
Requirements analysis can be a long and arduous process during which many delicate psychological skills are involved. New systems change the environment and relationships between people, so it is important to identify all the stakeholders, take into account all their needs and ensure they understand the implications of the new systems. Analysts can employ several techniques to elicit the requirements from the customer. Historically, this has included such things as holding interviews, or holding focus groups (more aptly named in this context as requirements workshops) and creating requirements lists. More modern techniques include prototyping, and use cases. Where necessary, the analyst will employ a combination of these methods to establish the exact requirements of the stakeholders, so that a system that meets the business needs is produced.
[edit] Requirements engineering

Systematic requirements analysis is also known as requirements engineering.[3] It is sometimes referred to loosely by names such as requirements gathering, requirements capture, or requirements specification. The term requirements analysis can also be applied specifically to the analysis proper, as opposed to elicitation or documentation of the requirements, for instance. Requirements Engineering can be divided into discrete chronological steps:
Requirements elicitation,
Requirements analysis and negotiation, Requirements specification, System modeling, Requirements validation, Requirements management.
Requirement engineering according to Laplante (2007) is "a subdiscipline of systems engineering and software engineering that is concerned with determining the goals, functions, and constraints of hardware and software systems."[4] In some life cycle models, the requirement engineering process begins with a feasibility study activity, which leads to a feasibility report. If the feasibility study suggests that the product should be developed, then requirement analysis can begin. If requirement analysis precedes feasibility studies, which may foster outside the box thinking, then feasibility should be determined before requirements are finalized.
[edit] Requirements analysis topics

This section does not cite any references or sources.
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (October 2009)
[edit] Stakeholder identification
See Stakeholder analysis for a discussion of business uses. Stakeholders (SH) are persons or organizations (legal entities such as companies, standards bodies) which have a valid interest in the system. They may be affected by it either directly or indirectly. A major new emphasis in the 1990s was a focus on the identification of stakeholders. It is increasingly recognized that stakeholders are not limited to the organization employing the analyst. Other stakeholders will include:

anyone who operates the system (normal and maintenance operators) anyone who benefits from the system (functional, political, financial and social beneficiaries) anyone involved in purchasing or procuring the system. In a mass-market product organization, product management, marketing and sometimes sales act as surrogate consumers (mass-market customers) to guide development of the product organizations which regulate aspects of the system (financial, safety, and other regulators) people or organizations opposed to the system (negative stakeholders; see also Misuse case) organizations responsible for systems which interface with the system under design those organizations who integrate horizontally with the organization for whom the analyst is designing the system
[edit] Stakeholder interviews
Stakeholder interviews are a common technique used in requirement analysis. Though they are generally idiosyncratic in nature and focused upon the perspectives and perceived needs of the stakeholder, very often without larger enterprise or system context, this perspective deficiency has the general advantage of obtaining a much richer understanding of the stakeholder's unique business processes, decision-relevant business rules, and perceived
needs. Consequently this technique can serve as a means of obtaining the highly focused knowledge that is often not elicited in Joint Requirements Development sessions, where the stakeholder's attention is compelled to assume a more cross-functional context. Moreover, the in-person nature of the interviews provides a more relaxed environment where lines of thought may be explored at length.
[edit] Joint Requirements Development (JRD) Sessions
Requirements often have cross-functional implications that are unknown to individual stakeholders and often missed or incompletely defined during stakeholder interviews. These cross-functional implications can be elicited by conducting JRD sessions in a controlled environment, facilitated by a trained facilitator, wherein stakeholders participate in discussions to elicit requirements, analyze their details and uncover cross-functional implications. A dedicated scribe and Business Analyst should be present to document the discussion. Utilizing the skills of a trained facilitator to guide the discussion frees the Business Analyst to focus on the requirements definition process. JRD Sessions are analogous to Joint Application Design Sessions. In the former, the sessions elicit requirements that guide design, whereas the latter elicit the specific design features to be implemented in satisfaction of elicited requirements.
[edit] Contract-style requirement lists
One traditional way of documenting requirements has been contract style requirement lists. In a complex system such requirements lists can run to hundreds of pages. An appropriate metaphor would be an extremely long shopping list. Such lists are very much out of favour in modern analysis; as they have proved spectacularly unsuccessful at achieving their aims; but they are still seen to this day.
[edit] Strengths

Provides a checklist of requirements. Provide a contract between the project sponsor(s) and developers. For a large system can provide a high level description.
[edit] Weaknesses

Such lists can run to hundreds of pages. It is virtually impossible to read such documents as a whole and have a coherent understanding of the system. Such requirements lists abstract all the requirements and so there is little context

This abstraction makes it impossible to see how the requirements fit or work together. This abstraction makes it difficult to prioritize requirements properly; while a list does make it easy to prioritize each individual item, removing one item out of context can render an entire use case or business requirement useless. This abstraction increases the likelihood of misinterpreting the requirements; as more people read them, the number of (different) interpretations of the envisioned system increase.
This abstraction means that it's extremely difficult to be sure that you have the majority of the requirements. Necessarily, these documents speak in generality; but the devil, as they say, is in the details.
These lists create a false sense of mutual understanding between the stakeholders and developers. These contract style lists give the stakeholders a false sense of security that the developers must achieve certain things. However, due to the nature of these lists, they inevitably miss out crucial requirements which are identified later in the process. Developers can use these discovered requirements to renegotiate the terms and conditions in their favour. These requirements lists are no help in system design, since they do not lend themselves to application.
[edit] Alternative to requirement lists
As an alternative to the large, pre-defined requirement lists Agile Software Development uses User stories to define a requirement in every day language.
[edit] Measurable goals Main article: Goal modeling
Best practices take the composed list of requirements merely as clues and repeatedly ask "why?" until the actual business purposes are discovered. Stakeholders and developers can then devise tests to measure what level of each goal has been achieved thus far. Such goals change more slowly than the long list of specific but unmeasured requirements. Once a small set of critical, measured goals has been established, rapid prototyping and short iterative development phases may proceed to deliver actual stakeholder value long before the project is half over.
[edit] Prototypes
In the mid-1980s, prototyping was seen as the best solution to the requirements analysis problem. Prototypes are Mockups of an application. Mockups allow users to visualize an application that hasn't yet been constructed. Prototypes help users get an idea of what the system will look like, and make it easier for users to make design decisions without waiting for the system to be built. Major improvements in communication between users and developers were often seen with the introduction of prototypes. Early views of applications led to fewer changes later and hence reduced overall costs considerably. However, over the next decade, while proving a useful technique, prototyping did not solve the requirements problem:

Managers, once they see a prototype, may have a hard time understanding that the finished design will not be produced for some time. Designers often feel compelled to use patched together prototype code in the real system, because they are afraid to 'waste time' starting again. Prototypes principally help with design decisions and user interface design. However, they can not tell you what the requirements originally were. Designers and end users can focus too much on user interface design and too little on producing a system that serves the business process.
Prototypes work well for user interfaces, screen layout and screen flow but are not so useful for batch or asynchronous processes which may involve complex database updates and/or calculations.
Prototypes can be flat diagrams (often referred to as Wireframes) or working applications using synthesized functionality. Wireframes are made in a variety of graphic design documents, and often remove all color from the design (i.e. use a greyscale color palette) in instances where the final software is expected to have graphic design applied to it. This helps to prevent confusion over the final visual look and feel of the application.
[edit] Use cases Main article: Use case
A use case is a technique for documenting the potential requirements of a new system or software change. Each use case provides one or more scenarios that convey how the system should interact with the end user or another system to achieve a specific business goal. Use cases typically avoid technical jargon, preferring instead the language of the end user or domain expert. Use cases are often co-authored by requirements engineers and stakeholders. Use cases are deceptively simple tools for describing the behavior of software or systems. A use case contains a textual description of all of the ways which the intended users could work with the software or system. Use cases do not describe any internal workings of the system, nor do they explain how that system will be implemented. They simply show the steps that a user follows to perform a task. All the ways that users interact with a system can be described in this manner.
[edit] Software requirements specification
A software requirements specification (SRS) is a complete description of the behavior of the system to be developed. It includes a set of use cases that describe all of the interactions that the users will have with the software. Use cases are also known as functional requirements. In addition to use cases, the SRS also contains nonfunctional (or supplementary) requirements. Non-functional requirements are requirements which impose constraints on the design or implementation (such as performance requirements, quality standards, or design constraints). Recommended approaches for the specification of software requirements are described by IEEE 830-1998. This standard describes possible structures, desirable contents, and qualities of a software requirements specification.
[edit] Types of Requirements

Requirements are categorized in several ways. The following are common categorizations of requirements that relate to technical management:[1]
Customer Requirements Statements of fact and assumptions that define the expectations of the system in terms of mission objectives, environment, constraints, and measures of effectiveness and suitability (MOE/MOS). The customers are those that perform the eight primary functions of systems
engineering, with special emphasis on the operator as the key customer. Operational requirements will define the basic need and, at a minimum, answer the questions posed in the following listing:[1]

Operational distribution or deployment: Where will the system be used? Mission profile or scenario: How will the system accomplish its mission objective? Performance and related parameters: What are the critical system parameters to accomplish the mission? Utilization environments: How are the various system components to be used? Effectiveness requirements: How effective or efficient must the system be in performing its mission? Operational life cycle: How long will the system be in use by the user? Environment: What environments will the system be expected to operate in an effective manner?
Architectural Requirements Architectural requirements explain what has to be done by identifying the necessary system architecture (structure + behavior) of a system. Structural Requirements Structural requirements explain what has to be done by identifying the necessary structure of a system. Behavioral Requirements Behavioral requirements explain what has to be done by identifying the necessary behavior of a system. Functional Requirements Functional requirements explain what has to be done by identifying the necessary task, action or activity that must be accomplished. Functional requirements analysis will be used as the toplevel functions for functional analysis.[1] Non-functional Requirements Non-functional requirements are requirements that specify criteria that can be used to judge the operation of a system, rather than specific behaviors. Performance Requirements The extent to which a mission or function must be executed; generally measured in terms of quantity, quality, coverage, timeliness or readiness. During requirements analysis, performance (how well does it have to be done) requirements will be interactively developed across all identified functions based on system life cycle factors; and characterized in terms of the degree of certainty in their estimate, the degree of criticality to system success, and their relationship to other requirements.[1] Design Requirements
The build to, code to, and buy to requirements for products and how to execute requirements for processes expressed in technical data packages and technical manuals.[1] Derived Requirements Requirements that are implied or transformed from higher-level requirement. For example, a requirement for long range or high speed may result in a design requirement for low weight.[1] Allocated Requirements A requirement that is established by dividing or otherwise allocating a high-level requirement into multiple lower-level requirements. Example: A 100-pound item that consists of two subsystems might result in weight requirements of 70 pounds and 30 pounds for the two lower-level items.[1]
Well-known requirements categorization models include FURPS and FURPS+, developed at Hewlett-Packard.
[edit] Requirements analysis issues

[edit] Stakeholder issues
Steve McConnell, in his book Rapid Development, details a number of ways users can inhibit requirements gathering:

Users do not understand what they want or users don't have a clear idea of their requirements Users will not commit to a set of written requirements Users insist on new requirements after the cost and schedule have been fixed Communication with users is slow Users often do not participate in reviews or are incapable of doing so Users are technically unsophisticated Users do not understand the development process Users do not know about present technology
This may lead to the situation where user requirements keep changing even when system or product development has been started.
[edit] Engineer/developer issues
Possible problems caused by engineers and developers during requirements analysis are:

Technical personnel and end users may have different vocabularies. Consequently, they may wrongly believe they are in perfect agreement until the finished product is supplied. Engineers and developers may try to make the requirements fit an existing system or model, rather than develop a system specific to the needs of the client. Analysis may often be carried out by engineers or programmers, rather than personnel with the people skills and the domain knowledge to understand a client's needs properly.
[edit] Attempted solutions
One attempted solution to communications problems has been to employ specialists in business or system analysis. Techniques introduced in the 1990s like prototyping, Unified Modeling Language (UML), use cases, and Agile software development are also intended as solutions to problems encountered with previous methods. Also, a new class of application simulation or application definition tools have entered the market. These tools are designed to bridge the communication gap between business users and the IT organization and also to allow applications to be 'test marketed' before any code is produced. The best of these tools offer:

electronic whiteboards to sketch application flows and test alternatives ability to capture business logic and data needs ability to generate high fidelity prototypes that closely imitate the final application interactivity capability to add contextual requirements and other comments ability for remote and distributed users to run and interact with the simulation
Non-functional requirement
(Redirected from Non-functional requirements) Jump to: navigation, search In systems engineering and requirements engineering, a non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. This should be contrasted with functional requirements that define specific behavior or functions. The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture. In general, functional requirements define what a system is supposed to do whereas nonfunctional requirements define how a system is supposed to be. Functional requirements are usually in the form of "system shall <do requirement>", while non-functional requirements are "system shall be <requirement>". Non-functional requirements are often called qualities of a system. Other terms for nonfunctional requirements are "constraints", "quality attributes", "quality goals", "quality of service requirements" and "non-behavioral requirements".[1] Qualities, that is, non-functional requirements, can be divided into two main categories: 1. Execution qualities, such as security and usability, which are observable at run time. 2. Evolution qualities, such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the software system.[2][3]
Contents

1 Examples 2 See also 3 References 4 External links
[edit] Examples
A system may be required to present the user with a display of the number of records in a database. This is a functional requirement. How up-to-date this number needs to be is a nonfunctional requirement. If the number needs to be updated in real time, the system architects must ensure that the system is capable of updating the displayed record count within an acceptably short interval of the number of records changing. Sufficient network bandwidth may also be a non-functional requirement of a system. Other examples:
Accessibility
Audit and control Availability (see service level agreement) Backup Capacity, current and forecast Certification Compliance Configuration management Dependency on other parties Deployment Documentation Disaster recovery Efficiency (resource consumption for given load) Effectiveness (resulting performance in relation to effort) Emotional factors (like fun or absorbing) Escrow Extensibility (adding features, and carry-forward of customizations at next major version upgrade) Failure management Legal and licensing issues or patent-infringement-avoidability Interoperability Maintainability Modifiability Network topology Open source Operability Performance / response time (performance engineering) Platform compatibility Price Privacy Portability Quality (e.g. faults discovered, faults delivered, fault removal efficacy) Recovery / recoverability (e.g. mean time to recovery - MTTR) Reliability (e.g. mean time between failures - MTBF) Resilience Resource constraints (processor speed, memory, disk space, network bandwidth etc) Response time Robustness Scalability (horizontal, vertical) Security Software, tools, standards etc. Compatibility Stability Safety Supportability Testability Usability by target user community
For example: online response times, timing of the company offices, security, back-up etc.
Usability testing
Usability testing is a technique used to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system.[1] This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users. Usability testing focuses on measuring a human-made product's capacity to meet its intended purpose. Examples of products that commonly benefit from usability testing are foods, consumer products, web sites or web applications, computer interfaces, documents, and devices. Usability testing measures the usability, or ease of use, of a specific object or set of objects, whereas general human-computer interaction studies attempt to formulate universal principles.
Contents
[hide]

1 History of usability testing 2 Goals of usability testing 3 What usability testing is not 4 Methods o 4.1 Hallway testing o 4.2 Remote testing o 4.3 Expert review o 4.4 Automated expert review 5 How many users to test? 6 See also 7 References 8 External links
[edit] History of usability testing

A Xerox Palo Alto Research Center (PARC) employee wrote that PARC used extensive usability testing in creating the Xerox Star, introduced in 1981.[2]] Only about 25,000 were sold, leading many to consider the Xerox Star a commercial failure. The Inside Intuit book, says (page 22, 1984), "... in the first instance of the Usability Testing that later became standard industry practice, LeFevre recruited people off the streets... and timed their Kwik-Chek (Quicken) usage with a stopwatch. After every test... programmers worked to improve the program."[1]) Scott Cook, Intuit co-founder, said, "... we did usability testing in 1984, five years before anyone else... there's a very big difference between doing it
and having marketing people doing it as part of their... design... a very big difference between doing it and having it be the core of what engineers focus on.[3]
[edit] Goals of usability testing

Usability testing is a black-box testing technique. The aim is to observe people using the product to discover errors and areas of improvement. Usability testing generally involves measuring how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response. The results of the first test can be treated as a baseline or control measurement; all subsequent tests can then be compared to the baseline to indicate improvement.

Performance -- How much time, and how many steps, are required for people to complete basic tasks? (For example, find something to buy, create a new account, and order the item.) Accuracy -- How many mistakes did people make? (And were they fatal or recoverable with the right information?) Recall -- How much does the person remember afterwards or after periods of non-use? Emotional response -- How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend?
[edit] What usability testing is not

Simply gathering opinions on an object or document is market research rather than usability testing. Usability testing usually involves systematic observation under controlled conditions to determine how well people can use the product.[4] Rather than showing users a rough draft and asking, "Do you understand this?", usability testing involves watching people trying to use something for its intended purpose. For example, when testing instructions for assembling a toy, the test subjects should be given the instructions and a box of parts. Instruction phrasing, illustration quality, and the toy's design all affect the assembly process.
[edit] Methods
Setting up a usability test involves carefully creating a scenario, or realistic situation, wherein the person performs a list of tasks using the product being tested while observers watch and take notes. Several other test instruments such as scripted instructions, paper prototypes, and pre- and post-test questionnaires are also used to gather feedback on the product being tested. For example, to test the attachment function of an e-mail program, a scenario would describe a situation where a person needs to send an e-mail attachment, and ask him or her to undertake this task. The aim is to observe how people function in a realistic manner, so that developers can see problem areas, and what people like. Techniques popularly used to gather data during a usability test include think aloud protocol and eye tracking.
[edit] Hallway testing
Hallway testing (or Hall Intercept Testing) is a general methodology of usability testing. Rather than using an in-house, trained group of testers, just five to six random people, indicative of a cross-section of end users, are brought in to test the product, or service. The
name of the technique refers to the fact that the testers should be random people who pass by in the hallway. [5]
[edit] Remote testing
Remote usability testing (also known as unmoderated or asynchronous usability testing) involves the use of a specially modified online survey, allowing the quantification of user testing studies by providing the ability to generate large sample sizes. Similar to an in-lab study, a remote usability test is task-based and the platforms allow you to capture clicks and task times. Hence, for many large companies this allows you to understand the WHY behind the visitors intents when visiting a website or mobile site. Additionally, this style of user testing also provides an opportunity to segment feedback by demographic, attitudinal and behavioural type. The tests are carried out in the users own environment (rather than labs) helping further simulate real-life scenario testing. This approach also provides a vehicle to easily solicit feedback from users in remote areas.
[edit] Expert review
Expert review is another general method of usability testing. As the name suggests, this method relies on bringing in experts with experience in the field (possibly from companies that specialize in usability testing) to evaluate the usability of a product.
[edit] Automated expert review
Similar to expert reviews, automated expert reviews provide usability testing but through the use of programs given rules for good design and heuristics. Though an automated review might not provide as much detail and insight as reviews from people, they can be finished more quickly and consistently. The idea of creating surrogate users for usability testing is an ambitious direction for the Artificial Intelligence community.
[edit] How many users to test?

In the early 1990s, Jakob Nielsen, at that time a researcher at Sun Microsystems, popularized the concept of using numerous small usability teststypically with only five test subjects eachat various stages of the development process. His argument is that, once it is found that two or three people are totally confused by the home page, little is gained by watching more people suffer through the same flawed design. "Elaborate usability tests are a waste of resources. The best results come from testing no more than five users and running as many small tests as you can afford." [6]. Nielsen subsequently published his research and coined the term heuristic evaluation. The claim of "Five users is enough" was later described by a mathematical model[7] which states for the proportion of uncovered problems U
U = 1 (1 p)n
where p is the probability of one subject identifying a specific problem and n the number of subjects (or test sessions). This model shows up as an asymptotic graph towards the number of real existing problems (see figure below).
In later research Nielsen's claim has eagerly been questioned with both empirical evidence[8] and more advanced mathematical models [9]. Two key challenges to this assertion are:
1. since usability is related to the specific set of users, such a small sample size is unlikely to be representative of the total population so the data from such a small sample is more likely to reflect the sample group than the population they may represent 2. Not every usability problem is equally easy-to-detect. Intractable problems happen to decelerate the overall process. Under these circumstances the progress of the process is much shallower than predicted by the Nielsen/Landauer formula [10].
Most researchers and practitioners today agree that, although testing 5 users is better than not testing at all, a sample size larger than five is required to detect a satisfying number of usability problems.
Compatibility testing
Jump to: navigation, search This article does not cite any references or sources.
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (February 2009) Software Testing portal
Compatibility testing, part of software non-functional tests, is testing conducted on the application to evaluate the application's compatibility with the computing environment. Computing environment may contain some or all of the below mentioned elements:

Computing capacity of Hardware Platform (IBM 360, HP 9000, etc.).. Bandwidth handling capacity of networking hardware Compatibility of peripherals (Printer, DVD drive, etc.) Operating systems (MVS, UNIX, Windows, etc.) Database (Oracle, Sybase, DB2, etc.) Other System Software (Web server, networking/ messaging tool, etc.) Browser compatibility (Firefox, Netscape, Internet Explorer, Safari, etc.)
Browser compatibility testing, can be more appropriately referred to as user experience testing. This requires that the web applications are tested on different web browsers, to ensure the following:

Users have the same visual experience irrespective of the browsers through which they view the web application. In terms of functionality, the application must behave and respond the same way across different browsers.
For more information please visit the link BCT

Carrier compatibility (Verizon, Sprint, Orange, O2, AirTel, etc.) Backwards compatibility. Hardware (different phones) Different Compilers (compile the code correctly) Runs on multiple host/guest Emulators
Certification testing falls within the scope of Compatibility testing. Product Vendors run the complete suite of testing on the newer computing environment to get their application certified for a specific Operating Systems or Databases.
Non-functional testing
(Redirected from Non-functional tests) Jump to: navigation, search This article does not cite any references or sources.
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (December 2009)
Non-functional testing is the testing of a software application for its non-functional requirements. The names of many non-functional tests are often used interchangeably because of the overlap in scope between various non-functional requirements. For example, software performance is a broad term that includes many specific requirements like reliability and scalability. Non-functional testing includes:

Baseline testing Compatibility testing Compliance testing Documentation testing Endurance testing Load testing Localization testing and mangi testing Performance testing Recovery testing Resilience testing Security testing Scalability testing Stress testing Usability testing Volume testing
Exception handling
From Wikipedia, the free encyclopedia (Redirected from Error handling testing) Jump to: navigation, search
Exception handling is a programming language construct or computer hardware mechanism designed to handle the occurrence of exceptions, special conditions that change the normal flow of program execution. Programming languages differ considerably in their support for exception handling (as distinct from error checking, which is normal program flow that codes for responses to adverse contingencies such as invalid state changes or the unsuccessful termination of invoked operations.) In some programming languages there are functions which cannot be safely called on invalid input data ... or functions which return values which cannot be distinguished from exceptionsfor example, in C the atoi (ASCII to integer conversion) function may return 0 (zero) for any input that cannot be parsed into a valid value. In such languages the programmer must either perform error checking (possibly through some auxiliary global variable such as C's errno) or input validation (perhaps using regular expressions). The degree to which such explicit validation and error checking is necessary is in contrast to exception handling support provided by any given programming environment. Hardware exception handling differs somewhat from the support provided by software tools, but similar concepts and terminology are prevalent. In general, an exception is handled (resolved) by saving the current state of execution in a predefined place and switching the execution to a specific subroutine known as an exception handler. Depending on the situation, the handler may later resume the execution at the original location using the saved information. For example, a page fault will usually allow the program to be resumed, while a division by zero might not be resolvable transparently. From the processing point of view, hardware interrupts are similar to resume-able exceptions, though they are typically unrelated to the user's program flow. From the point of view of the author of a routine, raising an exception is a useful way to signal that a routine could not execute normally. For example, when an input argument is invalid (e.g. a zero denominator in division) or when a resource it relies on is unavailable (like a missing file, or a hard disk error). In systems without exceptions, routines would need to return some special error code. However, this is sometimes complicated by the semipredicate problem, in which users of the routine need to write extra code to distinguish normal return values from erroneous ones. In runtime engine environments such as Java or .NET, there exist tools that attach to the runtime engine and every time that an exception of interest occurs, they record debugging information that existed in memory at the time the exception was thrown (call stack and heap
values). These tools are called automated exception handling or error interception tools and provide 'root-cause' information for exceptions. Contemporary applications face many design challenges when considering exception handling strategies. Particularly in modern enterprise level applications, exceptions must often cross process boundaries and machine boundaries. Part of designing a solid exception handling strategy is recognizing when a process has failed to the point where it cannot be economically handled by the software portion of the process.[1]
Contents
[hide]

1 Exception safety o 1.1 Verification of exception handling 2 Exception support in programming languages 3 Exception handling implementation 4 Exception handling based on Design by Contract 5 Checked exceptions o 5.1 Views on usage 6 Exception synchronicity 7 Condition systems o 7.1 Continuable exceptions o 7.2 Restarts separate mechanism from policy 8 See also 9 References 10 External links
[edit] Exception safety

A piece of code is said to be exception-safe, if run-time failures within the code will not produce ill effects, such as memory leaks, garbled stored data, or invalid output. Exceptionsafe code must satisfy invariants placed on the code even if exceptions occur. There are several levels of exception safety:
1. Failure transparency, also known as the no throw guarantee: Operations are guaranteed to succeed and satisfy all requirements even in presence of exceptional situations. If an exception occurs, it will not throw the exception further up. (Best level of exception safety.) 2. Commit or rollback semantics, also known as strong exception safety or no-change guarantee: Operations can fail, but failed operations are guaranteed to have no side effects so all data retain original values.[2] 3. Basic exception safety: Partial execution of failed operations can cause side effects, but invariants on the state are preserved. Any stored data will contain valid values even if data has different values now from before the exception. 4. Minimal exception safety also known as no-leak guarantee: Partial execution of failed operations may store invalid data but will not cause a crash, and no resources get leaked. 5. No exception safety: No guarantees are made. (Worst level of exception safety)
For instance, consider a smart vector type, such as C++'s std::vector or Java's ArrayList. When an item x is added to a vector v, the vector must actually add x to the internal list of objects and also update a count field that says how many objects are in v. It may also need to allocate new memory if the existing capacity isn't large enough. This memory allocation may fail and throw an exception. Because of this, a vector that provides failure transparency would be very difficult or impossible to write. However, the vector may be able to offer the strong exception guarantee fairly easily; in this case, either the insertion of x into v will succeed, or v will remain unchanged. If the vector provides only the basic exception safety guarantee, if the insertion fails, v may or may not contain x, but at least it will be in a consistent state. However, if the vector makes only the minimal guarantee, it's possible that the vector may be invalid. For instance, perhaps the size field of v was incremented but x wasn't actually inserted, making the state inconsistent. Of course, with no guarantee, the program may crash; perhaps the vector needed to expand but couldn't allocate the memory and blindly ploughs ahead as if the allocation succeeded, touching memory at an invalid address. Usually at least basic exception safety is required. Failure transparency is difficult to implement, and is usually not possible in libraries where complete knowledge of the application is not available.
[edit] Verification of exception handling
The point of exception handling routines is to ensure that the code can handle error conditions. In order to establish that exception handling routines are sufficiently robust, it is necessary to present the code with a wide spectrum of invalid or unexpected inputs, such as can be created via software fault injection and mutation testing (which is also sometimes referred to as fuzz testing). One of the most difficult types of software for which to write exception handling routines is protocol software, since a robust protocol implementation must be prepared to receive input that does not comply with the relevant specification(s). In order to ensure that meaningful regression analysis can be conducted throughout a software development lifecycle process, any exception handling verification should be highly automated, and the test cases must be generated in a scientific, repeatable fashion. Several commercially available systems exist that perform such testing.
[edit] Exception support in programming languages

See also: Exception handling syntax
Many computer languages, such as Actionscript, Ada, BlitzMax, C++, C#, D, ECMAScript, Eiffel, Java, ML, Object Pascal (e.g. Delphi, Free Pascal, and the like), Objective-C, Ocaml, PHP (as of version 5), PL/1, Prolog, Python, REALbasic, Ruby, Visual Prolog and most .NET languages have built-in support for exceptions and exception handling. In those languages, the event of an exception (more precisely, an exception handled by the language) searches back through the stack of function calls until an exception handler is found, with some languages calling for unwinding the stack as the search progresses. That is, if function f contains a handler H for exception E, calls function g, which in turn calls function h, and an exception E occurs in h, then functions h and g may be terminated, and H in f will handle E. An exception-handling language for which this is not true is Common Lisp with its Condition
System. Common Lisp calls the exception handler and does not unwind the stack. This allows to continue the computation at exactly the same place where the error occurred (for example when a previously missing file is now available). Mythryl's stackless implementation supports constant-time exception handling without stack unwinding. Excluding minor syntactic differences, there are only a couple of exception handling styles in use. In the most popular style, an exception is initiated by a special statement (throw, or raise) with an exception object (e.g. with Java or Object Pascal) or a value of a special extendable enumerated type (e.g. with Ada). The scope for exception handlers starts with a marker clause (try, or the language's block starter such as begin) and ends in the start of the first handler clause (catch, except, rescue). Several handler clauses can follow, and each can specify which exception types it handles and what name it uses for the exception object. A few languages also permit a clause (else) that is used in case no exception occurred before the end of the handler's scope was reached. More common is a related clause (finally, or ensure) that is executed whether an exception occurred or not, typically to release resources acquired within the body of the exception-handling block. Notably, C++ does not need and does not provide this construct, and the Resource-Acquisition-Is-Initialization technique should be used to free such resources instead.[3] In its whole, exception handling code might look like this (in Java-like pseudocode; note that an exception type called EmptyLineException would need to be declared somewhere):
try { line = console.readLine(); if (line.length() == 0) { throw new EmptyLineException("The line read from console was empty!"); } console.printLine("Hello %s!" % line); console.printLine("The program ran successfully"); } catch (EmptyLineException e) { console.printLine("Hello!"); } catch (Exception e) { console.printLine("Error: " + e.message()); } finally { console.printLine("The program terminates now"); }
As a minor variation, some languages use a single handler clause, which deals with the class of the exception internally. C supports various means of error checking but generally is not considered to support "exception handling." Perl has optional support for structured exception handling. The C++ derivative Embedded C++ excludes exception handling support as it can substantially increase the size of the object code. By contrast Python's support for exception handling is pervasive and consistent. It's difficult to write a robust Python program without using its support for try:ing certain operations and catching various except:ions.
[edit] Exception handling implementation
The implementation of exception handling typically involves a fair amount of support from both a code generator and the runtime system accompanying a compiler. (It was the addition of exception handling to C++ that ended the useful lifetime of the original C++ compiler, Cfront.[citation needed]) Two schemes are most common. The first, dynamic registration, generates code that continually updates structures about the program state in terms of exception handling.[4] Typically, this adds a new element to the stack frame layout that knows what handlers are available for the function or method associated with that frame; if an exception is thrown, a pointer in the layout directs the runtime to the appropriate handler code. This approach is compact in terms of space but adds execution overhead on frame entry and exit. It was commonly used in many Ada implementations, for example, where complex generation and runtime support was already needed for many other language features. Dynamic registration, being fairly straightforward to define, is amenable to proof of correctness.[5] The second scheme, and the one implemented in many production-quality C++ compilers, is a table-driven approach. This creates static tables at compile and link time that relate ranges of the program counter to the program state with respect to exception handling.[6] Then, if an exception is thrown, the runtime system looks up the current instruction location in the tables and determines what handlers are in play and what needs to be done. This approach minimizes executive overhead for the case where an exception is not thrown, albeit at the cost of some space, although said space can be allocated into read-only, special-purpose data sections that are not loaded or relocated until and unless an exception is thrown.[7] This second approach is also superior in terms of achieving thread safety. Other definitional and implementation schemes have been proposed as well.[8] For languages that support metaprogramming, approaches that involve no overhead at all have been advanced.[9]
[edit] Exception handling based on Design by Contract

A different view of exceptions is based on the principles of Design by Contract and is supported in particular by the Eiffel language. The idea is to provide a more rigorous basis for exception handling by defining precisely what is "normal" and "abnormal" behavior. Specifically, the approach is based on two concepts:
Failure: the inability of an operation to fulfill its contract. For example an addition may produce an arithmetic overflow (it does not fulfill its contract of computing a good approximation to the mathematical sum); or a routine may fail to meet its postcondition. Exception: an abnormal event occurring during the execution of a routine (that routine is the "recipient" of the exception) during its execution. Such an abnormal event results from the failure of an operation called by the routine.
The "Safe Exception Handling principle" as introduced by Bertrand Meyer in ObjectOriented Software Construction then holds that there are only two meaningful ways a routine can react when an exception occurs:
Failure, or "organized panic": the routine fails, triggering an exception in its caller (so that the abnormal event is not ignored!), after fixing the object's state by re-establishing the invariant (the "organized" part).
Retry: try the algorithm again, usually after changing some values so that the next attempt will have a better chance to succeed.
Here is an example expressed in Eiffel syntax. It assumes that a routine send_fast is normally the better way to send a message, but it may fail, triggering an exception; if so, the algorithm next uses send_slow, which will fail less often. If send_slow fails, the routine send as a whole should fail, causing the caller to get an exception.
send (m: MESSAGE) is -- Send m through fast link if possible, otherwise through slow link. local tried_fast, tried_slow: BOOLEAN do if tried_fast then tried_slow := True send_slow (m) else tried_fast := True send_fast (m) end rescue if not tried_slow then retry end end
The boolean local variables are initialized to False at the start. If send_fast fails, the body (do clause) will be executed again, causing execution of send_slow. If this execution of send_slow fails, the rescue clause will execute to the end with no retry (no else clause in the final if), causing the routine execution as a whole to fail. This approach has the merit of defining clearly what a "normal" and "abnormal" cases are: an abnormal case, causing an exception, is one in which the routine is unable to fulfill its contract. It defines a clear distribution of roles: the do clause (normal body) is in charge of achieving, or attempting to achieve, the routine's contract; the rescue clause is in charge of reestablishing the context and restarting the process if this has a chance of succeeding, but not of performing any actual computation.
[edit] Checked exceptions

The designers of Java devised[10][11] checked exceptions,[12] which are a special set of exceptions. The checked exceptions that a method may raise are part of the method's signature. For instance, if a method might throw an IOException, it must declare this fact explicitly in its method signature. Failure to do so raises a compile-time error. This is related to exception checkers that exist at least for OCaml.[13] The external tool for OCaml is both transparent (i.e. it does not require any syntactic annotations) and facultative (i.e. it is possible to compile and run a program without having checked the exceptions, although this is not suggested for production code).
The CLU programming language had a feature with the interface closer to what Java has introduced later. A function could raise only exceptions listed in its type, but any leaking exceptions from called functions would automatically be turned into the sole runtime exception, failure, instead of resulting in compile-time error. Later, Modula-3 had a similar feature.[14] These features don't include the compile time checking which is central in the concept of checked exceptions, and hasn't (as of 2006) been incorporated into major programming languages other than Java.[15] The C++ programming language introduces an optional mechanism for checked exceptions, called exception specifications. By default any function can throw any exception, but this can be limited by a throw clause added to the function signature, that specifies which exceptions the function may throw. Exception specifications are not enforced at compile-time. Violations result in the global function std::unexpected being called.[16] An empty exception specification may be given, which indicates that the function will throw no exception. This was not made the default when exception handling was added to the language because it would require too much modification of existing code, would impede interaction with code written in another language, and would tempt programmers into writing too many handlers at the local level.[16] Explicit use of empty exception specifications can, however, allow C++ compilers to perform significant code and stack layout optimizations that normally have to be suppressed when exception handling may take place in a function.[7] Some analysts view the proper use of exception specifications in C++ as difficult to achieve.[17] In the upcoming C++ language standard (C++0x), the use of exception specifications as specified in the current version of the standard (C++03), is deprecated.
[edit] Views on usage
Checked exceptions can, at compile time, reduce the incidence of unhandled exceptions surfacing at runtime in a given application; the unchecked exceptions (RuntimeExceptions and Errors) remain unhandled.[citation needed] However, checked exceptions can either require extensive throws declarations, revealing implementation details and reducing encapsulation, or encourage coding poorly-considered [citation try/catch blocks that can hide legitimate exceptions from their appropriate handlers. needed] Consider a growing codebase over time. An interface may be declared to throw exceptions X & Y. In a later version of the code, if one wants to throw exception Z, it would make the new code incompatible with the earlier uses. Furthermore, with the adapter pattern, where one body of code declares an interface that is then implemented by a different body of code so that code can be plugged in and called by the first, the adapter code may have a rich set of exceptions to describe problems, but is forced to use the exception types declared in the interface. It is possible to reduce the number of declared exceptions by either declaring a superclass of all potentially thrown exceptions or by defining and declaring exception types that are suitable for the level of abstraction of the called method,[18] and mapping lower level exceptions to these types, preferably wrapped using the exception chaining in order to preserve the root cause. In addition, it's very possible that in the example above of the changing interface that the calling code would need to be modified as well, since in some sense the exceptions a method may throw are part of the method's implicit interface anyway.
Using a minimal throws Exception declaration or catch (Exception e) is sufficient for satisfying the checking in Java. While this may have some use, it essentially circumvents the checked exception mechanism, and forces all calling code to catch (Exception e) or add throws Exception to its method signature. Unchecked exception types should not be handled except, with consideration, at the outermost levels of scope. These often represent scenarios that do not allow for recovery: [19] RuntimeExceptions frequently reflect programming defects, and Errors generally represent unrecoverable JVM failures. The view is that, even in a language that supports checked exceptions, there are cases where the use of checked exceptions is not appropriate.[by
whom?]
[edit] Exception synchronicity

Somewhat related with the concept of checked exceptions is exception synchronicity. Synchronous exceptions happen at a specific program statement whereas asynchronous exceptions can raise practically anywhere.[20][21] It follows that asynchronous exception handling can't be required by the compiler. They are also difficult to program with. Examples of naturally asynchronous events include pressing Ctrl-C to interrupt a program, and receiving a signal such as "stop" or "suspend" from another thread of execution. Programming languages typically deal with this by limiting asynchronicity, for example Java has lost thread stopping and resuming.[22] Instead, there can be semi-asynchronous exceptions that only raise in suitable locations of the program or synchronously.
[edit] Condition systems

Common Lisp, Dylan and Smalltalk have a Condition system [23] (see Common Lisp Condition System) which encompasses the aforementioned exception handling systems. In those languages or environments the advent of a condition (a "generalisation of an error" according to Kent Pitman) implies a function call, and only late in the exception handler the decision to unwind the stack may be taken. Conditions are a generalization of exceptions. When a condition arises, an appropriate condition handler is searched for and selected, in stack order, to handle the condition. Conditions which do not represent errors may safely go unhandled entirely; their only purpose may be to propagate hints or warnings toward the user.[24]
[edit] Continuable exceptions
This is related to the so-called resumption model of exception handling, in which some exceptions are said to be continuable: it is permitted to return to the expression that signaled an exception, after having taken corrective action in the handler. The condition system is generalized thus: within the handler of a non-serious condition (a.k.a. continuable exception), it is possible to jump to predefined restart points (a.k.a. restarts) that lie between the signaling expression and the condition handler. Restarts are functions closed over some lexical environment, allowing the programmer to repair this environment before exiting the condition handler completely or unwinding the stack even partially.
[edit] Restarts separate mechanism from policy
Condition handling moreover provides a separation of mechanism from policy. Restarts provide various possible mechanisms for recovering from error, but do not select which mechanism is appropriate in a given situation. That is the province of the condition handler, which (since it is located in higher-level code) has access to a broader view. An example: Suppose there is a library function whose purpose is to parse a single syslog file entry. What should this function do if the entry is malformed? There is no one right answer, because the same library could be deployed in programs for many different purposes. In an interactive log-file browser, the right thing to do might be to return the entry unparsed, so the user can see itbut in an automated log-summarizing program, the right thing to do might be to supply null values for the unreadable fields, but abort with an error if too many entries have been malformed. That is to say, the question can only be answered in terms of the broader goals of the program, which are not known to the general-purpose library function. Nonetheless, exiting with an error message is only rarely the right answer. So instead of simply exiting with an error, the function may establish restarts offering various ways to continuefor instance, to skip the log entry, to supply default or null values for the unreadable fields, to ask the user for the missing values, or to unwind the stack and abort processing with an error message. The restarts offered constitute the mechanisms available for recovering from error; the selection of restart by the condition handler supplies the policy.
Load testing
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a systems behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation. When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test. There is little agreement on what the specific goals of load testing are. The term is often used synonymously with software performance testing, reliability testing, and volume testing. Load testing is a type of non-functional testing.
Contents
[hide]

1 Software load testing o 1.1 Popular load testing tools 2 Mechanical load testing 3 Car charging system 4 See also 5 References 6 External links
[edit] Software load testing

Main article: Software performance testing
The term load testing is used in different ways in the professional software testing community. Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as web servers. However, other types of software systems can also be load tested. For example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. The most accurate load testing simulates actual use, as opposed to testing using theoretical or analytical modeling.
Load and performance testing analyzes software intended for a multi-user audience by subjecting the software to different amounts of virtual and live users while monitoring performance measurements under these different loads. Load and performance testing is usually conducted in a test environment identical to the production environment before the software system is permitted to go live. As an example, a web site with shopping cart capability is required to support 100 concurrent users broken out into following activities:

25 Virtual Users (VUsers) log in, browse through items and then log off 25 VUsers log in, add items to their shopping cart, check out and then log off 25 VUsers log in, return items previously purchased and then log off 25 VUsers just log in without any subsequent activity
A test analyst can use various load testing tools to create these VUsers and their activities. Once the test has started and reached a steady state, the application is being tested at the100 VUser load as described above. The applications performance can then be monitored and captured. The specifics of a load test plan or script will generally vary across organizations. For example, in the bulleted list above, the first item could represent 25 VUsers browsing unique items, random items, or a selected set of items depending upon the test plan or script developed. However, all load test plans attempt to simulate system performance across a range of anticipated peak workflows and volumes. The criteria for passing or failing a load test (pass/fail criteria) are generally different across organizations as well. There are no standards specifying acceptable load testing performance metrics. A common misconception is that load testing software provides record and playback capabilities like regression testing tools. Load testing tools analyze the entire OSI protocol stack whereas most regression testing tools focus on GUI performance. For example, a regression testing tool will record and playback a mouse click on a button on a web browser, but a load testing tool will send out hypertext the web browser sends after the user clicks the button. In a multiple-user environment, load testing tools can send out hypertext for multiple users with each user having a unique login ID, password, etc. The popular load testing tools available also provide insight into the causes for slow performance. There are numerous possible causes for slow system performance, including, but not limited to, the following:

Application server(s) or software Database server(s) Network latency, congestion, etc. Client-side processing Load balancing between multiple servers
Load testing is especially important if the application, system or service will be subject to a service level agreement or SLA.
[edit] Popular load testing tools Tool Name Company Name 'Open System Testing Architecture' Notes Open source web load/stress testing application, licensed under the Gnu GPL. Utilizes a distributed software architecture based on CORBA. OpenSTA binaries available for Windows. Eclipse based large scale performance testing tool primarily used for executing large volume performance tests to measure system response time for server based applications. Licensed.
OpenSTA
IBM Rational Performance Tester
IBM
JMeter
An Apache Jakarta Java desktop application for load testing and performance open source measurement. project Performance testing tool primarily used for executing large numbers of tests (or a large number or virtual users) concurrently. Can be used for unit and integration testing as well. Licensed. Performance testing tool that verifies functionality and performance under load. Supports SOAtest tests, JUnits, lightweight socket-based components. Detects concurrency issues. Performance testing in an open and sharable model which allows realistic load tests for thousands of users running business scenarios across a broad range of enterprise application environments. Visual Studio includes a load test tool which enables a developer to execute a variety of tests (web, unit etc...) with a combination of configurations to simulate real user load[1].
LoadRunner
HP
Load Test (included with Soatest
Parasoft
SilkPerformer
Micro Focus
Visual Studio Load Microsoft Test
[edit] Mechanical load testing

The purpose of a mechanical load test is to verify that all the component parts of a structure including materials, base-fixings are fit for task and loading it is designed for. The Supply of Machinery (Safety) Regulation 1992 UK state that load testing is undertaken before the equipment is put into service for the first time. Load testing can be either Performance,Static or Dynamic.
Performance testing is when the stated safe working load (SWL) for a configuration is used to determine that the item performs to the manufactures specification. If an item fails this test then any further tests are pointless. Static testing is when a load at a factor above the SWL is applied. The item is not operated through all configurations as it is not a requirement of this test. Dynamic testing is when a load at a factor above the SWL is applied. The item is then operated fully through all configurations and motions. Care must be taken during this test as there is a great risk of catastrophic failure if incorrectly carried out. The design criteria, relevant legislation or the Competent Person will dictate what test is required. Under the Lifting Operations and Lifting Equipment Regulations 1998 UK load testing after the initial test is required if a major component is replaced, if the item is moved from one location to another or as dictated by the Competent Person The loads required for a test are stipulated by the item under test, but here are a few to be aware off. Powered lifting equipment Static test to 1.25 SWL and dynamic test to 1.1 SWL. Manual lifting equipment Static test to 1.5 SWL For lifting accessories. 2 SWL for items up to 30 tonne capacity. 1.5 SWL for items above 30 tonne capacity. 1 SWL for items above 100 tonnes.
[edit] Car charging system

A load test can be used to evaluate the health of a car's battery. The tester consists of a large resistor that has a resistance similar to a car's starter motor and a meter to read the battery's output voltage both in the unloaded and loaded state. When the tester is used, the battery's open circuit voltage is checked first. If the open circuit voltage is below spec (12.6 volts for a fully charged battery), the battery is charged first. After reading the battery's open circuit voltage, the load is applied. When applied, it draws approximately the same current the car's starter motor would draw during cranking. Based on the specified cold cranking amperes of the battery, if the voltage under load falls below a certain point, the battery is bad. Load tests are also used on running cars to check the output of the car's alternator.
Volume testing

This article does not cite any references or sources.

Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (March 2008)
Volume Testing belongs to the group of non-functional tests, which are often misunderstood and/or used interchangeably. Volume testing refers to testing a software application with a certain amount of data. This amount can, in generic terms, be the database size or it could also be the size of an interface file that is the subject of volume testing. For example, if you want to volume test your application with a specific database size, you will expand your database to that size and then test the application's performance on it. Another example could be when there is a requirement for your application to interact with an interface file (could be any file such as .dat, .xml); this interaction could be reading and/or writing on to/from the file. You will create a sample file of the size you want and then test the application's functionality with that file in order to test the performance.
Security testing
Security testing is a process to determine that an information system protects data and maintains functionality as intended. The six basic security concepts that need to be covered by security testing are: confidentiality, integrity, authentication, availability, authorization and non-repudiation. Security testing as a term has a number of different meanings and can be completed in a number of different ways. As such a Security Taxonomy helps us to understand these different approaches and meanings by providing a base level to work from.
Contents
[hide]

1 Confidentiality 2 Integrity 3 Authentication 4 Authorization 5 Availability 6 Non-repudiation 7 Security Testing Taxonomy 8 See also
[edit] Confidentiality
A security measure which protects against the disclosure of information to parties other than the intended recipient that is by no means the only way of ensuring the security.
[edit] Integrity

A measure intended to allow the receiver to determine that the information which it is providing is correct. Integrity schemes often use some of the same underlying technologies as confidentiality schemes, but they usually involve adding additional information to a communication to form the basis of an algorithmic check rather than the encoding all of the communication.
[edit] Authentication

The process of establishing the identity of the user. Authentication can take many forms including but not limited to: passwords, biometrics, radio frequency identification, etc.
[edit] Authorization

The process of determining that a requester is allowed to receive a service or perform an operation. Access control is an example of authorization.
[edit] Availability

Assuring information and communications services will be ready for use when expected. Information must be kept available to authorized persons when they need it.
[edit] Non-repudiation

A measure intended to prevent the later denial that an action happened, or a communication that took place etc. In communication terms this often involves the interchange of authentication information combined with some form of provable time stamp.
[edit] Security Testing Taxonomy

Common terms used for the delivery of security testing;
Discovery - The purpose of this stage is to identify systems within scope and the services in use. It is not intended to discover vulnerabilities, but version detection may highlight deprecated versions of software / firmware and thus indicate potential vulnerabilities. Vulnerability Scan - Following the discovery stage this looks for known security issues by using automated tools to match conditions with known vulnerabilities. The reported risk level is set automatically by the tool with no manual verification or interpretation by the test vendor. This can be supplemented with credential based scanning that looks to remove some common false positives by using supplied credentials to authenticate with a service (such as local windows accounts). Vulnerability Assessment - This uses discovery and vulnerability scanning to identify security vulnerabilities and places the findings into the context of the environment under test. An example would be removing common false positives from the report and deciding risk levels that should be applied to each report finding to improve business understanding and context.
Security Assessment - Builds upon Vulnerability Assessment by adding manual verification to confirm exposure, but does not include the exploitation of vulnerabilities to gain further access. Verification could be in the form of authorised access to a system to confirm system settings and involve examining logs, system responses, error messages, codes, etc. A Security Assessment is looking to gain a broad coverage of the systems under test but not the depth of exposure that a specific vulnerability could lead to. Penetration Test - Penetration test simulates an attack by a malicious party. Building on the previous stages and involves exploitation of found vulnerabilities to gain further access. Using this approach will result in an understanding of the ability of an attacker to gain access to confidential information, affect data integrity or availability of a service and the respective impact. Each test is approached using a consistent and complete methodology in a way that allows the tester to use their problem solving abilities, the output from a range of tools and their own knowledge of networking and systems to find vulnerabilities that would/ could not be identified by automated tools. This approach looks at the depth of attack as compared to the Security Assessment approach that looks at the broader coverage. Security Audit - Driven by an Audit / Risk function to look at a specific control or compliance issue. Characterised by a narrow scope, this type of engagement could make use of any of the earlier approaches discussed (vulnerability assessment, security assessment, penetration test). Security Review - Verification that industry or internal security standards have been applied to system components or product. This is typically completed through gap analysis and utilises build / code reviews or by reviewing design documents and architecture diagrams. This activity does not utilise any of the earlier approaches (Vulnerability Assessment, Security Assessment, Penetration Test, Security Audit)
Scalability testing

Scalability Testing, part of the battery of non-functional tests, is the testing of a software application for measuring its capability to scale up or scale out [1]- in terms of any of its nonfunctional capability - be it the user load supported, the number of transactions, the data volume etc. Performance, scalability and reliability are usually considered together by software quality analysts.
Stress testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Stress testing may have a more specific meaning in certain industries, such as fatigue testing for materials.
Contents
[hide]

1 Computer software 2 Hardware o 2.1 Computer processors 3 Financial sector 4 See also 5 References
[edit] Computer software
Main article: stress testing (software)
In software testing, a system stress test refers to tests that put a greater emphasis on robustness, availability, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. In particular, the goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks. Examples:
A web server may be stress tested using scripts, bots, and various denial of service tools to observe the performance of a web site during peak loads.
Stress testing may be contrasted with load testing:
Load testing examines the entire environment and database, while measuring the response time, whereas stress testing focuses on identified transactions, pushing to a level so as to break transactions or systems. During stress testing, if transactions are selectively stressed, the database may not experience much load, but the transactions are heavily stressed. On the other hand, during load testing the database experiences a heavy load, while some transactions may not be stressed. System stress testing, also known as stress testing, is loading the concurrent users over and beyond the level that the system can handle, so it breaks at the weakest link within the entire system.
[edit] Hardware
Reliability engineers often test items under expected stress or even under accelerated stress. The goal is to determine the operating life of the item or to determine modes of failure.[1] Stress testing, in general, should put the hardware under exaggerated levels of stress in order to ensure stability when used in a normal environment.
[edit] Computer processors
When modifying the operating parameters of a CPU, such as in overclocking, underclocking, overvolting, and undervolting, it may be necessary to verify if the new parameters (usually CPU core voltage and frequency) are suitable for heavy CPU loads. This is done by running a CPU-intensive program (usually Prime95) for extended periods of time, to test whether the computer hangs or crashes. CPU stress testing is also referred to as torture testing. Software that is suitable for torture testing should typically run instructions that utilise the entire chip rather than only a few of its units. Stress testing a CPU over the course of 24 hours at 100% load is, in most cases, sufficient enough to determine that the CPU will function correctly in normal usage scenarios, where CPU usage fluctuates at low levels (50% and under), such as on a desktop computer.
[edit] Financial sector

See also: bank stress tests
Instead of doing financial projection on a "best estimate" basis, a company may do stress testing where they look at how robust a financial instrument is in certain crashes, a form of scenario analysis. They may test the instrument under, for example, the following stresses:

What happens if equity markets crash by more than x% this year? What happens if interest rates go up by at least y%? What if half the instruments in the portfolio terminate their contracts in the fifth year? What happens if oil prices rise by 200%?
This type of analysis has become increasingly widespread, and has been taken up by various governmental bodies (such as the FSA in the UK) as a regulatory requirement on certain financial institutions to ensure adequate capital allocation levels to cover potential losses incurred during extreme, but plausible, events. This emphasis on adequate, risk adjusted determination of capital has been further enhanced by modifications to banking regulations such as Basel II. Stress testing models typically allow not only the testing of individual stressors, but also combinations of different events. There is also usually the ability to test the current exposure to a known historical scenario (such as the Russian debt default in 1998 or 9/11 attacks) to ensure the liquidity of the institution. Stress testing reveals how well a portfolio is positioned in the event forecasts prove true. Stress testing also lends insight into a portfolio's vulnerabilities. Though extreme events are never certain, studying their performance implications strengthens understanding.
Defining stress tests
Stress testing defines a scenario and uses a specific algorithm to determine the expected impact on a portfolio's return should such a scenario occur. There are three types of scenarios:

Extreme event: hypothesize the portfolio's return given the recurrence of a historical event. Current positions and risk exposures are combined with the historical factor returns. Risk factor shock: shock any factor in the chosen risk model by a user-specified amount. The factor exposures remain unchanged, while the covariance matrix is used to adjust the factor returns based on their correlation with the shocked factor. External factor shock: instead of a risk factor, shock any index, macro-economic series (e.g., oil prices), or custom series (e.g., exchange rates). Using regression analysis, new factor returns are estimated as a result of the shock.
In an exponentially weighted stress test, historical periods more like the defined scenario receive a more significant weighting in the predicted outcome. The defined decay rate lets the tester manipulate the relative importance of the most similar periods. In the standard stress test, each period is equally weighted.
Sanity testing
(Redirected from Sanity test) Jump to: navigation, search A sanity test or sanity check is a basic test to quickly evaluate whether a claim or the result of a calculation can possibly be true. The point of a sanity test is to rule out certain classes of obviously false results, not to catch every possible error. In arithmetic, for example, when multiplying by 9, using the divisibility rule for 9 to verify that the sum of digits of the result is divisible by 9 is a sanity test - it will not catch every multiplication error, however it's a quick and simple method to discover many possible errors. In computer science, a sanity test is a very brief run-through of the functionality of a computer program, system, calculation, or other analysis, to assure that the system or methodology works as expected, often prior to a more exhaustive round of testing.
Contents
[hide]

1 Mathematical 2 Software development 3 See also 4 References
[edit] Mathematical
A sanity test can refer to various order-of-magnitude and other simple rule-of-thumb devices applied to cross-check mathematical calculations. For example:
If one were to attempt to square 738 and calculated 53,874, a quick sanity check could 2 2 2 show that this result cannot be true. Consider that 700 < 738, yet 700 = 7 100 = 490000 > 53874. Since squaring positive numbers preserves their inequality, 2 the result cannot be true, and so the calculation was bad. The correct answer, 738 = 544,644, is more than 10 times higher than 53,874, and so the result had been off by an order of magnitude. In multiplication, 918 155 is not 142135 since 918 is divisible by three but 142135 is not (digits add up to 16, not a multiple of three). Also, the product must end in the same digit as the product of end-digits 85=40, but 142135 does not end in "0" like "40", while the correct answer does: 918155=142290. An even quicker check is that the product of even and odd numbers is even, whereas 142135 is odd. When talking about quantities in physics, the power output of a car cannot be 700 kJ since that is a unit of energy, not power (energy per unit time). See dimensional analysis.
[edit] Software development

In software development, the sanity test (a form of software testing which offers "quick, broad, and shallow testing"[1]) determines whether it is reasonable to proceed with further testing. Software sanity tests are commonly conflated with smoke tests. [2] A smoke test determines whether it is possible to continue testing, as opposed to whether it is reasonable[citation needed]. A software smoke test determines whether the program launches and whether its interfaces are accessible and responsive (for example, the responsiveness of a web page or an input button). If the smoke test fails, it is impossible to conduct a sanity test. In contrast, the ideal sanity test exercises the smallest subset of application functions needed to determine whether the application logic is generally functional and correct (for example, an interest rate calculation for a financial application). If the sanity test fails, it is not reasonable to attempt more rigorous testing. Both sanity tests and smoke tests are ways to avoid wasting time and effort by quickly determining whether an application is too flawed to merit any rigorous testing. Many companies run sanity tests and unit tests on an automated build as part of their development process.[3] The Hello world program is often used as a sanity test for a development environment. If Hello World fails to compile or execute, the supporting environment likely has a configuration problem. If it works, the problem being diagnosed likely lies in the real application being diagnosed. Another, possibly more common usage of 'sanity test' is to denote checks which are performed within program code, usually on arguments to functions or returns therefrom, to see if the answers can be assumed to be correct. The more complicated the routine, the more important that its response be checked. The trivial case is checking to see that a file opened, written to, or closed, did not fail on these activities which is a sanity check often ignored by programmers. But more complex items can also be sanity-checked for various reasons. Examples of this include bank account management systems which check that withdrawals are sane in not requesting more than the account contains, and that deposits or purchases are sane in fitting in with patterns established by historical data large deposits may be more closely scrutinized for accuracy, large purchase transactions may be double-checked with a card holder for validity against fraud, ATM withdrawals in foreign locations never before visited by the card holder might be cleared up with him, etc.; these are "runtime" sanity checks, as opposed to the "development" sanity checks mentioned above.
Smoke testing
From Wikipedia, the free encyclopedia (Redirected from Smoke test) Jump to: navigation, search It has been suggested that this article or section be merged with Build_verification_test. (Discuss) This article needs additional citations for verification.
Please help improve this article by adding reliable references. Unsourced material may be challenged and removed. (December 2009)
Smoke testing refers to the first test made after assembly or repairs to a system, to provide some assurance that the system under test will not catastrophically fail. After a smoke test proves that "the pipes will not leak, the keys seal properly, the circuit will not burn, or the software will not crash outright," the assembly is ready for more stressful testing. The term smoke testing is used in several fields, including electronics, computer software development, plumbing, woodwind repair, infectious disease control, and the entertainment industry.
Contents
[hide]

1 History of the term 2 Smoke testing in various industries o 2.1 Electronics and electrical engineering o 2.2 Software development o 2.3 Plumbing o 2.4 Woodwind instrument repair o 2.5 Automotive repair o 2.6 Infectious disease control o 2.7 Entertainment 3 See also 4 References 5 External links
[edit] History of the term
"The phrase smoke test comes from hardware testing. You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don't have to do any more testing." [1]
[edit] Smoke testing in various industries

[edit] Electronics and electrical engineering
In electronics and electrical engineering the term smoke test or power on test is used to refer to the first time a circuit under development is attached to power, which will sometimes produce actual smoke if a design or wiring mistake has been made. Most often this smoke comes from burning resistors, which produce a unique smell familiar to many technicians. For certain circuits, overheating and burning due to circuitry that is still not properly operating can be avoided by slowly turning up the input voltage to the unit under test by using a variable autotransformer and watching the electric current consumption[citation needed]. As a poor-man's "autotransformer", a properly-sized incandescent light bulb in series with the power feed can provide a similar benefit: if the unit under test has a short circuit or other overload, the bulb will light up and provide a high resistance, limiting or preventing further damage to the unit being tested. Overloaded integrated circuits typically produce "blue smoke" (or magic smoke). "Blue smoke" is the subject of jokes among technicians who refer to it as if it were a genie in the circuit: It's the blue smoke that makes it worklet out the blue smoke and it won't do anything.
[edit] Software development
In computer programming and software testing, smoke testing is a preliminary to further testing, intended to reveal simple failures severe enough to reject a prospective software release. In this case the smoke is metaphorical. A subset of test cases that cover the main functionality of a component or system are selected and run, to ascertain if the most crucial functions of a program work correctly. For example, a smoke test may ask basic questions like "Does the program run?", "Does it open a window?", or "Does clicking the main button do anything?" The purpose is to determine whether the application is so badly broken that further testing is unnecessary. As the book "Lessons Learned in Software Testing" puts it, "smoke tests broadly cover product features in a limited time ... if key features don't work or if key bugs haven't yet been fixed, your team won't waste further time installing or testing"[1]. Smoke testing performed on a particular build is also known as a build verification test[1][2]. A daily build and smoke test is among industry best practices[citation needed]. Smoke testing is also done by testers before accepting a build for further testing. Microsoft claims that after code reviews, "smoke testing is the most cost effective method for identifying and fixing defects in software"[3]. In Microsoft's case a smoke test is the process of validating code changes before they are checked into source control. Smoke tests can either be performed manually or using an automated tool. When automated tools are used, the tests are often initiated by the same process that generates the build itself[citation needed].
Smoke tests can be broadly categorized as functional tests or unit tests. Functional tests exercise the complete program with various inputs. Unit tests exercise individual functions, subroutines, or object methods. Both functional testing tools and unit testing tools tend to be third-party products that are not part of the compiler suite[citation needed]. Functional tests may be a scripted series of program inputs, possibly even with an automated mechanism for controlling mouse movements. Unit tests may be separate functions within the code itself, or driver layer that links to the code without altering the code being tested.
[edit] Plumbing
In plumbing a smoke test forces non-toxic, artificially created smoke through waste and drain pipes under a slight pressure to find leaks[citation needed]. Plumes of smoke form where there are defects. This test can be performed when the plumbing is brand new, but more often it is used to find sewer gas leaks that may plague a building or an area[citation needed]. Any sign of smoke escaping can be considered a possible site for sewer gas to escape. Sewer gas typically has a rotten egg smell and can contain methane gas, which is explosive, or hydrogen sulfide gas, which is deadly. Plumbing smoke tests are also used to find places where pipes will spill fluid[citation needed], and to check sanitary sewer systems for places where ground water and storm runoff can enter. Smoke testing is particularly useful in places such as ventilated sanitary sewer systems, where completely sealing the system is not practical. When smoke testing a sanitary sewer system it is helpful to partially block off the section of sewer to be tested. This can be done by using a sand bag on the end of a rope. The sand bag is lowered into the manhole and swung into position to partially block lines. Completely blocking the line can cause water to back up and prevent smoke from escaping through defects. Smoke testing may not be done after rain or when ground water is unusually high as this may also prevent detection of defects. Large downdraft fans, usually powered by gasoline engines, are placed on top of open manholes at either end of the section to be tested. If possible all lines in the manholes except for the line between the manholes are partially blocked. Smoke is created using either a smoke bomb or liquid smoke. Smoke bombs are lit and placed on a grate or in a holder on top of each fan, while liquid smoke is injected into the fan via a heating chamber. The fans create a pressure differential that forces the smoke into the sewer at a pressure just above atmospheric. With properly installed plumbing, the traps will prevent the smoke from entering the house and redirect it out the plumbing vents. Defective plumbing systems or dry traps will allow smoke to enter the inside of the house. The area around the section being tested is searched for smoke plumes. Plumes coming from plumbing vents or the interface between the fan shroud and manhole rim are normal; however, smoke plumes outside of the manhole rim are not. Plumes are marked, usually with flags, and defects are noted using measurements from stationary landmarks like the corners of houses. The plumes or markers may also be photographed.
[edit] Woodwind instrument repair
In woodwind instrument repair, a smoke test involves plugging one end of an instrument and blowing smoke into the other to test for leaks. Escaping smoke reveals improperly seated
pads and faulty joints (i.e. leaks). After this test the instrument is cleaned to remove nicotine and other deposits left by the smoke. Due to tobacco smoke being used, this test may be hazardous to the health of the technician in the long run[citation needed]. Described in a repair manual written in the 1930s[citation needed], smoke testing is considered obsolete[citation needed], and is no longer used by reputable technicians[citation needed]. The usual alternative to smoke is to place a bright light inside the instrument then check for light appearing around pads and joints.
[edit] Automotive repair
In the same way that plumbing and woodwind instruments are tested, the vacuum systems of automobiles may be tested in order to locate difficult-to-find vacuum leaks. Artificial smoke is deliberately introduced into the system- under slight pressure and any leaks are indicated by the escaping smoke. Smoke can also be used to locate difficult-to-find leaks in the fuel evaporative emissions control (EVAP) system.
[edit] Infectious disease control
In infectious disease control a smoke test is done to see whether a room is under negative pressure. A tube containing smoke is held near the bottom of the negative pressure room door, about 2 inches in front of the door. The smoke tube is held parallel to the door, and a small amount of smoke is then generated by gently squeezing the bulb. Care is taken to release the smoke from the tube slowly to ensure the velocity of the smoke from the tube does not overpower the air velocity. If the room is at negative pressure, the smoke will travel under the door and into the room. If the room is not a negative pressure, the smoke will be blown outward or will stay stationary.
[edit] Entertainment
In the entertainment industry a smoke test is done to ensure that theatrical smoke and fog used during a live event will not set off the smoke detectors in a venue. To smoke test a venue the venue is filled to the full capacity with smoke to see if there are any smoke detectors still live, or if there are any leaks of smoke from the venue sufficient to set off detectors in other parts of the venue being tested.
Destructive testing
Jump to: navigation, search "Crash testing" redirects here. For the automotive use, see crash test. In destructive testing, tests are carried out to the specimen's failure, in order to understand a specimen's structural performance or material behaviour under different loads. These tests are generally much easier to carry out, yield more information, and are easier to interpret than nondestructive testing. Destructive testing is most suitable, and economic, for objects which will be mass produced, as the cost of destroying a small number of specimens is negligible. It is usually not economical to do destructive testing where only one or very few items are to be produced (for example, in the case of a building). Some types of destructive testing:

Stress tests Crash tests Hardness tests Metallographic tests
Contents
[hide]

1 Testing of large structures 2 Destructive software testing 3 See also 4 Notes
[edit] Testing of large structures
Snapshot from shake-table video of a 6-story non-ductile concrete building
Building structures or large nonbuilding structures (such as dams and bridges) are rarely subjected to destructive testing due to the prohibitive cost of constructing a building, or a scale model of a building, just to destroy it. Earthquake engineering requires a good understanding of how structures will perform at earthquakes. Destructive tests are more frequently carried out for structures which are to be constructed in earthquake zones. Such tests are sometimes referred to as crash tests, and they are carried out to verify the designed seismic performance of a new building, or the actual performance of an existing building. The tests are, mostly, carried out on a platform called a shake-table which is designed to shake in the same manner as an earthquake. Results of those tests often include the corresponding shake-table videos. As structural performance at earthquakes is better understood, testing of structures in earthquakes is increasingly done by modelling the structure using specialist finite element software.
[edit] Destructive software testing

Destructive software testing is a type of software testing which attempts to cause a piece of software to fail in an uncontrolled manner, in order to test its robustness
Exploratory testing
Jump to: navigation, search Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[2] While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.[3]
Contents
[hide]

1 History 2 Description 3 Benefits and drawbacks 4 Usage 5 See also 6 References 7 External links
[edit] History
Exploratory testing has always been performed by skilled testers. In the early 1990s, ad hoc was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner in his book Testing Computer Software[1] and expanded upon in Lessons Learned in Software Testing.[4] Exploratory testing can be as disciplined as any other intellectual activity.
[edit] Description
Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester's skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be. To further explain, comparison can be made of freestyle exploratory testing to its antithesis scripted testing. In this activity test cases are designed in advance. This includes both the individual steps and the expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures, operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting information that seems like to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort). In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context. According to Cem Kaner & James Bach, exploratory testing is more a mindset or "...a way of thinking about testing" than a methodology.[5] They also say that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).[6] The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs. During pair testing, two persons create test cases together; one performs them, and the other documents. Session-based testing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale. Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or tools to quickly help generate situations of interest, e.g. James Bach's Perlclip.
[edit] Benefits and drawbacks

The main advantage of exploratory testing is that less preparation is needed, important bugs are found quickly, and at execution time, the approach tends to be more intellectually stimulating than execution of scripted tests. Another major benefit is that testers can use deductive reasoning based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently. Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."
Disadvantages are that tests invented and performed on the fly can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run. Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.
[edit] Usage
Exploratory testing is particularly suitable if requirements and specifications are incomplete, or if there is lack of time.[7][8] The approach can also be used to verify that previous testing has found the most important defects.[7]
Ad hoc testing
Jump to: navigation, search Ad hoc testing is a commonly used term for software testing performed without planning and documentation (but can be applied to early scientific experimental studies).
The tests are intended to be run only once, unless a defect is discovered. Ad hoc testing is a part of exploratory testing,[citation needed] being the least formal of test methods. In this view, ad hoc testing has been criticized because it isn't structured, but this can also be a strength: important things can be found quickly. It is performed with improvisation, the tester seeks to find bugs with any means that seem appropriate. It contrasts to regression testing that looks for a specific issue with detailed reproduction steps, and a clear expected result. Ad hoc testing is most often used as a complement to other types of testing.
Reliability theory
(Redirected from Reliability testing) Jump to: navigation, search Reliability theory developed apart from the mainstream of probability and statistics. It was originally a tool to help nineteenth century maritime insurance and life insurance companies compute profitable rates to charge their customers. Even today, the terms "failure rate" and "hazard rate" are often used interchangeably. The failure of mechanical devices such as ships, trains, and cars, is similar in many ways to the life or death of biological organisms. Statistical models appropriate for any of these topics are generically called "time-to-event" models. Death or failure is called an "event", and the goal is to project or forecast the rate of events for a given population or the probability of an event for an individual. When reliability is considered from the perspective of the consumer of a technology or service, actual reliability measures may differ dramatically from perceived reliability. One bad experience can be magnified in the mind of the customer, inflating the perceived unreliability of the product. One plane crash where hundreds of passengers die will immediately instill fear in a large percentage of the flying consumer population, regardless of actual reliability data about the safety of air travel. Reliability period of any object is measured within the durability period of that object.
Installation testing
Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (December 2009) Software Testing portal
Implementation testing installation testing is a kind of quality assurance work in the software industry that focuses on what customers will need to do to install and set up the new software successfully. The testing process may involve full, partial or upgrades install/uninstall processes. This testing is typically done by the software testing engineer in conjunction with the configuration manager. Implementation testing is usually defined as testing which places a compiled version of code into the testing or pre-production environment, from which it may or may not progress into production. This generally takes place outside of the software development environment to limit code corruption from other future releases which may reside on the development network. The simplest installation approach is to run an install program, sometimes called package software. This package software typically uses a setup program which acts as a multiconfiguration wrapper and which may allow the software to be installed on a variety of machine and/or operating environments. Every possible configuration should receive an appropriate level of testing so that it can be released to customers with confidence. In distributed systems, particularly where software is to be released into an already live target environment (such as an operational website) installation (or software deployment as it is sometimes called) can involve database schema changes as well as the installation of new software. Deployment plans in such circumstances may include back-out procedures whose use is intended to roll the target environment back if the deployment is unsuccessful. Ideally, the deployment plan itself should be tested in an environment that is a replica of the live environment. A factor that can increase the organizational requirements of such an exercise is the need to synchronize the data in the test deployment environment with that in the live environment with minimum disruption to live operation. This type of implementation may include testing of the processes which take place during the installation or upgrade of a multitier application. This type of testing is commonly compared to a dress rehearsal or may even be called a dry run.
Maintenance testing
This article is an orphan, as few or no other articles link to it. Please introduce links to this page from related articles; suggestions may be available. (December 2009) Maintenance testing is that testing which is performed to either identify equipment problems, diagnose equipment problems or to confirm that repair measures have been effective. It can be performed at either the system level (e.g., the HVAC system), the equipment level (e.g., the blower in a HVAC line), or the component level (e.g., a control chip in the control box for the blower in the HVAC line).
Contents
[hide]

1 Technical basis for testing 2 In-service inspection and testing 3 Evaluation and trending of test results 4 Types of maintenance for which testing can be used 5 See also
[edit] Technical basis for testing

Maintenance testing uses system performance requirements as the basis for identifying the appropriate
[edit] In-service inspection and testing

One level of maintenance testing is the in-service inspection or inspection, which typically is a test or series of tests performed on a frequency established by the manufacturer based on prior experience with the system/equipment/component or upon engineering analysis of the probable failure rate for the equipment. Typical examples of inspections and tests include:
Periodic vehicle inspections mandated by regulatory authorities for features important to environmental protection (e.g., exhaust emissions) or for safety (e.g., brake and signal lights). Periodic inspections of nuclear power plant equipment important to the safe operation of the facility. Such inspections include a range of tests such as
Leak-rate tests for containment structures (the reactor boundary, the containment vessel, etc.). o Calibration tests to assure shutdown systems will activate on over-power or over-temperature conditions. The medical physical examination or clinical examination can be considered diagnostic testing of in-service inspection for the human body Performance tests such as turbine shaft vibration tests to identify wear and plan maintenance are a form of diagnostic testing.
[edit] Evaluation and trending of test results

A good testing program will maintain a record of test results and maintenance actions taken. These data will be evaluated for trends and serve as the basis for decisions on appropriate testing frequency, need to replace or upgrade equipment and performance improvement opportunities.
[edit] Types of maintenance for which testing can be used

Maintenance falls into the following four categories:

Preventive maintenance Changes to the existing system so as to reduce the risk of failure while operating. Corrective maintenance Correcting problems that arise while using the system. Perfective maintenance Enhancements (modifications) to improve the safety, reliability, efficiency or cost-effectiveness of operation. Adaptive maintenance Adaptations to address requirements that crop up due to changes in the environment or new regulations.
Recovery testing
Jump to: navigation, search In software testing, recovery testing is the activity of testing how well an application is able to recover from crashes, hardware failures and other similar problems. Recovery testing is the forced failure of the software in a variety of ways to verify that recovery is properly performed. Recovery testing should not be confused with reliability testing, which tries to discover the specific point at which failure occurs.Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications. It is basically testing how well a system recovers from crashes, hardware failures, or other catastrophic problems
Examples of recovery testing: 1. While an application is running, suddenly restart the computer, and afterwards check the validness of the application's data integrity. 2. While an application is receiving data from a network, unplug the connecting cable. After some time, plug the cable back in and analyze the application's ability to continue receiving data from the point at which the network connection disappeared. 3. Restart the system while a browser has a definite number of sessions. Afterwards, check that the browser is able to recover all of them.
Fault injection

Testing Basics

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Testing Basics

Uploaded by

Copyright:

Available Formats

Test case

From Wikipedia, the free encyclopedia

[edit] Formal test cases

[edit] Informal test cases

[edit] Typical written test case format

Test summary Configuration

[edit] Code-driven testing

[edit] Graphical User Interface (GUI) testing

[edit] What to test

[edit] Framework approach in automation

[edit] Popular Test Automation Tools

[edit] Testing the whole system

[edit] Types of tests to include in system testing

GUI software testing Usability testing Performance testing

Software development process

Software development process

[edit] Software development activities

[edit] Deployment and maintenance

[edit] Software Development Models

[edit] Process Improvement Models

[edit] Formal methods

Jump to: navigation, search

Software development process

1 Sample traceability matrix 2 See also 3 References 4 External links

[edit] Sample traceability matrix

1.1.1 1.1.2 1.1.3 1.1.4 1.1.5 1.1.6

Baseline (configuration management)

[edit] Capabilities of Baselines

[edit] Baseline Control

[edit] Big Bang

Software development process

[edit] Requirements engineering

[edit] Requirements analysis topics

[edit] Stakeholder identification

[edit] Stakeholder interviews

[edit] Alternative to requirement lists

[edit] Types of Requirements

[edit] Requirements analysis issues

[edit] Attempted solutions

1 Examples 2 See also 3 References 4 External links

[edit] History of usability testing

[edit] Goals of usability testing

[edit] What usability testing is not

[edit] How many users to test?

For more information please visit the link BCT

[edit] Exception safety

[edit] Exception support in programming languages

[edit] Exception handling implementation

[edit] Exception handling based on Design by Contract

[edit] Checked exceptions

[edit] Exception synchronicity

[edit] Condition systems

[edit] Restarts separate mechanism from policy

[edit] Software load testing

IBM Rational Performance Tester

Load Test (included with Soatest

Visual Studio Load Microsoft Test

[edit] Mechanical load testing

[edit] Car charging system

Jump to: navigation, search

This article does not cite any references or sources.

[edit] Security Testing Taxonomy

Jump to: navigation, search

[edit] Computer software