Test Methodologies

Aztecsoft has evolved a three-step approach to testing software, based on best practices observed
throughout a wide variety of successful testing projects.

Test Strategy

Test Test
Execution Planning

Test Strategy

A product's test strategy ties the product's release and sign-off criteria to its business objectives.

The overall testing strategy is defined in collaboration with the customer. In order to take key
dependencies into account, test planning, testcase design, test automation and test execution are
aligned with the development schedule. Meaningful test scheduling requires a clear understanding of
ETA's and sequencing for :

• Completion of low- and high-level specifications;

• Code-complete (coding for everything but bug-fixes stops);
• Completion of component unit-testing (when QA can begin interaction testing);
• UI-freeze (after which QA can be confident that (a) UI level automation will not break
repeatedly due to fluctuating UI and screen layouts, and (b) that API-level automation
will not be undermined by changes in interfaces/API's).
Identifying key trade-offs is essential, for it is impossible to test all scenarios, cover the full
configuration matrix, and automate all test cases, while remaining within the practical limits of time
and budget. Trade-offs on the development side need to be in concert with trade-off development
trade-offs; otherwise, development and test will have conflicting priorities. This is the stage in which
project focus is established.

We identify the features, components, sub-components, and items to be tested and the range of
tests to be carried out. In addition to available automation we also estimate other required and
possible automation. We catalog the tools used by the customer, potentially useful off-the-shelf
tools, and internal Aztecsoft tools that may be used for the project.

We identify which features/components will be tested manually, which will be tested via automation,
and what kind of automation tool is required (script-based, GUI-based, proprietary, off-the-shelf,
etc.).

To summarize, the key activities are as follows:

• Define project scope & commitments

• Define terms of reference
• Set customer expectations
• Tie together the business objectives of the STQE project with the release/sign off
criterion and associated testing activity
• Integrate the STQE processes with development lifecycle
• Partition the problem into manageable test plans
• Identify key dependencies & trade-offs
• Scope resource requirements
TOP
Test Planning

The next step is Test Planning, which defines the approach for testing. The first task is to establish,
and seek confirmation from the customer, a clear understanding of the project and its deliverables.

Exhaustive analysis ensures that there is no mismatch between our understanding and the
customer's requirements. For example we determine whether we should construct a clean test
environment for each test run, or we can use system imaging to shorten test-bed setup.

All relevant products interface, component, and other external dependencies are identified and the
timeframe for delivering the results is computed. The resulting plan is presented in industry-
standard format to the customer; further steps are not taken without customer acceptance.

Here are the key steps for Test Planning:

• Define release criteria (with the release manager)

• Outline and prioritize the testing effort.
• Chart test automation requirements
• Identify resource requirements at various stages of testing
• Set up calendar-based activity plan
• State reporting mechanism & establish communication model
• Configure team including number, type, and seniority of resources and length of time
required, mapped each resource onto the activity plan.
TOP
Test Execution

Execution of the test plan begins with fulfillment of test-start criteria, and ends with the fulfillment
of test-complete criteria. Although each project has different requirements, the intermediary steps
will resemble the following:

• Prepare comprehensive test plan specifications and test cases for each level of testing.
• Review all test plans and test cases
• Prepare test data and test logs.
• Set up the test environment so that all operations can be completed promptly, accurately,
and efficiently.
• Execute Error/Trap tests to ensure testers accuracy.
• Execute tests as described, noting where test cases require revision and updating.
• Report all bugs in the manner agreed upon with the customer, following all defect
management protocols, informing customer of current status, monitoring and driving to
resolution all red-flag conditions, and ensuring that communication between all parties is
complete and accurate.
• Run spot checks to ensure quality.
• Update weekly the Project Health Status document for Internal Audit & Tracking.
• When the project has been completed, review all aspects of the project and submit a
Project Retrospective report to the customer that objectively evaluates the project's
execution