Test - Strategy and Implementation

By jonasbn <jonasbn@io.dk>

Test - Strategy and Implementation

What this presentation will cover?

An outline of test methods How to design a test strategy from test methods How to implement it a test strategy using test methods Exemplified by all the mistakes humanly possible
Test - Strategy and Implementation

What this presentation will not give you?

A solution to all your problems A neat Solution::Simple for download from CPAN

Test - Strategy and Implementation

What this presentation try to give you?

A new mindset when thinking about testing A tool to make better tests A tool to make the right tests A tool to improve your development

Test - Strategy and Implementation

Test

QuickT ime and a T IF F (Uncompressed) decompressor are needed to see this picture.

Test - Strategy and Implementation

Are YOU a Tester?

Hello my name is jonasbn Im a tester. In my daily work I work as a developer and thats what it says on the paper, but Im really a tester. Every time my colleagues turn the other way I write one and every time my manager gives me a task - all I think about is tests and how to test often I work late so I can finish my tests without using valuable development time Yes Im a tester and I cannot help it. - jonasbn, developer (tester)
Test - Strategy and Implementation

What does test mean?

How do you define test?

Test - Strategy and Implementation

My test definition
Tests can assure me that what I am coding works correctly Tests are aiming at being automated Test is a tool to help my development Tests can be difficult to develop correctly

Test - Strategy and Implementation

What kind of test methods do you know?

What test methods do you currently use? Or have discarded? What test methods have you used?

Test - Strategy and Implementation

My list of Test Methods

Coverage Test Smoke Test Integration Test Acceptance Test System Execution Test Regression Test Stress Test Performance Test Unit Test

Test - Strategy and Implementation

Coverage Test
Simple Philosophy
All lines should be run at least once Testing all possible branches of the code Time consuming Large number of tests Difficult way of testing Tools to assist coverage test exist

Test - Strategy and Implementation

Smoke Test
From the world of electronics
Power it up - does it break? Not much detail Crude way of testing

Test - Strategy and Implementation

Integration Test
When things need to be assembled
Known from large projects For dispersed development When integration with 3rd. Party components

Test - Strategy and Implementation

Acceptance Test
When the Client/Customer are to approve
Should not replace Performance Test Should not replace automated testing Should not be automated - this is a manual process

Test - Strategy and Implementation

System Execution Test

From phase based development
When test is a phase Focus is changed Many aspects covered in other methods Has serious flaws Can be cut due to time constraints

Test - Strategy and Implementation

Regression Test
When old things are revisited
Used to ensure backwards compability

Test - Strategy and Implementation

Stress Test
When trying to break the system
Resembles smoke test Also known as monkey test Changing environments Garbage Feeding

Test - Strategy and Implementation

Performance Test
When speed is the need
For benchmarking Can be used to identify bottlenecks History is important

Test - Strategy and Implementation

Unit Test
The foundation of all testing
Can be used to implement almost all test methods Simplicity SMOP Can be automated Programmatic approach
Test - Strategy and Implementation

Test Methods Summarized

Automatic and manual test methods Black box and White box angles

Test - Strategy and Implementation

Black/White Box Testing

Nothing is known apart from what is public Everything is known Introspections are possible

Test - Strategy and Implementation

Test Strategy
Melchett: You look surprised, Blackadder. Edmund: I certainly am, sir. I didn't realise we had any battle plans. Melchett: Well, of course we have! How else do you think the battles are directed? Edmund: Our battles are directed, sir? Melchett: Well, of course they are, Blackadder -directed according to the Grand Plan. Edmund: Would that be the plan to continue with total slaughter until everyone's dead except Field Marshal Haig, Lady Haig and their tortoise, Alan? Melchett: Great Scott! Even you know it! Guard! Guard! Bolt all the doors; hammer large pieces of crooked wood against all the windows! This security leak is far worse than we'd imagined!
Test - Strategy and Implementation

Project Goals
We need to examine the projects goals to make the right test strategy The test strategy should support the projects goals not work against them

Test - Strategy and Implementation

Defining a strategy
We apply a selection the listed test methods to form a strategy
Which ones should we pick?

Test - Strategy and Implementation

An example - eXtreme Programming

Unit Test during development Integration Test when applicable Acceptance Test and the end of each cycle

Test - Strategy and Implementation

An example - Waterfall/Phasedivided
Unit Test during development Smoke Test when building Integration Test when partially done System Execution Test when done Acceptance Test

Test - Strategy and Implementation

Other Strategies?
What strategy do YOU use?

Test - Strategy and Implementation

Select the right strategy

Is your application a GUI? Is your application performance critical? Is quality important? Is data correctness essential? Is data processing essential? Is backwards compability important?

Test - Strategy and Implementation

Test Implementation

Test - Strategy and Implementation

Basic Unit Tests

use strict; use Test::More tests => 6; use lib qw(lib); BEGIN { use_ok( 'Geneva::Product' ); } require_ok( 'Geneva::Product' ); my $p = Geneva::Product->new(); ok(! (ref $p), 'failing constructor test'); $p = Geneva::Product->new( p_customerRef => 'argle', ); ok((ref $p) eq 'Geneva::Product', 'succeeding constructor test'); ok($p->isa('Geneva'), 'Testing isa of Geneva'); ok($p->can('new'), 'Testing can Geneva::Product::new');

Test - Strategy and Implementation

Unit Test Tactics

Should I go for black box or white box? Can I trust the units

Test - Strategy and Implementation

Advanced Unit Tests 1

sub move2done ($) { my $file = shift ; if ( -e "$mail_processed/$file") { unlink ("$mail_processed/$file"); } my $rv = rename("$mail_spool/$file","$mail_processed/$file") || warn "Unable to move file: $mail_spool/$file to $mail_processed/$file - $!\n"; return $rv; }

Test - Strategy and Implementation

Advanced Unit Test 2

sub DSL0281ReceiveTDCRejected ($$) { my ($dbh, $filename) = @_; my ($dslid, $order_copper_id, $reason) = RejectedHandler($filename); if ($dslid and $order_copper_id) { eval { my $status = getstatus($dbh, $order_copper_id); if ($status ne DB_STATUS_RECEIVED) { die ("Status of order_copper_id $order_copper_id is $status, expected ".DB_STATUS_RECEIVED) ; } updateOrderCopper($dbh, $order_copper_id, DB_STATUS_REJECTED, undef, undef, \$reason); addtolog($dbh, DB_STATUS_REJECTED, $dslid, $order_copper_id); }; if ($@) { warn $@ ; return "Error: $@" ; } else { return "OK" ; } } else { return "Error: DSL_ID or Order_Copper_Id could not be extracted from mail" ;

Test - Strategy and } Implementation

Acceptance Test
Acceptance Test is a manual process and should therefore not be automated Unit Test reports can add some color to the delivery though underlining the fact that the product/system has been tested

Test - Strategy and Implementation

Coverage Test
Devel::Coverage Devel::Cover

Test - Strategy and Implementation

Smoke Test
Perl smoke testing worth a study

Test - Strategy and Implementation

Integration Test
You can easily test this if you take a Smoke Test approach

Test - Strategy and Implementation

System Execution Test

This Test method is merely a phase, but it should be implemented using the test methods/strategy mentioned in this presentation

Test - Strategy and Implementation

Regression Test
Regression Test will be necessary at some point Save and use old tests, dont throw anything away

Test - Strategy and Implementation

Stress Test
Just bombard your application with calls from your test suite Your test suite will cover all aspects of your application Do it structured Examine the output

Test - Strategy and Implementation

Performance Test 1
Benchmark is your friend Save reports, historic data is essential Examine output And compare output

Test - Strategy and Implementation

Performance Test 2
use SOAP::Lite; use Benchmark; my $t0 = new Benchmark; my $proxy = "http://somemachine/tempuri/Service1.asmx"; my $uri = "http://service.domain.dk/"; my $soap = SOAP::Lite ->uri($uri) ->proxy($proxy) ->on_action(sub {join '', @_}); my $method = SOAP::Data->name('getcustomer')->attr({xmlns => 'http://service.domain.dk/'}); my @params = (); push(@params, SOAP::Data->name('dslid' => 'dsl89305')->attr({'xsi:type' => 'xsd:string'})); my $rc = $soap->call($method => @params)->valueof("//getcustomerResponse"); my $t1 = new Benchmark; my $td = timediff($t1, $t0); print "the code took:",timestr($td),"\n";

Test - Strategy and Implementation

Databases
Database encapsulations can be tricky to test Data is often volatile and a baseline is difficult to optain

Test - Strategy and Implementation

Solution 1: The Sims

The volatile aspect disappears Data is known and controlled The data can be misleading

Test - Strategy and Implementation

Solution 2: The Others

Real-life data

Test - Strategy and Implementation

The Others Example

use strict; use Test::More tests => 2; use LWP::UserAgent; my $phoneno = '32545697'; my $uri = 'http://hostname.org/cgi-bin/proxy1.cgi'; $uri .= "?phoneno=$phoneno"; my $ua = LWP::UserAgent->new(env_proxy => 1, keep_alive => 1, timeout => 30, ); my $request = HTTP::Request->new('GET', $uri); my $response = $ua->request($request); my $content = $response->content; is($response->message, "OK", "Testing connection to URL: $uri"); like($content, qr/Amagerbro/i, "Testing content of response");

Test - Strategy and Implementation

The Frame-work
Too many tests Redundant code all over the place Programmatic Practice SMOP

Test - Strategy and Implementation

The Frame-work 2
my %data_map = ( message => { type => 'xsd:string', value => kdplg', }, ); my $test_target = 'addtolog'; my $proxy = "http://webservice/Service1.asmx"; my $uri = "http://webservice.host.org/"; my @api_params = qw(message); my $t = Test::Accept::SOAP->new(); $t->test( 'proxy' => $proxy, 'uri' => $uri, 'test_target' => $test_target, 'api_params' => \@api_params, 'data_map' => \%data_map, );

Test - Strategy and Implementation

The Frame-work 3
sub test ($%) { my ($self, %args) = @_; my my my my my my $proxy $uri $test_target $api_params $data_map $soap = = = = = = $args{'proxy'}; $args{'uri'}; $args{'test_target'}; $args{'api_params'}; $args{'data_map'}; $args{'soap'};

unless ($proxy and $uri and $test_target and $api_params and $data_map) { warn "Unable to perform test, insufficient parameters\n"; return; } my $method = $self->methodize($uri, $test_target); my @params = (); foreach my $p (@{$api_params}) { $self->parameterize(\@params,$data_map, $p); } my $rc = $self->exercize($soap, $test_target, $method, \@params, $debug); ok($rc, 'Checking we got something back from the webservice'); my $test_result = $test_target.'Result'; ok($rc->{$test_result}, 'Testing the returned result (shallow)');

Test Strategy and return 1; Implementation }

Test Related Perl Modules

Test Test::Simple Test:More Devel::Cover Devel::Coverage

Test - Strategy and Implementation

The Power of Perl

SOAP/Web Services SNMP SMTP HTTP Databases Whatever you can think of

Test - Strategy and Implementation

References
The C++ Programming Language 2nd. Ed., Bjarne Stroustrup Extreme Programming Explained embrace change, Kent Beck Code Complete, Steve McConnell Testing and Code Coverage abstract, Paul Johnson Test::More POD, Michael Schwern Test - Strategy and Implementation article, jonasbn
Test - Strategy and Implementation