Agile Automation Detailed

Successfully Automating Functional and Regression Testing with Open Source Tools Detailed Notes
By John Lockhart, john@webtest.co.nz, www.webtest.co.nz Successfully Automating Functional and Regression Testing with Open Source Tools Detailed Notes.....................................................................................................................................................1 Part 0: Overview...............................................................................................................................1 What we will cover.......................................................................................................................1 Part 1: The profession.......................................................................................................................2 Why test? .....................................................................................................................................2 Perceptions of testing....................................................................................................................2 Task/Job/Career/Profession/Specialty...........................................................................................3 Other disciplines within software development.............................................................................3 The broader perspective: Software Engineering...........................................................................5 A Software Engineer Specialising in Testing.................................................................................6 Part 2 Key principles from other areas to apply.............................................................................8 Lessons for testers from Software Engineerings other disciplines...............................................8 Lessons for testers from outside Software Engineering................................................................8 All about Agile..............................................................................................................................9 Part 3 Examples...........................................................................................................................12 Example 1 Small successful development team from the 90's.................................................12 Example 2 Internet booking application with development team of 20...................................12 Example 3 Corporate struggle.................................................................................................12 Example 4 Corporate success, so far anyway..........................................................................14 Part 4 Automating........................................................................................................................16 The two main causes of failure and keys to success...................................................................16 Principles and Architecture.........................................................................................................16 Part 5 Overview of Fitnesse and associated tools........................................................................20 The Fitnesse Architecture............................................................................................................20 Fitnesse Features.........................................................................................................................20 New Fitnesse Developments and Add-Ons.................................................................................20 Part 6: Additional useful skills and tools of the trade for testers.....................................................22 Part 7 The path to success and happiness....................................................................................24
Part 0: Overview
What we will cover
Testing is perceived as the least important / least professional discipline and lags behind others in software development. How can we beat this emotionally and practically? We will examine our beliefs and views of testing as these control our level of success. We will apply principles of these mature areas to testing and learn from them. Essential to effective and satisfying testing is the appropriate approach to automation. Examples will demonstrate the importance of the right people with the right priority and demonstrate some successful and unsucessful dynamics. We then delve more deeply in functional automation using open source tools in an agile manner with a focus on architecture and maintenance and look at Fitnesse as an example. We will see that the biggest risk for automation is failure in maintenance and that the keys to avoiding this are firstly having the right people with the right priority and secondly having the correct test architecture.
Successfully Automating Functional and Regression Testing with Open Source Tools Detailed Notes John Lockhart - john@webtest.co.nz STANZ 2008, Wellington, New Zealand Page 1 of 24
We combine all this with a suggested model of software testing as a specialty within the general profession of software engineering in order to define what we need to do to deliver the most value and satisfaction. After reviewing the overall attitudes as well as the specific tools and skills required, you should be well placed to know what you need to do to succeed individually or with a team that you manage. You should have a clear vision of what success is and the knowledge, distinctions and drive to create the change required.
Part 1: The profession

What you believe about yourself, your work, the world and what will give you pleasure or pain drives your level of enjoyment and satisfaction and also drives your behaviour. Your attitude and behaviour in turn drive, or limit, your success. So it is critical to examine these beliefs rather than to just absorb them unquestioningly as you go through life.
Why test?
There are many possible answers, some more useful than others: Ensure no bugs. Impossible you can prove bugs exist by finding them but you can't prove there are no bugs, so sets you up for failure when one slips through. Ensure all functionality is delivered correctly. Sounds good but according to what standard? Usually this is a functional specification or requriements document but I've yet to see one that was interpreted identially by all readers and had no omissions. Ensure key workflows are not totally broken. This doesn't sound so smooth, but is the most basic and important of regression and integration testing principles. If you test that some examples of the most critical scenarios through the system work end to end, you have mitigated the highest risk that the release is broken in some fundamental way. Stop defects entering production. Testing as a quality gate for production is traditional e.g. I heard it from an IBM consultant recently. Problem is the Japanese quality revolution which was based on Demings work had as a key concept that quality is the responsibility of the whole team. If you accept the quality police role it creates an us and them situation which is not useful. It also has the same failing as the first reason in that some defects will always enter production. Assess risk or provide information regarding quality. This is a balanced view of what testers do within the larger team and reflects the fact that it is not usually their decision how much testing can be done or when the software should be deployed. Assist the team to improve quality. This is the agile model and works very well in close teams encouraging a culture of quality throughout the development process and lets the testers add value in the maximum possible ways. Legal or contractual reasons. This and other specific test requirements will often exist and can be covered in most of the models given above.
Perceptions of testing
We will pull people in who are not busy from elsewhere in the business to do the testing. Implies anyone can do it which implies no skills required. Even supermarkets require some training before staff are let loose in any role! Testing is a necessary evil and a cost centre that does not deliver business value. Often this is a management view but sometimes exists in the development team as well. Testers are pedants with no business sense or with no technical sense. Testers are failed developers or are learning to be a developer or a business analyst. Sometimes testing is not even recognised as worthy of a title so testers have to be classified as a junior business analyst etc.
If testing really is like that and doesn't require specific skills or experience then it is by definition of little value unskilled labour. Do you believe any of the above? Do people around you? How can we correct this? What would the opposite belief be? Perhaps: Testers have general knowledge of both business and technical areas and specific skills that produce great value in terms of quality with impressive efficiency and business focus reducing overall product risk and cost and find it a satisfying specialty providing opportunities unmatched by other roles. If you don't believe in yourself or your role, and have sound and specific reasons for those beliefs, others certainly won't.
Task/Job/Career/Profession/Specialty
Think about the importance of a name. Say to yourself a job, a career, a profession. For me very different things come to mind. A job is something that never changes and you probably don't enjoy but just do for the money. That pay probably isn't much and is unlikely to increase unless you work overtime. You might train for maybe a day or a week when you start as defined by the boss but are unlikely to have ongoing training or development. A career has a path or ladder of progress up a salary scale but training and how you proceed is as much or more your bosss responsibility as your own. It is likely to require a degree and salary range might be $40 000 to $80 000. Some job satisfaction is likely. A profession has connotations of being responsible for your own progress, level of expertise, income and of being something you are really interested in. Salaries are probably higher and getting well into the $100 000+ range. A specialist takes it further with an expectation that you do what you need to do to maintain world class expertise including travelling to attend training, reading professional journals, and perhaps even doing research in a particular niche. There is no hard limit on earning potential. Think of a doctor who then does two additional years of training to specialise. Which sets of perceptions is true of you? Which would you like to be? Whichever of those labels you choose, do you have goals that will get you there?
Other disciplines within software development

I believe testing is perhaps the most interesting area in software development, because it is at the cusp of a revolution. That is why I moved to it from the more mature area of project management. Let's compare it to some of the other roles in software engineering: Architect: Started around 1970. Service Oriented Architecture (SOA) perhaps represents the maturing of a generic architectural model that is not hardware dependant as previous ones were. $95,000 to $135,000. UML, EABOK, Zachman framework. SOA. Microsoft has a stringent qualification for this (see http://www.microsoft.com/learning/mcp/architect/) among others. Others exist such as http://www.opengroup.org/itac/ and http://www.sei.cmu.edu/architecture/certificate_program.html. The ultimate technical position partly due to the responsibility of managing the technical complexity. Agile has a somewhat different model for the architect role but is still the technical leader.
Project Manager: Started in the 1950s and matured in the 1960s. Pay $80,000 to $130,000 Defined by Project Management Institute (PMI) (1969) with body of knowledge PMBOK and professional qualification (PMP) with stringent initial and ongoing requirements and professional support. Many university have Masters courses etc. Prior to the 1950s, projects were managed on an ad hoc basis using mostly Gantt Charts and informal techniques and tools sound like testing? Is testing 50 years behind project management in its development? Scrum (1986/91) is challenging traditional project management theory for software. Part of agile and uses a rugby game vs. a relay race metaphor so work together rather than passing the baton. Programmer/Developer/DBA: In 1954 FORTRAN was invented and was the first high level programming language. Matured in 1990s with Object Oriented styles standardising on Java/C#. $65,000 to $95,000 Sun and Microsoft have Certifications and some IT degree usually present. Communities but no professional body. Similarly to other Engineers most serious software developers these days would be expected to have four years of university training. Object Oriented approach led to standardisation on Java / C# for most large scale development. Extreme Programming (XP) and Agile have revitalised teams and can be seen as the discipline of programming taking responsibility for its own standards and methods rather than relying on managers to tell them what to do with some Software Development Life Cycle (SDLC). Business Analyst: History and origins not clear. Maybe maturing with UML etc but often not used in practice by BAs. $65,000 to $95,000 Certification in initial stages. UML provides some shared language. The International Institute of Business Analysis (http://www.theiiba.org/) provides a certification program for business analysts (Certified Business Analyst Professional or CBAP), as well as providing a body of knowledge for the field ([Business Analysis Body of Knowledge or BABOK). Like testing often perceived as something with no specific skills that anyone from the business or with an IT degree can do and also as a path into IT say for a support or call centre person. Like testing many BAs lack the tools and skills they need. Tester/ Test Analyst / Test Manager: The separation of debugging from testing was introduced by Glenford J. Myers in 1979. Not even names standardised. Originally testing was not even included as a role within XP but that has now reversed in Agile. Test Driven Development (TDD) and the more recent Acceptance Test Driven Development (ATDD) are fascinating paradigm shifts with massive implications for the role and value of testing. $50,000 to $95,000 Certification in initial stages. ISTQB Foundation and just this year Advanced certification (equivalent to ISEB). Perhaps more importantly provide a BOK but controversy over this from competing models e.g. AST (http://www.associationforsoftwaretesting.org/). CSTE and
CSQA also have a BOK: http://www.softwarecertifications.org/cstebok/cstebok.htm. CSTP (see http://www.iist.org/cstp.php, and http://www.testinginstitute.com/certification.php and http://www.qaiworldwide.org) has prerequisites and ongoing requirements for recertification but they are extremely minimal e.g. 1 year's experience and 10 days of training to be certified. None of these have a professional body of any significance. Perceived as something with no specific skills that anyone from the business can do and also as a path into IT say for a support or call centre person. Tools and methods for automation unchanged from the 90s using outdated Computer Aided Software Engineering (CASE) model with large custom propriatory tool suites based around databases. The tool vendors as a rule don't even use their own tools. The agile community has found an alternative to this. Note: The above are my estimates for salary. See
http://www.absoluteit.co.nz/absolute/absoluteweb.nsf/SalarySurvey/$FILE/Remuneration%20Survey %20AbsoluteIT.pdf and http://www.robertwalters.com/resources/salarysurvey/New%20Zealand %202008.pdf for some more credible data and http://www.payscale.com/mypayscale.aspx for an
interesting site that calculates your current salary range among other interesting stuff.
The broader perspective: Software Engineering

Software Engineer: Term appeared in late 50's with conferences in late 60's. The profession grew in response to the software crisis of those and subsequent decades. Dijkstra's 1968 article "A Case against the GO TO Statement", is regarded as a major step towards the widespread deprecation of the GOTO statement and its effective replacement by structured control constructs, such as the while loop. This methodology was also called structured programming. This was the birth of the profession. Is test automation struggling to learn these lessons of 40 years ago? $70,000 to $90,000 (omitting architects and high value specialists) As of the 2004 edition, the SWEBOK guide defines ten knowledge Areas (KA) within the field of "software engineering". Testing and Quality are two of them. Software Engineering 2004 (SE2004) provides a competing view. Both are IEEE sponsored. There are plenty of degree and post grad courses. Lack a central certification but some well established professional bodies e.g. Computer Society. http://www2.computer.org/portal/web/csda/prep has a list of resources for their software engineer certification. For example have a look at the sample questions at http://www2.computer.org/portal/web/csda/test to get some idea of how much you may already know. Let's look into Software Engineering to get a better context for testing: Some quotes from the SWEBOK introduction: WHAT IS A RECOGNIZED PROFESSION? For software engineering to be fully known as a legitimate engineering discipline and a recognized profession, consensus on a core body of knowledge is imperative. This fact is well illustrated by Starr when he defines what can be considered a legitimate discipline and a recognized profession. In his Pulitzer Prize-winning book on the history of the medical profession in the USA, he states, The legitimization of professional authority involves three distinctive claims: first, that the knowledge and competence of the professional have been validated by a community of his or her peers; second, that this consensually validated knowledge rests on rational, scientific grounds; and third, that the professionals judgment and advice are oriented toward a set of substantive values, such as health. These aspects of legitimacy correspond to the kinds of attributescollegial, cognitive, and moralusually
embodied in the term profession. WHAT ARE THE CHARACTERISTICS OF A PROFESSION? Gary Ford and Norman Gibbs studied several recognized professions, including medicine, law, engineering, and accounting. They concluded that an engineering profession is characterized by several components: An initial professional education in a curriculum validated by society through accreditation Registration of fitness to practice via voluntary certification or mandatory licensing Specialized skill development and continuing professional education Communal support via a professional society A commitment to norms of conduct often prescribed in a code of ethics This Guide contributes to the first three of these components.
A Software Engineer Specialising in Testing

Page 31 and 32 of the 234 page SWBOK Guide at http://www.swebok.org/ lists the 11 Knowledge Areas of SW Engineering of which Testing is one and Quality is another. The full list is: Software requirements Software design Software construction Software testing Software maintenance Software configuration management Software engineering management Software engineering process Software engineering tools and methods Software quality Perhaps to really be a professional tester you need understanding and some skill in each of these. For example: Do you understand configuration management to the level that you can manage at least test resources effectively, and can access other resources as required? If not start using subversion or some other similar tool so you do understand the concepts. Do you understand requirements enough to validate them and to test again even technical ones such as UML? If not see the excellent STANZ talk on this 2 years ago. Do you understand project management enough to capably run a testing project or the testing component of a project? Can you define scope and deliverables, manage budget, timelines, staffing, suppliers, reporting and communications? Do you understand software construction enough to properly construct a test suite that is maintainable? And programming well enough that you can automate simple tasks in some language? Do you have skills with XML/XSL, Databases, text files and tools such as RegEx to work
with each of these, sufficient to use log files and other sorts of results effectively? Do you understand traditional waterfall SDLC and differences from Agile? All of these skills can be learnt in a degree course, in commercial courses at considerable cost, or in your own time at next to no expense, or perhaps even on your bosss time. But should your boss pay for the time for you to become more highly skilled, so you can ask for a higher salary and he has to replace your current role, or are you doing that primarily for your benefit so should you take responsibility for it? To be a software engineer specialising in testing implies general skills such as those above, an understanding of the SDLC and Agile as well as specific testing expertise. Can you provide general value in software development as well as specific value in testing that others can't? Look at your self-image: Tester. Now imagine you meet a medical professional at the top of their field. How might they describe themselves? As a people fixer? More likely he might say he is an Ophthalmologist and if asked to explain say that he is a Doctor who specialises in eyes. So are you a software tester or are you a Software Engineer or Software Developer with specialist skills and qualifications in Testing? Or perhaps you are a software quality specialist? It may sound silly but how you think of yourself is the most important predictor of what you achieve and how others view you.
Part 2 Key principles from other areas to apply

Specialists like to look broadly and think freely to get new ideas and insights and inspiration. Fresh perspectives give useful ideas, keep work interesting and broaden skills. As an exercise in this develop the habit of being contrary and questioning beliefs. Consider the opposites of what you currently think e.g. Do you need bug tracking software, management software, release cycles, test scripts? You can start to collect your own ideas but below are some to get you started:
Lessons for testers from Software Engineerings other disciplines

Be familiar with at least one relevant body of knowledge preferably a broad one e.g. Software Engineering, and a specific one e.g. ISTQB. Stay up to date. Read books! Consider certification e.g. ISTQB or AST or a more general one. As an automator a developer certification e.g. Java might be more useful than a testing course. Consider values PMI has ethical standards and that is one thing that distinguishes professionals and specialists from cowboys. Consider joining a professional group e.g. Computer Society and Test Professionals Network. Be part of a group where you are exposed to ideas and can learn from peers. There are many suitable online forums. See Lee Copelands Forgettings talk from STANZ 2006 which covered much of this.
Lessons for testers from outside Software Engineering

A couple of principles from Just in Time theory: Preventative maintenance (Application: Run automated tests every night even if no build so that any problems with the environment for any reason are resolved before the pressure is on and without confusing test results) Reduce Setup Times (Applications: Automate deployments, smoke test) And from Total quality control: Automatic inspection (Applications: Automated execution but also automated source inspection for errors or excessive complexity or coverage) Quality measures (Application: How do we know testing worked? How do we know quality is improving?) Fail-safe methods (Applications: System should respond to even unexpected errors in a way that is likely to avoid catastrophe. System should prevent errors where possible e.g. Provide list instead of free form, limit entry length.) Various management theories mostly from a manufacturing context that focus on constraints: They talk about how the only process worth optimising is the one weak link in the chain that constrains your delivery (Application: What is the bottleneck? Is it debugging? If so find bugs earlier. Is it test environment instability? If so make that number one priority to
improve.) It is very likely that the constraint(s) are policy in nature and not physical. (Application: Does policy of separating test and coding staff improve outcomes of does it slow down cycle time and reduce quality with only theoretical benefits?) Anything that reduces smoothness of production causes waste. (Application: Why do we have the release cycle we have? Does it cause waste or resource? If regression is automated could we release more often to reduce lumpiness? Daily?) Continuous improvement and Quality Circles: If we improve by 1% per week we will be working twice as efficiently within 18 months (due to compounding). Could you do that? The best people to improve processes are the people doing them, not the managers.
All about Agile

http://en.wikipedia.org/wiki/PM_Declaration_of_Interdependence says "We ... Increase return on investment by -- making continuous flow of value our focus. Deliver reliable results by -- engaging customers in frequent interactions and shared ownership. Expect uncertainty and manage for it through -- iterations, anticipation and adaptation. Unleash creativity and innovation by -- recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference. Boost performance through -- group accountability for results and shared responsibility for team effectiveness. Improve effectiveness and reliability through -- situationally specific strategies, processes and practices." For testing the above might suggest: Try to ensure you are actually doing testing each day manual or automated - and minimising documentation. Don't wait for releases. Document your tests in such a way that it improves communication with customers in a useful way. The most useful way is if they find problems before development. Accept uncertainty. This will make you very popular. Doesn't mean you do other's jobs for them, just that you don't assume they are doing things wrong. Creativity and innovation help us avoid just looking for the same bugs over and over. Let automation do that, not people! Innovate in all areas. It doesn't matter if most things you try fail as long as you manage the risks appropriately. Group accountability: Don't let yourself be the police. Instead work with the team and help them.
STANZ 2008, Wellington, New Zealand Page 9 of 24
Successfully Automating Functional and Regression Testing with Open Source Tools Detailed Notes John Lockhart - john@webtest.co.nz
Use situationally specific strategies: Work out your priorities and find the best ways in your situation to tackle them.
Agile (evolved from XP in mid-90's and core principles in Agile Manifesto): Short iterations with full cycle typically from 2 to 4 weeks at full release quality Co-location, teams of 5 to 9, cross-functional and self-organising Scrum: Daily stand-up meetings. Pigs and chickens. Customer rep in team and prioritise per iteration (including do they want a document or a working screen!). Time boxing: Sacrifice scope not schedule, budget or quality. Test early and often (TDD and automated unit tests etc). Only code to make tests pass then stop. No development for hypothetical requirements, but do architect the required code to reduce smells or anti-patterns. Core metric is working software (stories). Late changes in requirements are welcome. Motivated expert trusted staff. Attention to technical excellence and good design and simplicity. Refactoring to reduce technical debt, sometimes as separate iteration. Implications for testing: If you are a team member you need to understand software development and be technically competent to earn respect. Understand the role of unit testing and what JUnit is. Take responsibility instead of asking developers each time to give you extracts from logs, ask them how to access them and learn to extract the key data yourself. Also understand the various environments and configuration systems. Make sure your test cases can be validated with the customer rep and the developers. Use test cases that can then be executed directly as automated tests to reduce cycle time. Automate all regression tests that you can, including load and performance tests, to reduce cycle time. Don't grizzle when things change understand the paradigm.
Don't hesitate to ask the programmers to enhance testability of the code. Understand good test design, smells or anti-patterns, refactoring and technical debt. Understand load and performance (sometimes called SVP) testing as well as functional and exploratory testing.
Part 3 Examples
Example 1 Small successful development team from the 90's
The only job I've ever had where neither management nor customers complained about the cost of testing or the standard of quality. Product was an application used by 1500 dairy farmers to monitor every aspect of their herds. Reasons the testing was so successful included: The business expert (a vet and researcher) was intimately involved every day with the team, including exploratory testing and real use of the application. This is very similar to current agile practice. Quality was recognised as critical due to the high cost of bug fix releases on CD to thousands of users. This is also now a standard assumption of agile. One full time staff member fully allocated to automated regression hence able to keep up with current cycle. The was done using Rational Robot so is a good counter-example to my general belief that open source tools are best, but: It was his only priority so he could understand all the nuances of the tool and follow up any issues quickly with the vendor and user community, becoming a real expert in the test tool. It was a Windows application and Robot worked best in that space i.e. The tool suited the job. Both the application being tested and the testing tool used variants of Visual Basic so it was a good match to the in-house skill-set. The architecture was simple in that just one application was being tested and it was basically a standalone Windows application. Suitable open source tools were not available to my knowledge in the 1990s.
Example 2 Internet booking application with development team of 20

Management only funded two testers and they were so busy that we could not get even one of them to focus on automation full time. The culture was one of crisis management. The testers had no technical or development skills. Result was poor test architecture and maintainability. Manual testing was poorly defined and there were no standard manual regression tests so automation required definition of what would be tested, not just automation of existing tests. The complex multi-tier distributed application with a Windows client and web clients customised for individual customers let to many more technical challenges with the test automation and that complexity was not well managed. The entire Rational suite was used, and this time the cost for the large group including licences, servers, administration, upgrades (which took weeks) and issue resolution was impossible to justify from the minimal successes achieved. A low cost offshore subsidiary was created and allowed for manual testing using staff that knew the application well and had very low cost. The development team use primarily open source tools (PHP) and Borland products so the Rational technology was not a good match and it was a struggle to generate enthusiasm outside the test team.
Example 3 Corporate struggle

Here there was an excellent permanent manual testing team of approximately six staff, and good test processes, but none of the testers were technical except the team leader and there was no significant reuse of test assests and no standard regression tests hence no quick win targets for automation..
A standard big bang project for automation was run with RFI, RPF, demo's from competing vendors, proof of concept etc so the business case was for > $100 000 dollars, which meant there were beurocratic hurdles and that a lot of money was spent before any benefit could accrue. Rational was selected again, but only one person in NZ was qualified to maintain the custom code to link the test tool to the 'green screen' system. This was a complex architecture with Web, Windows, Database, Mainframe and Web Services and again the scope was not limited to reduce risk and produce measurable benefits in a reasonable timeframe. No staff were allocated full time to automation nor had it as their primary responsibility. The ccntractor who drove the project moved on to other things in the company then left and the test team leader while technically excellent and well qualified had many other conflicting responsibilities, primarliy to ensure the large manual testing workload was managed. Keyword (or DSL) automation approaches were poorly understood, if at all, by the staff and tool vendor and only functional decomposition was recommended as a model. That is an 80's approach to developing software and not suitable for a complex environment. Some years on automation is still not delivering good results in this case.
Example 4 Corporate success, so far anyway

This business also has diverse systems and a very complex architecure. Previous approaches to automation with large tools from mainstream vendors had ended up shelfware and left the company burnt and sceptical and afraid of test automation. Hence a low risk approach was selected that could fly under the radar and show that results could be delivered quickly and maintained . To do that we focussed just on the online channel which was bringing in almost half of their revenue - over $1 billion per year and had the highest visibility and testing cost. We also restricted the scope was to automating 80% of the existing documented manual regression scripts. This helped as a big cost in automation that is often not allowed for is the analysis and processing of the many defects, suspected defects or requirements confusion uncovered in the process. The remaining 20% were either not cost-effective to automate for various reasons, or were kept manual to ensure a human eye scanned all key screens for each release. That is of course necessary to pick any defects not specifically tested by the automated scripts, which must be very narrow in their validations or they become too brittle and hard to maintain. Even with these existing scripts as a basis there was a fair amount of clarification requried as they were quite inconsistent and out of date despite being used on each release! This is common with manual scripts and a big advantage of automated regression test scripts is that they can't be out of date because then they will not pass! Surprisingly we also found a number of defects that had slipped past the manual testing. Now (automated) scripts are always up to date (otherwise they fail!) while previously (manual) tests were notoriously poorly maintained as it was left till after the project then not done as people moved on to next urgent project. Canoo WebTest (webtest.canoo.com) was chosen. It is a tool written in java and using XML/Ant/Groovy and is specifically designed for testing web applications. It drives a pretend browser which is a java library called HTMLUnit, and while fast and powerful and supporting most AJAX libraries etc, does mean the business had to get over their reservation that the tests were not seeing exactly the environment the users had (in this case usually but not limited to Internet Explorer). Of course no test environment ever exactly matches any user environment! Because of limited scope and use of a simple free tool in mostly stable areas of the web application the business case had just the cost of one person to do the automation for four months at about $45,000 hence below governance radar and meaning there was inherently low risk. A competent and committed person with this as #1 priority and who would be around to guide later evolution of suite was allocated. Many principles of keyword architecture and agile approach followed e.g. attempting to avoid any code duplication and to encapsulate access to a page in one place. Result was that goal was achieved and a team of 1 test coordinator and 4 testers per release was replaced with one coordinator/automator and one tester for same period of time. This also enabled a move to a dedicated tester whereas when needed large team had to pull from business, and this led to increased efficiency. This group is now sold on the benefits of using professional testers, as well as on automation. The cost of maintaining scripts was covered by savings on management of staff and of manual scripts. Use of java-based tool in java team allowed us to leverage off existing expertise, though this was not really required in initial phase. Simple free tool meant negligible tool admin or maintenance cost. Now they are looking to add a different toolset (Fitnesse + WebDriver) to:

Give better use of the keyword-driven and executable requirements models Be more flexible by allowing driving the application stack at any layer rather than primarily at
the GUI.
Support a variety of real and simulated browsers through open source libraries such as WebDriver. Leverage off the technical skills of the in-house java development team (who were not available when the testing was initially begun).
Another interesting lesson from this organisation was that when keyword-style automation was first suggested, the java developers decided they would take that on themselves, and attempted multiple approaches using java testing tools and also writing their own. After more than 6 months of this their manager realised it was not succeeding. There are two main reasons for this: The developers are not expert in application level testing (as against unit testing) and they do not have it as their core priority. I have seen this pattern in other companies too, and it is perhaps the one risk of the agile approach that is not present with the big tool, big vendor traditional approach.
Part 4 Automating
The two main causes of failure and keys to success
We can see from the examples above, and I think any person with enough experience is likely to agree, that the biggest risk to test automation and most common cause of failure is the issue of maintenance of the test suite as it becomes complex. There is a technical and a business reason for this. The business reason is failure to obtain or maintain the right expertise, or to have the right people but without automation as their top priority. If it is not the top priority then it will sooner or later for some release be omitted due to the pressure to deliver on time. Unfortunately the next release is likely to have the same pressure, and a backlog of work to bring the automation up to date, and so is more likely to omit the automation. Gradually the asset that was developed degrades and becomes worthless and automation is seen to have failed. The technical reason is related to architecture, which we discuss further below. Without a sound architecture, the complexity of the suite as it evolves and grows and the cumulative changes to the application being tested as it also evolves and grows more complex, get to a point where maintenance is such a problem that it can not be done in the time between releases. This leads to the same outcome as above. If we understand that we are developing a complex software application, in other words that a complex test suite (not the tool used to build it but the automated tests themselves) is a software application in itself all this becomes obvious. No one would expect to be able to develop successfully using spare time efforts by people with poor skills and no attempt to apply sound architecture or programming principles.
Principles and Architecture

Agile principles for automation revolve around simplicity and maintainability: 1 2 3 4 5 6 7 1 2 Just enough: Set an objective and do just what you need to in order to achieve it. Don't angst over anything else. Horses for Courses: Choose the simplest tool that can achieve your goals in your context. Quick and dirty if don't want reuse, otherwise architect. Architect means use a layered approach to manage complexity and provide maintainability, which we will discuss below. Use tools that leverages off the skills of yourself and team e.g. VB/.Net/Java/RegEx/XPath for scripting and ideally to allow modification of source code for tools. Don't get involved with vendors or licences or closed source unless strong reason to do so. Clarify the layers you want to test and ensure toolset supports them. Minimise documentation and maximise its use using the executable requirements model. Separate concerns in layers Avoid duplication and redundancy
The keys to any sound architecture - test or otherwise - are to:
The first of these is the essence of architecture and the second follows on from it and is the essence of development. Let's look at them in turn. Layered architectures are used for networking e.g. Web sites use HTTP which defines the web pages and sits on top of TCP which sits on top of IP etc; and of course for applications we have Ntier architectures e.g. For a web application might have client/browser, web server, application server, database and that is a relatively simple example.
For testing we want the top layer to express the tests in business terminology. This means that tests should be expressed in the language of requirements, using the business/user terminology in a structured but easily readable way. It also means they should express the tests at a level above and independent of the test technology, test level (e.g. GUI or database) and of the GUI structure such as buttons and pages. For example Add a new user with name of 'John Smith' and check that they appear in the user list. is good. Insert a user record in the database... is bad as is Enter 'John' in the first name field then click..... A common example of this model is Keyword (or Action word) driven testing where we define high level actions like Add user or Search for account which may take parameters and can be reused in multiple test not only with different data but also with the keywords mixed and matched in different orders to create flow style tests that can express use case scenarios or stories. Associated to this is the concept of Domain Specific Languages (DSLs). This is the programming level equivalent to keywords, where methods or procedures are defined in the programming language, which map to these business level actions. This may be actually used to code the application, or may be separate to the application code, and used just to wrap the application in such a way that the test keywords can drive the application, and validate or reuse data from the application responses. In the latter case these are sometimes (e.g. With FIT/Fitnesse) called fixtures. Fixtures are the second level of the architecture and provide the glue from the business level keywords which might be expressed in a spreadsheet or a wiki, to the application at some suitable level of its stack. For example the fixture might drive the application through the GUI or might drive it at the web service or database or code object layer or even at a command line layer. These fixtures might in turn use another layer, usually a generic one, to connect to the application at the appropriate level. For a web GUI this might be a library or tool such as Selenium or WebDriver that knows how to enter data into web forms, click links, and extract data from web pages. In the case of database or web service fixtures it might be a general java or .Net library class using JDBC or ADO to hit a database or using Axis or some other classes to access a web service. Finally we have the application itself, which can be considered part of the test stack for two reasons: you can't run tests without it; and ideally the developers will optimise it for testability through hooks or naming conventions for example making sure key elements in a web page that the automation will need to access have unique and meaningful Ids. Each of these layers should have the ability to log and report appropriately. Ideally the first business layer will show results directly in the context of the tests. They may colour them to show success or failure and insert actual results beside the expected ones. The other layers should support modes of logging for various purposes, including debugging, ideally without having to write specific code for this, and allowing the level of detail of logging to be specified for the run. This is best accomplished using a library such as log4j. These layers should also provide logical tools to identify key elements in the application such as for a web page: form elements, buttons and links, and content in the page. RegEx and XPath would be two key ones to expect, as you do not want to have to learn techniques specific to the tool, that are neither available nor useful outside the context of that tool Versioning and source control must be provided at all these levels. It is best to use the most standard way of versioning things at the level in question so if using a wiki at the top level, use the build in wiki versioning, but with a note which version of the application the test was validated against. If using a spreadsheet then use whatever configuration control is used for office type documents in your organisation, and for fixtures and other code-level artefacts use the same source control that the developers use for their code, or if you don't have in house developers but are testing 3 rd party code, use a standard tool that suits code written in the language you are writing your tests in. In summary we have four levels:
1 2 3
The test is defined in sequences of keywords with data as parameters using terminology that is meaningful to the business and independent of the technology. The tests are then mapped to code modules that link each keyword to the appropriate part of the application. This code may use other code or tools to map generically to the particular layer of the application in questions such as running a stored procedure in a database. This may not be necessary if the application is coded using a model that maps well to the business. The application itself should be at least set up and ideally optimised to support the testing.
Each layer should be as independent as possible so that: Optimal tools can be swapped in and out at each level according to the desired purpose or the suitability or availability of tools, without disrupting the entire test architecture. This minimises tool and vendor lock-in and allows multiple tools to be used depending on the type of testing most suited to particular tests. People with the most appropriate skills can work at each level. The person writing the tests themselves may be a tester, a business analyst, or a business customer or user of the system. The person writing the fixtures may be one of the developers using the same language and tools they are already skilled in, and just having the job of implementing the agreed keywords, or could be a tester with basic technical skills, or could be outsourced. The keywords are core to this and should be agreed between all interested parties. Changes to the application or tests should be able to be made with minimal impact on the other layers. A change to the structure of a web page should only require changes to the fixture that drives that page. A change to a test cases should require either no changes to other layers, or the addition or modification of usually a very few keywords and hence fixtures, to support functionality that had not previously been set up for testing. A change to the tool used to drive web pages for selenium to WebDriver would only require changes to the fixture layer and of course the swapping of these tools in the third layer, but not to the test scripts themselves. Changing a fixture to support an additional parameter can easily be done in such a way that none of the existing tests break. Avoiding duplication and redundancy is a key to quality code development and applies just as much to testing. This principle has been around almost as long as coding, but has continually evolved. We have already talked about how each fixture should drive and be able to parse the parts of the application that are used for a specific keyword. Generally only one keyword will access any given part of the application for example only the Log in keyword will access the username, password, forgot password parts of a web system, even if they appear on multiple pages. Often variations on a given keyword are required, for example some taking all the detailed parameters and others being high level and making some default assumptions for easy and readability of tests where the specific details of data used are not important. There may be specific tests that cover all aspects of the payment step in an online flow, but outside of these tests many other end to end scenario tests will require a payment to be made but not care about the details. So we may have a Pay keyword with parameters for payment type, card type, expiry date etc but may want to also create another keyword, either also called Pay or perhaps Make default payment that just uses default values. This principle means that the simple one would just call the detailed one internally, passing in default values, so behind the scenes they are variations on one keyword and don't involve duplicating code. The team can decide on a rule of thumb such as whenever you write more than 5 lines of code that repeat code from elsewhere, it should be abstracted into a method that can be called from each place, but in reality it is not this simple and requires judgement. If there are say 3 lines but repeated in 50 places, then that is more important to factor out than 5 lines used in two places. Similarly if different parameters are used it is slightly more work to factor out so maybe 7 lines is appropriate, but if no parameters it is easy to do, so maybe 3 lines. The key is to have time allocated and an agreed priority
to continually improve the quality of the code as you work, and never to sacrifice that for slightly quicker development.
Part 5 Overview of Fitnesse and associated tools

The Fitnesse Architecture
Fitnesse Features

Fitnesse is a wiki test management and execution framework built on top of FIT and extended by FitLibrary which are all free java open source applications. It allows tests to be written in business language using simple tables in a wiki and supports most functional testing patterns such as tables to do data-driven style testing of business rules and keyword style tests to exercise end to end scenarios. It automatically executes against fixtures written in java, C# or various other languages thus allowing leverage of existing skills and libraries and reporting back in real time in the same wiki tables that the tests are defined in. It separates the non-technical process of test case writing which can and should be done in conjunction with the business as part or requirement definition ahead of coding, from the technical process of creating code fixtures that connect each action to the application being tested which should be done in conjunction with development. As a wiki it contains built in version control, security, distributed access and ability to mix annotations among the test cases making then true executable requirements .\ The wiki structure is enhanced by Fitnesse with features to support test suite management and basic refactoring. Fitnesse supports the use of variables so that data definitions can be centralised to avoid duplication.
New Fitnesse Developments and Add-Ons

The new Macro feature due out this month allows actions to be defined in terms of a series of lower level actions; passing parameters in the same way as through they were fixture code. New SpiderFixture feature due out this month provides built in actions for all basic web
functionality e.g. type into a text field, click something, validate some page content. It drives Firefox or IE or HTMLUnit (which is a headless i.e. pretend java browser) and displays test results directly in the wiki. New XMLUnit support to allow easy calling of web services and validation of responses using XPath etc. The one notable omission in Fitnesse is the ability to pass back values from the application in one test step, and reuse them in later steps. I expect this to be added soon but until it is that logic can easily be handled at the fixture level but does require basic coding skills. Database access is provided by a database fixture.
Part 6: Additional useful skills and tools of the trade for testers
There are core skill areas you must cover to be a testing professional and tools are relevant to some of these. Here is a summary: 1 Test planning i.e. given a testing situation have a feel for the types of testing that might be required and the time, tools and resources required. Specific tools are not required at this level, but make sure you understand the range of testing e.g. From the ISTQB Foundation Syllabus (http://www.istqb.org/syllabi.htm) and have access to the testing heuristics sheet (http://testobsessed.com/wordpress/wpcontent/uploads/2007/02/testheuristicscheatsheetv1.pdf) and good example test plans. Test analysis i.e. given a testing challenge and a specific approach, be able to define and prioritise the specific tests you recommend. Again tools are not so relevant but you need to get these skills one way or another. If you are afraid of maths and logic work through it and get past that you can't be afraid of core parts of your job and be successful or a useful resource! Manual test execution: You must be able to execute tests; record issues well, follow them up appropriately, report on progress, and carry out exploratory testing. If not good at any of this then get good! Performance testing: This is a specialised sub-field but many of the same principles apply and if you want to increase your value and broaden your opportunities a good start is to read the free book by Scott Barber (http://www.codeplex.com/PerfTestingGuide/Release/ProjectReleases.aspx? ReleaseId=6690). Tools are essential here and depend on the architecture you are testing. A great general purpose tool that you can teach yourself to use is JMeter. Make sure you know how to use this to performance and load test web services and you can easily extend that knowledge to other contexts. Automated functional testing: we cover this in detail elsewhere in this talk but you need to understand the underlying principles, at least one good framework like Fitnesse, and some way to connect to the architectures you are likely to have to test e.g. selenium/WebDriver for web, or equivalent tools to test driver web services, databases, windows GUIs, java GUIs, Flash/Flex, databases etc. You may wish to also download the trial version of one of the big 4 commercial tools and spend a month of your spare time getting a feel for one of them, but don't think that will impress on your CV as all the graduates from overseas universities say on their resumes that they have experience with the Mercury suite, but none of them understand how to use automation effectively.
Now we get down to more specific supporting skills that are equally essential and tools are even more important here. When working as a tester you need to know how to use the tools of the trade. You don't want to call yourself a builder then ask your customer to find you a drill, or find someone to hammer in a nail for you! What tools do you need? I would suggest you need the following utility skills with some suggestions of tools required: 1 Quick and dirty testing automation of repetitive tasks. For this you need a macro tool for the environment you usually work in e.g. Windows, and a macro tool for any key applications you use in that environment e.g. Text editors or Excel. You should be the person others can come to for this sort of thing and then you become more valuable. Examples are iMacros or better Selenium IDE for web actions, AutoIt for windows GUI actions (http://www.autoitscript.com/autoit3/), Groovy with Scriptom for Windows COM interface automation (http://groovy.codehaus.org/COM+Scripting). For quick and dirty testing of web applications I highly recommend the Selenium IDE Firefox plug-in. It installs in less than one minute, can be learnt in less than 10 minutes and gives free easy, powerful test record/playback with ability to handle almost all requirements and validations.
Data and log manipulation: You must be able to prepare large volumes of data for input to tests, and analyse large amounts of output data, and even convert one to the other. Again you should be the go-to person for any data manipulation and you need a good set of tools: You must be competent in SQL and able to work with a database of your choice e.g. MS Access or MySQL. Find a "For Dummies" book and add this to your training list. A scripting tool such as Ruby or Groovy or Visual Basic and be familiar with all the text manipulation commands particularly RegEx (regular expressions) which let you do text matching and replacing using very powerful and complex rules very easily. There are plenty of online references for RegEx and you will earn the respect of the developers. Finally it is worth getting reasonably competent with the data manipulation capabilities of Excel. XML transformation either to drive web services or to extract information from web pages or web service responses or to manipulate XML from one format to another or to display XML data in pretty HTML formats etc. You need two tools: XPath and XSLT. XPath lets you query an xml document in an analogous way to using SQL for a database. This means you can validate content or rules or extract any subset of information from xml. XSLT extends this to a framework to transform from one XML format to any other XML, HTML or text format using rules themselves specified in XML so that you don't have to do any coding. Many tools and applications support these e.g. Most browsers support XSLT and most open source testing tools support XPath (and Regular Expressions) as a way of identifying data in a response so you can either validate or extract that data to a variable. There are plenty of online tutorials if you Google these. The old-fashioned but still industry-standard automated functional testing tools: download a trial version for a 15 or 30 days and get a feel for at the commercial tools: IBM/Rational, HP/Mercury, Borland/Silk, Compuware, TestComplete. If one is used in your company then pick that one of course.
That's all! Remember - you don't have to do all this in one year. Doctors take a while to go through their training to the point where they are high-earning specialists so you have to be in for the long haul. Commit yourself to getting confident with one of the above each year. If you're a test manager rather than a tester: Use the guidelines above to develop your test team. Look for these skills in people you employ or have employed, or look for a willingness to develop them. Pick at least one person and preferably two in your team to focus on developing world class expertise in each of the areas above. There is one other skill that deserves a quick mention: Don't ever go back to people to ask the same question you have already asked because you didn't record the information. To be professional you must be organised and keep track of information that is relevant to your work well. Find a tool you like a wiki or a loose leaf notebook with an index or whatever I haven't found a perfect solution although wikis come close if you always have internet access. You must be able to instantly record important data and re-order it so it stays useful and ideally access it from anywhere. Again, you should be the guru that people come to regarding test environment access, configuration, troubleshooting etc, NOT the other way round! If you are a tester, think about how a test manager would react to seeing some or all of the above on your CV. Test managers get so sick of seeing resumes that look like they have all been copied and pasted from each other: Little meaningless bits of experience with the Mercury tools but no understanding at all; Lots of details of the applications that were tested but no details of the specific tools, techniques, successes, failures or lessons learnt. Anyone can write a CV by copying stuff from the ISTQB syllabus but most employers can see through that! If you can put the items we've discussed above on your CV they show real value you can add, and you don't need anyone's permission or help it's completely up to you.
Part 7 The path to success and happiness

We said we would map out a path to success and happiness in testing for managers and analysts by: Being clear on what it means to succeed as a test team not just getting through each release. Vision, knowledge and distinctions so can lead from above or below and re-frame testing. Developing the drive to follow through: fears, growth, contribution and fun. Creating change. Understanding and being able to use the tools of the trade. Creating change personally and within a team. Hopefully we've done this, but lets wrap up with some vital general principles. All of these are keys to life in general, but (you probably won't be surprised to hear) have specific applicability to the great science and art of software testing :

Develop passion and leadership and vision. Continually learn and grow in knowledge and skills. Seek continual incremental improvement in everything you do. Build assets architect and reuse. Support and work with the team. Dont be the police. Be flexible adaptive and proactive. Automate the boring bits!
As always and by way or review let's have some examples to demontrate this . To build a successful test team or to become a successful test professional we need to:

Provide leadership by being passionate about testing and up with the play thus providing leadership in the area. Otherwise find something else to do! Ensure each test cycle we go through should not only accomplish the short term goal of testing the application for this release, but should incrementally improve the test assets of the organisation, and the skill level of the team. Repetitive work that is deterministic i.e. you can describe exactly what to do, and what to check for, should as a rule be automated. Work with and support the team however you can from a quality perspective. Always be improving our skills and techniques and be proactive and adaptive. Avoid taking on a production police role and never take on the decision to go live or not. Encourage root cause analysis even if initially you have to do it yourselves. Make sure you understand all the core testing paradigms, especially the modern, agile ones, and be prepared to demonstrate how well they work. Make sure you understand how to use a little bit of automation at a time to demonstrate its usefulness, but always bearing in mind sound test architecture and be able to explain this.
If you do all the above the testing team will earn respect as a group with unique value that can not be replaced by pulling a some people from the business for a few weeks or offshoring or outsourcing or downsizing. More importantly you as an individual will find your work exciting, interesting and lucrative.

Agile Automation Detailed

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Agile Automation Detailed

Uploaded by

Copyright:

Available Formats

Successfully Automating Functional and Regression Testing with Open Source Tools Detailed Notes

Part 1: The profession

Other disciplines within software development

The broader perspective: Software Engineering

A Software Engineer Specialising in Testing

Part 2 Key principles from other areas to apply

Lessons for testers from Software Engineerings other disciplines

Lessons for testers from outside Software Engineering

All about Agile

Example 2 Internet booking application with development team of 20

Example 3 Corporate struggle

Example 4 Corporate success, so far anyway

Principles and Architecture

The keys to any sound architecture - test or otherwise - are to:

Part 5 Overview of Fitnesse and associated tools

New Fitnesse Developments and Add-Ons

Part 7 The path to success and happiness

You might also like