Radical Test Portability

Ian Dees - texagon.blogspot.com

FOSCON 2007

Hi, I'm Ian Dees. I write software for a test equipment manufacturer in the Portland, Oregon area.
This talk isn't really related to work, but our jumping-off point is sort of inspired by something that
happened on the job once: I had the chance to evaluate some really crappy GUI testing toolkits.
 Where there is a stink of [ ],
there is a smell of being.

Antonin Artaud

But a funny thing about crap is that it can be fantastic fertilizer for growth.

That's not what Artaud meant by this quote, by the way. But that's okay -- he was insane anyway.
 Radical Test Portability

So, in keeping with FOSCON's "Really Radical Ruby" theme, let's talk about Radical Test Portability.

It shouldn't be a revolutionary concept that tests can be portable. It should be old news.
But you wouldn't know that from reading the websites of the big-time test toolkit vendors. They
say, "No programming." What that really means is, "No programming yet." These fire-and-forget
tools typically generate a pile of code that you then have to go and customize.
 MoveMouse(125,163);
Delay(0.65);
LeftButtonDown();
Delay(0.074);
LeftButtonUp();
GetWindowText(hCtrl, buffer, bufferSize);
if (0 != lstrcmp(buffer, L"Some text"))
LOG_FAILURE("Text didn't match\n");
Delay(0.687);
MoveMouse(204,78);

//... ad nauseam

And as often as not, you get a mess like this. Good luck trying to figure out where to insert your
pass/fail tests, or maintaining this zoo a month from now, when the GUI designer moves a button.
 def press_button(caption)
send_msg 'PressButton ' + caption
end

press_button 'OK'

If the vendors would just implement clean protocols and document them well, you could write your
test script in your choice of language instead of theirs.

In this hypothetical case, it just so happens that the Ruby function name looks like the message we
want to send. So the code can get even simpler....
 def_msg :press_button, [:caption]

press_button 'OK'

...by using Ruby's metaprogramming capabilities.

And when the language gets clear, a funny thing starts to happen.
 conversation

The tests become the centerpiece of a conversation between designers, developers, and testers.

And in that spirit, I'd like to show you some sample code.
 require 'spec_helper'

describe 'A text editor' do

it_should_behave_like 'a document-based app'

it 'supports cut and paste' do

@editor.text = 'chunky'

@editor.select_all
@editor.cut
@editor.text.should == ''

2.times {@editor.paste}
@editor.text.should == 'chunkychunky'
end
end

Imagine that we're discussing a text editor's cut/paste feature. Using the RSpec test description
language built on Ruby, we come up with something like this. The heart of the test is the third line
from the end, where we express the intended behavior with "should."
 require 'spec_helper'

describe 'A text editor' do

it_should_behave_like 'a document-based app'

it 'can undo the most recent edit' do

@editor.text = 'bacon'

@editor.text = ''
@editor.undo

@editor.text.should == 'bacon'
end
end

Here's another example, testing the undo feature. See, you can show this stuff to anyone, no
matter how much Ruby code they've seen in their lives.

The setup code happens in that spec_helper file. We won't see the code for that -- it's on my blog.
All we need to know for now is that it creates a TextEditor object.
 require 'Win32API'

class TextEditor
@@keybd_event = Win32API.new 'user32',
'keybd_event', ['I', 'I', 'L', 'L'], 'V'

KEYEVENTF_KEYDOWN = 0x0000
KEYEVENTF_KEYUP = 0x0002
VK_BACK = 0x08

def keystroke(*keys)
keys.each do |k|
@@keybd_event.call k, 0, KEYEVENTF_KEYDOWN, 0
sleep 0.05
end

keys.reverse.each do |k|
@@keybd_event.call k, 0, KEYEVENTF_KEYUP, 0
sleep 0.05
end
end
end

And here's a small piece of the TextEditor object's implementation for Windows Notepad. Here,
we're teaching Ruby how to type a single character, possibly using modifier keys like Shift or Ctrl.
We use the Win32API library to call the keybd_event function -- we press the keys down, then
release them in reverse order.
 class TextEditor
def text=(string)
unless string =~ /^[a-z ]*$/
raise 'Lower case and spaces only, sorry'
end

select_all

keystroke VK_BACK

string.upcase.each_byte {|b| keystroke b}

end
end

But the top-level test script never calls keybd_event directly. It calls text= instead, which breaks
the message up into characters and types each one in turn. On Windows, a character is usually
different than a keystroke (the "A" key might type an upper-case "A" or a lower-case "a"). So we're
just going to deal with the easiest subset of messages where the character and keystroke are the
same.
And here's what it looks like when you run it. I promise that's Ruby doing the typing!

Now, on to portability. Since we didn't put any Windows-specific calls into our top-level test script,
we can port it to other platforms, just by supplying a new implementation of the TextEditor class.
 class TextEditor
def initialize
@window = JFrameOperator.new 'FauxPad'
@edit = JTextAreaOperator.new @window
@undo = JButtonOperator.new @window, 'Undo'
end

def text=(string)
@edit.set_text ''
@edit.type_text string
end

%w(text select_all cut paste).each do |m|

define_method(m) {@edit.send m}
end

def undo; @undo.do_click end

end
Here's a test for a trivial editor for the Java runtime called FauxPad. You can run our same Ruby test
for it using JRuby. I was going to show you just a piece of the TextEditor class for JRuby, but it
turns out the whole thing fits on one slide. The text= method, which took up an entire slide in
Windows, is just four lines here, thanks to the Jemmy GUI test library from NetBeans.
Now on to OS X. Of course there's AppleScript, but not every app exposes an AppleScript interface.
Fortunately, you can turn on a setting in your preferences that will let control any app through its
menus and buttons through the System Events interface.
tell application "System Events"
script.
tell process "TextEdit"
application("System Events").
keystroke "b"
process("TextEdit").
end tell
keystroke!("b")
end tell

We can send AppleScript to the OS X command line pretty easily, but building the script up first can
be cumbersome. All the commands except the last one start with "tell," and afterward there's an
entire chain of "end tell" lines.

But with a little Ruby magic, we can generate that long script from a little more terse starting point.
 class AppleScript
def initialize
@commands = []
end

def method_missing(name, *args)

arg = args[0]
arg = '"' + arg + '"' if String === arg

command = name.to_s.chomp('!').gsub('_', ' ')

@commands << "#{command} #{arg}"

return name.to_s.include?('!') ? run! : self

end
end

The technique involves chaining a bunch of method calls together, and keeping them around in an
array. When we encounter a method with a bang in it, we scoop up all those commands and pass
them off to OS X.
 class AppleScript
def to_s
inner = @commands[-1]
rest = @commands[0..-2]
rest.collect! {|c| "tell #{c}"}

rest.join("\n") +
"\n#{inner}" +
"\nend tell" * rest.length
end

def run!
clauses = to_s.collect do |line|
'-e "' + line.strip.gsub('"', '\"') + '"'
end.join ' '

`osascript #{clauses}`.chomp
end
end

The OS X command to run an arbitrary chunk of AppleScript is "osascript." It doesn't like long lines,
so we break long commands up into pieces.
 class TextEditor
def initialize
script.application("TextEdit").activate!
end

def text=(string)
select_all
delete

string.split(//).each do |c|
script.
application("System Events").
process("TextEdit").
keystroke!(c)
end
end
end

Now that we can control Mac programs, we can write the TextEditor class yet again for a new
application, Apple's TextEdit.

Here's that same text= method, written for simplicity and not speed. We're spawning an osascript
instance for every single keystroke -- definitely not something you'd want to do in a real project.
 portability across apps

So far, we've seen one test script that can test three different apps: Notepad, FauxPad, and TextEdit.
 portability across platforms

And that same test script runs on Windows, Mac, and other platforms.

So what's next?
 portability across languages

How about portability across languages?

Our top-level test is in RSpec, but the underlying TextEditor object is just as at home in RBehave,
another test language built with Ruby.
 Story "exercise basic editor functions",

%(As an enthusiast for editing documents

I want to be able to manipulate chunks of text
So that I can do my job more smoothly) do

# one or more Scenarios here...

end

While RSpec is geared toward small, self-contained examples, RBehave's basic unit of testing is the
story. Stories are typically a little longer and more representative of how a real user would interact
with the program.

For now, we're just going to do a straight port from our RSpec test to RBehave.
 Scenario "a chunky document" do
Given "I start with the text", "chunky" do |text|
@editor = TextEditor.new
@editor.text = text
end

When "I cut once and paste twice" do

@editor.select_all
@editor.cut
2.times {@editor.paste}
end

Then "the document's text should be", "chunkychunky" do |text|

@editor.text.should == text
end
end

Each Story consists of multiple Scenarios, which are phrased as Given... When... Then.... It's a little
more verbose, but at least RBehave remembers its scenarios.

So I can reuse the phrase "I start with the text" or "the document's text should be" in other scenarios
without repeating the code that implements it.
 Scenario "a bacon document" do
Given "I start with the text", "bacon"

When "I cut and then undo the cut" do

@editor.select_all
@editor.cut
@editor.undo
end

Then "the document's text should be", "bacon"

end

Like this. The only new behavior we've had to define for this scenario is "I cut and then undo the
cut." The other two were already defined on the previous slide.
For more explorations like these, keep your eyes peeled for my book coming out in early 2008 from
the Pragmatic Programmers.
 thank you
Ian Dees - texagon.blogspot.com

All of the source code for this exercise is available on my blog. Thanks for your time.