SIKULI

USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING

Tsung-Hsiang (Sean) Chang, Rob Miller MIT CSAIL

Collaborator: Tom Yeh

1
2010611 1

How to tell computers what to do?

2
2010611 2

case $rtval in 1) rm -f /tmp/dirip.$$ ; return ;; 255) rm -f /tmp/dirip.$$ ; return ;; esac mfile=`cat /tmp/dirip.$$` if [ -z $mfile ] then mfile=`pwd`/* else grep "*" /tmp/dirip.$$ if [ $? -eq 1 ] then ! mfile=$mfile/* fi fi for i in $mfile do if [ -f $i ] then ! echo "$i Delete?" >> /tmp/finallist.$$ fi! done

SHELL SCRIPT

3
2010611 3

4
2010611 4

GUI AUTOMATION

5
2010611 5

2 3 1
6
2010611 6

7
2010611 7

click

click

7
2010611 7

SIKULI SCRIPT
8
2010611 8

SIKULI TEST
9
2010611 9

OUTLINE
Problem Sikuli

- How to refer to a GUI widget?

Script - examples and demos - demo

Sikuli Test Test

By Demonstration of testing practices

Support

Conclusions
10
2010611 10

HOW TO REFER TO A GUI WIDGET?

11
2010611 11

HOW TO REFER TO A GUI WIDGET?

<AWTTestScript> Abbot <launch classpath="src/demo" args="[]" class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> <terminate/> by pre-programmed references </AWTTestScript>

11
2010611 11

HOW TO REFER TO A GUI WIDGET?

<AWTTestScript> Abbot <launch classpath="src/demo" args="[]" class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> <terminate/> by pre-programmed references </AWTTestScript>

Autoit

by absolute locations

11
2010611 11

Use Screenshots!

12
2010611 12

TEMPLATE MATCHING
click( )

13
2010611 13

TEMPLATE MATCHING
click( )

13
2010611 13

EXAMPLE 1: EMPTY TRASH CAN

2 3 1
14
2010611 14

EX2: FACEBOOK STALKER

15
2010611 15

EX2: FACEBOOK STALKER

15
2010611 15

16
2010611 16

RESTRICT SEARCH REGION

17
2010611 17

EX3: MUTE SPEAKERS

18
2010611 18

EX3: MUTE SPEAKERS

18
2010611 18

SPATIAL OPERATORS

19
2010611 19

SPATIAL OPERATORS

19
2010611 19

SPATIAL OPERATORS (2)

20
2010611 20

EX4: SET IP ADDRESS

21
2010611 21

VISUAL-EVENT-DRIVEN PROGRAMMING

22
2010611 22

VISUAL-EVENT-DRIVEN PROGRAMMING

22
2010611 22

SIKULI TEST

23
2010611 23

VISUAL ASSERTION
Verify

appearance

24
2010611 24

VISUAL ASSERTION
Verify

replacement

25
2010611 25

TEST BY DEMONSTRATION
detect changes click

26
2010611 26

Demo

27
2010611 27

GENERATE ACTION COMMANDS

mouse mouse down up

A A b b down up down up

28
2010611 28

GENERATE ACTION COMMANDS

low-level events
mouse mouse down up A A b b down up down up

28
2010611 28

GENERATE ACTION COMMANDS

low-level events
mouse mouse down up A A b b down up down up

high-level commands

click

type Ab

28
2010611 28

GENERATE ACTION COMMANDS

low-level events
mouse mouse down up A A b b down up down up

click

high-level commands

click

type Ab

combine with screenshots

28
2010611 28

GENERATE ASSERTIONS

click

29
2010611 29

GENERATE ASSERTIONS

click

1. nd differences / manually hint

29
2010611 29

GENERATE ASSERTIONS

click

1. nd differences / manually hint 2. group together

29
2010611 29

GENERATE ASSERTIONS

click

1. nd differences / manually hint 2. group together 3. combine with assertExist

29
2010611 29

GENERATE ASSERTIONS

click

1. nd differences / manually hint 2. group together 3. combine with assertExist

29
2010611 29

jUNIT INTEGRATION

30
2010611 30

TEST-DRIVEN DEVELOPMENT

31
2010611 31

READABILITY OF CODE

32
2010611 32

PLATFORM INDEPENDENCE
Works

on ANY GUI that can be displayed on Windows/Linux/Mac

Virtual

machines desktop

Remote Mobile

simulators: Android, iPhone

Web: Flash, HTML+Javascript

33
2010611 33

34
2010611 34

34
2010611 34

LIMITATIONS
Theme Variations
Visibility Constraints

35
2010611 35

DISCUSSION
Sikuli programs are written against the user interface instead of an API

UI: visible, familiar, always exists API: faster, probably more stable

36
2010611 36

FUTURE WORK

fast

and accurate OCR on screen robust against different skins

more

37
2010611 37

Thanks!

http://sikuli.csail.mit.edu or, just google sikuli

New Sikuli 0.10 released. Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
38
2010611 38