Pycon 2009: Functional Testing of Desktop Applications

Here is a link to Michael Foord talk at Pycon 2009 in Chicago about functional testing of desktop applications:

Functional Testing of Desktop Applications
Many advocates of unit testing believe that testing desktop (GUI) applications is so difficult as to not be worth it. Not only is it possible, but 'functional testing' (testing an application from the point of view of the user) is an important part of an agile development process.

I watched it yesterday and it is very good!

Testing an unknown application

From time to time you may need to test an unknown application. I have encounter it when I needed to test our installation program. The task was simple - install the application for automated smoke tests. In such cases I gladly return to Accessibility.

Let's pretend our testing application is not written in .NET and the only way how to explore it is through the accessibility (AccExplorer):



All we need to do is to access Accessibility objects from IronPython. There is a project called Managed Windows API that nicely wraps accessibility for .NET. It also wraps other Win32 API calls (mouse clicks etc.) but I stay with Win32API.dll because of my laziness :-).

Utilizing the Managed Windows API you can control the accessibility objects. Here is the snippet:

import clr
# Win32API provide access to Win32 API functions
clr.AddReference('Win32API')
from Win32API import Win32API
# ManagedWinapi provide access to Accessibility objects
clr.AddReference('ManagedWinapi')
import ManagedWinapi.Accessibility as ma
import ManagedWinapi.Windows as mw

def GetGUIATWindow():
   """ Return GUIAT_PoC window accessibility object. """
   def callback(aWindow):
       return aWindow.Title == 'GUIAT - Proof of Concept'
   guiat_window = mw.SystemWindow.FilterToplevelWindows(callback)
   # assume guiat_window is list with just one object
   return ma.SystemAccessibleObject.FromWindow(guiat_window[0],
       ma.AccessibleObjectID.OBJID_WINDOW)

guiat_acc = GetGUIATWindow()
# position of the 'New listbox item' text box
pos = guiat_acc.Children[3].Children[1].Location
# focus the text box
Win32API.MouseClick(pos.X + pos.Width/2, pos.Y + pos.Height/2)
Win32API.SendString('Accessibility test')
# position of the 'Add Item' button
pos = guiat_acc.Children[3].Children[2].Location
# click the button
Win32API.MouseClick(pos.X + pos.Width/2, pos.Y + pos.Height/2)

To run it, download managedwinapi-0.3.zip and extract the ManagedWinapi.dll to the same folder as the source code.

When you run the code, it enters Accessibility test text into New listbox item text box and clicks Add Item button. The controls are always on the same position within the Children enumeration so we can use direct referencing: e.g. guiat_acc.Children[3].Children[2] is the button accessibility object.

Enhancing this example is up to you - I have created several functions that take care about entering the text into text boxes, selecting buttons or a checking check boxes. That's all I need to create a script that installs our application.

Building the framework (3)

After a while, it's time to continue building the GUI Automated Testing framework. Today, I focus on a list box. You can download the source as usual.

I assume standard windows list box component where only unique items are stored and only one item can be selected. These two conditions are set only because of simplicity of the example. If you don't like them, enhance the example by yourself as your homework :-)

The list box component is again a subclass of BaseComponent. It is quite simple - it has one method and two properties:

Select(self, aItem)
The method selects aItem in the list box.

def Select(self, aItem):
  """ select an item from list box
  @aItem - string with item to select
  """
  if aItem not in self.items:
    raise Exception("Item '%s' not in list box" % aItem)
  self.guiat.Activate()
  self._CheckVisibility()
  pos = self.location
  # click on the first item to focus the list box
  Win32API.MouseClick(pos[0] + (pos[2]/2), pos[1] + 3)
  # send Home to be sure we are on the first item
  # (we could be scrolled down a little)
  Win32API.SendKey(Win32API.VK_HOME)
  # simulate pressing down arrow until we find the item
  # we should find it because it is among self.items
  while self.value != aItem:
    Win32API.SendKey(Win32API.VK_DOWN)

First, it checks whether aItem is in the list box and raises an exception if not. Then it clicks onto the first visible list box item, simulates pressing Home key to focus the first item and repeatedly press Down key until the value equals aItem.
This approach has the advantage we don't need to care about scroll box. The basic windows list box does not provide nice properties or methods to return its state. We would have to dive into Win32 API to find it. For example, the DevExpress ListBoxControl has method GetViewInfo that returns information about internal list box state.

This brings us to the important note:

We do not test the list box component. We test the application.

One way of selecting item in a list box is enough. Of course, we must be aware of its limitations and side effects. Selecting an item with our GUIAT component fires one OnClick and several OnChange events.

items
The property contains read only list of all items in the list box.

value
The property contains the selected item in the list box. User can assign a string to it to select the string in the list box.

Incorporating the new GUIAT ListBox class into the framework is easy - just add it into the RecognizableComponents dictionary of the Form class.

This is the last post about controlling components. You know the idea so developing a new GUIAT class controlling your component should be easy.

Next time, I show how to control application that is already started.

GUI Testing at Resolver Systems

I'm currently quite busy at work and with my other activities (like photos). That's why instead of writing another chapter of the GUI Automated Testing framework, I point you to the great article of Michael Foord:

Functional Testing of GUI Applications

Michael describes his experiences with GUI automated testing in Resolver Systems.

They use quite similar approach to run the tested application. It is run in another thread but the thread invoking is different. I am not a specialist in threading but I will try to find out more about which way is better or whether there are any catches[1].

They also do not use win32api for simulating user's input. They usually call directly the .NET method responsible for generating the requested event (e.g. PerformClick). That can be dangerous because your test can pass even when user is unable to click the button - the button could be hidden behind another component.

[1] My current problem can be caused by threading. When I run our application from automated testing tool, two buttons overlap. When I run it normally, it is OK. To be it stranger, it occurs only on my PC. I have no idea why...

Update 10/30/2008: I have not found any problem with my way of starting the tested application. I asked my co-workers, searched for articles about threading and found both ways should be similar. [1] is still mystery to me, though.

Building the framework (2)

Last time we have started building the GUI Automated Testing framework. Today we are going to enhance it. Download the source to be able to follow the text.

The first version implemented the BaseComponent class to represent windows components. We could not do anything but find out location of the component. It would be nice to be able write text into a text box or click a button by simple method, wouldn't it?

To do that, we need to know the type of each component. The type name of .NET components is stored in component.GetType().Name. Knowing the type, we create subclass of BaseComponent for each type. In the subclass, we provide nice methods and properties.

First, we enhance BaseComponent class. We add _CheckVisibility(self) method that checks whether the component is visible. If not, it raises an exception. We need it for checking if button is visible so we can click it or text box is visible to write it. And we add guiat property to be able to activate the tested application from a component by calling self.guiat.Activate().

Let's look on a button. The Button class is very simple:

from time import sleep
from Win32API import Win32API
from BaseComponent import BaseComponent

class Button(BaseComponent):
   """interface to the Button component"""

   def Click(self):
       """ perform left mouse click on the center of the button, no parameters"""
       self.guiat.Activate()
       self._CheckVisibility()
       pos = self.location
       Win32API.MouseClick(pos[0] + (pos[2]/2), pos[1] + (pos[3]/2))
       sleep(0.1)

The Click(self) method activates the tested application, checks whether the component is visible, finds out the position of the button, clicks into the middle of the button position, and finally waits a little bit. I do not like the waiting but it is here for safety. Windows may repainting some areas or do some other cool thinks, so it is better to give them some time for it.

The TextBox class provides three methods and one property:

• FocusEditor(self)
  The method essentially does the same as Button's Click method. It activates the tested application, checks whether the component is visible, and clicks into the middle of the TextBox position to focus the editor.
• Clear(self)
  The method focuses the text box, moves cursor to the start position simulating pressing Home key, and simulates pressing Delete key until the text in text box is empty. If the text is not empty then, it raises exception.
  
• Write(self, aString)
  The method clears the text box and uses Win32API method SendString to simulate typing aString. Then checks whether the text box contains the aString value and raises exception if not.
  
• value
  The property contains the actual value of the text box. When user assigns a string to it, the string is written to the text box.

Now look on the creating instances of GUIAT components in the Form class. We have simply added dictionary with known component types (RecognizableComponents). When we go through all components on the form in the _AnalyzeStructure method, we first check whether we know the component type or not. If so, we create instance of the appropriate component type class (e.g. Button or TextBox). If not, we create instance of the BaseComponent class.

Let's try a small example:

>>> import GUIAT
>>> g = GUIAT.GUIAT()
>>> g.Run()
Starting GUIAT...
"frmGUIAT" (frmGUIAT)
  "btnAddItem" (Button)
  "lblNewItem" (Label)
  "txtNewItem" (TextBox)
  "lbxItems" (ListBox) - UNKNOWN
  "btnQuit" (Button)
Starting GUIAT done.
>>> txt = g.mainForm._GetComponentByName('txtNewItem')
>>> btn = g.mainForm._GetComponentByName('btnAddItem')
>>> txt.value = 'Hello world!'
>>> btn.Click()

The result of the above small script is a new line in the list box. You see that the button, label, and text box are know component types. The label is treated as BaseComponent (see RecognizableComponents in Form.py). The list box is unknown component.

Today, we have shown how to create button and text box GUIAT classes that control respective .NET components. We can already script our small tested application!

Next time, we add list box class and logging.

Building the framework (1)

Today we start building the automated test framework. We have prepared tools in previous parts. See them to learn how to run tested application in a separate thread or how to simulate user's input.

The foundation of our test framework are IronPython classes that control Windows components. Simply said, everything in Windows is a component. So we create IronPython layer that allows controlling each Windows component easily. Then we create classes that will control forms with many components. And finally, we build test scripts and test suites for the whole application.

We start with just three simple classes. The first class, GUIAT, is the core class taking care of running, activating (focusing), and inspecting the tested application. The second class, BaseComponent, is ancestor of all component classes. The last one, Form, descendant of BaseComponent, is ancestor of all forms.

The BaseComponent class provides access to common properties of all components. There is just one in the first version:

• location - size and position of the component on the screen (not within the parent form)

The BaseComponent class has also several private variables and methods:

• _name - name of the component, not all components have this field filled
• _control - reference to the .NET component instance
• _guiat - reference to the main GUIAT object (see below)
• _guiatComponents - dictionary with all child GUIAT components
• _GetComponentByName - method that searches for component with given name

Note: The .NET component means instance of the Windows component in the tested application. The GUIAT component means instance of the IronPython class that controls the .NET component.

The Form class extends the BaseComponent class. Its purpose is to find and store all components on itself. It has one private method that does it:

• _AnalyzeStructure - method that searches and stores components on a form. It is recursively called for each member of the Controls collection. It prints out the name, type, and depth level of the found components. Then it creates BaseComponent instance and stores it into the _guiatComponents dictionary of the parent component.

The GUIAT class as the core class of the framework contains in the first version only one property and two methods:

• mainForm - GUIAT representation of the main form of the tested application (Form instance)
  
• Run - method that runs the tested application in separate thread and creates the Form instance of the main form
• Activate - method that activates (focuses) the tested application

Let's try a small example:

>>> import GUIAT
>>> g = GUIAT.GUIAT()
>>> g.Run()
Starting GUIAT...
"frmGUIAT" (frmGUIAT)
  "btnAddItem" (Button)
  "lblNewItem" (Label)
  "txtNewItem" (TextBox)
  "lbxItems" (ListBox)
  "btnQuit" (Button)
Starting GUIAT done.
>>> print g.mainForm._GetComponentByName('txtNewItem').location
(212, 177, 96, 16)
>>> g.Activate()
True

The above code runs the tested application, lists name and type of all components, then prints location of the text box, and finally activates the tested application.

To sum up the process again:

• Create instance of the GUIAT class - let's call it g
• g.Run() runs the tested application
• g.Run() creates instance of IronPython Form class for the main form of the tested application
• The Form instance searches during its initialization all components on the main form, prints information about them, and creates BaseComponent instance for each of them
• To find location of "txtNewItem" component, execute:
  g.mainForm._GetComponentByName("txtNewItem").location
• To activate tested application, execute:
  g.Activate()

Today I presented the core of my GUI automated testing framework (source). I'm going to extend it and make it more user friendly in next parts.

Simulate user's input

We have learned in the previous part how to explore the tested application and read its values. To test it, we also need set values to its fields.

The first idea may be to utilize the .NET objects. When we can read values from the tested application fields, we can also set them. Use the code snippet from the previous part to run the tested application and follow it by command

App.Controls[2].Text = 'GUIATtext'

The tested application is started and GUIATtext appears in the New listbox item text box.

Well, it does not appear as a proper testing to me. I'd rather simulate user's interaction with mouse and keyboard. Unfortunately, sending key strokes and mouse events is not job for .NET framework. We have to dive deep into Windows and use Win32 API to perform these tasks.

I have created a simple DLL to make Win32 API functions available for IronPython [1]. Moreover, I have added functions to simulate user's input. Here is the list of functions in the Win32API namespace with a short description:

GetForegroundWindow()

        return handle of the active window

SetForegroundWindow(handle)

        set the active window to the window with handle
        return boolean

ShowWindow(handle, state)

        set window with handle to state (minimalized, maximalized, ...)
        return boolean

GetWindowText(handle, title, capacity)

        return title of window with handle, title variable type is String of capacity size

MouseClick(X, Y)

        simulate click with left mouse button on position X, Y

SendKey(virtual_key)

        simulate pressing virtual_key

SendString(string)

        simulate typing a string

I will not go into details here. Anyone interested can explore the sources of the Win32API.dll.

Using these functions is straightforward in IronPython. Just download the Win32API.dll and look at the example below how to perform a mouse click on the specific position:

import clr
clr.AddReference('Win32API')
from Win32API import Win32API
Win32API.MouseClick(10, 10)

The code snippet moves mouse to the coordinates 10, 10 (that is to the top-left corner of the screen) and perform a click with the left mouse button. More useful examples will follow in next parts.

Finally, couple of warnings. The simulated mouse click is caught by the top most window on the position. And the simulated key strokes are sent to the active window. That means we have to pay attention to position and active state of the tested application.

In the next part, we start building the GUIAT testing framework.

[1] I do not know how to access Win32 API directly from IronPython.