Building the framework (3)

After a while, it's time to continue building the GUI Automated Testing framework. Today, I focus on a list box. You can download the source as usual.

I assume standard windows list box component where only unique items are stored and only one item can be selected. These two conditions are set only because of simplicity of the example. If you don't like them, enhance the example by yourself as your homework :-)

The list box component is again a subclass of BaseComponent. It is quite simple - it has one method and two properties:

Select(self, aItem)

The method selects aItem in the list box.

def Select(self, aItem):
  """ select an item from list box
  @aItem - string with item to select
  """
  if aItem not in self.items:
    raise Exception("Item '%s' not in list box" % aItem)
  self.guiat.Activate()
  self._CheckVisibility()
  pos = self.location
  # click on the first item to focus the list box
  Win32API.MouseClick(pos[0] + (pos[2]/2), pos[1] + 3)
  # send Home to be sure we are on the first item
  # (we could be scrolled down a little)
  Win32API.SendKey(Win32API.VK_HOME)
  # simulate pressing down arrow until we find the item
  # we should find it because it is among self.items
  while self.value != aItem:
    Win32API.SendKey(Win32API.VK_DOWN)

First, it checks whether aItem is in the list box and raises an exception if not. Then it clicks onto the first visible list box item, simulates pressing Home key to focus the first item and repeatedly press Down key until the value equals aItem.

This approach has the advantage we don't need to care about scroll box. The basic windows list box does not provide nice properties or methods to return its state. We would have to dive into Win32 API to find it. For example, the DevExpress ListBoxControl has method GetViewInfo that returns information about internal list box state.

This brings us to the important note:

We do not test the list box component. We test the application.

One way of selecting item in a list box is enough. Of course, we must be aware of its limitations and side effects. Selecting an item with our GUIAT component fires one OnClick and several OnChange events.

items

The property contains read only list of all items in the list box.

value

The property contains the selected item in the list box. User can assign a string to it to select the string in the list box.

Incorporating the new GUIAT ListBox class into the framework is easy - just add it into the RecognizableComponents dictionary of the Form class.

This is the last post about controlling components. You know the idea so developing a new GUIAT class controlling your component should be easy.

Next time, I show how to control application that is already started.

GUI Testing at Resolver Systems

I'm currently quite busy at work and with my other activities (like photos). That's why instead of writing another chapter of the GUI Automated Testing framework, I point you to the great article of Michael Foord:

Functional Testing of GUI Applications

Michael describes his experiences with GUI automated testing in Resolver Systems.

They use quite similar approach to run the tested application. It is run in another thread but the thread invoking is different. I am not a specialist in threading but I will try to find out more about which way is better or whether there are any catches[1].

They also do not use win32api for simulating user's input. They usually call directly the .NET method responsible for generating the requested event (e.g. PerformClick). That can be dangerous because your test can pass even when user is unable to click the button - the button could be hidden behind another component.

[1] My current problem can be caused by threading. When I run our application from automated testing tool, two buttons overlap. When I run it normally, it is OK. To be it stranger, it occurs only on my PC. I have no idea why...

Update 10/30/2008: I have not found any problem with my way of starting the tested application. I asked my co-workers, searched for articles about threading and found both ways should be similar. [1] is still mystery to me, though.

Update 8/20/2009: Mystery [1] solved - it was bug in our application :-)

Building the framework (2)

Last time we have started building the GUI Automated Testing framework. Today we are going to enhance it. Download the source to be able to follow the text.

The first version implemented the BaseComponent class to represent windows components. We could not do anything but find out location of the component. It would be nice to be able write text into a text box or click a button by simple method, wouldn't it?

To do that, we need to know the type of each component. The type name of .NET components is stored in component.GetType().Name. Knowing the type, we create subclass of BaseComponent for each type. In the subclass, we provide nice methods and properties.

First, we enhance BaseComponent class. We add _CheckVisibility(self) method that checks whether the component is visible. If not, it raises an exception. We need it for checking if button is visible so we can click it or text box is visible to write it. And we add guiat property to be able to activate the tested application from a component by calling self.guiat.Activate().

Let's look on a button. The Button class is very simple:

from time import sleep
from Win32API import Win32API
from BaseComponent import BaseComponent

class Button(BaseComponent):
   """interface to the Button component"""

   def Click(self):
       """ perform left mouse click on the center of the button, no parameters"""
       self.guiat.Activate()
       self._CheckVisibility()
       pos = self.location
       Win32API.MouseClick(pos[0] + (pos[2]/2), pos[1] + (pos[3]/2))
       sleep(0.1)

The Click(self) method activates the tested application, checks whether the component is visible, finds out the position of the button, clicks into the middle of the button position, and finally waits a little bit. I do not like the waiting but it is here for safety. Windows may repainting some areas or do some other cool thinks, so it is better to give them some time for it.

The TextBox class provides three methods and one property:

• FocusEditor(self)
  The method essentially does the same as Button's Click method. It activates the tested application, checks whether the component is visible, and clicks into the middle of the TextBox position to focus the editor.
• Clear(self)
  The method focuses the text box, moves cursor to the start position simulating pressing Home key, and simulates pressing Delete key until the text in text box is empty. If the text is not empty then, it raises exception.
  
• Write(self, aString)
  The method clears the text box and uses Win32API method SendString to simulate typing aString. Then checks whether the text box contains the aString value and raises exception if not.
  
• value
  The property contains the actual value of the text box. When user assigns a string to it, the string is written to the text box.

Now look on the creating instances of GUIAT components in the Form class. We have simply added dictionary with known component types (RecognizableComponents). When we go through all components on the form in the _AnalyzeStructure method, we first check whether we know the component type or not. If so, we create instance of the appropriate component type class (e.g. Button or TextBox). If not, we create instance of the BaseComponent class.

Let's try a small example:

>>> import GUIAT
>>> g = GUIAT.GUIAT()
>>> g.Run()
Starting GUIAT...
"frmGUIAT" (frmGUIAT)
  "btnAddItem" (Button)
  "lblNewItem" (Label)
  "txtNewItem" (TextBox)
  "lbxItems" (ListBox) - UNKNOWN
  "btnQuit" (Button)
Starting GUIAT done.
>>> txt = g.mainForm._GetComponentByName('txtNewItem')
>>> btn = g.mainForm._GetComponentByName('btnAddItem')
>>> txt.value = 'Hello world!'
>>> btn.Click()

The result of the above small script is a new line in the list box. You see that the button, label, and text box are know component types. The label is treated as BaseComponent (see RecognizableComponents in Form.py). The list box is unknown component.

Today, we have shown how to create button and text box GUIAT classes that control respective .NET components. We can already script our small tested application!

Next time, we add list box class and logging.

Building the framework (1)

Today we start building the automated test framework. We have prepared tools in previous parts. See them to learn how to run tested application in a separate thread or how to simulate user's input.

The foundation of our test framework are IronPython classes that control Windows components. Simply said, everything in Windows is a component. So we create IronPython layer that allows controlling each Windows component easily. Then we create classes that will control forms with many components. And finally, we build test scripts and test suites for the whole application.

We start with just three simple classes. The first class, GUIAT, is the core class taking care of running, activating (focusing), and inspecting the tested application. The second class, BaseComponent, is ancestor of all component classes. The last one, Form, descendant of BaseComponent, is ancestor of all forms.

The BaseComponent class provides access to common properties of all components. There is just one in the first version:

• location - size and position of the component on the screen (not within the parent form)

The BaseComponent class has also several private variables and methods:

• _name - name of the component, not all components have this field filled
• _control - reference to the .NET component instance
• _guiat - reference to the main GUIAT object (see below)
• _guiatComponents - dictionary with all child GUIAT components
• _GetComponentByName - method that searches for component with given name

Note: The .NET component means instance of the Windows component in the tested application. The GUIAT component means instance of the IronPython class that controls the .NET component.

The Form class extends the BaseComponent class. Its purpose is to find and store all components on itself. It has one private method that does it:

• _AnalyzeStructure - method that searches and stores components on a form. It is recursively called for each member of the Controls collection. It prints out the name, type, and depth level of the found components. Then it creates BaseComponent instance and stores it into the _guiatComponents dictionary of the parent component.

The GUIAT class as the core class of the framework contains in the first version only one property and two methods:

• mainForm - GUIAT representation of the main form of the tested application (Form instance)
  
• Run - method that runs the tested application in separate thread and creates the Form instance of the main form
• Activate - method that activates (focuses) the tested application

Let's try a small example:

>>> import GUIAT
>>> g = GUIAT.GUIAT()
>>> g.Run()
Starting GUIAT...
"frmGUIAT" (frmGUIAT)
  "btnAddItem" (Button)
  "lblNewItem" (Label)
  "txtNewItem" (TextBox)
  "lbxItems" (ListBox)
  "btnQuit" (Button)
Starting GUIAT done.
>>> print g.mainForm._GetComponentByName('txtNewItem').location
(212, 177, 96, 16)
>>> g.Activate()
True

The above code runs the tested application, lists name and type of all components, then prints location of the text box, and finally activates the tested application.

To sum up the process again:

• Create instance of the GUIAT class - let's call it g
• g.Run() runs the tested application
• g.Run() creates instance of IronPython Form class for the main form of the tested application
• The Form instance searches during its initialization all components on the main form, prints information about them, and creates BaseComponent instance for each of them
• To find location of "txtNewItem" component, execute:
  g.mainForm._GetComponentByName("txtNewItem").location
• To activate tested application, execute:
  g.Activate()

Today I presented the core of my GUI automated testing framework (source). I'm going to extend it and make it more user friendly in next parts.

Simulate user's input

We have learned in the previous part how to explore the tested application and read its values. To test it, we also need set values to its fields.

The first idea may be to utilize the .NET objects. When we can read values from the tested application fields, we can also set them. Use the code snippet from the previous part to run the tested application and follow it by command

App.Controls[2].Text = 'GUIATtext'

The tested application is started and GUIATtext appears in the New listbox item text box.

Well, it does not appear as a proper testing to me. I'd rather simulate user's interaction with mouse and keyboard. Unfortunately, sending key strokes and mouse events is not job for .NET framework. We have to dive deep into Windows and use Win32 API to perform these tasks.

I have created a simple DLL to make Win32 API functions available for IronPython [1]. Moreover, I have added functions to simulate user's input. Here is the list of functions in the Win32API namespace with a short description:

GetForegroundWindow()

        return handle of the active window

SetForegroundWindow(handle)

        set the active window to the window with handle         return boolean

ShowWindow(handle, state)

        set window with handle to state (minimalized, maximalized, ...)         return boolean

GetWindowText(handle, title, capacity)

        return title of window with handle, title variable type is String of capacity size

MouseClick(X, Y)

        simulate click with left mouse button on position X, Y

SendKey(virtual_key)

        simulate pressing virtual_key

SendString(string)

        simulate typing a string

I will not go into details here. Anyone interested can explore the sources of the Win32API.dll.

Using these functions is straightforward in IronPython. Just download the Win32API.dll and look at the example below how to perform a mouse click on the specific position:

import clr
clr.AddReference('Win32API')
from Win32API import Win32API
Win32API.MouseClick(10, 10)

The code snippet moves mouse to the coordinates 10, 10 (that is to the top-left corner of the screen) and perform a click with the left mouse button. More useful examples will follow in next parts.

Finally, couple of warnings. The simulated mouse click is caught by the top most window on the position. And the simulated key strokes are sent to the active window. That means we have to pay attention to position and active state of the tested application.

In the next part, we start building the GUIAT testing framework.

[1] I do not know how to access Win32 API directly from IronPython.

Exploring test application: IronPython (2)

We demonstrated how to run a .NET application from IronPython in the previous part. However, we couldn't control it while it was running. The solution for this problem is to run the tested application in a separate thread. Thus we will be able to enter commands in IronPython console while the tested application is running. Here is a snippet from the source:

import clr
clr.AddReference('System')
clr.AddReference("System.Windows.Forms")
from System import *
from System.Reflection import *
from System.Threading import *
from System.Windows.Forms import Application
from time import sleep

def RunMeCallBack(var):
    global App
    asm = Assembly.LoadFrom('GUIAT_PoC.exe')
    asm_type = asm.GetType('GUIAT_PoC.frmGUIAT')
    App = Activator.CreateInstance(asm_type)
    Application.Run(App)

App = None
ThreadPool.QueueUserWorkItem(WaitCallback(RunMeCallBack))
while not App:
    sleep(0.2)

The RunMeCallBack function starts the tested application the same way as we showed in the previous part. We create an independent thread and run this function in it so it does not block the console. The thread finishes its work when the tested application terminates (the main form of the application is closed) or when the console is closed.

The important part is line

App = Activator.CreateInstance(asm_type)

Here we remember the instance of the main form in the variable App. We have access to the whole application thanks to the App variable! The while cycle at the end ensures waiting until the App variable is not None. Which only happens when our tested application is up and running.

The App variable is our Holy Grail. Let's explore what is inside:

>>> App
<GUIAT_PoC.frmGUIAT object at 0x000000000000002B
[GUIAT_PoC.frmGUIAT, Text: GUIAT - Proof of Concept]>
>>> App.Text
'GUIAT - Proof of Concept'

Basically, we have access to all public properties and methods. Try dir(App) and you'll see. With a trick, we can even access private properties and methods (using the power of reflection).

To find what components are on the main form, iterate through the Controls collection:

>>> for c in App.Controls:
...   print c.Name, c.GetType()
...
btnAddItem System.Windows.Forms.Button
lblNewItem System.Windows.Forms.Label
txtNewItem System.Windows.Forms.TextBox
lbxItems System.Windows.Forms.ListBox
btnQuit System.Windows.Forms.Button

To find out what text is in the text box, try the following:

>>> App.Controls[2].Text
''

Now, write something directly into the text box in the tested application and call the statement again:

>>> App.Controls[2].Text
'something'

Cool, isn't it? ;-)

Next time, I show you how to simulate user interaction programatically - how to send a text or click to the tested application.

Exploring test application: IronPython (1)

How can IronPython help with exploring the test application? The answer is .NET Reflection:

Reflection is the mechanism of discovering information solely at run time.

We can discover both class and instance information. That's important. Knowing class information, we know what the object is capable of. We know whether it is a button having click method or a text edit having value property. And knowing instance information, we can find out what text the text edit displays.

That's nice but to be able to utilize the reflection, we need access to the tested application's objects from our testing framework. That's easy. The tested application is a .NET application so we can run it directly from IronPython.

When you look to Program.cs (source), you see how the tested application is started. If you don't have the sources of tested application available, you can use Lutz Roeder's .NET Reflector to find the information.

static void Main()
{
   Application.EnableVisualStyles();
   Application.SetCompatibleTextRenderingDefault(false);
   Application.Run(new frmGUIAT());
}

Let's mimic the start in IronPython. We are not interested in the first two rows. The third row is the core. The Application class is part of System.Windows.Forms namespace. So we just need to create an instance of frmGUIAT class. To create it, we use method CreateInstance of Activator class from System namespace. The CreateInstance method needs to know the type of the future instance. We find the type utilizing the reflection.

The whole code looks like this:

import clr
clr.AddReference('System')
clr.AddReference("System.Windows.Forms")
from System import *
from System.Reflection import *
from System.Windows.Forms import Application

asm = Assembly.LoadFrom('GUIAT_PoC.exe')
asm_type = asm.GetType('GUIAT_PoC.frmGUIAT')
App = Activator.CreateInstance(asm_type)
Application.Run(App)

When you run the above code from IronPython console, the tested application is displayed. Note the IronPython must be started form the directiory where GUIAT_PoC.exe is located. The disadvantage is we cannot do anything in IronPython console while the tested application is running. The solution is to run it in separate thread. I will show it in the next article.

Exploring test application: Accessibility

Accessibility looks more promising:

Active Accessibility (AA) is technology developed by Microsoft to make the Windows platform more accessible to people with disabilities. It consists of a set of COM interfaces and methods which allow an application to expose information to third parties, making it easier to implement tools such as magnifiers and screen readers. It proves very useful also for testing utilities. Microsoft has implemented the accessibility interfaces for the standard Windows controls.

AccExplorer (download) shows you the test application structure:



You see nice hierarchical window structure. Every object has several properties:

• Name - e.g. caption of button
• Value - e.g. value of list box
• Role Text - type of the component
• Location - position on the screen
  
• State - e.g. disabled, normal, etc.
• Def Action - default action for the component
  

The properties provides more information than Win32 API. But when you look closer, you see it is not enough:

• We still cannot distinguish the similar controls. Both buttons have role push buttons and can be distinguished only by displayed name. Which is problematic when names change or you need to test localized applications.
• Some components do not have name at all. In our example it is the text box - its name in AccExplorer is NAMELESS.
• The third party components usually lack support for the accessibility in the same quality as Microsoft. For example, DevExpress grid only tells you "I am a grid". It does not tell you how many rows it contains, what cells are there, etc...
  

The good news is developers can change this behavior. They can define AccessibleName property for a component so we can later easily find exactly what we need. If I have set AccessibleName for the text box to myTextBox, we would have seen myTextBox instead of NAMELESS in AccExplorer now. For grid, they can define the whole structure of accessible objects to represent the grid (see Exposing Data Tables through Microsoft Active Accessibility). Unfortunately, when your application does not have sufficient support for accessibility, it is quite lot of work...

Microsoft tries to address testing issues with his new framework in .NET 3.5: Microsoft UI Automation. I have not tried it as our applications do not run on .NET 3.5.

Also DevExpress promised to enhance their components with better support for UI testing tools - see the 2008 roadmap.

But we don't want to wait, right? We want to test our .NET application right now! I will show how in the next article.

Exploring test application: Win32 API

Let's look on what we can discover about our simple test application when it is running. Before IronPython, we had two options:

• Win32 API
• Accessibility

Win32 API provides several functions that return information about an application. First, we have to find the appliction handle. We utilize functions EnumWindows and GetWindowText from Python for Windows extensions project:

def enum(hwnd, extra):
   global app_handle
   if winxpgui.GetWindowText(hwnd).find('GUIAT - Proof') >= 0:
       app_handle = hwnd

app_handle = None
winxpgui.EnumWindows(enum, '')
print 'Test application handle:', app_handle

When we know the application handle, we can enumerate its children:

def enum_child(hwnd, extra):
   print '%s %s:    %s' % (hwnd, winxpgui.GetClassName(hwnd),
           winxpgui.GetWindowText(hwnd))

winxpgui.EnumChildWindows(app_handle, enum_child, '')

When you run the whole script (source), you get:

Test application handle: 2949744
2621886 WindowsForms10.BUTTON.app.0.378734a:    Add Item
7668314 WindowsForms10.STATIC.app.0.378734a:    New listbox item
4194952 WindowsForms10.EDIT.app.0.378734a:
2753142 WindowsForms10.LISTBOX.app.0.378734a:
3342924 WindowsForms10.BUTTON.app.0.378734a:    Quit

Here is the catch. How shall we identify Add Item button from Quit button? Both have the same class name WindowsForms10.BUTTON.app.0.378734a. Sure, the text differs, but this isn't always true. Imagine two text boxes - both having the same class name WindowsForms10.EDIT.app.0.378734a. What then? Of course, you can check positions. But this is leads you back to record/play tools and that is exactly what I want to avoid.

So this is the terminus for Win32 API. Next time - Accessibility.

GUI Application to Test

Let's start with something easy. We have a Windows GUI application that looks like this:


The application contains a text box, two buttons, and a list box. It simply adds a text from text box as the last item to the list box (exe, source).

The testing steps might be as follows:

1. Run the application.
2. Check the 'New listbox item' fields is empty.
3. Check the list box contains three rows with texts: 'This is', 'GUIAT', 'demo.'.
4. Enter a text (e.g. 'new line') into text box, press 'Add Item' button. Verify your text ('new line') appears as the fourth line/item in the list box.
5. Press 'Quit' button. Verify the application terminates.

Any record/play testing tool can do the test up to point 4. There it cannot verify whether the text was added to the correct place in the list box. These tools also cannot handle moving 'Add Item' button closer to the text box. They would then click to empty place. Sure there are more sophisticated tools that can handle the above problems. But I do not know about any for free.

Next time we try some dead ends from my beginnings :-)

The beginning

Testing GUI application can be tedious work. Usually, you have to manually click through the interfaces. The first round is fun, the second is OK, and from the third round on it is boring.

When I joined Radiant, I started to think about automation. I searched on Internet but I did not find anything satisfying. All tools were basically record/play applications. That is not sufficient for testing. There are many concerns, for example:

• How do you verify the test result?
• How easy can you change the script (add testing of a new field, new form, etc.)
  
• Scalability - how easy can you join basic scripts and create bigger (regression) script?
• How the test application works with your amended components?
• etc.
  

I know the testing tool cannot do everything. So I set up several goals I'd like to achieve:

• Allow to control GUI application.
  That means to start it, navigates through it, close it. To know its status.
• Allow to read values from the screen.
  You have to know what is on the screen to be able to test it.
• Allow to set values.
  Also, you have to be able to set/write values into the application to be able to test it.
• Easy to create.
  Testers are not experienced programmers. The tool must be easy to use. Anybody, who knows what a program is, should be able to write a test script.
  
• Easy to maintain.
  When the tested application changes, amending a test script should not be necessary or be a piece of cake.
  
• Scalability.
  Joining test scripts together, creating regression scripts.

In next few weeks, you'll find out on this blog how I succeed in this task.