Automated GUI testing with Autopilot

I spent a few days learning to use Autopilot. This is a tool originally designed to test Unity, which has been extended to test Qt and GTK applications. In this article I will explain how it works especially in the case of GTK applications.

Autopilot is twofold: on the one hand, it is a set of modules built into each toolkit (GTK, Qt, etc…) to get information about the widgets displayed (such as the position, the content…). On the other hand, it is a Python library which can be used to write efficient tests. Yes, you will need to know Python to write tests, but it is a wonderful, fast, and easy to learn language. The choice of Python is especially good to write tests: they do not need to be super fast, and it is more practical to not re-compile them every time you do a change in your test code.

How to install Autopilot

I will only cover Ubuntu here, but the method to install it must be similar on other platforms.

If you are lucky, there are already the packages in the repositories (for Raring and Saucy), so just install autopilot-desktop and python-autopilot:

sudo apt-get install autopilot-desktop python-autopilot

If you are using Precise (or Quantal, but it has not been tested on it), just install these packages on your computer, copied from the Raring ppa there: https://launchpad.net/~autopilot/+archive/ppa (just adjust the names if you are using an amd64 or an i386 computer): autopilot-desktop, python-autopilot-trace, python-autopilot-vis, python-autopilot, libautopilot-gtk, libxpathselect. Download all these files in a folder, then do:

sudo dpkg -i *.deb

(Yes, this is not the cleanest way to install packages, but it works and backporting everything properly would just be a waste of time.)

Running the tests

I wrote two small tests suites for the elementary project. The first one is for Scratch, but I will not show it here (it is on Launchpad tough), because it needs a patch in autopilot-gtk that I am trying to get upstream.The other one is the Pantheon Terminal test. Currently, it only tests the tabs, but it aims to test more part in the future. The code is in lp:~xapantu/pantheon-terminal/tests.

cd elementary # if you have a clean elementary folder with all your development stuff
mkdir tests
cd tests
bzr branch lp:~xapantu/pantheon-terminal/tests pantheon-terminal # it is important to name the branch pantheon-terminal
autopilot run pantheon-terminal # if you do not use JHBuild
jhbuild run autopilot pantheon-terminal # for people who use JHBuild

You can then read the code to learn how extend them.

Why not LDTP or dogtail?

LDTP and dogtail are the two other softwares used to write GUI test that I am aware of. Both use the accesibility API only (via D-BUS), without any modules linked to the toolkit. While it is theorically more flexile, in my experience, it can lead to some nasty refresh bugs (especially in a GtkNotebook), that I did not encounter with Autopilot (it does not mean it is bug free though).
Moreover, after having tested these three solutions, I eventually prefered the Autopilot API (which is entirely subjective).

 

I will write an article to explain how to write the tests in detail (and how to use autopilot vis, etc…) in the following days.

Build Pantheon and elementary applications with JHBuild

JHBuild is a software that download several VCS repositories to build them together. Everything is sandboxed, so it does not interact with the rest of the system. How can elementary applications be built with JHBuild? What are the advantages of JHBuild for a developer or a tester? It is explained below!

JHBuild is not intended as a replacement for the distribution’s package management system. Instead, it makes it easy to build software into a separate install prefix without interfering with the rest of the system.

See this page for more general information about JHBuild.  It is mostly used to build the GNOME plateform, but we wrote a few oncifuration files so as most elementary code can be built automatically with it.

Build

Get JHBuild

As said above, you don’t need any root account, everything is installed in your /home folder. So, first, let’s get JHBuild. The version in Ubuntu 12.04 repositories is too old (it doesn’t support the new system module type). However, since it is installed in ~/.local, it shouldn’t be a problem. But wihle a root account is not required to build JHBuild and the modules, a few dependencies are still needed (make sure you installed yelp-tools from the repositories before compiling JHBuild, for instance):

$ git clone git://git.gnome.org/jhbuild
$ cd jhbuild
$ ./autogen.sh
$ make
$ make install

Now, make sure you can launch it correctly:

$ which jhbuild
/home/user/.local/bin/jhbuild

If this last command does not return anything, add ~/.local/bin/ to your PATH (so as binaries can also be loaded from the command line there).

Open ~/.bashrc, and add, at the end:

export PATH=~/.local/bin:$PATH

Now, the command above should work correctly.

Elementary moduleset

We wrote several configuration files so as everything is compiled directly. Everything is in a bazaar branch so as you don’t have to configure anything.

bzr branch ~elementary-apps/+junk/jhbuild/ elementary
cd elementary
jhbuild -f jhbuild_elementary.rc

Launch an application, or the desktop

Then you can launch an application using jhbuild (it set some environment variable, so it is not a good idea to launch the binaries directly):

jhbuild -f jhbuild_elementary.rc scratch-text-editor

Go further

Build more modules

While building all elementary modules is very interesting to debug and test Pantheon as well as the other elementary applications, JHBuild was usually used to build all GNOME components, including gtk+, glib, etc…

So, if you want to build all your sandboxed applications with the latest gtk+ from git, you can just comment the line skip = [‘gtk+’] in the jhbuild_elementary.rc file. It will also build all gtk+ dependencies, including glib, gdk-pixbuf… So it may be quite long (around 25 modules to build, some with huge git repossitories) and space consuming, that is why we disabled it by default in the configuration file.

Use the dependency management 

JHBuild can detect missing dependencies, and it will tell you:

Required packages:
  System installed packages which are too old:
    (none)
  No matching system package installed:
    indicator (indicator3-0.4.pc)
jhbuild build: Required system dependencies not installed. Install using the command 'jhbuild sysdeps --install' or to ignore system dependencies use command-line option --nodeps

However, using directly jhbuild sysdeps –install will not work if you do not have app-file installed.

$ sudo apt-get install apt-file
$ apt-file update # as normal user
$ jhbuild -f jhbuild_elementary.rc sysdeps --install

Using -f jhbuild_elementary.rc is tiring!

To keep everything sandboxed, in this tutorial, we used jhbuild with a custom configuration file, so as if you run jhbuild alone, it will try to run th default configuration file. This behavior is interesting if you want to build elementary, and the GNOME plateform in several sandbox. But it is also tiring to type -f jhbuild_elementary.rc.

Open home_jhbuildrc, and replace the line current_dir = “…” by the appropriate path to your elementary sandbox folder (if you followed this how-to, it should be something like /home/username/elementary). Then do cp home_jhbuildrc ~/.jhbuildrc.

With a GUI, please!

To use the GUI, we strongly recommend you to make elementary the default configuration of JHBuild, so, in the bzr branch you checked out, just do:

cp home_jhbuildrc ~/.jhbuildrc

Then, search in slingshot for JHBuild, or just launch jhbuild gui.

Links

• The bazaar branch with the elementary moduleset: lp:~elementary-apps/+junk/build
• Manual and introduction of JHBuild on the GNOME dev center.
• JHBuild on Freedesktop.org

Writing tests using LDTP for elementary

This is an unfinished article.

Testing is a topical issue in elementary. There are more and more features in the applications, and we can’t test everything by hand. That’s why there may be some regressions when a big merge is done.

Unfortunately, most of the code has not been written with testing in mind, that’s why a unit testing approach is difficult to use. Here is another means, using LDTP: a GUI Automation tool which uses accessibility features to discover the interface.

For instance, let’s write a simple test for Scratch: check wether a file is well saved and that it content is correct.

Beware, erverything is in Python.

Unit testing in Python

Everything is already included in standard Python libraries for unit testing.
Here is a stub we are going to use:

import unittest
class ScratchSave(unittest.TestCase):
    def setUp(self):
        pass
    def test_save(self):
        pass

if __name__ == '__main__':
    unittest.main()

Let’s see some LDTP related code to launch Scratch and get a window object:

import ldtp, ooldtp
ldtp.launchapp('scratch-text-editor')

window = ooldtp.context('*Scratch')
window.waittillguiexist()

print window.getobjectlist()

When this code is executed, we get:

['flr8', 'flr4', 'flr5', 'flr6', 'flr7', 'flr0', 'flr1', 'flr2', 'flr3', 'btnSave', 'btnOpenfile', 'tbar0', 'lblOpenafiletobeginediting', 'ptl0', 'lblNofilesareopen', 'splt2', 'splt0', 'btnNewdocument', 'btnRevert', 'tbar1', 'pnl6', 'pnl7', 'pnl4', 'pnl5', 'pnl2', 'pnl3', 'pnl0', 'btnOpen', 'tbtn1', 'tbtn0', 'frmScratch', 'ptl1', 'pnl8', 'pnl9', 'btnUndo', 'btnRedo', 'pnl1', 'btnNewfile', 'ptl2', 'splt1', 'pnl10', 'pnl11']

Then we want to get a button to click it:

window.getchild("btnNewdocument").click()

Hacking on LibreOffice – Get a build

This is how I got a build of LibreOffice with Gtk 3 on Ubuntu Oneiric.

First, let’s clone the core git repository :

git clone git://anongit.freedesktop.org/libreoffice/core

Then, we need to use the maximum of libraries already built on the system to have a quick build. So,  use with autogen.sh --with-system-libs:

./autogen.sh --with-system-libs

LibreOffice will complain because you won’t have all required libraries. Install the missing ones, and if it is not available in the repository of your distribution, ask autogen to build it with LibreOffice (it will take less time than if you had to build it manually). So, for instance, I couldn’t get libcmis, graphite, sampleicc, lpsolve, libexttextcat, libcdr and libvisio:

./autogen.sh --with-system-libs --without-system-libcmis --without-system-graphite --without-system-sampleicc --disable-extensions --without-system-lpsolve --without-system-libexttextcat

Then we want to build LibreOffice with Gtk 3 support, so, we add –enable-gtk3 and –with-system-cairo. Maybe –with-system-cairo is not required since –with-system-libs should enable it, but it looks like it is a good idea to remember autogen that we absolutely want the system cairo, it won’t build with the cairo built in LibreOffice.

To speed up the build, I also disabled java, mozilla, the help and the dictionaries and I added –with-num-cpus=2, but it shouldn’t be needed if you have a dual core. So my final autogen.sh call is:

./autogen.sh --with-system-cairo --enable-gtk3 --without-help --disable-mozilla --with-system-libs --without-system-libcmis --without-java --without-junit --without-system-graphite --without-system-sampleicc --disable-extensions --without-system-lpsolve --without-system-libexttextcat --without-doxygen --with-num-cpus=2 --without-myspell-dicts

On my computer, with ccache, 4GB of RAM, and an Intel Core 2 Duo (E8400), the build took 3-4 hours.