Acceptance testing with FitNesse and PyFIT

Author: Grig Gheorghiu

Last Revision: 12/14/2004

What follows is a step-by-step tutorial on using PyFIT within the FitNesse framework. PyFIT, written by John Roth, is the Python port of Ward Cunningham’s FIT framework

Step 1: Install FitNesse

I installed FitNesse locally, on a Windows box, but the examples should work just as well in any environment supported by FitNesse.

• Download the full distribution of FitNesse from its site's download page and unzip it. In the following discussion, I will assume it was unzipped in C:\fitnesse
• Start FitNesse by running the run.bat script in C:\fitnesse. By default, FitNesse runs a Web server on port 80, but you can specify a different port, say 8080, like this:

run.bat -p 8080

• Go to http://localhost in your browser and look around to get a feel of what FitNesse looks like (it's a Wiki) -- the User Guide is especially helpful

Step 2: Install PyFIT

• Download the latest PyFIT distribution (currently PyFIT-0.6a1.zip) from the Files section of the FitNesse Yahoo group. You need to become a member of the group in order to be able to download the file
• Unzip the PyFIT distribution (I unzipped it in C:\Python23\PyFIT-0.61a), then install it by running: python setup.py install
• Go to the FitNesse FrontPage and edit it by clicking on the blue Edit button on the left-hand side of the page. Add a WikiWord somewhere in the page (for example PythonTest) and click Save. The FrontPage should now show PythonTest followed by a ? character, which means it does not yet have a page called PythonTest

• To create content for the PythonTest page, click on the ? link, then enter the following lines and click Save (Note for Linux users: you need to omit the double quotes around %m):

!define COMMAND_PATTERN {python "%m" %p}

!define TEST_RUNNER {C:\Python23\PyFIT-0.6a1\fit\FitServer.py}

|eg.Division|

|numerator|denominator|quotient?|

|10|5|2|

|10|2|5.001|

• The COMMAND_PATTERN variable tells FitNesse to start python instead of java when running test fixtures. The TEST_RUNNER variable tells FitNesse to replace the default fit.FitServer Java test runner with FitServer.py, which is its Python equivalent. If you unzipped PyFIT somewhere else, you need to set TEST_RUNNER to the full path to FitServer.py
• The | | notation is a special Wiki convention for describing tables:

the cell in the first row tells FitNesse to run a fixture called eg.Division

the cells in second row are the names of variables to be set or get by FitNesse

a ? mark next to a variable name means that FitNesse will retrieve the value for that variable and will compare it agains the values entered by the user

the next 2 rows are examples of input (10 and 5 in the second row, 10 and 2 in the third row) and expected output (2 in the second row and 5.001 in the third row)

• Make the PythonTest page a test page by clicking on the blue Properties button, then clicking the Test checkbox, then clicking on the Save button on the Properties page
• Run the test by clicking on the Test button, which should have appeared on top of the Edit button. If everything goes well, the row containing 10, 5 and 2 should be colored green, and the next row should be colored red, with 2 cells for quotient: one containing "5.001 expected" and the other containing "5.0 actual". Of course, the correct answer is 5, but I put 5.001 just so we can see how a failed test looks like. Also, on top of the table, FitNesse prints a summary of the test run. In this case, it is:

Assertions: 1 right, 1 wrong, 0 ignored, 0 exceptions

• One question you may have at this point is how did FitNesse find the eg.Division fixture. If you installed everything like I did, you will see a directory called eg under C:\Python23\PyFIT-0.6a1\fit and a file called Division.py in that directory. There is also an important file called __init__.py in that directory; this is an empty file which tells Python to treat that directory as a package, thus making it possible to invoke FitServer.py with the argument eg.Division
• It is instructive at this point to look at the Division.py file. Here it is:

 
from fit.ColumnFixture import ColumnFixture
 
class Division(ColumnFixture):
   _typeDict={
       "numerator": "Float",
       "denominator": "Float",
       "quotient": "Float",
       "quotient.charBounds": "99",
   }
   numerator = 0.0
   denominator = 0.0
 
   def quotient(self):
       return self.numerator / self.denominator

• The Division class is derived from the fit.ColumnFixture class. A ColumnFixture is the most common type of fixture and is most useful when you need a way to specify inputs for the acceptance test, and then verify outputs.
• A PyFIT-specific caveat is that you need to have the _typeDict dictionary in any fixture you write (actually anywhere you need to use a TypeAdapter). Here is what the PyFIT documentation (PyFIT-0.6a1/fit/Doc/FIT_TypeAdapters.htm) has to say about Type Adapters:

"FIT, as distributed, requires the use of a Type Adapter to convert the text format used in the tables to and from the actual data type needed by the various fields, methods and properties in the fixture. This practice came from the Java version, where manifest typing makes it easy to find the expected data type by reflection. Since Python does not have manifest typing, there is no way that the reflection capability can determine the proper type. Type information must be provided another way. Since types need to be declared separately, TypeAdapter contains a more general metadata mechanism. This consists of a dictionary named _typeDict that must be located in the class whose fields, methods or properties are to be referenced. It's also possible to pass a metadata dictionary to the type adapter factory function; this is useful for unusual requirements."

• In Division.py the _typeDict dictionary specifies that the numerator and denominator variables, as well as the return value of the quotient method, are of type Float.

Step 3: Write the application that you want to test with FitNesse

• I wrote a simple Blog Management application, based on the feedparser Python module written by Mark Pilgrim of Dive Into Python fame. You might wonder what this has to do with the day-to-day activities of a tester. If you read Mike Clark’s Pragmatic Project Automation book, you know that one of his many ideas for build automation and continuous integration is to publish the results of a new build (done for example via CruiseControl) as an RSS or Atom feed that can then be read by all the members of the team. It seemed to me that a good way to prepare for implementing something like this would be to learn more about RSS and Atom, hence the Blog Management application. I will use the same application later in this article to show how to write FitNesse tests in Python.
• To follow along the example, you need to register with Blogger and create a test blog (I called mine fitnessetesting). The most important parameters to remember are your user name, your password and your blog's Atom Feed URL (which in my case is http://fitnessetesting.blogspot.com/atom.xml). You also need to install the feedparser Python module. You can download it from the project's SourceForge page. Unzip it, and then run: python setup.py install
• I implemented the blog management functionality in a class called Blogger. You can get the source code from here: Blogger.py
• If you created your own blog at Blogger, you need to assign the appropriate values to the BlogParams class variables FEED_URL, USER and PASSWORD
• The main functionality of the Blogger class is exported via the following methods:

• post_new_entry: posts a new entry to the blog, takes a title and a content as parameters
• get_nth_entry_title: returns the title of the nth entry (note that the entries are ordered most-recent first, so the entry entered last will appear first in the blog)
• get_nth_entry_content: returns the HTML content of the nth entry
• get_nth_entry_content_strip_html: strips HTML tags from the content (partially implemented so far)
• delete_nth_entry: deletes the nth entry from the blog
• delete_all_entries: self-explanatory

• Consumers of the Blogger class get an instance of a Blogger object via a get_blog() function that returns a global variable defined at the Blogger module level
• I also wrote a unit test suite for my Blogger class. You can get the source code from here: testBlogger.py
• After making sure that all the unit tests pass, we're ready to tackle the FitNesse integration

Step 4: Write a FitNesse acceptance test suite for the Blogger application

• Now let's go back to the FitNesse FrontPage (http://localhost). We will create a FitNesse test suite page called BlogMgmtSuite. You need to edit the FrontPage and add the BlogMgmtSuite WikiWord. Save, then click on the ? link next to BlogMgmtSuite in order to edit that page. For now, enter the following content and save it:

!define COMMAND_PATTERN {python "%m" %p}

!define TEST_RUNNER {C:\Python23\PyFIT-0.6a1\fit\FitServer.py}

!path C:\eclipse\workspace\blogger

!2 ''Blog Management acceptance test suite'' |

^DeleteAllEntries|''Delete all blog entries''|

• The first 2 lines are similar to the ones in our PythonTest example. The third line tells FitNesse to look for our fixtures in the C:\eclipse\workspace\blogger directory. The line starting with !2 will be interpreted as a header. The ^ character before DeleteAllEntries is a special FitNesse convention signifying that the DeleteAllEntries page is a "child" page of the current BlogMgmtSuite page
• We need to tell FitNesse that this page is a suite; to do this, click on the Properties button of the BlogMgmtSuite page, then click on the Suite checkbox and save
• You will see a question mark next to DeleteAllEntries. This is of course because that page does not exist. Let's create it: click on the ? link, then enter the following content and save it:

!3 We test deleting all entries from the blog

We delete all blog entries and we verify that we have 0 entries.

!|BloggerFixtures.DeleteAllEntries|

|num_entries?| |0|

• If you look at the URL of this page, you will note that it appears as http://localhost/BlogMgmtSuite.DeleteAllEntries -- this is because FitNesse made it a "child" page of the BlogMgmtSuite page
• We also need to make the DeleteAllEntries page a test page, by going to its Properties and clicking on the Test checkbox
• Now we are ready to run the DeleteAllEntries acceptance test by clicking on the Test button of that page. The first first 2 rows of the table should be colored yellow and you should see the following text:

Fixture 'BloggerFixtures.DeleteAllEntries' not found

Also, if you click on the Output Captured link in the right upper corner, you should see this (on a single line):

 
template: 'Fixture '%s' not found' args: '('FixtureNotFound', u'BloggerFixtures.DeleteAllEntries')'

Of course, we did not write the DeleteAllEntries fixture yet. Let's first create a directory called BloggerFixtures under C:\eclipse\workspace\blogger (if you create this directory somewhere else, you need to replace the !path variable in the BlogMgmtSuite page with the parent directory of BloggerFixtures). We also need to create an empty filed called __init__.py in the BloggerFixture directory, otherwise FitServer.py will not consider it a package and thus will not know how to interpret BloggerFixtures.DeleteAllEntries

Now let's write the actual DeleteAllEntries fixture. It resembles the Division fixture discussed previously as an example. Here it is, in its entirety:

 
from fit.ColumnFixture import ColumnFixture
import sys
blogger_path = "C:\\eclipse\\workspace\\blogger"
sys.path.append(blogger_path)
import Blogger
 
class DeleteAllEntries(ColumnFixture):
    _typeDict={
        "num_entries": "Int",
    }
    blogger = Blogger.get_blog()
 
    def num_entries(self):
        return self.blogger.get_num_entries()
 
    def execute(self):
        self.blogger.delete_all_entries()

• The DeleteAllEntries class is derived from fit.ColumnFixture. It gets a Blogger object via the get_blog() function call and assigns it to a class variable called blogger. Note that you need to replace blogger_path at the top of the file with the actual path to your Blogger.py class
• The table we entered in the DeleteAllEntries page contains only one column, called num_entries, followed by a question mark, which means it is a method, as opposed to a variable.We need to implement a method called num_entries in DeleteAllEntries.py. In our case, this method will simply call the get_num_entries() method of the blogger object. Note also that we need to define the _typeDict dictionary and declare the type of the value returned by the num_entries method. In our case, that type is an integer
• The real action happens in a special method called execute(), which is inherited from fit.ColumnFixture. This method is called by the FitNesse framework once for every row of the table, before the other elements of that row are processed. In our case, the execute() method simply calls delete_all_entries() method of the blogger object. Generally speaking, all the FitNesse fixtures you will write will serve as a simple 'wiring' to the actual application objects. All they need to do is to define the variables and the methods named in the FitNesse table, then return the appropriate values by calling the application object's methods
• Let's run the DeleteAllEntries test again by clicking the Test button. This time, the row containing the number 0 should turn green, and the test summary at the top of the page should read:

Assertions: 1 right, 0 wrong, 0 ignored, 0 exceptions

We wrote our first FitNesse fixture. Time to celebrate by going back to the BlogMgmtSuite page and then clicking on the Suite button. This will run all the tests found on the page -- in our case only DeleteAllEntries. You should see the following summary colored green at the top of the page:

Test Pages: 1 right, 0 wrong, 0 ignored, 0 exceptions Assertions: 1 right, 0 wrong, 0 ignored, 0 exceptions

The summary is followed by individual test results with links to the tests. In our case, we have: 1 right, 0 wrong, 0 ignored, 0 exceptions DeleteAllEntries

We made BlogMgmtSuite a test suite (as opposed to a test page) so that we can easily add more test pages to it. Let's add another acceptance test which will test posting and then deleting a blog entry. Edit the BlogMgmtSuite page, add the following line and save:

|^PostDelete1Entry|''Post single blog entry''|

Now click on the ? link next to PostDelete1Entry, enter the following text, save the page and don't forget to make it a Test page by clicking the Test checkbox in its Properties:

!3 We test posting a single new entry to the blog

First we delete all entries from the blog and we verify that we have 0 entries.

!|BloggerFixtures.DeleteAllEntries|

|num_entries?| |0|

Then we post the new entry and we verify that we have 1 entry.

!|BloggerFixtures.PostNewEntry|

|title|content|valid?|num_entries?|

|BloggerFixtures.PostSingleEntry Title|BloggerFixtures.PostSingleEntry Content|true|1|

We verify that the entry has the title and the content we indicated.

!|BloggerFixtures.GetEntryTitleContent|

|entry_index|title?|content?|

|1|BloggerFixtures.PostSingleEntry Title|BloggerFixtures.PostSingleEntry Content|

We delete the entry and we verify that we have no entries left.

!|BloggerFixtures.DeleteEntry|1|

|valid?|num_entries?|

|true|0|

Note how convenient it is to write down an acceptance test in FitNesse. We simply explain what we want to do, then we put together tables with inputs and desired outputs. FitNesse will ignore anything that is not part of the table, and will invoke the fixtures defined in the tables

The page contains many fixtures that we haven't defined yet. After calling DeleteAllEntries, we call PostNewEntry, which is another ColumnFixture with 2 member variables (title and content) and 2 methods (valid and num_entries). We need to create another Python module in the BloggerFixtures directory and call it PostNewEntry.py. Here is my version of it:

from fit.ColumnFixture import ColumnFixture
import sys
blogger_path = "C:\\eclipse\\workspace\\blogger"
sys.path.append(blogger_path)
import Blogger
 
class PostNewEntry(ColumnFixture):
    _typeDict={
        "title": "String",
        "content": "String",
        "num_entries": "Int",
        "valid": "Boolean"
    }
 
    title = ""
    content = ""
    blogger = Blogger.get_blog()
 
    def num_entries(self):
        return self.blogger.get_num_entries()
 
    def valid(self):
        return self.blogger.post_new_entry(self.title, self.content)

• This fixture defines the all-important _typeDict dictionary, then defines 2 class variables, title and content, whose values will be assigned by the FitNesse framework. The valid() method does all the work here, by invoking the Blogger object's post_new_entry method and returning true or false, depending on the success or failure of this operation. The num_entries() method again serves as only a wiring to the Blogger object's get_num_entries method
• The other 2 fixtures invoked on the PostDelete1Entry page are similar. For your reference, here is DeleteEntry.py (all the fixtures are also available at http://agile.unisonis.com/blogger/BloggerFixtures):

 
from fit.ColumnFixture import ColumnFixture
import sys
blogger_path = "C:\\eclipse\\workspace\\blogger"
sys.path.append(blogger_path)
import Blogger
 
class DeleteEntry(ColumnFixture):
    _typeDict={
        "num_entries": "Int",
        "valid": "Boolean"
    }
    blogger = Blogger.get_blog()
 
    def num_entries(self):
        return self.blogger.get_num_entries()
 
    def valid(self):
        entry_index = int(self.getArgs()[0])
        return self.blogger.delete_nth_entry(entry_index)

An interesting thing to note in the DeleteEntry fixture is that it gets a parameter passed via the FitNesse table cell next to the fixture name:

!|BloggerFixtures.DeleteEntry|1|

The parameter is available in the DeleteEntry.py class via the self.getArgs() list, which in this case contains only 1 element

Here is GetEntryTitleContent.py:

 
from fit.ColumnFixture import ColumnFixture
import sys
blogger_path = "C:\\eclipse\\workspace\\blogger"
sys.path.append(blogger_path)
import Blogger
 
class GetEntryTitleContent(ColumnFixture):
    _typeDict={
        "entry_index": "Int",
        "title": "String",
        "content": "String",
    }
 
    entry_index = 0
    blogger = Blogger.get_blog()
 
    def title(self):
        return self.blogger.get_nth_entry_title(self.entry_index)
 
    def content(self):
        return self.blogger.get_nth_entry_content_strip_html(self.entry_index)

• Now we can go back to the BlogMgmtSuite page and click the Suite button. FitNesse will run both acceptance tests that we have defined so far: DeleteAllItems and PostDelete1Entry. If for some reason you notice that only DeleteAllItems has been executed, this usually means that you forgot to make PostDelete1Entry a test page
• If everything ran fine, you will see the following summary:

Test Pages: 2 right, 0 wrong, 0 ignored, 0 exceptions Assertions: 9 right, 0 wrong, 0 ignored, 0 exceptions

1 right, 0 wrong, 0 ignored, 0 exceptions DeleteAllEntries

8 right, 0 wrong, 0 ignored, 0 exceptions PostDelete1Entry

• I wrote a few more acceptance tests for the Blogger application. They expand on PostDelete1Entry by posting several entries, deleting them one by one and verifying at each step that the expected entries are kept. They are all available at http://agile.unisonis.com/blogger/BlogMgmtSuite.

Concluding thoughts

1. I was able to see how FitNesse acceptance tests exercise the application in ways that unit tests do not.

In a typical unit test scenario, a single application object (for example a Blogger object) is instantiated in the setUp method and then used throughout the test case class methods

In a typical FitNesse acceptance test, there are several fixture objects created during the execution of a test page, each object using potentially different instances of the application object. This can result in inconsistencies between the states of these objects, and thus in seemingly mysterious failures. Case in point: I started getting failures after posting three entries and deleting them one by one via the PostDelete3Entries test page. This page is using 4 different fixtures: DeleteAllEntries, PostNewEntry, GetEntryTitleContent and DeleteEntry. Each of these fixtures appears in several tables, but the PyFIT framework only instantiates one object per fixture, so for example the same GetEntryTitleContent object is used in every row of every table corresponding to the GetEntryTitleContent fixture. When I first wrote the fixtures, each fixture object had its own copy of a Blogger object, so there was a disconnect between the number and order of entries reported by each fixture. For example, when an entry was deleted via DeleteEntry, the fact was not reflected in the GetEntryTitleContent object.

Here is an example of how DeleteEntry looked initially:

 
class DeleteEntry(ColumnFixture):
    _typeDict={
        "num_entries": "Int",
        "valid": "Boolean"
    }
    blog_params = Blogger.BlogParams()
    blogger = Blogger.Blogger(blog_params)
 
    def num_entries(self):
        return self.blogger.get_num_entries()
    etc...

I rewrote the fixture so that now it looks like this:

class DeleteEntry(ColumnFixture):
    _typeDict={
        "num_entries": "Int",
        "valid": "Boolean"
    }
    blogger = Blogger.get_blog()
 
    def num_entries(self):
        return self.blogger.get_num_entries()
    etc...

Note the call to Blogger.get_blog(), which returns a global variable at the Blogger.py module level, thus guaranteeing that all fixture objects will share a single instance of a Blogger object. The FitNesse documentation recommends using static instances of Singleton objects for sharing an object among tables on the same page. I have not done so yet and instead I took the easy way out by resorting to a shared global variable. I will experiment with implementing the Singleton pattern.

2. I had problems with the HTTP connection to blogger.com, hence many times both the unit tests and the FitNesse acceptance tests failed. I intend to isolate this type of failures by using Mock Objects (for example the pMock Python module) in order to simulate posting and deleting blog entries without actually using the Blogger API.

3. FitNesse advocates the use of SetUp and TearDown pages that are automatically invoked at the beginning and the end of a test suite run. I have not used them so far, but I intend to use them, for example for instantiating a Singleton object in the SetUp page. I realize that all this is just scratching the surface of what FitNesse can do, and I intend to further explore its functionality. My next goal is to experiment with RowFixture and ActionFixture tables. The GetEntryTitleContent fixture in particular is a query, so it's especially suitable for being expressed as a RowFixture. I expressed it as a ColumnFixture mainly because it seemed pretty hard to write RowFixture-derived classes in PyFIT, but I intend to remedy this.