Wednesday, 28 November 2007

Testing C++ applications

Software testing is one of those things that you can't really nail down. Thanks to a lot of advertising and ease-of-use due to the *Unit libraries, unit testing is quite popular nowadays. But really, unit testing is just the tip of the iceberg.

A unit test tests the tiniest part of a program that can be tested. This can be a free function, a class function or sometimes even a whole class. But thats it. However, most people are really writing component tests. A component is the combination of multiple units to give some larger functionality. A component test therefore tests this combination. Then there are integration tests, then regression tests, etc, etc. There is a whole spectrum of testing. You can find more information in Code Craft.

In a lot of projects, there is a huge gap between developer testing and QA testing. Usually the only type of testing that takes place including both groups is unit testing and usability testing, if you are lucky! And in a lot of these, neither the unit tests nor the tests QA run are automated; they usually require some manual intervention.

With a little bit of work, you can really boost the way you test your C++ code. There are a couple of things you need to do first:

  • Automated build

  • Automated deployment

That is, you need to be able to push a button to build the distributable from scratch and push another button to deploy the application. And yes, as you guessed, you need to wire one button to the other. If you used batch files to accomplish the above, do it again using either SCons, Ant, Final Builder or the bazillion tools created especially for automating builds. I shouldn't have to tell you this, but you would be surprised!

Anyway, the next step is to choose a tool for writing tests in C++. I have used Boost.Test with great success. For whatever this is worth, I highly recommend this library. You want to write your unit and component tests in this library, for obvious reasons.

Automate the running of these tests and add them to the automated build.

If you have been keeping up, and since you are a C++ guru, you are also proficient in a language like Python. Now pay attention: you will be writing at least half of your tests in Python! Really. Just because all your tests are either unit tests or component tests, does not mean that you don't need to be doing this! In my opinion, this is where most teams miss out. They write their tests in CPPUnit or Boost.Test and then it is off to QA. And the main reason is that C++ is not a glue language. It is an application/library development language. Python (or other languages with a REPL) shine at gluing components together. You can effortlessly write a whole bunch of portable tests by choosing to go this way. The Boost.Python library with some code generation will get you started with a minimum of fuss.

Whether you are shipping a library or an application, the addition of a Python API can be very useful: for a library, it is another checkbox in the "we support these languages" box. For an application it is a bullet point: "Fully portable scripting." Not to mention the benefits you'll reap.

The addition of this simple functionality means that now there is now no excuse not to automate all your non-UI tests. And if you are on Windows, with a choice few Windows API functions, you can do your UI testing in the same language. Quite handy.

Chances are, now you are going to have a bunch of Python scripts lying around which still need to be invoked manually. To close the loop, I would recommend QMTest to manage your tests and their dependencies.

Now don't lie, you aren't testing as much as you can in C++. You need Python for that :-)

Update: I forgot one very important aspect of this whole thing: continuous integration. I've used BuildBot with great success and would recommend it's use. One little thing you might want to add onto the bot is the ability to create builds for each platform, for each branch, automatically. I've done this using Subversion triggers and it is beautiful to watch in action.

No comments: