Tuesday, June 24, 2008

MbUnit with NAnt support for TeamCity 3.x

This week I've switched to TeamCity 3.1 after using CruiseControl.NET for a while. CC.NET is ok, but feels like something built on an off day to get automated builds working. TeamCity on the other hand feels much more mature. It's a lot easier to configure (no more XML fiddling for basic setup!). The UI is smooth and easy to work with. It's just a friendly tool.

No MbUnit support
TeamCity 3.1 does however have one big drawback for me; it only supports NUnit unit testing on .Net. There is no support for MbUnit, my current unit testing framework of choice. Fortunately, TeamCity can be extended quite easily. So, I was exprecting someone had already found a solution for this shortcoming. After some googling around I quickly found there is no easy solution available yet. I've tried some stuff like writing an XSL for the XML generated by MbUnit, but that did not yield any usable results.

The fix
TeamCity will monitor the build log and capture all sorts of information from it (described here), including test results. This is the easiest way to relay information from a build to TeamCity. MbUnit tests are executed by a custom NAnt task that comes with MbUnit (MbUnit.Tasks assembly). Since MbUnit is open source the source code is available and I can expand the task to do as I want. So, I've added support for an additional report type for TeamCity. Instead of logging to a file, it uses the build log to directly report results to TeamCity.

Not quite perfect
Unfortunatly, due to the way the MbUnit test runner works it's not possible to log test results in real-time as TeamCity expects. The results are only available after all tests in the task have completed. At that time the results are exported. This means timing information is not valid. The test results however are quite detailed, including console output, error output and a stack trace.

Using the task in a build
To use the new MbUnit task in a NAnt build script there are 3 options:

  1. If you copied the MbUnit.Tasks.dll into the NAnt bin folder, replace it with Alanta.MbUnit.Tasks.dll.
  2. Load the tasks from your build script: <loadtasks assembly="path-to-custom-tasks/Alanta.MbUnit.Tasks.dll">
  3. Load the tasks from the command line.
I use NAnt build scripts to build from the command line as well as for automated builds. So, I've defined the report type for MbUnit in a property: <property name="mbunit-report-type" value="Html" overwrite="false" /> ... <target name="test"> ... <mbunit types="${mbunit-report-type}" report-output-directory="TestResults" halt-on-failure="false"> <assemblies> <include name="MyProject.*.dll" /> </assemblies> </mbunit> </target>In TeamCity I can now set the mbunit-report-type property on the command line ( on the Runner tab ):
-D:mbunit-report-type=html;teamcity

This will generate both an html report and output the test results to the log file for TeamCity.

Update This solution is now part of the NAnt-Extensions project. You can download it here.

7 comments:

Alex said...

Hi Marnix,

what build runner do you use for building/running the unit tests?

I've been using the cmdline runner: The ##teamcity messages show up in the log, but unfortunately they don't get parsed by TeamCity itself. (That is, the "Tests" tab does not appear.)

If I use the NAnt runner everythings works ("Tests" tab is shown) but the build log is very verbose. Do you also see this problem on your build server?

Thanks,

Alex

Marnix said...

Hi Alex,

Since this solution uses a NAnt task, I'm using the NAnt runner.

From what I've seen TeamCity taps into NAnt logging at a low level by hooking up monitoring tools on the command line.
Every message that is logged through the NAnt API is sent directly to the TC server.
The command line runner taps in at the same level and will filter that info for the console based on the log level setting.
This may also explain why the command line runner is not working for you; invoking NAnt from the command line will prevent TeamCity from hooking up it's log monitoring tools.

Anyway, as far as I'm concerned a build log can't be detailed enough. I want to be able to go back and see exactly what has been build, especially if something fails.

If you need specific information from a build, you could output it to a separate file and pull it into TC as a build artifact. That way you don't need to search through the log file.

Alex said...

Hi Marnix,

I've taken your binaries and extended them with the help of Max Feldman from JetBrains to make them work with both the NAnt and the Console runner.

I've also created other tasks for TC status report etc. based on these tasks for MSBuild.

Would you like to join forces and create an OSS project out of this. If you don't want to, would you mind me publishing the source code and adding a reference to your blog post?

Alex

Chakri said...

Hi, help me anyone to run MbUnit reports from NAnt at a specific time...

Thanks in advance.

Marnix said...

Hi Chakri,

If you're using TeamCity you should configure a Schedule Trigger in the build configuration. If you want to run the build directly from windows you should setup a scheduled task.

Marnix

Chakri said...

Thank you Marnix,
But Teamcity is not open source i think. can u plz suggest me any other opensource tool, which solves my problem.

Marnix said...

Hi Chakri,

TeamCity is not open source but the professional license is available free of charge.
The open source alternative is CruiseControl.NET (hosted on SourceForge). I have used both and must say that TeamCity is by far the easiest to setup, configure and use.

Marnix