Integrate xUnit tests into your daily Team Build

May 7, 2009

General
Test driven development
tfs
xunit

Doing proper Test Driven Development with the default visual studio test runner (mstest) is like playing Gears of War 2 on an old fashioned tube television, it’s just wrong ;).. There are several alternatives out there like mbUnit and nUnit, but my favorite is xUnit. xUnit is heavily extensible and there is a project called xunitbddextensions, which extends xunit making it possible to do specication based testing (BDD). Using resharper or testdriven.net this makes for an excellent experience when developing in visual studio. xUnit tests can also be integrated into your daily TFS (Microsoft Team Foundation Server) build by using the xUnit build task. However, the details of the testresults are not displayed in the build overview and the results are not used in the reporting functionality of TFS (Like the Quality Indicators report).

On the xUnit site there is some discussion about this functionality, but at the moment there is no implementation. For nUnit there is a project on codeplex called nUnit for Team Build. This consists of a sample build script that uses an XSLT file to translate the xml results of nUnit into a visual studio test result file and publishes this result to TFS. This way integrating the nUnit test results into TFS.

I figured I could modify the XSLT file to translate the xUnit tests results instead. However, having a better look at the xUnit build task I noticed a NunitXml property, which makes it possible to output the xUnit results into a nUnit compatible xml file. So I could just use the XSLT file and integrate the xUnit results into our TFS build.

To make this work I only had to modify the build script to use the xUnit build task instead of the nUnit build task. This resulted in the following script (This just needs to be added to the end of your build definition):

<!-- Xunit Build task--><PropertyGroup><XunitBuildTaskPath>$(ProgramFiles)\MSBuild\Xunit\</XunitBuildTaskPath></PropertyGroup><UsingTask AssemblyFile="$(XunitBuildTaskPath)\xunit.runner.msbuild.dll"
    TaskName="Xunit.Runner.MSBuild.xunit"/><Target Name="AfterCompile"><!-- Create a Custom Build Step --><BuildStep TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Name="XUnitTestStep" Message="Running Xunit Tests"><Output TaskParameter="Id" PropertyName="XUnitStepId"/></BuildStep><CreateItem Include="$(OutDir)\*.Test.dll"><Output TaskParameter="Include" ItemName="TestAssemblies"/></CreateItem><!-- Run the tests --><xunit ContinueOnError="true" Assembly="@(TestAssemblies)" NUnitXml="$(OutDir)nunit_results.xml"><Output TaskParameter="ExitCode" PropertyName="XUnitResult"/></xunit><!-- Set the status of the buildstep in the build overview --><BuildStep Condition="'$(XUnitResult)'=='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(XUnitStepId)" Status="Succeeded"/><BuildStep Condition="'$(XUnitResult)'!='0'" TeamFoundationServerUrl="$(TeamFoundationServerUrl)" BuildUri="$(BuildUri)" Id="$(XUnitStepId)" Status="Failed"/><!-- Regardless of NUnit success/failure merge results into the build --><Exec Command="&quot;$(XunitBuildTaskPath)\nxslt3.exe&quot; &quot;$(OutDir)nunit_results.xml&quot; &quot;$(XunitBuildTaskPath)\nunit transform.xslt&quot; -o &quot;$(OutDir)xunit_results.trx&quot;"/><Exec Command="&quot;$(ProgramFiles)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe&quot; /publish:$(TeamFoundationServerUrl) /publishbuild:&quot;$(BuildNumber)&quot; /publishresultsfile:&quot;$(OutDir)xunit_results.trx&quot; /teamproject:&quot;$(TeamProject)&quot; /platform:&quot;%(ConfigurationToBuild.PlatformToBuild)&quot; /flavor:&quot;%(ConfigurationToBuild.FlavorToBuild)&quot;"/><Error Condition="'$(XUnitResult)'!='0'" Text="XUnit Tests Failed"/></Target>

This script assumes you have copied the xunit build task, the XSLT file and the nxslt3.exe file to a \MSBuild\Xunit\ folder under the program files folder on your build server.

Whether the tests fail or not, the resuls should always be published. However, the xUnit build task does not have an ExitCode and I could not find another way to get the result from the xUnit task. So I have modified the xUnit task to provide an ExitCode. Open up the xUnit code from codeplex. Locate the xunit.runner.msbuild project and open up the xUnit class. Add the following property and modify the Excute method to modify the value:

        [Output]
        publicint ExitCode { get; privateset; }

        publicoverridebool Execute()
        {
            try
            {
                string assemblyFilename = Assembly.GetMetadata("FullPath");

                if (WorkingFolder !=null)
                    Directory.SetCurrentDirectory(WorkingFolder);

                using (ExecutorWrapper wrapper =new ExecutorWrapper(assemblyFilename, ConfigFile, ShadowCopy))
                {
                    Log.LogMessage(MessageImportance.High, "xUnit.net MSBuild runner (xunit.dll version {0})", wrapper.XunitVersion);
                    Log.LogMessage(MessageImportance.High, "Test assembly: {0}", assemblyFilename);

                    IRunnerLogger logger =
                        TeamCity ? (IRunnerLogger)new TeamCityLogger(Log) :
                        Verbose ?new VerboseLogger(Log) :
                        new StandardLogger(Log);

                    List<IResultXmlTransform> transforms =new List<IResultXmlTransform>();

                    using (Stream htmlStream = ResourceStream("HTML.xslt"))
                    using (Stream nunitStream = ResourceStream("NUnitXml.xslt"))
                    {
                        if (Xml !=null)
                            transforms.Add(new NullTransformer(Xml.GetMetadata("FullPath")));
                        if (Html !=null)
                            transforms.Add(new XslStreamTransformer(htmlStream, Html.GetMetadata("FullPath"), "HTML"));
                        if (NUnitXml !=null)
                            transforms.Add(new XslStreamTransformer(nunitStream, NUnitXml.GetMetadata("FullPath"), "NUnit XML"));

                        TestRunner runner =new TestRunner(wrapper, logger);
                        if(runner.RunAssembly(transforms) == TestRunnerResult.Failed)
                        {
                            ExitCode =-1;
                            returnfalse;   
                        }

                        ExitCode =0;
                        returntrue;
                    }
                }
            }
            catch (Exception ex)
            {
                Exception e = ex;

                while (e !=null)
                {
                    Log.LogError(e.GetType().FullName +": "+ e.Message);

                    foreach (string stackLine in e.StackTrace.Split(new[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries))
                        Log.LogError(stackLine);

                    e = e.InnerException;
                }

                ExitCode =-1;

                returnfalse;
            }
        }

Now build the project and deploy the xUnit build task to the server. To realize the xUnit build integration just follow the steps you find on the nunit4teambuild site and use the previous mentioned modifications. After this, you should be set to go.

BTW: In the nunit4teambuild they use nxslt2, I used nxslt3. So depending on which one you use, modify the script accordingly.

Jonne Kats

Integrate xUnit tests into your daily Team Build

May 7, 2009

Comments