Mathias Brandewinder on .NET, F#, VSTO and Excel development, and quantitative analysis / machine learning.
22. March 2014 13:11

A couple of days ago, I got into the following Twitter exchange:

So why do I think FsCheck + XUnit = The Bomb?

I have a long history with Test-Driven Development; to this day, I consider Kent Beck’s “Test-Driven Development by Example” one of the biggest influences in the way I write code (any terrible code I might have written is, of course, to be blamed entirely on me, and not on the book).

In classic TDD style, you typically proceed by writing incremental test cases which match your requirements, and progressively write the code that will satisfy the requirements. Let’s illustrate on an example, a password strength validator. Suppose that my requirements are “a password must be at least 8 characters long to be valid”. Using XUnit, I would probably write something along these lines:

namespace FSharpTests

open Xunit
open CSharpCode

module Password validator tests =

[<Fact>]
let length above 8 should be valid () =
let validator = Validator ()

… and in the CSharpCode project, I would then write the dumbest minimal implementation that could passes that requirement, that is:

public class Validator
{
{
return true;
}
}

Next, I would write a second test, to verify the obvious negative:

namespace FSharpTests

open Xunit
open CSharpCode

module Password validator tests =

[<Fact>]
let length above 8 should be valid () =
let validator = Validator ()

[<Fact>]
let length under 8 should not be valid () =
let validator = Validator ()

This fails, producing the following output in Visual Studio:

… which forces me to fix my implementation, for instance like this:

public class Validator
{
{
{
return false;
}

return true;
}
}

Let’s pause here for a couple of remarks. First, note that while my tests are written in F#, the code base I am testing against is in C#. Mixing the two languages in one solution is a non-issue. Then, after years of writing C# test cases with names like Length_Above_8 _Should_Be_Valid, and arguing whether this was better or worse than LengthAbove8 ShouldBeValid, I find that having the ability to simply write “length above 8 should be valid”, in plain old English (and seeing my tests show that way in the test runner as well), is pleasantly refreshing. For that reason alone, I would encourage F#-curious C# developers to try out writing tests in F#; it’s a nice way to get your toes in the water, and has neat advantages.

But that’s not the main point I am interested here. While this process works, it is not without issues. From a single requirement, “a password must be at least 8 characters long to be valid”, we ended up writing 2 test cases. First, the cases we ended up are somewhat arbitrary, and don’t fully reflect what they say. I only tested two instances, one 7 characters long, one 8 characters long. This is really relying on my ability as a developer to identify “interesting cases” in a vast universe of possible passwords, hoping that I happened to cover sufficient ground.

This is where FsCheck comes in. FsCheck is a port of Haskell’s QuickCheck, a property-based testing framework. The term “property” is somewhat overloaded, so let’s clarify: what “Property” means in that context is a property of our program that should be true, in the same sense as mathematically, a property of any number x is “x * x is positive”. It should always be true, for any input x.

Install FsCheck via Nuget, as well as the FsCheck XUnit extension; you can now write tests that verify properties by marking them with the attribute [<Property>], instead of [<Fact>], and the XUnit test runner will pick them up as normal tests. For instance, taking our example from right above, we can write:

namespace FSharpTests

open Xunit
open FsCheck
open FsCheck.Xunit
open CSharpCode

module Specification =

[<Property>]
let square should be positive (x:float) =
x * x > 0.

Let’s run that – fail. If you click on the test results, here is what you’ll see:

FsCheck found a counter-example, 0.0. Ooops! Our specification is incorrect here, the square value doesn’t have to be strictly positive, and could be zero. This is an obvious mistake, let’s fix the test, and get on with our lives:

[<Property>]
let square should be positive (x:float) =
x * x >= 0.

More...

26. January 2013 19:08

Phil Trelford recently released Foq, a small F# mocking library (with a very daring name). If most of your code is in F#, this is probably not a big deal for you, because the technique of mocking isn’t very useful in F# (at least in my experience). On the other hand, if your goal is to unit test some C# code in F#, then Foq comes in very handy.

So why would you want to write your unit tests in F# in the first place?

namespace CodeBase
{
using System;

public class Translator
{
public const string ErrorMessage = "Translation failure";

public Translator(ILogger logger, IService service)
{
this.logger = logger;
this.service = service;
}

public string Translate(string input)
{
try
{
return this.service.Translate(input);
}
catch (Exception exception)
{
this.logger.Log(exception);
return ErrorMessage;
}
}
}

public interface ILogger
{
void Log(Exception exception);
}

public interface IService
{
string Translate(string input);
}
}
We have a class, Translator, which takes 2 dependencies, a logger and a service. The main purpose of the class is to Translate a string, by calling the service. If the call succeeds, we return the translation, otherwise we log the exception and return an arbitrary error message.

This piece of code is very simplistic, but illustrates well the need for Mocking. If I want to unit test that class, there are 3 things I need to verify:

• when the translation service succeeds, I should receive whatever the service says is right,
• when the translation service fails, I should receive the error message,
• when the translation service fails, the exception should be logged.

In standard C#, I would typically resort to a Mocking framework like Moq or NSubstitute to test this. What the framework buys me is the ability to create cheaply a fake implementation for the interfaces, setup their behavior to whatever my scenario is (“stubbing”), and in the case of the logger, where I can’t observe through state whether the exception has been logged, verify that the proper call has been made (“mocking”).

This is how my test suite would look:

namespace MoqTests
{
using System;
using CodeBase;
using Moq;
using NUnit.Framework;

[TestFixture]
public class TestsTranslator
{
[Test]
public void Translate_Should_Return_Successful_Service_Response()
{
var input = "Hello";
var output = "Kitty";

var service = new Mock<IService>();
service.Setup(s => s.Translate(input)).Returns(output);

var logger = new Mock<ILogger>();

var translator = new Translator(logger.Object, service.Object);

var result = translator.Translate(input);

Assert.That(result, Is.EqualTo(output));
}

[Test]
public void When_Service_Fails_Translate_Should_Return_ErrorMessage()
{
var service = new Mock<IService>();
service.Setup(s => s.Translate(It.IsAny<string>())).Throws<Exception>();

var logger = new Mock<ILogger>();

var translator = new Translator(logger.Object, service.Object);

var result = translator.Translate("Hello");

Assert.That(result, Is.EqualTo(Translator.ErrorMessage));
}

[Test]
public void When_Service_Fails_Exception_Should_Be_Logged()
{
var error = new Exception();
var service = new Mock<IService>();
service.Setup(s => s.Translate(It.IsAny<string>())).Throws(error);

var logger = new Mock<ILogger>();

var translator = new Translator(logger.Object, service.Object);

translator.Translate("Hello");

logger.Verify(l => l.Log(error));
}
}
}

More...

13. November 2011 11:03

We were doing some pair-programming with Petar recently, Test-Driven Development style, and started talking about how figuring out where to begin with the tests is often the hardest part. Petar noticed that when writing a test, I was typically starting at the end, first writing an Assert, and then coding my way backwards in the test – and that it helped getting things started.

I hadn’t realized I was doing it, and suspected it was coming from Kent Beck’s “Test-Driven Development, by Example”. Sure enough, the Patterns section of the book lists the following:

Assert First. When should you write the asserts? Try writing them first.

So why would this be a good idea?

I think the reason it works well, is that it helps focus the effort on one single goal at a time, and requires clarifying what that goal is. Starting with the Assert forces you to imagine one single fact that should be true once you have implemented the feature, and to think about how you are going to verify that the feature is indeed working.

Once the Assert is in place, you can now write the story backwards: what is the method that was called to get the result being checked, and  where does it belong? What classes and setup is required to make that method call? And, now that the story is written, what is it really saying, and what should the test method name be?

In other words, begin with the Assert, figure out the Act part, Arrange the actors, and (re)name the test method.

I think what trips some people is that while a good test will look like a little story, progressing from a beginning to a logical end, the process leading to it follows a completely opposite direction. Kent Beck points the Self-Similarity in the entire process: write stories which describe what the application will do once done, write tests which describe what the feature does once the code is implemented, and write asserts which will pass once the test is complete. Always start with the end in mind, and do exactly what it takes to achieve your goal.

12. December 2010 17:00

I recently began playing with ASP.NET MVC (never too late), and so far I really enjoy it. One aspect I really appreciate is its testability – I can write fairly straightforward unit tests to verify that the application behaves as I believe it should, as well as make sure I understand what is going on.

One point which got me stumped was how to test for authorization. A controller, or some of its methods, can be decorated with the attribute [Authorize], restricting users who can access the method by role or name. In the default ASP.NET MVC 2 template, when a user isn’t authorized to a specific area, he gets re-directed to the LogOn method on the AccountController.

So far so good. However, I ran into unexpected issues when I attempted to unit test that. Suppose there are two roles in our web application, Chicken and Pigs, and that we have a Controller that leads to a pigs-only area of the web site:

[Authorize(Roles = "Pig")]
public class PigsOnlyController : Controller
{
public ActionResult Index()
{
return View("Index");
}
}

My first thought was to mock the ControllerContext and do something along these lines:

[Test]
public void OnlyPigsShouldAccessIndex()
{
var context = new Mock<ControllerContext>();
context.SetupGet(p => p.HttpContext.Request.IsAuthenticated).Returns(true);
context.Setup(p => p.HttpContext.User.IsInRole("Pig")).Returns(true);

var controller = new PigsOnlyController();
controller.ControllerContext = context.Object;

// check what controller.Index() returns
}

However, while the web application itself behaved properly (Users in the Pig role get to the Index page, whereas Chicken get redirected to the Logon page), the test wasn’t doing what I expected: both Pigs and Chicken were happily reaching the Pigs-Only area.

As is often the case, I found out why on StackOverflow; the reason is that the re-direction is not the responsibility of the Controller. If the controller is properly decorated, the Index method won’t even be invoked, and where the call gets redirected to is handled in a different part.

So how do you unit test this behavior? In this case, we trust the framework to handle the redirection, so the only functionality we need to ascertain is that the Controller has the proper attribute.

Instead of validating the redirection, we can verify that the Controller class has an Authorize attribute, with the proper roles specified:

[Test]
public void ControllerShouldAuthorizePigsButNotChicken()
{
var information = typeof(PigsOnlyController);
var attributes = information.GetCustomAttributes(typeof(AuthorizeAttribute), false);
Assert.AreEqual(1, attributes.Length);

var authorization = attributes[0] as AuthorizeAttribute;
var authorizedRoles = authorization.Roles;

var roles = authorizedRoles.Split(',');
Assert.IsTrue(roles.Contains("Pig"));
Assert.IsFalse(roles.Contains("Chicken"));
}

I am not totally happy with the hard-coded strings “Pig” and “Chicken” in the test, but I don’t see a way around it; maybe it’s a sign that this test is more of an integration test than a unit test? If you know of a better way to test for that aspect, I am all ears!

6. November 2010 16:21

The question of how to unit test VSTO projects has been bugging me (terrible pun, I know) for a while. I am a big fan of automated tests, if only because they remind you when new features you added wrecked havoc in existing functionality. But whenever I work on Excel VSTO add-ins, I end up writing very little tests, because, quite frankly, these projects are a massive pain to test. Excel behaves in many respects both like a user interface and a database, two notoriously hard-to-test areas – and on top of that, you cannot directly instantiate and tear down the add-in, because that happens through Office.

I am still very far from a satisfactory solution, but recently I began organizing my projects differently, and this is showing some potential. I limit as much as possible the role of the add-in project itself, and move the application logic, including interactions with Excel, into a separate project, using the add-in only for "quick-starting” things. The benefit is that unlike the add-in itself, the other project is perfectly suitable for unit testing.

As an illustration, imagine that your add-in, among other things, kept track of the current worksheet you are in, as well as the previous worksheet that was active. You could implement that along these lines:

{
public Worksheet PreviousSheet
{
get; set;
}

public Worksheet CurrentSheet
{
get; set;
}

private void ThisAddIn_Startup(object sender, System.EventArgs e)
{
var excel = this.Application;
excel.SheetActivate += SheetActivated;
excel.SheetDeactivate += SheetDeactivated;
}

private void SheetDeactivated(object sheet)
{
var worksheet = sheet as Worksheet;
this.PreviousSheet = worksheet;
}

private void SheetActivated(object sheet)
{
var worksheet = sheet as Worksheet;
this.CurrentSheet = worksheet;
}

You could easily debug that project and check that it works; however, you won’t be able to write an automated test for that.

More...