In my previous article, we have seen how testing and refactorizing legacy code with Golden Master Pattern. Now, we will continue to talk about testing, introducing a new paradigm with a simple example.

We are familiar with writing tests based on examples. In literature we talk about “Example test”: given an input we have an expected output.

The spread of techniques like TDD and the growing relevance of tests in software building could create a misunderstanding, that a good test is such if it maximizes the coverage. Writing test for each statement, doesn’t make our codebase error-free. Indeed, the test development time become too long and makes it difficult to maintain them when we have to add or modify a feature.

Kent Beck spoke on this topic on Stack Overflow.

Let’s take a simple case of study. Our company is conducting many interviews for developer position and, to carry out first skim, we assign candidates a kata.

Manually check the result involves big effort, so, we develop a test suite that checks automatically the proposed solutions, in order to reduce as many as possible our work. You know that laziness is one of the virtues of the developer: Laziness is a virtue!

The kata we assign to out candidates is a classic: The FizzBuzz. It comes to write an algorithm that, given a number, prints:

• “Fizz” for multiples of three.
• “Buzz” for multiples of five.
• “FizzBuzz” for multiples of both.

The simplicity of exercise give us the opportunity to focus on concepts, without worrying about implementation problem. In the fsharp branch of the repo you can find the complete solution in F#.

Let’s write a test suite with 100% of coverage using TDD with example based tests.

[<Tests>]
let tests =
      testList "Example tests" [
      testCase "Three should be Fizz" <| fun _ ->
      Expect.equal (FizzBuzz.fizzbuzz 3) "Fizz" "Is not Fizz."
      ]

The simple implementation:

module FizzBuzz  =
      let fizzbuzz (x:int) = "Fizz"

Let’s add more tests:

testCase "Five should be Buzz" <| fun _ ->
      Expect.equal (FizzBuzz.fizzbuzz 5) "Buzz" "Is not Buzz."

The implementation becomes:

let fizzbuzz (x:int) =
      match x with
      | 3 -> "Fizz"
      | 5 -> "Buzz"
      | _ -> string x // Complete pattern matching

Let’s go on faster and add the remaining tests covering all possible cases.

testCase "Nine should be Fizz" <| fun _ ->
      Expect.equal (FizzBuzz.fizzbuzz 9) "Fizz" "Is not Fizz."
 
testCase "Twentyi-Five should be Buzz" <| fun _ ->
      Expect.equal (FizzBuzz.fizzbuzz 25) "Buzz" "Is not Buzz."
 
testCase "Fifteen should be FizzBuzz" <| fun _ ->
      Expect.equal (FizzBuzz.fizzbuzz 15) "FizzBuzz" "Is not FizzBuzz."

In this way we got the 100% of statement coverage!

We deliver the exercises to the candidates and, at the end, we launch the suite on the received papers. Among the various solutions proposed, the following stands out:

module FizzBuzz  =
      let fizzbuzz (x:int) =
      match x with
      | 3 -> "Fizz"
      | 5 -> "Buzz"
      | 9 -> "Fizz"
      | 15 -> "FizzBuzz"
      | 25 -> "Buzz"
      | _ -> string x

The solution is obviously wrong, even if all the tests are passed.

We must therefore make our tests more robust. We add a test with random numbers generated at runtime. Here is a possible implementation:

let actualList =
      randomList |> List.map (FizzBuzz.fizzbuzz)
 
let expectedList =
      randomList
      |> List.map (fun i ->
      match i with
      | n when i % 3 = 0 && i % 5 = 0 -> "FizzBuzz"
      | n when i % 3 = 0 -> "Fizz"
      | n when i % 5 = 0 -> "Buzz"
      | _ -> string i)
 
testCase "Random number test"
      <| fun _ -> Expect.sequenceEqual actualList expectedList "Not equal"

The problem in using random numbers in tests comes out when we have to generate the expected results. To build it we have introduced the algorithm solution in tests.

Let’s introduce now the concept of Property Based Testing become famous in function programming with QuickCheck for Haskell. The basic idea is straightforward: rather than using example for tests, we test property.

A property can be tested with random inputs and covers even borderline cases. If the solution is correct the property is always valid; in this way we don’t need to build expected result.

To implement property tests, let’s use FsCheck, a F# porting of QuickCheck. In FsCheck the data for test will be generated from a Generator module, which is able to build random data. As we will see, you can define a custom Generator and use it in tests.

In addition to the Generator, another feature of Fscheck is the Shrinker that comes into play when a test fails. The Shrinker is able to return the least possible input for which the test may fail.

To implement the test suite, the first step is to define algorithm’s properties:

• The multiples of three should contain Fizz.
• The multiples of five should contain Buzz.
• The both multiples should be FizzBuzz.
• The non multiple of three or five should be the same.

Once the properties have been defined, we can develop our Generator. Let’s see how to define a multiples of three generator.

let multipleOfThree n = n * 3
 
let gen3 =
      Arb.generate<NonNegativeInt>
      |< Gen.map (fun (NonNegativeInt n) -> multipleOfThree n)
      |< Arb.fromGen

The Arb module lets you generate automatically a sequence of random number: it specifies the type NonNegativeInt the module generates only non negative numbers.

With the Gen module we can execute operations on Generator: initialize, filter, conver ecc.

We has used the function Gen.map for mapping the random number generated by Arb for obtain multiples of three.

A this point, the generator has to be registered before we can use it in test. To do that, we must specify a new type.

type ThreeGenerator =
      static member ThreeMultiple() =
      Arb.generate<NonNegativeInt>
      |> Gen.map (fun (NonNegativeInt n) -> multipleOfThree n)
      |> Gen.filter(fun n->  n> 0)
      |> Arb.fromGen

and register it using, for example, the library Expecto (often used together with FsCheck)

let multipleOfThreeConfig =
      { FsCheckConfig.defaultConfig with
              arbitrary = [ typeof<ThreeGenerator> ] }

At this point we can write the test using our Generator

let tests =
      testList "Property based tests"
      [ testPropertyWithConfig multipleOfThreeConfig "Multiple of three should contain Fizz"
      <| fun x -> Expect.containsAll (FizzBuzz.fizzbuzz x) "Fizz" "Not contain Fizz" ]

In the same way, we can write the remaining generators.

let multipleOfFive n = n * 5
let multipleOfBoth n = n * 15
let noMultiple n = (multipleOfBoth n) - 1
 
type FiveGenerator =
      static member FiveMultiple() =
      Arb.generate<NonNegativeInt>
      |> Gen.map (fun (NonNegativeInt n) -> multipleOfFive n)
      |> Gen.filter (fun n -> n > 0)
      |> Arb.fromGen
 
type BothGenerator =
      static member BothMultiple() =
      Arb.generate<NonNegativeInt>
      |> Gen.map (fun (NonNegativeInt n) -> multipleOfBoth n)
      |> Arb.fromGen
 
type NoMultipleGenerator =
      static member BothMultiple() =
      Arb.generate<NonNegativeInt>
      |> Gen.map (fun (NonNegativeInt n) -> noMultiple n)
      |> Arb.fromGen

Testing property give us the ability to use random input, this input can cover also edge cases with no need to build expected value.

We have seen how we can obtain robust test suites for lazy developers, just by changing the paradigm used to the code!

You can find references below

https://fsharpforfunandprofit.com/posts/property-based-testing

https://fsharpforfunandprofit.com/posts/property-based-testing-2

https://fscheck.github.io/FsCheck/TestData.html

I hope I’ve intrigued you.
See you at the next article!