In the last article about Actor Model, we have shown how to implement this pattern by using Akka Actors.
Today we will see how to use Coyote, a framework implementing the Actor Model pattern but mainly used for testing.
Deterministic programs are easy to test because you know the program executes in the same way if provided the same input. Still, it could be more complex when they’re not: maybe a simple test cannot get all execution cases to find some error or perhaps an unexpected result.
In our scenario, we are developing a subsystem to manage product thumbnails on an e-commerce platform; our ProductsController contains the following method:
[HttpPut]
public async Task<ActionResult<Image>> CreateOrUpdateImageAsync(string productId, Image image)
{
var imageItem = image.ToItem();
await this.BlobContainer.CreateOrUpdateBlobAsync(“products”, productId, image.Content);
imageItem = await this.ImageContainer.UpsertItem(imageItem);
await this.MessagingClient.SubmitMessage(new GenerateProductThumbnailMessage()
{
ProductId = productId,
ImageStorageName = image.Name
});
return this.Ok(imageItem.ToImage());
}
In this method, firstly, we upload the image contents to Azure Storage using this.BlobContainer.CreateOrUpdateBlobAsync. Then we write the image metadata to an Azure Cosmos DB by using this.CosmosContainer.UpsertItem, and finally, the handler submits a GenerateProductThumbnailMessage to Azure Service Bus by calling this.MessagingClient.SubmitMessage, and returns an HTTP 200 Ok status code.
The generation of thumbnails could take some time, so we prefer to get it asynchronously with a background worker. We enqueue a request to generate a thumbnail in an Azure Service Bus messaging queue, which are then asynchronously dequeued by the worker service.
MessagingClient is the Azure Service Bus messaging queue interface that allows clients to submit asynchronous messages to the queue.
Let’s write a unit test to verify that the method runs properly:
[Test]
public static async Task TestProductImageUpdate()
{
var cosmosContainer = new CosmosContainerMock();
var blobContainer = new BlobContainerMock();
var productsControllerClient = new TestProductsControllerClient(cosmosContainer, blobContainer);
var productId = Guid.NewGuid().ToString();
var image = new Image()
{
Name = "image.jpg",
ImageType = "JPEG",
Content = new byte[] { 0, 1, 2, 3, 4 }
};
var result = await productsControllerClient.CreateOrUpdateImageAsync(productId, image);
if (result.StatusCode == HttpStatusCode.OK)
{
var imageResponse = await productsControllerClient.GetImageContentsAsync(productId);
Assert.IsTrue(imageResponse.StatusCode == HttpStatusCode.OK);
}
}
The test is formally correct and will always give a positive result, but is it enough to test our method?
The main questions are: what happens if operators update the product image two or more times?
What is the final result? How are we able to manage concurrency in our platform?
To answer these questions, we can use Coyote.
Coyote is an integrated tool, useful while developing asynchronous software, that helps you both with testing and developing, by providing high-level programming abstractions and support for writing detailed system specifications and a very effective high-coverage testing tool. We find testing with Coyote so useful because it helps to find concurrency bugs, and any of these bugs can be replayed. That testing tool can also be integrated with a .NET standard unit-testing framework.
L’Installation of the Coyote framework is easy because we just need to execute two commands:
dotnet install Microsoft.Coyote
dotnet install Microsoft.Coyote.Test
We can also install the dotnet coyote tool by executing the following command:
dotnet tool install --global Microsoft.Coyote.CLI
In this way, we can use the Coyote tool to perform rewriting and testing our DLL.
Thus, starting from the base test previously written, we write a concurrency test, using Coyote, to exercise the scenario where two requests race on updating the image of the same product:
[TestMethod]
public async Task TestConcurrentProductImageUpdate()
{
var cosmosState = new MockCosmosState();
var database = new MockCosmosDatabase(cosmosState);
var imageContainer = (MockCosmosContainer)await database.CreateContainerAsync(Constants.ImageContainerName);
var blobContainer = new MockBlobContainerProvider();
var messagingClient = new MockMessagingClient(blobContainer);
var productsControllerClient = new TestProductsControllerClient(imageContainer, blobContainer, messagingClient);
var productId = Guid.NewGuid().ToString();
var image1 = new Image()
{
Name = "image1.jpg",
ImageType = "JPEG",
Content = new byte[] { 0, 1, 2, 3, 4 }
};
var image2 = new Image()
{
Name = "image2.jpg",
ImageType = "JPEG",
Content = new byte[] { 5, 6, 7, 8, 9 }
};
var task1 = productsControllerClient.CreateOrUpdateImageAsync(productId, image1);
var task2 = productsControllerClient.CreateOrUpdateImageAsync(productId, image2);
await Task.WhenAll(task1, task2);
Assert.IsTrue(task1.Result.StatusCode == HttpStatusCode.OK);
Assert.IsTrue(task1.Result.StatusCode == HttpStatusCode.OK);
var imageResult = await productsControllerClient.GetImageAsync(productId);
Assert.IsTrue(imageResult.StatusCode == HttpStatusCode.OK);
byte[] image = imageResult.Resource;
byte[] thumbnail;
while (true)
{
var thumbnailResult = await productsControllerClient.GetImageThumbnailAsync(productId);
if (thumbnailResult.StatusCode == HttpStatusCode.OK)
{
thumbnail = thumbnailResult.Resource;
break;
}
}
Assert.IsTrue(image.SequenceEqual(thumbnail));
}
If we try to run this concurrent test, the assertion will most likely fail. The failure is not guaranteed because there are some interconnected tasks where it passes, and others where it fails.
It’s an important activity to configure mocks when writing unit tests because they have to simulate the actual behavior and to help to find out concurrency bugs.
For instance, our MockBlobContainer, introduced in the previous test, contains a dictionary of containers and some utility methods to add and delete containers:
internal class MockBlobContainerProvider : IBlobContainer
{
private readonly ConcurrentDictionary<string, ConcurrentDictionary<string, byte[]>> Containers;
internal MockBlobContainerProvider()
{
this.Containers = new ConcurrentDictionary<string, ConcurrentDictionary<string, byte[]>>();
}
public Task CreateContainerAsync(string containerName)
{
return Task.Run(() =>
{
this.Containers.TryAdd(containerName, new ConcurrentDictionary<string, byte[]>());
});
}
//…
public Task DeleteBlobAsync(string containerName, string blobName)
{
return Task.Run(() =>
{
this.Containers[containerName].TryRemove(blobName, out byte[] _);
});
}
}
So, the previous test is correct but probably useless, since it can only find bugs through sheer luck. If we modify the test by adding a delay of one millisecond between the two CreateOrUpdateImageAsync calls, and if we run this test in a hundred times, it probably won’t fail once:
var task1 = productsControllerClient.CreateOrUpdateImageAsync(productId, image1);
await Task.Delay(1);
var task2 = productsControllerClient.CreateOrUpdateImageAsync(productId, image2);
That’s because the race condition is still there, but our concurrency unit test suddenly became ineffective at catching it. Developers often write stress tests, where the system is bombarded with thousands of concurrent requests to find out some nondeterministic bug. Still, stress testing can be complex to set up and it doesn’t always find the trickiest bugs.
To use Coyote on your task-based program is very easy in most cases. All you need to do is to invoke the coyote tool to rewrite your assembly so that Coyote can inject logic that allows it to take control of the schedule of C# tasks. Then, you can invoke the coyote test tool which systematically explores interconnected tasks to uncover the bug. Even better is that if a bug is uncovered, Coyote allows you to reproduce it every single time deterministically.
Now we can run your test under the control of Coyote, but , as a first step, we need to rewrite our DLL.
Rewriting is a process that loads one or more assemblies and rewrites them for testing. The rewritten code maintains exact semantics with the production version, but has several stubs and hooks injected that allow Coyote to take control of concurrent execution and various sources of nondeterminism in a program.
To rewrite our DLL, we type:
coyote rewrite .\ProductManager.dll
This command injects logic that allows Coyote to take full control of the schedule of C# tasks and other sources of nondeterminism in your program.
Finally, we can invoke the coyote tool to test our concurrent program by systematically exploring different task inteconnections.
In our case we test the method TestConcurrentProductImageUpdate for 100 iterations.
test coyote .\ ProductManager.dll -m TestConcurrentProductImageUpdate -i 100
Coyote finds the race condition in the concurrent method almost immediately, in 3 test iterations.
... Elapsed 2.3226862 sec.19
... Testing statistics:20
..... Found 1 bug21
... Scheduling statistics:22
..... Explored 3 schedules: 3 fair and 0 unfair.23
..... Found 100.00% buggy schedules24
..... Number of scheduling points in fair terminating schedules 34 (min), 34(avg), 34 (max).25
In each test iteration, Coyote will execute your unit test from start to completion multiple times, each time exploring potentially different task inteconnections interleavings and other non-deterministic choices, to try to find bugs.
Perhaps more importantly, it outputs a .schedule file, which allows us to replay the exact same path and reproduce the concurrency bug.
After reproducing the bug, we can fix it.
In our case, we modify the CreateOrUpdateImageAsync method in this way:
[HttpPut]
public async Task<ActionResult<Image>> CreateOrUpdateImageAsync (string productId, Image image)
{
var imageItem = image.ToItem();
var uniqueId = Guid.NewGuid().ToString();
imageItem.StorageName = uniqueId;
await this.BlobContainer.CreateOrUpdateBlobAsync("products", imageItem.StorageName, image.Content);
imageItem = await this.ImageContainer.UpsertItem(imageItem);
await this.MessagingClient.SubmitMessage(new GenerateProductThumbnailMessage()
{
ProductId = productId,
ImageStorageName = imageItem.StorageName
});
return this.Ok(imageItem.ToImage());
}
To fix this bug, the controller should generate a unique GUID for each request. The image contents and thumbnail should use this GUID as a key when they are stored in the Azure Storage blob container, instead of a user-provided name.
This approach would guarantee that two CreateOrUpdateImageAsync handlers would never interfere with each other.
Conclusions
In this article, we showed you how to use Coyote to test a concurrent method.
We wish we raised your interest in the topic.
The sample project with the code used in this article is available here.
See you at the next article!
