Tag Archives: unit testing

Integration Testing your Asp .Net Core app with an in memory database

Parts:

  1. Integration testing your asp .net core app with an in memory database (this)
  2. Integration testing your asp .net core app dealing with anti request forgery csrf formdata and cookies

Revisions:

  • 14th august 2016 – updated for .net core 1.0
  • 29th april 2016 – first version covering RC1

Recently I am working with .Net (C#). And we’re working in .Net Core, which is awesome. (new stuff, wooh yeah!). I wanted to set up integration testing and it was tough to find resources to make it all happen. Which is pretty obvious considering how new some stuff is.

I found some articles scattered around this topic. But, there is not a full guide from “start till ‘full integration testing’ + in memory database”. So because of the lack of it, here is my take.

I couldn’t have made it this far without some notable resources (see below) and the answer to my Github question (with a friendly and very constructive response, thanks Asp.Net guys!).

Overview: What this blog post covers

  1. Setting everything up – your first integration test
  2. Run your tests against an in memory database + making sure the memory database has its tables set up.
  3. Then make it as fast as possible

Do note: An in memory database is NOT the same as your SQL Server. But if that is not bothering you (or not in these test cases), no worries there.

Step 1: First make it work – setting everything up

I assume you don’t have any integration test running yet. I am using xUnit. Read this well written article[#1] how to set up your integration test base. I summarise here quickly, but if you get stuck read the article. Then get back here.

Hook up your dependencies, here are mine (taken from project.json):

...
"dependencies": {
    ... // my project dependencies
    "FluentAssertions": "4.2.1",
    "xunit": "2.1.0",
    "xunit.runner.dnx": "2.1.0-rc1-build204",
    "Microsoft.AspNetCore.TestHost": "1.0.0",
  }
...

Within your test class, define:


	public TestServer server { get; }

	public HttpClient client { get; }

Then in the constructor of your test class:

var builder = new WebHostBuilder().UseStartup<Startup>();

server = new TestServer(builder);

client = server.CreateClient();

Now also create a test case. Something along the lines of:


[Fact]
public async void TestVisitRoot() {
    var response = await client.GetAsync("/");
    response.EnsureSuccessStatusCode();
}

This is basically the example from original article, but stripped down (where applicable).

Try running the test case first. It should run the app as if you ran it normally and it would visit the homepage. It also connects to your real database, webservices and whatnot.

Congrats, you completed step one. On to the next. Where we will be…

Step 2: Replacing database with an in-memory SQLite database

I assume you use a SQL Server in your ‘real world’ scenario. For integration testing you want to have a predictable state before running the test. An empty database is pretty predictable (after you fill it up with test data ;-)).

Also an in-memory database saves you the hassle of dealing with files, permissions, removing (temp) files, etc.

In order to inject our in memory database, we need to override the Startup class. We need to create a seam in our class so we can write our test-specific (ie override) database setup code there.

We start by creating a class TestStartup which extends from Startup. Then we make sure we use TestStartup in our constructor in our test class:

var builder = new WebHostBuilder()
	.UseStartup<TestStartup>() // use our testStartup version


In your Startup class you need a method where you are setting up your database. You probably do this within a Configure or ConfigureServices method. Instead of wiring up the database within that method, extract that code in a separate method and call it SetupDatabase.

The code in that method might look a bit like this:

public virtual void SetUpDataBase(IServiceCollection services)
{
	services
		.AddEntityFramework()
		.AddSqlServer()
		.AddDbContext<YourDatabaseContext>(options =>
			options.UseSqlServer(
				Configuration["Data:DefaultConnection:ConnectionString"]
		));
}

Make sure you define this method as virtual. This allows us to override it within our TestStartup class.

Before we override the method, we need to make sure we have the appropiate SQLite dependency defined in our project.json. So add it. This should make the dependencies look like:

...  
"dependencies": {
    ...
    "FluentAssertions": "4.2.1",
    "xunit": "2.1.0",
    "xunit.runner.dnx": "2.1.0-rc1-build204",
    "Microsoft.AspNetCore.TestHost": "1.0.0",
    "Microsoft.EntityFrameworkCore.InMemory": "1.0.0",
    "Microsoft.EntityFrameworkCore.Sqlite": "1.0.0",
  }

Now, in your TestStartup override method SetupDatabase and let it set up your SQLite in memory database:

public override void SetUpDataBase(IServiceCollection services)
{
	var connectionStringBuilder = new SqliteConnectionStringBuilder { DataSource = ":memory:" };
	var connectionString = connectionStringBuilder.ToString();
	var connection = new SqliteConnection(connectionString);

	services
		.AddEntityFrameworkSqlite()
		.AddDbContext<CmsDbContext>(
			options => options.UseSqlite(connection)
		);
}

Try running your app. See how it behaves.

You might run into problems where it complains about not having a database setup, or no tables being found. No worries, there are a few things left to do.

Ensure creation of database

At some place in your webapp you most likely create your dbContext, along the lines of:

//Create Database
using (var serviceScope = app.ApplicationServices.GetRequiredService<IServiceScopeFactory>()
	.CreateScope())
{
	var dbContext = serviceScope.ServiceProvider.GetService<YourDatabaseContext>();

	// run Migrations
	dbContext.Database.Migrate();
}

For making sure your in-memory database has a database setup (with tables, etc). In general you want to do this…:

//Create Database
using (var serviceScope = app.ApplicationServices.GetRequiredService<IServiceScopeFactory>()
	.CreateScope())
{
	var dbContext = serviceScope.ServiceProvider.GetService<CmsDbContext>();

	dbContext.Database.OpenConnection(); // see Resource #2 link why we do this
	dbContext.Database.EnsureCreated();

	// run Migrations
	dbContext.Database.Migrate();
}

Of course you want this only for integration tests. So don’t leave it like that. Again, create a seam. So you get:

// method in Startup.cs
public virtual void EnsureDatabaseCreated(YourDatabaseContext dbContext) {
	// run Migrations
	dbContext.Database.Migrate();
}

// within your Configure method:
using (var serviceScope = app.ApplicationServices.GetRequiredService<IServiceScopeFactory>()
	.CreateScope())
{
	var dbContext = serviceScope.ServiceProvider.GetService<YourDatabaseContext>();
	EnsureDatabaseCreated(dbContext);
}

And in your TestStartup you override it like so:

// method in TestStartup.cs
public override void EnsureDatabaseCreated(YourDatabaseContext dbContext) {
	dbContext.Database.OpenConnection(); // see Resource #2 link why we do this
	dbContext.Database.EnsureCreated();

	// now run the real thing
	base.Migrate(dbContext);
}

Same trick. Now re-run your test. It should work now. You could leave it like this. There are a few challanges up ahead, like dealing with cookies, anti-request forgery and so on. I might blog about those too.

Note: overriding like this might not be the only/best case after RC1, as there are changes that should make it way easier to add your own dependencies/setup that will be coming in RC2.

Now, the downside of integration tests is that b ooting them up is very slow compared to unit tests. So you want to do that only once (preferably) and then run all your tests. Yes, that also has downsides, your tests should be careful when sharing state throughout one webapp run. So make sure you keep your tests isolated.

Step 3: Speed up your integration tests! Use a TestFixture + xUnit collection

Inspired by another article[#3] and xUnit’s ability to use xUnit’s CollectionDefinition you can make sure your webapp is only booted once.

Sharing webapp between test cases using a TestFixture

This solves the problem: creating a web app for each test case.

To do this, in short, create a new class. For instance TestServerFixture. Move your client/server setup in this class. So it looks like this:

public class TestServerFixture : IDisposable
{
	public TestServer server { get; }

	public HttpClient client { get; }

    public TestServerFixture()
    {
		// Arrange
		var builder = new WebHostBuilder()
			.UseEnvironment("Development")
			.UseStartup<TestStartup>();
			// anything else you might need?....

		server = new TestServer(builder);

		client = server.CreateClient();
	}

	public void Dispose()
    {
		server.Dispose();
		client.Dispose();
    }
}

Note the differences, the testFixture implements an IDisposable interface. Your setup which was in your test class constructor, has moved to the constructor of the fixture.

Now, to make things easier (as you will create more and more integration test classes), create an abstract test class, which will be setup to receive the TestServerFixture. Then you can extend from this abstract class in your concrete test classes.

The abstract class would look like this:

public abstract class AbstractIntegrationTest : IClassFixture<TestServerFixture>
{
	public readonly HttpClient client;
    public readonly TestServer server;


    // here we get our testServerFixture, also see above IClassFixture.
    protected AbstractIntegrationTest(TestServerFixture testServerFixture)
	{
		client = testServerFixture.client;
		server = testServerFixture.server;
	}
}

As you can see we use an IClassFixture. Which is used for shared context between test cases..

This little boilerplate code will now allow us to get our concrete test class to look like:

public class MyAwesomeIntegrationTest : AbstractIntegrationTest
{
	public MyAwesomeIntegrationTest(TestServerFixture testServerFixture) : base(testServerFixture)
	{
	}

 [Fact]
public async void TestVisitRoot() {
    var response = await client.GetAsync("/");
    response.EnsureSuccessStatusCode();
}

// etc more tests...
}

Sharing your webbapp between test classes using xUnit’s CollectionFixture

This solves the problem: creating a web app (using a TestFixture) for each test class.

So awesome you have multiple test classes and you notice that you boot up your webapp everytime. And you want to solve this for (some) classes. Well that is possible. For obvious pro’s and con’s which I won’t dive into. (beware of state! ;-))

Setting up a CollectionFixture is also explained here. But for completeness sake, let me rephrase:

First create a class that we will use to define a collection. Like so:

[CollectionDefinition("Integration tests collection")]
public class IntegrationTestsCollection
{
	// This class has no code, and is never created. Its purpose is simply
	// to be the place to apply [CollectionDefinition] and all the
	// ICollectionFixture<> interfaces.
}

Now above all test classes you want to include in the Integration tests collection, simply add this line above the class definition:

[Collection("Integration tests collection")]

Which makes, in our above example, it like this:

[Collection("Integration tests collection")]
public class MyAwesomeIntegrationTest : AbstractIntegrationTest
{
	public MyAwesomeIntegrationTest(TestServerFixture testServerFixture) : base(testServerFixture)
	{
	}

 // your tests here...
}

Now run your tests again and note you boot up your webapps for each collection only once. Hence if you put everything in one collection, your webapp will only boot once.

Conclusion

We can run integration tests. If we want we can run them against an in memory database. Using seams we can inject our own test setup code, which probably changes in RC2 or further. If we want to speed up our tests we can put them in one “Integration collection” and use a single TestFixture.

Resources:

1. Asp.net docs – Integration testing
2. SQlite in memory database create table does not work
3. Fast testing
4. My original question/ticket at Github

First coderetreat of 2012 in Amsterdam – Retrospective

At the end of 2011 I started organizing a coderetreat. It started on twitter around October. I’ve also posted about it in my last mini blog. The original event can be found here.

If anyone was interested, they could sign up (max 25 people) for free. All you needed to do was bring your best humor and if possible a laptop with your preferred dev environment set up. (Its not hard to organize one, check here if you’re interested)

If you want to know more about what a coderetreat is, click here. Even better: join a coderetreat somewhere near you and experience it. It is way better than just reading about it 🙂

Honing the craft together

Coderetreat

Lets start with a management summary:

It was awesome!

It reminded me of my experience with the bowling game kata last year. Since you’re repeating the exercise over and over again, you will find different approaches. Even better, because you’re switching pairs, you will have a different mindset literally to approach the problem presented by the coderetreat. Instead of writing a bowling game, you will be working on Conway’s Game Of Life.

The most notable things of that day where:

  • In the very first session we where let ‘free’. We could approach this problem how we wanted. Me and my pair where able to implement the first three rules. However we where not able to implement the fourth rule. Our design was not easy enough to revive dead cells. (gosh, this reminds me of the bowling game code kata first attempt…)
  • The second session we got to choose from different constraints. I picked the “no conditionals” one, because I can get my methods under 4 lines without pain. Programming without no conditions is a whole different story though.
  • The third session with ‘only check in within 2 minutes, else revert everything’ was an eye opener! It really forced you into thinking how to make all (baby) steps. Hence, I am using this at work now and it really works. I commit 10 times more often. Although I don’t make the 2 minute mark yet at work (5 minutes is easy though now).
  • The fourth session was fun, as we where able to implement *all rules* (opposed to the first session), but without the code we had implemented in the first session! We totally isolated the behaviour (this session was called “tdd as if you meant it”) and it blew our minds.

Will I attend more coderetreats? You bet! Just need to take a look at the list of events and pick an appropiate one. If I attend one, I will let you know (on twitter surely, perhaps on this blog).

If you want to know how it looked like, click here to see som pictures of the coderetreat.

I loved the coderetreat, and I’ll surely organize one again in the future. I would recommend anyone who loves his profession to join a coderetreat and practice. You’ll learn new things for sure!

How hard can it be, right? 😉

The ‘unit’ in unit testing; and kinds of unit tests.

Recently I had the oppertunity to explain some students about what unit testing was. I started off with the question of “What does unit testing mean?”. They gave different types of answers. One of them talked about the smallest piece of code. And even though he is ‘right’. I asked him to apply this knowledge to his current code where he said “But I don’t want to test my get/set methods, that is useless!”. And so, our definition of ‘unit testing’ became unclear again.

So what is a unit test? According to this article on wikipedia “A unit is the smallest testable part of an application”. But what does that mean? What is the ‘smallest testable part’? Do you need to test get/set methods? Do you need to test assigning values? Yes and no.

Even though the smallest testable part is related to lines of code, I believe it also is related to behavior. I think a better description for unit testing would be to test the smallest piece of behavior of an application. Most of the time you probably will be testing very little pieces of code with it. As long as there is some behavior (context) you want to test.

So do get/set methods fall into this category? Depends… Get/Set methods in themselves are, without context, useless. Returning a value or setting a value is not much worth of testing. However, if you test a method that calls a get/set method and does something with it, that is another story. When code is executed within a certain context, it is perfectly valid to unit test it. An example: getTotalPrice that just returns a value “price” is not worth to be testing. A getTotalPrice that does some calculation is a good candidate to test!

With software development, it is the expected behavior of the software that matters. I compare this with designing interfaces (seperating what and how), where you’re busy thinking what you would expect from a certain object when using it. The behavior of the software; how it presents itself and how the user can interact with it (the how), is different from how the software realizes this behavior (the what). When thinking in terms of behavior when writing code, we force ourselves to think of what the software should be doing, and not *how* it should be coded to do that. Test Driven Development is a way of forcing you to write code that has a good design. It clearly a functional (behavior) point of view.

With that said, a unit test should aim to test the smallest possible piece of behavior. When a total price is shown. There are a few steps taken before it is shown on the screen. One of the important behaviors is that the totalprice is calculated somewhere. Testing the calculation of the totalprice is a unit test. This is a State based test. (Does the class give expected output X when using input Y?)

The totalPrice is calculated from certain input. The output is shown on the screen.
A controller class is putting input and output together, making sure the totalPrice is being calculated and pushed to the view. Making sure that the controller uses its collaborators to do that is also a unit test. This is called a collaboration test. (Does my class call the collaborators that I expect, with the parameters I expect?, yes you do that by Mocking)

Even though with these two tests, you still miss an important type of test. You still need to test if your interfaces behave as they say they should. Should you test interfaces of external (3rd party) API’s? Probably not. Unless you have a reason to distrust your supplier of the API. Testing interfaces are called Contract Tests and look a lot like State based tests. These tests are making sure your expectations about the interfaces are validated! If you don’t do that you will get defects, even though you test states and behaviors.

Last but not least. It is useful to test broader pieces of behavior. For example, if you use Spring to bootstrap your webapp. You probably need to make sure everything is autowired correctly. That is an integration test. There are multiple types of integration tests. An often used integration test is to connect to a database, put data in, fire up some methods and test their behavior. All of them are integration tests. Hence, even testing whole pages (end-to-end tests) using Selenium, those are also integration tests, although at a much higher level.

In time I will blog more about the type of unit tests.

EasyMock and Mockito, a little comparison

Someone mentioned a mocking framework called Mockito some time ago to me. I am familiar with mocking frameworks, as I work with EasyMock quite a lot. I really like EasyMock, but I am curious about Mockito. I thought of trying it out a bit and write down my experiences along with a comparison with EasyMock.

This is by all means not a very in-depth comparison, but I did find out some interesting things.

In order to test out the mocking frameworks I have set up some code in place to test. The code to be tested is an ‘itemRepository’ (It is however, not a full implementation of the Repository Pattern). Typically you use it to get some stuff (Items). In this case the Item is retrieved from a Dao object used by the repository. Just for easiniess the DAO returns the same object as the repository. (The repository pattern makes a distinction between the object the DAO returns and the Domain objects the Repository should return).

Here is the code of the ItemRepository:
[sourcecode language=”java”]
public class ItemRepositoryImpl implements ItemRepository {

private ItemDao itemDao;

@Override
public Item getItem(String someInput) {
return itemDao.getById(Long.valueOf(someInput));
}

public void setItemDao(ItemDao itemDao) {
this.itemDao = itemDao;
}

public ItemDao getItemDao() {
return itemDao;
}

}
[/sourcecode]

My goal is to test the ItemRepository’s getItem method. And all I care about is that the itemDao is correctly called with the correct parameters. Hence this is what a collaboration test is all about.

Lets start with my comfort zone: EasyMock

The test class I write talks to interfaces. I find it a good habit to talk against interfaces and not concrete implementations. Doing that however, forces me to cast to an implementation to set collaborators in the tests. Normally such things happen in the @Before. But since I will also create a test with the Mockito framework I have it done in the test method itself.

The begin of the test:
[sourcecode language=”java”]
public class ItemRepositoryTest {

private ItemRepository itemRepository;
private ItemDao itemDao;

@Before
public void setUp() {
itemRepository = new ItemRepositoryImpl();
}
[/sourcecode]

In line of the tripple A testing guidelines I have an arrange, act and assert:
[sourcecode language=”java”]
@Test
public void mustReturnItemFromMockedItemDao_EasyMock_Behaviour() {
ItemRepositoryImpl impl = getItemRepository();
itemDao = EasyMock.createMock(ItemDao.class);
impl.setItemDao(itemDao);

// this is weird: I don’t *want* to bother about returning stuff here, just that it is called!
// mocking and stubbing is combined here
EasyMock.expect(itemDao.getById(1L)).andReturn(null);
EasyMock.replay(itemDao);

// Act
itemRepository.getItem("1");

// Verify
EasyMock.verify(itemDao);
}
[/sourcecode]

As you can see I have pointed out one thing of EasyMock’s behaviours with a comment. Whenever you mock a method that has a return type, EasyMock expects you to fill in that behaviour. If you don’t do that, you’ll end up with an exception like this:

java.lang.IllegalStateException: missing behavior definition for the preceding method call getById(1)

In this case I have ‘solved’ it by returning null. While writing the test, I did not want to be bothered about it, but I have to satisfy EasyMock. Doing so reduces the readability of my code. The intention of the code remains unclear at this point, and that’s something I’d like to avoid as much as I can.

Another well known behaviour of EasyMock is the need to replay your mock after you have set the expectations. Using a ‘recorder’ metaphor it sounds reasonable. But basically you’re writing the behaviour ‘twice’: First you write down the expectations, then you actually run them. Writing the expectations is often found to be a cumbersome job to do, especially with multiple collaborators.

And of course at the end it all needs to be verified. Which is done by EasyMock.verify. We could even check the outcome. Most of the time you’re tempted to do so, you had to write the return value anyway. Better test if the return value being returned is the same as the method returns right? I think not. Reason: your intention of your test becomes unclear if you assert this as well. And as a bonus you introduce your tests to be more fragile. The unit test becomes now a collaborator test and a value-based test in one. Mockito makes this distinction very clear.

Mockito:
[sourcecode language=”java”]
@Test
public void mustReturnItemFromMockedItemDao_Mockito_Behaviour() {
// Arrange
ItemRepositoryImpl impl = getItemRepository();
itemDao = Mockito.mock(ItemDao.class);
impl.setItemDao(itemDao);

// Act
itemRepository.getItem("1");

// Assert / verify
Mockito.verify(itemDao).getById(1L);
}
[/sourcecode]

Plus:
– no need to mock return etc, as we dont care
– seperation of mocking and stubbing (stubbing is done by using the static method when, mocking by using verify).

So how about stubbing in Mockito? Here is how you stub the itemDao:

[sourcecode language=”java”]
@Test
public void mustReturnItemFromMockedItemDao_Mockito_Stubbing() {
// Arrange
ItemRepositoryImpl impl = getItemRepository();
itemDao = Mockito.mock(ItemDao.class);
impl.setItemDao(itemDao);

Mockito.when(itemDao.getById(1L)).thenReturn(null);

// Act
itemRepository.getItem("1");
}
[/sourcecode]

For more information about Mockito, they also have a more extensive comparison between EasyMock and Mockito.

Do you have any experience using Mockito and EasyMock and you would like to share some insights? Please leave a comment.

Someone pointed at RhinoMocks also being available for Java. I’m not so sure about that. However, if you do know about a Java variant, let me know so I can compare it as well.