There are those who believe that a developer should not test their own code. This may sound logical, but I’m not sure I’m buying it.

This statement typically refers to QA testing, and doesn’t mean that a developer shouldn’t write unit tests. The thinking here is that a second person testing the features that you’ve developed might think of things a different way and find a problem that you didn’t think of when you wrote the code.

There are lots of commonly accepted statements and ideas like this in software development. But I’ve found that in many cases, these ideas are usually based on certain assumptions, and if you can challenge those assumptions, you might open yourself up to things that you didn’t think were possible.

The assumption that I see here is that a developer writing the code is not sufficiently capable of thinking of all of the test cases. Imagine you write code for a feature, and now you have to test it. At this point, you’ve gone through a certain mental thought process when you implemented the feature. This makes it much harder to think outside of the box to come up with the edge cases. Not to mention that when you see that the feature appears to be working overall, it’s really really tempting to do some basic manual testing and then move on to the next feature without really doing your due diligence. A independent QA tester, however, will look at the feature objectively because their thought process isn’t clouded by past experience of having written the code.

OK, so what if the developer figured out all of the tests cases before writing the code? Now their thinking isn’t clouded by the implementation of the feature because they haven’t wrote the code yet. Maybe a QA person helps define the test cases, but this post is about developers testing their own code, so let’s assume that QA people aren’t involved. I would argue that now we’ve removed the main reason that developers are not good at testing their own code (thinking of the test cases after writing the code), so they should be able to think of test cases just as well as a QA tester, and therefore they should be able to test their own code.

Don’t misconstrue what I’m saying there – I’m not saying that we don’t need our QA teams. I’m saying that developers need to be responsible for testing. QA teams can add more testing help, but developers need to be responsible for their own code.

This opens you up to new possibilities. It enables developers to be confident about the quality of their code. It limits the wasted time you incur when you have the back-and-forth that comes with QA finding bugs and developers having to go back and fix things. It can reduce the amount of “checking” that QA people need to do because they might be comfortable knowing that developers are writing quality code.

If you’re a developer, this is some thing you can start doing today. Before you implement a feature, come up with all of the test scenarios before you write your code. If you have a QA team, have them review your test cases to see if you’ve missed anything. Then go write some bug free code!

Lately I’ve been thinking about a whole team approach to testing, where we decide as a team how features will be tested and where we use the skillsets of the whole team to automate testing. We do this on our project, and this has led to a regression testing suite of ~2500 SpecFlow acceptance tests that automate almost all scripted QA testing and regression testing for our application.

We didn’t always do this. Originally there was no automated acceptance testing, but developers were diligently writing unit tests. Those unit tests are still around, but we don’t write many unit tests anymore. We start with acceptance tests now, and the acceptance tests cover all of the testing scenarios that need to be covered. Our application has well-defined design patterns that we follow, so the idea of TDD driving the design of our code doesn’t really apply. If the unit tests fail, we often just delete them because it’s not worth fixing all of the mocks in the unit tests that are causing them to fail, and we have acceptance testing coverage around all of it.

This approach does not line up with the conventional wisdom on automated testing. They say that you’re supposed to write lots of unit tests that run really fast to give you fast feedback, help design your code, and ensure the internal quality of your code. In the past, this is how I’ve always done it. In fact, many of them dislike Cucumber.

Cucumber makes no sense to me unless you have clients reading the tests. Why would you build a test-specific parser for English?

— DHH (@dhh) March 29, 2011

While TDD isn’t as mainstream as I would like, TDD is nothing new. Kent Beck was writing about it 10 years ago, and the original XP guys valued such things as unit testing, the SOLID principles, and things like that.

Automated acceptance testing still feels like a relatively new phenomenon. I’m sure people were doing it 10 years ago, but back then we didn’t have Cucumber and SpecFlow and the Gherkin language. Now I see a lot more people using tools like that to automate QA testing in way that uses business language and more maintainable code, rather than the old “enterprise” solutions like QTP.

Here’s what I’m getting at – I wasn’t there 10 years ago when Kent Beck was writing his books and the XP movement was starting, but it seems to me to be primarily an effort by developers to ensure the quality of their code through the effort of developers. I see very little talk of where QA fits into that process. There is some mention of QA for sure, but the general gist seems to be that developers need to write tests in order to ensure quality, and the best way to do that is to write unit tests. QA people typically don’t say that unit testing is enough because it doesn’t test end-to-end, so then what do they do? Manually test? Use QTP?

My question is this – if we think of testing as whole-team activity and not just a QA activity or a developer activity, will we arrive at the same conclusions as we did before?

I’m not ready to discount unit testing as a valuable tool, and I’m also not ready to say that everyone should do it my way because it worked for us on one project. But we have largely abandoned unit testing in favor of acceptance testing, and other teams in our department are doing it too. I write unit tests for things like extension methods and some classes that have important behavior on their own and I want to ensure that those classes work independent of the larger system.

We have 3 Amigos meetings in which one of the things we do is develop a set of acceptance tests for a feature before any code is written. We usually decide at this point (before any code is written) that most or all of these scenarios will be automated. We write the acceptance tests in SpecFlow, I watch them all fail, and them I write the code to make them pass. I follow the patterns and framework that we have set up in our application, so there aren’t many design decisions to make. When my acceptance tests pass, I am done.

Where do unit tests fit in there? If my acceptance tests pass, then I’m done, so why spend more time writing duplicate tests? Also, with acceptance tests, I’m not dealing with mocks, and more importantly, I’m not fixing broken unit tests because of broken mocks. If you follow the Single Responsibility Principle (which we try to do), you end up with lots of small classes, and unit tests for those classes would be mostly useless because those classes do so little that it’s hard to write bugs and each class does such a small part of the larger activity.

There is an obvious trade-off here – my acceptance tests are not fast. I’m just testing web services (no driving a browser), so all ~2500 tests will run in about an hour. But we accepted this trade-off because we were able to get things done faster by just writing the acceptance tests, which we were going to do anyway to automate QA testing. The end result is high quality software with few bugs, not just because we have tests, but also because we communicate as a team and decide on the best way to test each feature and what it is that needs to be tested, and then we find the best way to automate the testing as a team.

Again, I’m not ready to say that this way is the best way for every project, and I’ve seen each approach work extremely well. I just wonder if the conventional wisdom on testing would be the same if we thought of it from the perspective of the whole team.

In my last post, I talked about how acceptance tests can be more important than unit tests. But clearly there is another side of the coin.

The conventional wisdom is that you should have more unit tests than acceptance tests. This is how I’ve done it in the past (on pretty much every project other than my current one).

A common scenario is a web application that uses an ORM for data access with as few stored procedures as possible. Because I’m using an ORM, my data layer has very little code. I’ll probably write a few basic unit tests and integration tests to test what data layer I have, and I’ll write integration tests to test the stored procs that I have (usually only for queries that need custom SQL).

My data layer primarily contains a really simple repository class that lets me do CRUD operations and get sort of IQueryable so that I can write LINQ queries. (If you’re using Rails, you don’t have to build any of that because it’s all there for you out of the box.) I don’t want my repository methods to have any concept of business logic or specific LINQ queries. Those belong in the business layer. The exception would be any stored procs that I have to have. I’m want to have a really thin data access layer and a fat business layer (yet another reason to use an ORM instead of having lots of SQL code in stored procs).

The business layer is going to contain the meat of the code. This is where I’m going to have a lot of unit tests. Because I kept my data layer so simple, I’m not going to have as many classes to stub out, and when I do, I’m stubbing out simple methods like Save() and Get(), which have no business logic in them.

I also have controllers, which take information from the business layer and return some sort of view model. I’ll write unit tests that verify that that translation happens correctly.

If I have some hardcore JavaScript, I’ll write unit tests for that. If it’s really simple stuff like, “when this radio button is clicked, disable this text box”, I don’t write tests for that. But when you build something that has more logic, you’ll want to test it.

At this point, I’ve tested pretty much everything very extensively. I could add some acceptance tests, which could help me communicate with the business and automate some QA testing. But at the same time, I’ve been on projects where I had excellent unit test coverage and no acceptance tests and it came out just fine. Now, that project was a two-person project with no QA team, so we just trusted our coding abilities and accepted the risk and things came out just fine. The point of saying this is not to say that you should fire your QA team, it just an example of how you can structure your code in a way that will allow you to unit test the code and get it right pretty much all the time.

This is just one example of one project. Your project might be different, with a different architecture, a different team, and a different business. You need to figure out what works best for you.

The conventional wisdom on TDD and testing is that unit tests are the preferred kind of tests to write. They help you design your code, and they run fast. Integration tests and acceptance tests are important too, but you should have far fewer of these. They exist so that you can have some end to end tests and to help test things that you can’t easily test with a unit test.

You may have also seen the automated testing triangle diagram, which illustrates how you should have more unit tests than the other kinds of tests and how the unit tests are the foundation of your test suite.

The implication of all of these diagrams and ideas is that this is the best way to do it on every project. But is that true?

I would argue that there are situations where integration tests and acceptance tests are more important than unit tests.

But wait, you say, aren’t you the guy who is always speaking about TDD and the value of unit tests?!? Was that all a lie?

Once again, the correct answer is… it depends!

Take my current project for example. We are writing web services that do back end transaction processing. Our system is very configurable, so we can make the system work very differently based on some configuration tables in the database, and different configurations are used for different types of transactions.

We have a very well defined framework. Even before I start coding a feature, I know what classes I need to create because they were defined up front with a systems analyst. So I’m not really going to get any benefit from TDD designing my code because the design work is already done.

Because the system is so configurable, we can handle new kinds of transactions by inserting new configurations into the database. There is very little coding involved. Sure, I could write unit tests for the configurable components of the application, but unit tests can’t tell me if the system works correctly for my new configuration. The best way to do that is with acceptance tests.

Not only is our framework well defined, we follow the SOLID principles and have lots of small classes that work together. I could write unit tests for all of the small classes, and in some cases I do, but in
most cases I find that the unit tests aren’t testing very much. Not only that, they tend to have lots of stubs and mocks, which makes the unit tests very brittle. Usually the integration tests and acceptance tests are more brittle, in our case, it seems to be the other way around. Nothing seems like a waste of time more than updating an existing unit test that broke because you refactored a seemingly unrelated class, especially when you have a nice acceptance test that is working just fine.

I’ve just shot down many of the positive arguments for unit tests on our project. Because it’s very important for us to test that our system is configured correctly, acceptance tests become very
important.

Well, if I already have a big suite of acceptance tests, do I really need unit tests to test things that are well covered by acceptance tests? Maybe if I have some class that has many different permutations, but if there’s only a couple paths through a class and I’ve covered it with acceptance tests, then I’m just going to skip the unit tests.

We also use stored procedures to do most of our data access. You can’t easily unit test a stored procedure, so we need more acceptance tests and integration tests to take care of that.

We still have about twice as many unit tests as acceptance tests, but that tide is turning. In a couple months, we will probably have twice as many acceptance tests as unit tests.

It honestly took us awhile to convince ourselves that we weren’t doing it wrong based on the conventional wisdom. But when we thought about it it just made sense to do it the way we’re doing it.

The moral of the story is that there are always exceptions to the so-called “best practices”.

When I write unit tests, I use the BDD-style GIven/When/Then format so that the tests are descriptive and explain the business functionality that is being implemented. But I’ve recently changed the way that I do it.

Let’s say that I’m implementing bank account functionality and I am writing code to implement the following:

Given a bank account
When I deposit money into the account
Then the balance should increase by the amount of the deposit

The old way

I used to write them like this:

[TestFixture]
public class When_I_deposit_money_into_the_account : Specification
{
    private Account _account;

    public void Establish_context()
    {
        _account = Given_a_bank_account();
    }

    public void Because_of()
    {
        _account.Deposit(10m);
    }

    [Test]
    public void Then_the_balance_should_increase_by_the_amount_of_the_deposit()
    {
        _account.Balance.ShouldEqual(10m);
    }

    private Account Given_a_bank_account()
    {
        return new Account();
    }
}

This style is used by RSpec and many people in the .NET community. I’ve had a lot of success doing it this way, but I’ve always had some complaints.

• The Given/When/Then phrases are spread out and aren’t in order.
• We have the Specification base class, which isn’t bad but I think it might confuse new people who know NUnit but don’t know my special base class.
• If you have a lot of “given” statements, you’re tempted to use inheritance or nested contexts, each with their own level of setup, virtual properties and methods, etc. This is very unreadable and gets unwieldy very quickly.

The new way

This isn’t really a new way, but it’s new to me. I’ve seen it before over the years but I didn’t really start doing it until I joined my current team where they did it this way and I’ve come to like it a lot better than the old way.

[TestFixture]
public class Bank_account_tests
{
    [Test]
    public void Deposit()
    {
        Given_a_bank_account();
        When_I_deposit_money_into_the_account();
        Then_the_balance_should_increase_by_the_amount_of_the_deposit();
    }

#region Helper methods  

    private void Given_a_bank_account()
    {
        _account = new Account();
    }

    private void When_I_deposit_money_into_the_account()
    {
        _account.Deposit(10m);
    }

    private void Then_the_balance_should_increase_by_the_amount_of_the_deposit()
    {
        _account.Balance.ShouldEqual(10m);
    }

#endregion
}

The best thing about doing it this way is that the business functionality is clearly specified at the top of the class and the Given/When/Then statements aren’t spread out all over the place. There are no crazy inheritance hierarchies, base classes, or big setup methods. When I write tests, I just write out the Given/When/Then scenarios in plain text and then use Specs2Tests to generate the test code for me. Then all I have to do is fill in the private helper methods. This is really easy, like filling in the blanks.

Also, I typically hate regions but in this case I find that they work quite well because they hide the helper methods that I typically don’t want to see when I open the class file.

It depends

Obviously there are situations where this is all overkill and you can just write simple tests without Given/When/Then methods all over the place. Just do whatever makes sense. I’ve found that this new-to-me way leads to very readable and easy to maintain tests.

Most people I know put their unit tests in files that mirror the folder structure and filename of the actual class that is being tested. So if I have app/models/order.rb, I’ll have spec/models/order_spec.rb. The tests in the order_spec.rb file will test the code in the order.rb file. Every project I have been on, whether Ruby or .NET, has done it this way.

But have you ever thought about why we do it this way?

We do it this way on our project and I keep running into two problems. The first problem is when I am about to modify some existing code and I want to know what tests will test that portion of the code. The first place I look is the corresponding test file (based on the folder structure or filename), but that doesn’t always give me all of the tests for that functionality.

The second problem is when I refactor some existing code and the refactoring spans multiple classes. Now I have broken tests all over the place and it’s hard to reconstruct the tests so that they test the same business concepts that they were testing before. Often times the tests were testing a portion of the system of classes that I am refactoring, but were not as encompassing as they should’ve been.

Here’s the thing — when I write tests, I’m typically using the Given/When/Then style of writing tests, even in unit tests using frameworks like RSpec. I’ll have test code that looks like this:

describe "When completing an order" do
  it "should set the status to In Process" do
     # test code here
  end
end

That code tests functionality having to do with orders and is going to help me ensure that my code performs some business function correctly. But if you look at that code snippet, you don’t know what classes I’m testing. Yeah I know, it’s just an example and I left out those details. But the point is that it doesn’t matter what classes I’m testing. What matters is that my tests are testing that my code performs a certain business function.

That being said, why are we grouping our tests by file? Wouldn’t it make much more sense to group our tests by business function instead?

I already have ways to find tests for a given class. I can search my code for the class name, or if I’m in .NET I can do Find Usages and have a little window pop up that tells me everywhere a class (or method) is used.

If my classes were grouped in folders by business function instead, I get the following benefits:

• I can see what tests exist for a given business function
• It encourages me to write tests that test business functionality, not test data structures (classes, methods, etc.)
• I can put all kinds of tests in there (unit tests, Cucumber tests, even manual test plans) — all in one place, all checked into source control
• My tests document business functionality instead of documenting a class

Remember, code constructs like classes and methods are just a means to an end, our goal is to write software that provides business value and performs specific functions. So I might as well organize my tests accordingly.

(Disclaimer: I have never actually tried organizing tests this way. It makes sense to me and I think it would work great, but I might try it and find out that it doesn’t work. But if anything, maybe I’ll start a good discussion.)

Here are the slides, code, and links from my Testable, Object Oriented JavaScript talk at Stir Trek.

Slides
Code
JSView
Jasmine

Have fun with it, I’m always open for suggestions with this stuff, so let me know if find ways to improve it.

It seems that the accepted way to test in Ruby is to use Rspec for unit tests and to use Cucumber for acceptance tests (higher level functional testing). After doing a little bit of Cucumber, I’ve started to fall in love with the format of Cucumber tests.

Most Rubyists would probably agree that behavior-driven development is good (in other words, writing tests in a Given/When/Then format). We obviously do this in Cucumber (there isn’t much choice), but I’ve also written tests in this format in Rspec and in .NET.

I like BDD for two main reasons. First, I believe that software development is a series of translations. I want to translate business requirements into readable, executable specifications, then translate that into tests, then translate that into implementation code. Second, before I implement a feature and even before I write my tests, I try to write out what I want the code to do in English. If I can’t write out what I want to do in English, how and I supposed to know what I’m supposed to write in code?

Here’s my theory: if we agree that BDD is good, why don’t we write our unit tests in a format that is more amenable to BDD, that being the Cucumber format of tests? I’m not saying that we write acceptance level tests instead of unit tests, I’m saying that maybe we should write unit tests in a different format. Not only that, Cucumber tables give us a nice way to write more readable, data-driven tests. Here are a couple examples from the supermarket pricing kata (in Rspec and Cucumber).

Cucumber:

Feature: Checkout

  Scenario Outline: Checking out individual items
    Given that I have not checked anything out
    When I check out item 
    Then the total price should be the  of that item

  Examples:
    | item | unit price |
    | "A"  | 50         |
    | "B"  | 30         |
    | "C"  | 20         |
    | "D"  | 15         |

  Scenario Outline: Checking out multiple items
    Given that I have not checked anything out
    When I check out 
    Then the total price should be the  of those items

  Examples:
    | multiple items | expected total price | notes                |
    | "AAA"          | 130                  | 3 for 130            |
    | "BB"           | 45                   | 2 for 45             |
    | "CCC"          | 60                   |                      |
    | "DDD"          | 45                   |                      |
    | "BBB"          | 75                   | (2 for 45) + 30      |
    | "BABBAA"       | 205                  | order doesn't matter |
    | ""             | 0                    |                      |

  Scenario Outline: Rounding money
    When rounding "" to the nearest penny
    Then it should round it using midpoint rounding to ""

    Examples:
      | amount | rounded amount |
      | 1      | 1              |
      | 1.225  | 1.23           |
      | 1.2251 | 1.23           |
      | 1.2249 | 1.22           |
      | 1.22   | 1.22           |

Rspec:

require 'spec_helper'

describe "Given that I have not checked anything out" do
  before :each do
    @check_out = CheckOut.new
  end

  [["A", 50], ["B", 30], ["C", 20], ["D", 15]].each do |item, unit_price|
  describe "When I check out an invididual item" do
    it "The total price should be the unit price of that item" do
      @check_out.scan(item)
      @check_out.total.should == unit_price
    end
  end
end

  [["AAA", 130], # 3 for 130
    ["BB", 45],  # 2 for 45
    ["CCC", 60],
    ["DDD", 45],
    ["BBB", 75], # (2 for 45) + 30
    ["BABBAA", 205], # order doesn't matter
    ["", 0]].each do |items, expected_total_price|
    describe "When I check out multiple items" do
      it "The total price should be the expected total price of those items" do
        individual_items = items.split(//)
        individual_items.each { |item| @check_out.scan(item) }
        @check_out.total.should == expected_total_price
      end
    end
  end
end

class RoundingTester
  include Rounding
end

[[1, 1],
  [1.225, 1.23],
  [1.2251, 1.23],
  [1.2249, 1.22],
  [1.22, 1.22]].each do |amount, rounded_amount|
  describe "When rounding an amount of money to the nearest penny" do
    it "Should round the amount using midpoint rounding" do
      RoundingTester.new.round_money(amount).should == rounded_amount
    end
  end
end

A couple things stand out to me when you compare these two. First, if I want to run data-driven tests with different values, the Cucumber syntax is so much cleaner and more descriptive. Second, the “Given I have not checked anything out” section in the Rspec version is really long and contains two nested “describe” sections (many times you end up with many more than this). When you nest sections like this, it’s really hard to see the context of things or read the tests because the “Given” text is nowhere near the nested “When” sections in the code.

Rspec follows in the footsteps of previous unit testing frameworks that write test methods in test classes (or in the case of Rspec, something that resembles test classes and methods. But is this the best way, or just the way that we’re used to? We have been writing unit tests this way for years and years because we had no other choice. But that doesn’t mean that it’s the best way.

Here are the benefits I see of using the Cucumber syntax over Rspec:

• The tests are much easier to read (especially when doing data-driven “scenario outline” tests).
• The Given/When/Then text is all in one place (as opposed to spread out and mixed in with code).
• It forces me to be able to write out in English what I want the code to do.
• Any step definition that I write can easily be reused anywhere in any other Cucumber test.
• The code just looks cleaner. I’ve seen a lot of messy Rspec tests.
• Rspec doesn’t have a method that corresponds to the “When” step (unless I’m missing something), so you have to shoehorn it into before(:each) or the “it” method. (I’m not sure why this is, we figured this out in the .NET world long ago.)

To be fair, there are more BDD-friendly flavors of Rspec (like rspec-given). This helps you write tests in Given/When/Then format, but I still feel like all of the underscores and symbols and syntax is getting in the way of the actual test verbiage.

Favoring Cucumber is my personal preference and I know that there are some people that would probably disagree with my opinion on this, and that’s fine. But I’m really enjoying what Cucumber brings to the table, both in terms of functionality and the syntax.

I had a simple problem today. I had to turn this:

… into this:

[TestFixture]
public class When_depositing_money_into_an_account : Specification
{
    [Test]
    public void should_add_the_specified_amount_into_the_account()
    {

    }
}

[TestFixture]
public class When_withdrawing_money_from_an_account : Specification
{
    [Test]
    public void should_withdraw_the_specified_amount_from_the_account()
    {

    }

    [Test]
    public void should_specify_that_the_withdrawal_was_successful()
    {

    }
}

I really didn’t want to do all of this typing every time, so I wrote code to do it for me. It works like this:

1) Copy the acceptance criteria text onto the clipboard
2) Run Specs2Tests.exe (will put the code onto the clipboard for you)
3) Paste the code into your code file

Now I can get all of my acceptance criteria nailed down in a text file where I don’t have the ceremony of a programming language, then let something convert them into tests. Since I just cranked this out today, I’m guessing that it doesn’t handle every scenario that you might think of, and it only writes out C# code. There is no reason that it couldn’t spit out VB.NET, RSpec, or anything else. If it doesn’t do what you want, get on github, create a fork, and send me a patch!

The code is here: http://github.com/jonkruger/specs2tests. Have fun.

I wanted to come up with a way to show people how to use Rhino Mocks (other than telling them to read the documentation). What better way to do this than by showing you how it works through a bunch of simple unit tests that document how Rhino Mocks works?

So that’s what I did. You can view the code here, or if you want to download the whole project and run the tests, you can get the whole thing here.

(If you’re interested in Moq, how it compares to Rhino Mocks, and to see Moq documented through tests, check out Steve Horn’s post.)

UPDATE: Fixed the test that was incorrectly showing how to use Expect() and VerifyAllExpectations(). Thanks to Sharon for pointing this out.