Jon Kruger

As soon as you write code, bad things can happen. Your code can be defective, and it can break existing code. It’s our job to make sure that we don’t break things.

Not breaking things is not easy. You might end up with only one bug that sneaks through into production but that can be one too many.

The team I’m on is currently trying to figure out what our plan is for testing and QA. We have two concerns:

• How can we wrote code without defects?
• How can we make sure that our code changes don’t break existing code?

The second one is proving to be the difficult one. Our application is 4 years old, and a good chunk of the code was written by people who are no longer on the team. It’s really difficult at times to not break something when you don’t know enough about the system to know the ramifications of what you’re about to do.

When I develop new code, I try to figure out how we can validate that everything is working and complete. When possible, I want to write automated tests. In the Ruby world, I’m using Rspec and Cucumber to do this. I want to make sure that all of the code that I write is covered by a test if possible. Thankfully Rspec and Cucumber do not make this very hard at all, and we tend to test the crap out of things as a result.

But some things are hard to validate. For example, if you’re pulling a subset of data from an external database using an SQL query, how do you know that you’re pulling back the correct data? It’s hard (if not impossible) to write an automated test for this, and in some cases you have to coordinate with someone on the team that owns the database in order to test it. It’s really important not to ignore these tests and just say that it’s going to be too hard to test. Unless you’re OK with it being broken, you have to validate it.

As a developer, don’t think that this is just the responsibility of your QA team to figure out how to test things. Certainly your QA team should be involved in coming up with the plan for testing, but you might know of edge cases that a QA person wouldn’t know about, and you might know of technical constraints that might make testing hard. Too many defects are created because developers put the onus on QA to find everything. Remember, we’re all on the same team.

So now you go into production and you move on to something else, but you’ve created a lot of features that will need to be regression tested. Here is where I think it’s really difficult. How do you keep these regression tests up to date? For example:

• When you add a new security role to the system, do you update old test plans to test whether or not the new security role should be able to do something?
• If you change a validation rule, do you update old test plans that could be affected by the new validation rule? (This could mean adding new tests to an old test plan because you might now have two paths through something where you might’ve only had one test before.)
• When things change like this, how do you even know where you might need to update test plans?

I’d be interested in hearing how you all handle these situations. This is a situation where I’m really glad that I have automated tests (yesterday I was doing a big refactoring and my Cucumber tests pointed out a whole bunch of things that I had broken). But automated tests only solve the problem when you have a test for something. If you create new paths through the code that now don’t have a test, you need a way to find those scenarios and write tests for them.

• Leave a comment... (0)

It seems that the accepted way to test in Ruby is to use Rspec for unit tests and to use Cucumber for acceptance tests (higher level functional testing). After doing a little bit of Cucumber, I’ve started to fall in love with the format of Cucumber tests.

Most Rubyists would probably agree that behavior-driven development is good (in other words, writing tests in a Given/When/Then format). We obviously do this in Cucumber (there isn’t much choice), but I’ve also written tests in this format in Rspec and in .NET.

I like BDD for two main reasons. First, I believe that software development is a series of translations. I want to translate business requirements into readable, executable specifications, then translate that into tests, then translate that into implementation code. Second, before I implement a feature and even before I write my tests, I try to write out what I want the code to do in English. If I can’t write out what I want to do in English, how and I supposed to know what I’m supposed to write in code?

Here’s my theory: if we agree that BDD is good, why don’t we write our unit tests in a format that is more amenable to BDD, that being the Cucumber format of tests? I’m not saying that we write acceptance level tests instead of unit tests, I’m saying that maybe we should write unit tests in a different format. Not only that, Cucumber tables give us a nice way to write more readable, data-driven tests. Here are a couple examples from the supermarket pricing kata (in Rspec and Cucumber).

Cucumber:

Feature: Checkout
 
  Scenario Outline: Checking out individual items
    Given that I have not checked anything out
    When I check out item <item>
    Then the total price should be the <unit price> of that item
 
  Examples:
    | item | unit price |
    | "A"  | 50         |
    | "B"  | 30         |
    | "C"  | 20         |
    | "D"  | 15         |
 
  Scenario Outline: Checking out multiple items
    Given that I have not checked anything out
    When I check out <multiple items>
    Then the total price should be the <expected total price> of those items
 
  Examples:
    | multiple items | expected total price | notes                |
    | "AAA"          | 130                  | 3 for 130            |
    | "BB"           | 45                   | 2 for 45             |
    | "CCC"          | 60                   |                      |
    | "DDD"          | 45                   |                      |
    | "BBB"          | 75                   | (2 for 45) + 30      |
    | "BABBAA"       | 205                  | order doesn't matter |
    | ""             | 0                    |                      |
 
  Scenario Outline: Rounding money
    When rounding "<amount>" to the nearest penny
    Then it should round it using midpoint rounding to "<rounded amount>"
 
    Examples:
      | amount | rounded amount |
      | 1      | 1              |
      | 1.225  | 1.23           |
      | 1.2251 | 1.23           |
      | 1.2249 | 1.22           |
      | 1.22   | 1.22           |


Rspec:

require 'spec_helper'
 
describe "Given that I have not checked anything out" do
  before :each do
    @check_out = CheckOut.new
  end
 
  [["A", 50], ["B", 30], ["C", 20], ["D", 15]].each do |item, unit_price|
  describe "When I check out an invididual item" do
    it "The total price should be the unit price of that item" do
      @check_out.scan(item)
      @check_out.total.should == unit_price
    end
  end
end
 
  [["AAA", 130], # 3 for 130
    ["BB", 45],  # 2 for 45
    ["CCC", 60],
    ["DDD", 45],
    ["BBB", 75], # (2 for 45) + 30
    ["BABBAA", 205], # order doesn't matter
    ["", 0]].each do |items, expected_total_price|
    describe "When I check out multiple items" do
      it "The total price should be the expected total price of those items" do
        individual_items = items.split(//)
        individual_items.each { |item| @check_out.scan(item) }
        @check_out.total.should == expected_total_price
      end
    end
  end
end
 
class RoundingTester
  include Rounding
end
 
[[1, 1],
  [1.225, 1.23],
  [1.2251, 1.23],
  [1.2249, 1.22],
  [1.22, 1.22]].each do |amount, rounded_amount|
  describe "When rounding an amount of money to the nearest penny" do
    it "Should round the amount using midpoint rounding" do
      RoundingTester.new.round_money(amount).should == rounded_amount
    end
  end
end


A couple things stand out to me when you compare these two. First, if I want to run data-driven tests with different values, the Cucumber syntax is so much cleaner and more descriptive. Second, the “Given I have not checked anything out” section in the Rspec version is really long and contains two nested “describe” sections (many times you end up with many more than this). When you nest sections like this, it’s really hard to see the context of things or read the tests because the “Given” text is nowhere near the nested “When” sections in the code.

Rspec follows in the footsteps of previous unit testing frameworks that write test methods in test classes (or in the case of Rspec, something that resembles test classes and methods. But is this the best way, or just the way that we’re used to? We have been writing unit tests this way for years and years because we had no other choice. But that doesn’t mean that it’s the best way.

Here are the benefits I see of using the Cucumber syntax over Rspec:

• The tests are much easier to read (especially when doing data-driven “scenario outline” tests).
• The Given/When/Then text is all in one place (as opposed to spread out and mixed in with code).
• It forces me to be able to write out in English what I want the code to do.
• Any step definition that I write can easily be reused anywhere in any other Cucumber test.
• The code just looks cleaner. I’ve seen a lot of messy Rspec tests.
• Rspec doesn’t have a method that corresponds to the “When” step (unless I’m missing something), so you have to shoehorn it into before(:each) or the “it” method. (I’m not sure why this is, we figured this out in the .NET world long ago.)

To be fair, there are more BDD-friendly flavors of Rspec (like rspec-given). This helps you write tests in Given/When/Then format, but I still feel like all of the underscores and symbols and syntax is getting in the way of the actual test verbiage.

Favoring Cucumber is my personal preference and I know that there are some people that would probably disagree with my opinion on this, and that’s fine. But I’m really enjoying what Cucumber brings to the table, both in terms of functionality and the syntax.

• Leave a comment... (4)

Your team can become much more productive today! And the cost is very small relative to the cost of your developers!

You think that managers would be very interested in this sales pitch, yet almost none of them will buy it.

What am I talking about? Buying your development team new hardware. This especially applies if you’re developing in a language in .NET or Java where you have to compile your project.

I was excited to read this post today because this team gets it. One of their big improvements was to get solid-state drives for their code.

I’m trying to figure out how this doesn’t make sense. If you’re paying your developers $70-90k a year, why wouldn’t you spend a couple hundred dollars for a solid-state drive to help them get things done faster? If your team needs a profiler or ReSharper or some other software package, that request is almost always approved.

Compilation time on big .NET projects (even medium-sized .NET projects) is brutal. It totally slows down your momentum. There really isn’t much else you can do during this time (most developers check email, check Twitter, surf the web, etc.). Wouldn’t you like that to be productive time?

• Leave a comment... (4)