We had a week-long debate awhile back about whether or not it’s OK to modify your production code in order to enable automated acceptance testing. I’m not talking about using dependency injection, interfaces, etc. to allow you to mock things in unit tests. I’m talking about modifying application code solely to help your automated acceptance tests.
There are many ways this can be done, some of which we’ve done:
- Creating a SystemTime class, which is like DateTime except that we can set what “Now” is, so we can change time in tests
- Adding optional parameters to stored procedures solely so that we can have them only operate on a subset of data in an acceptance test instead of operating on the entire set of data in the database
- Adding extra HTML attributes so that automated tests can find elements on a page easily
To me, modifying production code to help us do automated testing is no big deal. First, if our goal is quality, I don’t think it matters how we get there. After all, we own the code base and tests so there aren’t any real restrictions on what we can do with the code or the tests as long as the end product is good.
Second, developers and QA are on the same team, and we work together quite closely, so we should do what we can to help each other out. So if we can make a minor change to the application code to save us a lot of time developing or running automated tests, then to me it makes sense to do so.
This goes back to my assertion that we need to stop thinking of QA like external auditors that have to take the application just as it is without talking to the developers and act as the independent quality police. We need to all work together to ensure quality, both developers and QA. Developers are just as responsible for quality as QA. If we place all of the responsibility for quality on QA, then developers will care less and less about quality, and you end up with shoddy code with lots of bugs (and usually no tests). I’d rather treat testing as a whole-team activity and structure the application to make testing as easy as possible.
Lately I’ve been thinking about a whole team approach to testing, where we decide as a team how features will be tested and where we use the skillsets of the whole team to automate testing. We do this on our project, and this has led to a regression testing suite of ~2500 SpecFlow acceptance tests that automate almost all scripted QA testing and regression testing for our application.
We didn’t always do this. Originally there was no automated acceptance testing, but developers were diligently writing unit tests. Those unit tests are still around, but we don’t write many unit tests anymore. We start with acceptance tests now, and the acceptance tests cover all of the testing scenarios that need to be covered. Our application has well-defined design patterns that we follow, so the idea of TDD driving the design of our code doesn’t really apply. If the unit tests fail, we often just delete them because it’s not worth fixing all of the mocks in the unit tests that are causing them to fail, and we have acceptance testing coverage around all of it.
This approach does not line up with the conventional wisdom on automated testing. They say that you’re supposed to write lots of unit tests that run really fast to give you fast feedback, help design your code, and ensure the internal quality of your code. In the past, this is how I’ve always done it. In fact, many of them dislike Cucumber.
Cucumber makes no sense to me unless you have clients reading the tests. Why would you build a test-specific parser for English?
— DHH (@dhh) March 29, 2011
While TDD isn’t as mainstream as I would like, TDD is nothing new. Kent Beck was writing about it 10 years ago, and the original XP guys valued such things as unit testing, the SOLID principles, and things like that.
Automated acceptance testing still feels like a relatively new phenomenon. I’m sure people were doing it 10 years ago, but back then we didn’t have Cucumber and SpecFlow and the Gherkin language. Now I see a lot more people using tools like that to automate QA testing in way that uses business language and more maintainable code, rather than the old “enterprise” solutions like QTP.
Here’s what I’m getting at – I wasn’t there 10 years ago when Kent Beck was writing his books and the XP movement was starting, but it seems to me to be primarily an effort by developers to ensure the quality of their code through the effort of developers. I see very little talk of where QA fits into that process. There is some mention of QA for sure, but the general gist seems to be that developers need to write tests in order to ensure quality, and the best way to do that is to write unit tests. QA people typically don’t say that unit testing is enough because it doesn’t test end-to-end, so then what do they do? Manually test? Use QTP?
My question is this – if we think of testing as whole-team activity and not just a QA activity or a developer activity, will we arrive at the same conclusions as we did before?
I’m not ready to discount unit testing as a valuable tool, and I’m also not ready to say that everyone should do it my way because it worked for us on one project. But we have largely abandoned unit testing in favor of acceptance testing, and other teams in our department are doing it too. I write unit tests for things like extension methods and some classes that have important behavior on their own and I want to ensure that those classes work independent of the larger system.
We have 3 Amigos meetings in which one of the things we do is develop a set of acceptance tests for a feature before any code is written. We usually decide at this point (before any code is written) that most or all of these scenarios will be automated. We write the acceptance tests in SpecFlow, I watch them all fail, and them I write the code to make them pass. I follow the patterns and framework that we have set up in our application, so there aren’t many design decisions to make. When my acceptance tests pass, I am done.
Where do unit tests fit in there? If my acceptance tests pass, then I’m done, so why spend more time writing duplicate tests? Also, with acceptance tests, I’m not dealing with mocks, and more importantly, I’m not fixing broken unit tests because of broken mocks. If you follow the Single Responsibility Principle (which we try to do), you end up with lots of small classes, and unit tests for those classes would be mostly useless because those classes do so little that it’s hard to write bugs and each class does such a small part of the larger activity.
There is an obvious trade-off here – my acceptance tests are not fast. I’m just testing web services (no driving a browser), so all ~2500 tests will run in about an hour. But we accepted this trade-off because we were able to get things done faster by just writing the acceptance tests, which we were going to do anyway to automate QA testing. The end result is high quality software with few bugs, not just because we have tests, but also because we communicate as a team and decide on the best way to test each feature and what it is that needs to be tested, and then we find the best way to automate the testing as a team.
Again, I’m not ready to say that this way is the best way for every project, and I’ve seen each approach work extremely well. I just wonder if the conventional wisdom on testing would be the same if we thought of it from the perspective of the whole team.
Recently I had the privilege of hearing Uncle Bob Martin talk at the Columbus Ruby Brigade. Among the many nuggets of wisdom that I learned that night, my favorite part was the Automated Testing Triangle. I don’t know if Uncle Bob made this up or if he got it from somewhere else, but it goes something like this.
At the bottom of the triangle we have unit tests. These tests are testing code, individual methods in classes, really small pieces of functionality. We mock out dependencies in these tests so that we can test individual methods in isolation. These tests are written using testing frameworks like NUnit and use mocking frameworks like Rhino Mocks. Writing these kinds of tests will help us prove that our code is working and it will help us design our code. They will ensure that we only write enough code to make our tests pass. Unit tests are the foundation of a maintainable codebase.
But there will be situations where unit tests don’t do enough for us because we will need to test multiple parts of the system working together. This means that we need to write integration tests — tests that test the integration between different parts of the system. The most common type of integration test is a test that interacts with the database. These tests tend to be slower and are more brittle, but they serve a purpose by testing things that we can test with unit tests.
Everything we’ve discussed so far will test technical behavior, but doesn’t necessarily test functional business specifications. At some point we might want to write tests that read like our technical specs so that we can show that our code is doing what the business wants it to do. This is when we write acceptance tests. These tests are written using tools like Cucumber, Fitnesse, StoryTeller, and NBehave. These tests are usually written in plain text sentences that a business analyst could write, like this:
As a user
When I enter a valid username and password and click Submit
Then I should be logged in
At this point, we’re are no longer just testing technical aspects of our system, we are testing that our system meets the functional specifications provided by the business.
By now we should be able to prove that our individual pieces of code are working, that everything works together, and that it does what the business wants it to do — and all of it is automated. Now comes the manual testing. This is for all of the random stuff — checking to make sure that the page looks right, that fancy AJAX stuff works, that the app is fast enough. This is where you try to break the app, hack it, put weird values in, etc.
I find that the testing triangle on most projects tends to look more like this triangle. There are some automated integration tests, but these tests don’t use mocking frameworks to isolate dependencies, so they are slow and brittle, which makes them less valuable. An enormous amount of manpower is spent on manual testing.
Lots of projects are run this way, and many of them are successful. So what’s the big deal? Becuase what really matters is the total cost of ownership of an application over the entire lifetime of the application. Most applications need to be changed quite often, so there is much value in doing things that will allow the application be changed easily and quickly.
Many people get hung up on things like, “I don’t have time to write tests!” This is a short term view of things. Sometimes we have deadlines that cannot be moved, so I’m not denying this reality. But realize that you are making a short term decision that will have long term effects.
If you’ve ever worked on a project that had loads of manual testing, then you can at least imagine how nice it would be to have automated tests that would test a majority of your application by clicking a button. You could deploy to production quite often because regression testing would take drastically less time.
I’m still trying to figure out how to achieve this goal. I totally buy into Uncle Bob’s testing triangle, but it requires a big shift in the way we staff teams. For example, it would really help if QA people knew how to use automated testing tools (which may require basic coding skills). Or maybe we have developers writing more automated tests (beyond the unit tests that they usually write). Either way, the benefits of automated testing are tremendous and will save loads of time and money over the life of an application.
I’ll be speaking on the SOLID software design principles at the Quick Solutions office (440 Polaris Pkwy., Suite 500, Westerville) on Wednesday, Sept. 16 from 5:30pm-to 7pm. In this talk, we’ll go through Uncle Bob’s “SOLID” software design principles, separation of concerns, a brief overview of inversion of control containers like StructureMap and Ninject, and more object-oriented goodness that will help you write better code.
There’s free food too. If you’re coming, please RSVP to amorey at quicksolutions dot com so that we can have enough food.
If you’re going to the Software Engineering 101 conference, this is basically the same talk that I’ll be giving there. So if you’re going to go to one, go to the Software Engineering conference since you’ll get lots of other good stuff there too.
At some point, we all have to write some small apps. I’m talking about things like…
- Some small utility or diagnostic app
- Something to help with your deployment process
- Other small applications or websites (i.e. something that takes a month or less)
In these cases, we often throw good software design principles out the window. We say that we don’t need to write unit tests, we don’t need dependency injection, we can put data access code in our code-behinds, and things like that. Since it’s a small app, we think we can get away with it.
Just because your app didn’t take you long to write doesn’t mean that you get off easy. The consequences of your decisions just aren’t as severe, but that doesn’t mean that the pain is gone!
How many times have you written some small utility to help with something and then your boss sees it and really likes it, and then he asks you to add more functionality to it. Eventually you’ve spent a couple months on the app. If you cheated at the beginning, that code is going to hard to change and it’s going to start becoming more of a pain.
Maybe you’re writing a console app to help with your deployment process. Now this code better work, because if it doesn’t, your app might not deploy correctly. This is very important code! Doesn’t code this important warrant some extra attention (i.e. tests)?
Look, I’m not saying that every little app has to be this big, blown-out, enterprise application. I’m just saying that you should be careful when you cheat, because you don’t want that to come back to haunt you.
Many times you can take simple steps to make things easier to change. Take dependency injection, for example. It is really easy to set up an IoC container like StructureMap, it doesn’t overly complicate your code, and you don’t have to write tons of extra code to use it. But if you want to write tests for you code, dependency injection will make it a lot easier. You’re just putting code in better places.
Again, there are times to use dependency injection and there are times where it’s superfluous. There are times when unit testing is essential and times when you can get away with it (personally I always like writing tests if I can, that way I know my code works). But you need to be careful. Software is software, and many of the same development principles that apply to big apps still apply to small apps.
As I was writing my post on aspiring to be a Software Craftsman, I was thinking about why we have such a low standard of quality in our industry. Clearly there is a problem when there are so many software projects out there that are rewrites of previous projects, and businesses are hamstrung by software that is too difficult to change.
I’m very much an optimist most of the time, honest! But I can’t help notice all kinds of things that don’t sit well with me.
Poor university training
I had 5 classes in my 4 years of college that were relevant to software development. I graduated in 2002. They started us out on C++, which was a fair choice in 1998. I was not able to take a database course until my senior year. During my last semester they finally introduced an intro to Java course which I was able to take. No courses on .NET, no courses in PHP, no courses in software design, no courses on techniques that could’ve made me successful (like unit testing).
The software industry is constantly changing, and many university programs are not keeping up. When you graduate you know just enough to be dangerous but not enough to really know what you’re doing. 4 years is a long time, you should be able to learn a lot during that time.
Post-college training is not encouraged
If you want to learn about software development, there is always a user group you can go to, a conference you can attend, thousands of blogs you can read, podcasts to listen to, and on and on. There is a wealth of information out there, but the average developer is not taking advantage of it. Some of these people don’t care to learn. But I think that most people care about doing a good job. The problem is that their employer does not encourage it. If someone doesn’t want to spend their time outside of work reading blogs and attending conferences, that’s their choice. There is nothing wrong with leaving your work at work and spending time with your family or doing other things. But most employers don’t offer enough training during work hours, whether that is organizing lunch and learns, paying for their devs to go to CodeMash, bringing in guest speakers, etc. It costs a lot of money to build software. Shouldn’t employers be willing to spend a little more money to train their employers so that they can build better quality software?
People don’t know what they don’t know
There are a lot of developers out there don’t know software design techniques and concepts like unit testing, what SOLID stands for, why you should use an IoC container, and why you should design with interfaces. I would guess that a majority of these developers would love to know this stuff, but they don’t know that they don’t know it. A year ago I had never used Rhino Mocks, and I didn’t know what SOLID stood for or what an IoC container was. I thought I was a good developer, and I was in charge of a fairly large project, but I didn’t know that I didn’t know these things. Now that I’ve learned all that stuff I am a much better developer than I was a year ago. But now I’m wondering what other important stuff I don’t know.
My point is that I’m sure that it would really help developers if there was some way that they could go through some kind of training that would teach advanced software development topics like the ones that I mentioned. I see a lot of junior developers take Microsoft certification tests. It is unlikely that Microsoft certification tests will teach you any of the concepts that I mentioned. You will, on the other hand, memorize a lot of stuff that you can find on Google. I’m not saying that studying for and taking those tests won’t teach you anything, but they’re not going to teach you how to write quality software.
Low business expectations
Many IT managers and executives have very low expectations for their software. Having to rewrite software is widely accepted as normal these days. Wouldn’t it be so much better and cheaper to change your existing software when you need it to do something different?
The problem is that so much of our software is too difficult to change due to bad design, lack of testing, and many other factors. Many businesses do very little to address the root problem — that their developers don’t have the adequate training or don’t have the ability to write good software that can last.
Many IT managers believe that the best that they can do is this disposable software, so they decide that they might as well pay less money if they’re going to get low quality, so they turn to offshoring. Either way, they’re still getting the same low quality software and they’re wasting money rewriting their software every several years.
Short term thinking
Many business get sucked into short term thinking. I’m sure there are many ways that this can work, but here’s one way I can see this happening.
- Executives want to show results because they’re worried about their stock price. They demand results this year.
- IT Manager is under pressure to deliver something this year, possibly with a limited budget.
- Developers are under pressure to deliver something and have really tight deadlines.
- Developers throw something together and finish the project. IT Managers show it to executives who like what they see. Executives show it at a trade show and shareholders are happy.
- Two years later the software does not meet the needs of the business because the app is unmaintainable and they start budgeting money to bring in consultants to rewrite it.
It would’ve made much more sense to invest more time and money into the project the first time so that it wouldn’t have to be rewritten two years later. But because the business needs to show progress to the shareholders they won’t do it this way. This is a tough situation, and I can understand why executives make some of these decisions, but it doesn’t seem to make sense to me.
Little or no accountability
If you’re a developer and you write bad code, you probably aren’t going to lose your job unless there is a recession, you do a really really bad job, or just quit showing up for work. If a civil engineer builds a bridge that fails after 5 years, I guarantee you he won’t be working on bridges anymore, at least not at the same company. There is very little pressure in the software industry to write quality software.
In closing…
I don’t know what the solution to this problem is because it is a problem on so many fronts. But I do have control over what’s around me. I can aspire to become a software craftsman — someone who designs and writes quality software. I can hold myself accountable and make sure that I have high standards. And I can try and teach other people to do the same.
There has a bunch of talk recently on blogs and in the open spaces at CodeMash about what other industries or professions we can compare software development to. The point of this was to look at these other industries and see what we can learn from them and how their practices can be applied to software development, particularly in the area of quality.
The title “software engineer” is frequently used to describe software developers. Let’s look at what an engineer in another discipline, civil engineering, might go through in their career.
Building a bridge leaves very little margin for error. Numerous people are involved in the construction of a bridge, including civil engineers, who will design the bridge, ensure that it will be safe, be able to handle winds and earthquakes, and on and on.
How did the civil engineer get to this point? He or she probably spent 4 years in college, then at some point had to study to take the PE exam, and probably went through extra study and training to learn the intricacies of bridge construction. That’s a good amount of training.
Quality is of the utmost importance to a civil engineer for obvious reasons – when you drive across a bridge, you expect to make it to the other side. If a civil engineer builds a bridge that fails, he is going to be in big trouble for that mistake, possibly to the point that he loses his job (if the offense is egregious enough).
In our discussions, the title that we preferred over “software engineer” is “software craftsman”. What’s the difference?
A craftsman might go through a different kind of training, and the work is slightly different. Take a carpenter for example. A carpenter will probably go through some sort of schooling, and then do an apprenticeship with a master carpenter. If he is good enough, eventually the apprentice will learn enough to become a master carpenter himself.
Let’s say that our master carpenter is building kitchen cabinets. There are certain basic design principles that he will follow, and he will be as precise and accurate as possible. You expect your cabinets to stay attached to the wall, you expect the doors to be on straight, and you expect the shelves to hold 20 plates and 15 bowls without collapsing. At the same time, there is an aesthetic element to kitchen cabinets, which is the criteria that you probably look at the most when you go to the store to buy kitchen cabinets.
Let’s compare the software development industry to these two disciplines. The average software developer goes to school for 4 years. Personally I think that university Computer Science programs are a joke (at least in the U.S., can’t speak for other countries). In my 4 years at the U. of Toledo I think I had 5 classes that taught me something relevant to real world software development. The words “unit test” were never uttered in my classrooms <shudder>.
Once the developer graduates from college, he immediately gets a job in the workplace and starts working on projects. In my case, I had 5 relevant classes in my 4 years, which equates to about one semester. This developer may receive little or no formal training after this for the rest of his career. Over the course of his career, this developer will write sub-standard quality code, but he’ll continue to get his standard raise every year and might even become a manager someday. Trust me, I have seen it happen.
Are we surprised at the quality of the software that is being written based on this analysis? A civil engineer builds a bridge that collapses, and he will probably not build another bridge again. If a carpenter builds cabinets using shoddy materials that don’t hold up, and no one will buy his cabinets. A software developer writes poor quality code, and he’s on the new team a few years later that is rewriting the legacy app in a different language.
There are so many problems here that I don’t even know where to start. You may not know everything about software development, and that’s fine. But there is no excuse for our lack of quality in our industry.
Let’s look at one area — testing.
Many developers would agree that unit testing and test driven development are good ideas. Even people who aren’t good at testing or TDD would say that. If you’re one of those people, learn how to do it! Ideally, find someone you know that knows how to do it and pair program with them for a week and you’ll be on your way. If you do know how to test, stop cheating and write your tests!
We’ve all caught ourselves cheating. We know that we should write tests but for whatever reason we don’t do it. That’s not holding yourself accountable. You are sacrificing quality, period.
Are you an aspiring Software Craftsman?
A software craftsman has a higher standard. A software craftsman cares about his craft and takes pride in doing quality work that will stand the test of time. A software craftsman does not cut corners or do shoddy work just to cross something off their to do list. Is that your standard?
I saw a very interesting chart today…
Source: http://www.riceconsulting.com/public_pdf/STBC-WM.pdf
If this doesn’t convince you of the value of unit tests, I don’t know what will. The cost to fix bugs in production could be dramatically higher than the cost to fix them in development, which is why having a suite of unit tests that you can run when you make changes is invaluable because it will prevent you from getting into these costly scenarios where you have to fix nasty bugs in production.
