In my last post, I talked about the kinds of metrics that I don’t like. But sometimes even good metrics can go wrong.

When I worked for a previous employer, we used “load sheets” on our Agile card wall. These were pieces of paper divided into 10 blocks (10 days in a 2 week iteration) and then we would cut out cards for the features that were sized to take up as many blocks as the estimate (so a 3 day estimate card would take up 3 blocks). We would use different colored dot stickers to indicate when we had discussed the requirements, discussed the design, completed the development, and when QA was complete. Each developer on the team would have one of these sheets on the wall, so you could see who was working on what and where we were pretty easily.

I didn’t come up with the system, but I’m guessing that they designed it for at least two reasons (probably more than that):

• Anyone can come and see what we’re working on and what the status is of a particular feature
• We would meet with the project sponsor every other week and they would pick out features for the next two weeks, and the differently-sized cards gave a good visual representation of the cost associated with each feature

This system did an excellent job of both of those things. The problem is that the development team interpreted it in a completely different way.

When the developers looked at the sheets, these are some of the things that they thought:

• The most important thing is for me to get my features to QA in the estimated time
• Completing my tasks in the estimated time is important (more important than things like writing tests or helping other developers, because there are no metrics for those)
• Completing my tasks this iteration is more important than helping others complete their tasks (even if helping someone else would be better for the team). This indirectly discouraged pairing on tasks.

As a result, the following things actually happened (I’m not making this stuff up):

• One developer segregated himself from the team and focused on completing as many features as he possibly could. He did a good job of staying focused and he did get a lot done, but there was little collaboration or design discussion and virtually no unit tests. He used his productivity metric to ask for a raise (because he was completing twice as much work).
• Developers would work on features, and as soon as the estimated time passed and they weren’t done, they got openly flustered and complained about the estimates being wrong (which was correct), and focused on getting it done instead of getting it done right (which led to technical debt).
• Developers would get the work most of the way done and then say it was done, saying, “It’s far enough along that QA can take a look at it and let me know what’s wrong.” (That way they could move on to the next task.)

Whoa, what happened?!? I’m sure this wasn’t what the people who came up with this system intended. But that’s the problem — while the system provided a lot of positive benefits, it also indirectly encouraged individualistic behavior and cutting corners and discouraged collaboration because of implied metrics. I was leading the team, and I wasn’t going into TFS and spitting out reports to compare developers, and I wasn’t tracking who was completing tasks within the estimated time, because it didn’t matter at all. It didn’t matter what I was or wasn’t doing — the data was being tracked, so developers assumed that something was (or could) be done with it. (I’m not using this example to criticize the people who came up with the system, because no system is perfect, and there were many very successful projects completed using this approach.)

What could we have done differently? We could figure out what message we want our card wall to convey to the developers and structure it that way. For example:

• I care much more about how much the team accomplishes in an iteration than how much each individual developer gets done, so don’t organize the cards by team member (if you want to write developers’ names on cards, that’s fine, just don’t organize them in a way that it’s easy to compare developers).
• I care much more about a feature being done and ready to be deployed than I care about when a developer hands it over to QA, so instead of measuring how long it takes developers to complete their first pass at development, I’ll measure how long it takes from the time we start working on a feature to the time that it’s ready to be deployed. We need to break down the traditional silos of development and have BAs, QAs, and developers work together to achieve one goal.
• I value communication and collaboration, so when I’m talking to the team, I’ll emphasize working together, being willing to help each other out, pairing, and things like that. I don’t want developers to think that they should be smart enough to figure something out when they’re stuck, I want them to ask for help and I want people to solve each other’s problems.

Every team, company, and IT department wants to collect metrics these days. Sometimes metrics can be used for good purposes (e.g. are we on schedule), and some can be used for bad (e.g. let’s come up with some way that we can attempt to quantify how good our teams are so that we can compare them).

Let’s just get this out of the way — I really dislike metrics when they are used incorrectly. I think metrics are a good way of measuring your own progress internally. For example, I want to know if I’m on schedule to meet a target date. If the team is estimating work, I want to know how the actual time spent on the features compares to the estimates so that we can get better at estimating. I want to know how much time team members are spending on different activities (e.g. requirements gathering, development, production support) so that I can make future plans and staff the team accordingly. I might want to measure cycle time (the amount of time from when we start working on a feature to the time when the feature is deployed or ready for deployment). In all of these cases, two things are true: I’m not publicizing the metrics outside our team, and all of the metrics are measuring the performance of the team, not the performance of individuals.

I may use the metrics in order to know what message to send to a project sponsor. For example, if I can determine from our metrics that we’re ahead or behind, I would let the project sponsor know how ahead or behind we are. I might use cycle time to come up with a date when we think we can have a feature completed.

What I don’t want to do as a team lead is share any of the raw data with the people above me (unless I can trust them). Why? Because I don’t want to give them an opportunity to misinterpret the metrics and manage my team for me. If I’m leading a team, it’s my job to lead the team and it’s the team’s job to deliver. Too often, the higher-ups will use metrics in the wrong way.

This is what happens when your manager comes to you and says that you have to cram in some extra scope and cut out testing. They saw a metric (time for a developer to complete a feature) and made a decision based on it. They essentially are saying that “development time” is the most important metric, when in reality, what’s more important is the time to do all of the work (requirements, development, and testing). They are stepping in and managing the team and choosing a metric that you gave them (a development estimate).

This is why you have to choose your metrics wisely. What if you gave estimates to your project sponsor in terms of how many person-hours it takes? Let’s say that feature X will take 4 hours of BA work, 2 days of development, and 1 day of testing. In the end, that means that 3.5 days of people’s time will be needed. You probably know how much available BA and QA capacity you will have just like you know how much development capacity you will have. So why not portray it as one big bucket of capacity? Maybe you still break down the time and give that estimate by role. You run the risk of the sponsor trying to lop off the testing, at which point you have to earn your salary and explain to them why you need that much testing time (we wouldn’t say that we needed it if we didn’t need it, right?). If they don’t make QA people available to you, then use developers or BAs to do the QA. Either way, you need 3.5 hours of people’s time to do it.

Notice what I did differently — I gave a whole-team estimate. I didn’t just say how long it takes to do the development. That opened the door for your project sponsor (who probably doesn’t know how to manage your team) to staff the BA and QA team for you.

To me, the kinds of metrics I want to expose outside my team are things like features delivered, money saved/earned for the company, and things that show business value. Ultimately, this should be what matters to the business. These aren’t always easily quantifiable (which makes it difficult), but this should be what project sponsors and executives should ultimately care about. In fact, I would keep a list of achievements that you know have business value and report that up. Your project sponsor/manager would love that because they’ll turn around and report those same things to their boss. Now you’re speaking in the language that they should be caring about (cost savings, business value), and you’re free to manage the team the way that you know how.