Five Simple Guidelines to Agile Metric Bliss

Over the past couple years, I’ve had the opportunity to work with many great teams in many industries. I often work with teams and their managers to generate reports. In doing so, I quickly realize that although teams may be working to adapt and leverage one of the frameworks that fall under the Agile moniker, they are not yet adopting or clearly understanding the Agile Principles and Values. This comes through clearly when we look at the kinds of reports people are using.

I’ve seen dashboards that compare multiple team’s velocity. Or, the classic utilization report that shows time worked by team member. Or, the quality reporting that focuses on who versus what. And then there is the singular snapshot that represents percentage of backlog done – doesn’t sound that bad until you have a manager have kittens because the percentage is not what as expected.

Now, I have to admit — I’ve made all these reports before and used them myself. Sometimes the purpose of the report had an intended outcome, but my best intentions sometimes resulted in gamesmanship of the numbers or fear by the team and in one case, a team member pulled me aside worried that he may be culled.

Let’s face it, all metrics and reports can be used for bad. But what do you need to do to create good metrics? There are some great resources all over the internet that help answer this question, but let me give you my top five things you can do to make your metrics effective while fostering an agile environment:

  1. Make them Transparent. This is obvious, but often I see people create reports and don’t share them. I get that there are some reports for “their eyes only”; however, in most cases, if not all, unless it has salary information — make the reports visible to everyone – the Team, Stakeholders, Managers, and even Customers.
  2. Make them Visual. Use charts, shapes, colors, and or pictures over a table of figures. We do this for three reasons – easy to read, reduces the likelihood that people focus solely on outlier values, and in many cases — creates conversations. By the way, use colors wisely — just like words, colors mean stuff.
  3. Follow the Trends. Goes hand-in-hand with visualize — a good metric should be informative provided indicators that make it easy to see if the needle is pointing up or pointing down. Trends generally allow you to decide if what we are doing is good or bad, and reduce snap decisions.
  4. Make them Relevant and Timely. There are the out-of-the-box metrics — burn downs, cumulative flow, burn ups, cycle time, and or velocity – that should be maintained on a daily, weekly, or iteration basis — updating them in arrears does no good. This is the same for all agile metrics. Since the goal of any metric should be to continuously improve in some way, reports/metrics that are created or updated weeks or months after the fact does us no good. And, couldn’t we better utilize the time and brain power on something current.
  5. Have a Purpose. Every report or metric needs to be leveraged for importance. If you cannot answer two questions about a report or metric, maybe you should stop spending time and money to create the report. First, why are we creating this report? And, second, what will we do if the report is indicating a need to adjust or change?

Now, these are my five simple (and in some cases, general) guidelines – what are your’s? Do you have any suggestions?

Always keep in mind …

Working software is the primary measure of progress.

Continuous attention to technical excellence and good design enhances agility.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

This entry was posted in Agile Development, Agile Management, Agile Metrics, Agile Project Management. Bookmark the permalink.

6 Responses to Five Simple Guidelines to Agile Metric Bliss

  1. Nick Henson says:

    Hi Matthew
    Good posting. I am currently working on an agile transformation at a large UK bank. There is a huge appetite for agile, but there is also a lot of resistance from traditional waterfall roles such as Project Managers and Programme Management.
    To fight this resistance I have had to adapt the project reporting we provide from our agile teams. The reporting till now has been prescriptive and hard to produce. Many reports have a low ROI and we have tended not to produce these. Gradually we have changed the quality and quantity of reports produced from agile teams. Each sprint we plan to give a sprint report showing product increments delivered, number of automated unit tests ran and number of failures, number of functional tests run and failures and number of new tests added.
    Your posting will help me refocus on how we produce reports

  2. Srinath says:

    Great article. “Agile” and “Metrics” seem to be contradictory terms – with metrics being primarily used to know the health of the project in the “Pre-Agile” world. In Agile, what ever metrics one can conjure up, working software is the PRIMARY measure of progress.

  3. Brian Hall says:

    Matthew,

    Great post! Your Guidelines serve to remind that the value of metrics is in what we do with them, not that you have them. It makes no sense to gather metrics if you are not going to use them to improve.

    In a response to Srinath, ultimately, you are correct, but there are so many more things that you can measure to help improve either your progress, or your predictability. For instance, if you have the right metrics in place you might be able to predict the following:
    - What is the expected productivity hit (time and duration) for bringing a new member of the team on board?
    - How often is a user story/work item broken into multiple items, and how many items usually result?
    - What is the expected productivity hit if a member of the team is out for a specific period of time?
    - How accurate are our story points (if you use them)?
    - Is the rate of defects found in the field increasing or decreasing each iteration or release?

    That’s just a small sample of some of the things that metrics can help a team be more productive and predictable about.

    Please notice that not one of them I mentioned singled out an individual team member. Although individual metrics can be quite alluring, I urge great caution in using them. If you measure how many user stories, or even story points, an individual works on you could destroy team dynamics because developers won’t want to spend time helping teammates because it might affect his/her personal metric. Or maybe your senior developers have low metrics in comparison to the rest of the team, but that could be because you are giving them the most critical and complex tasks.

    Just remember that whatever you measure will affect your team in some way. The team will naturally adjust itself to improve whatever metrics you are gathering, so choose those metrics that actually matter towards the end goal, which, as Srinath stated, is “working software.”

  4. Matt Badgley says:

    Great point Srinath, and often folks look at metrics and Agile not going together — I have a few fellow coaches that have echoed that. However, you cannot improve if you don’t know where you are at in certain respects. Burndown charts are metrics, code coverage is a metric, # of days without a broken build is a metric. There are a bunch of them out there and we use them generally for a single purpose and then throw them away when we are done with them.

    Nonetheless, I agree with the idea that the best metric for quality is what the customers are saying. When we all measure productivity, listening to the customer’s perspective is key — and in these cases this equals “working” software.

    • Srinath says:

      Thanks Matt. I agree – we should have the basic metrics in place to see where we can improve. Metrics such as burn down charts, defect metrics, test / code coverage are all good to keep track of – if it is used as a means of improvement for the project and does not involve too much additional effort. My only concern is we shouldn’t slip back to the old Waterfall model, where we had a separate team to cull all sorts of metrics and the Project team to respond why certain parameters were above / below limits – after the project was dead and buried.

  5. Suesh Kumar says:

    Thanks everyone for you excellent inputs on metrics & agile, what I think is that it is good to measure to improve but we should also be conscious to the cost of quality & teams must be sensitized about it even before they are asked to produce these metrics. If teams starts taking them as an over burden to them, it will defeat the whole purpose of producing the metrics.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>