Introduction

At software architects we live and breathe agile development. We started with Scrum in the early days of our company when we built the first bits of our flagship product time cockpit. Over time we changed to Kanban but the agile principles have been our constant companion for many years now.

Besides developing time cockpit, we also license our base technology called Cockpit Framework (short CoFX) to other ISVs and help them building cloud-based SaaS solutions. Of course we work agile in such projects, too. But one thing differs: Project reporting. Although most our customers understand the ideas of agile development, the simple facts of real-world business life force them to think in budgets, milestones, project plans, etc.

Over the years we have developed a solid practice for reporting in agile projects. It tries to bridge the (apparent) gap between classic project management with rigid, upfront project plans and ever changing environments in agile projects. In this article we would like to share our most important learnings. Additionally, we describe the KPIs on which our reporting practice relies.

The Core Question: Are We on Track?

“Are we on track?” This is the core questions our customers want to have answered by our regular - typically monthly - project reporting. We break down this question into two aspects:

1. How are we doing in terms of budget?
2. Will we be able to finish on time?

These questions are not specific to agile software projects. They have been playing an important role as long as project-oriented work exists. Therefore, we did not have to invent a completely new reporting and KPI framework. We could build on something that’s already there: Earned Value Management (EVM) (see also Earned Value Analysis for further information in German)

Earned Value Management (EVM)

The basic idea of EVM is that you have to “earn” your project’s budget by completing scheduled work. At the start of the project the Earned Value EV is 0 because you have not completed anything. At the end of the project you have earned the entire Budget at Completion (BAC) assuming that you have completed all planned work.

The following chart illustrates the relationship of Earned Value, Planned Value and Actual Cost. At milestone t1 - could be e.g. a sprint meeting, end-of-month-report, etc. - the project team has completed more work than planned (EV > PV, Schedule Variance SV > 0). Therefore the team is performing better than planned in terms of schedule. At milestone t2 the situation is different. More work should have been completed at t2 (EV < PV, SV < 0). Additionally the team spent more money than it should have until t2 (AC > EV, CV < 0).

Let me describe the basics of EVM with a very simple example. Think back when you were a little kid and your parents promised you 10$ for mowing the lawn. From experience you knew that this job would take you 2 hours. So your hourly rate would be 5$.

• EVM would call the 10$ Budget at Completion (BAC).
• You plan to get something to drink after having mowed half of the lawn. EVM would call this a Milestone.
• The Planned Value (PV) for reaching your milestone would be PV = 10$ * 50% = 5$.
• Your Schedule according to EVM would be to reach your milestone after 1 hour and finish your work after 2 hours.

Equipped with this knowledge you get the mower and start your work. After 1.5 hours you finished 50% of the lawn. In this extremely simplified example, EVM could help you to answer the question mentioned above: Am I on track?

According to EVM you would calculate the Earned Value (EV) when you have reached your milestone:

• EV = 50% (completed work) * 10$ (Budget) = 5$

Now you can compare EV with your Actual Cost (AC) to get the Cost Variance (CV):

• AC = 1.5 hours (time need to reach milestone) * 5$ (hourly rate) = 7.5$
• CV = EV - AC = 5$ - 7.5$ = -2.5$

A negative value for CV indicates that you are not on track in terms of budget! Your goal would be to have a CV >= 0.

You can also get an indication whether you are likely to finish in time. For this you could calculate the Schedule Variance (SV) after working for 1 hour:

SV = 0$ (EV after 1 hour because you have not reached any milestone)
         - 5$ (Planned Value PV after 1 hour) = -5$

Again the negative value is an indication that you are not on track in terms of schedule. As a manager you would need to dig deeper, find the root cause of the problem, and take the appropriate actions.

Big Upfront Planning for EVM in Agile Projects?

At a first glance EVM looks horribly old-style for someone who values the principles of agile development. In order to be able to calculate EVM’s KPIs, you need an overall budget, a milestone plan, work items with cost estimations, network schedules, etc. All this assumes some upfront planning. But isn’t agile all about avoiding big upfront plans and designs?

Agile does not mean that upfront planning is unwanted or even impossible. In his book The Scrum Field Guide[4] the Author Mitch Lacey describes how release planning works in Scrum. The starting point is the product backlog. Whenever we start a project with a customer, we start with scoping workshops in which we work out high-level requirements (aka Epic Stories). They make up the initial product backlog.

Next, we set the scope for the initial release together with the customer. The product backlog for the initial release gives us all we need to start with EVM. It does not even matter if you choose relative (e.g. story points, T-Shirt sizes) or absolute numbers (e.g. person days, working hours, Dollars) to estimate the effort for the items in the product backlog. You can always calculate the EV and compare it to the PV.

Use Ranges Instead of Single-Value Estimates

Regular readers of our blog know that we believe in range estimations when it comes to effort estimation. This influences the way we handle the PV, as the following figure illustrates. Our customers always get estimations in form of lower and upper bounds. In many cases the upper bound acts as the cap of the project’s cost.

Practical Example

So how does this look like in practice? Let’s assume a rather simple project:

1. The Product Backlog consists of five work items (=Epic Stories).
2. For every backlog item we have an estimation in person days (lower and upper bound).
3. The employees record their working time per work item.
4. Assumptions (simplifications):
   1. Linear resource investment (if the plan would assume less resources at the beginning and more resources at the end, the model would be more complex).
   2. Only direct costs, indirect costs are not taken into account.
   3. Only personnel costs, no material costs.

The following table shows how a simple project report could look like. It starts with the effort estimation, the actual costs, and an estimation of the completion per work item. Based on these KPIs, EV, PV, CV, and SV are calculated. As you can see in the table, the project is on track in terms of schedule (SV > 0). In terms of budget the project is not perfectly ok as CV is < 0 compared to the lower bound but still > 0 in comparison to the upper bound. For us this would be a clear indication that costs and budget has to be closely monitored and managed in this project.

Effort Tracking in Time Cockpit

Analyzing effort each month is important but in our experience not sufficient. Therefore we monitor the effort continuously using dedicated time cockpit lists. They show the estimated effort per work item (lower bound because this is what we always try to achieve) and compare it to the actual effort. The following figure shows such a time cockpit list. It is taken from a rather small project directly out of our productive system (click to enlarge):

Where is Agility?

As mentioned above, EVM can definitively be applied to agile projects. However, there is a certain risk of losing agility. It is important that all stakeholders understand that the plan which underlies the EVM calculation is not fixed. It has to be adopted to the learnings during the project. If the product backlog changes, the EVM calculation has to be changed accordingly.

In our experience the secret of success it to make sure that people responsible for budgets understand the agile approach. They have to learn that agile projects do not have an upfront plan that is frozen and must not change. We have seen that EVM-based reporting helps e.g. product owners in a Scrum project to recognize budget issues in an early stage of the project.

Today many peoples apply EVM to agile projects and write about their experiences. If you want to dig deeper into this topic I encourage you to read the articles Earned Value for Agile Development and AgileEVM – Earned Value Management in Scrum Projects.

Lessons Learned

Over the years we had to learn some lessons regarding project reporting the hard way. Here are our top learnings which we want to share with you:

1. Establishing a good project reporting practice is important and many teams struggle with it[6]. EVM is a good extension to e.g. Scrum as it helps to identify budget issues in early project stages.
2. Agile project management does not eliminate the need for upfront planning. However, you should carefully manage the effort for upfront planning and delay any planning work that is not absolutely necessary for the initial budgeting and scoping phase.
3. Make sure that all stakeholders (including budget owners) understand the idea of agile development.
4. The quality of the EVM-based reporting largely depends on the estimation of the degree of completion. Therefore a solidDone-Done checklistis crucial for EVM. Otherwise you will have unpleasant surprises at the end of your project.
5. With the cloud, computing costs change from indirect to direct costs. Traditionally, costs for development or testing servers are not calculated as project costs. The variable cost nature of the cloud enables you to assign computing costs to specific projects easily. In larger projects this can be a noteworthy cost block.
6. Don’t forget to calculate and report apportioned effort (e.g. coordination overhead, sprint meetings, etc.).
7. Don’t calculate too meticulously. Often it is enough to calculate KPIs only on project-level. If the results indicate that there is a problem, you can drill into the details.
8. Scrum burn charts are somewhat similar to EVM. However,burn charts on sprint-level are not enough. EVM as described in this article helps the product owners to manage costs and schedule across sprint boundaries.

Bibliography

[1] Electronic Industries Alliance EIA, ANSI/EIA-748-1998: Earned Value Management Systems, http://www.srs.gov/general/EFCOG/02GovtReferences/03NDIAANSI/ANSIEIA748.pdf, 1998

[2] National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), ANSI/EIA-748-A Standard for Earned Value Management Systems Intent Guide, http://www.srs.gov/general/EFCOG/02GovtReferences/03NDIAANSI/NDIAIntentGuide.pdf, 2005

[3] US Department of Defense, Earned Value Management Implementation Guide, http://guidebook.dcma.mil/79/EVMIG.doc

[4] Lacey M, The Scrum Field Guide – Practical Advice for Your First Year, Chapter 11. Release Planning, Addison-Wesley, ISBN 0-321-55415-9

[5] Withall S., Software Requirements Patterns (Best Practices), Microsoft Press, ISBN 0-735-62398-8

[6] Pfister E. et al, Key Performance Indicators, in The Swiss Project Management Review Magazine, 6/2010, page 19ff, http://www.pmi-switzerland.ch/pm@ch/201008/PM@CH-Edition6_Summer-2010.pdf