Effort estimation is hard, especially when managing agile projects. At the beginning of a project you do not exactly know what you have to build. Nevertheless, CFOs, project managers, customers, teams that you have to work with, and many other stakeholders want and need estimations. No matter if you are a project manager or a software developer, at some point you will be asked for expected costs, roadmaps, timelines, necessary resources, and so on. The bad news is that people are typically very weak estimators. The good news is that you can improve your effort estimation skills with training.
I don't want to read the introduction, take me directly to the quiz
How Does the Effort Estimation Quiz Work?
The Goal: 90% Confidence
Before we go any further into estimation, I invite you to the following test. Below you find a list of 10 questions. They should not test your general knowledge. Instead, they should test your ability to estimate. Your job is to answer the questions using an interval. You should choose the interval so that you are 90% confident that the correct answer lies within. Some may argue that you could just enter such large numbers that it is absolutely sure that the answer is within the interval. Well, of course you could, but that wouldn’t be much fun. Try to set the intervals as narrow as possible but still in a way that makes you confident that 9 out of 10 values are within the specified ranges.
Before you start, try this example:
How wide [meters/ft] is an Airbus A380, the largest civil commercial aircraft ever built in series, from one wing tip to the other?
What would be the interval you would answer (confidence 90%)?
Fermi Method Can Help
Maybe some of you answer that they have absolutely no idea. Well, I bet that’s not 100% correct. Do you think that the plane is wider than let’s say 1km? Probably not. See, you have at least a rough idea. You could try to make your estimation better by using e.g. the Fermi method. Enrico Fermi, the Italian physicist who amongst other things invented the first nuclear reactor, was known for his ability to make good educated guesses with little or no actual data. The basic idea is to start from the few things that you might know or which you can at least reasonably estimate. Now you use basic math to get an estimation for the problem you have to solve.
Let’s try to apply this idea to the question mentioned above:
- Let’s start by concentrating on the body of the plane. How many people do you think sit together in a single row in economy class side by side? Did you ever fly in a large airplane? How many people did you count? I will tell you the answer for an Airbus A380: It’s 10.
- How wide is a single seat in an airplane? Those of you who did longer flights in economy class before know that the answer is “much too slim”. Seriously, what do you think? The answer is approx. 50cm (=20in).
- How many isles are in the body of the aircraft? All large long-distance airplanes have two. How wide are they? Probably similar to a seat’s width, that is 50cm (=20in).
- Now we can estimate the width of the aircraft’s body:
10 seat * 50cm + 2 isles * 50cm = 6m
(10 seats * 20in + 2 isles * 20in = 240in = 20ft).
In fact the max. cabin width of an Airbus A380 is 6.54m (=21ft 7in). We were quite close. - Now we know that the plane cannot be smaller than 6m (=20ft) in width. You could now try to recall from memory the relation between cabin and wing size from airplanes you know. What do you think? 1:3, 1:4, or maybe even more? Maybe the picture on the right can help. Probably you have an idea for a lower bound and an upper bound. Do the math …
You see that from only a few things that we know or reasonably estimate we can come up with an estimation about the range. Now what is your estimation? I will tell you the answer: Incredible 79.80m (=261ft 8in)!
If you want to learn more about Fermi problems and other estimation and measurement methods, I encourage you to read the book How to Measure Anything: Finding the Value of "Intangibles" in Business by D. W. Hubbard.
Equipped with the Fermi tool you can now do the quiz. Have fun!
Test Your Skills
Provide an interval which contains the correct answer. Set the interval so that you are 90% certain. Don't use sources like Wikipedia - the test is about estimation not about your ability to research on the web.
Analysis
Did you reach the goal of 9 correct intervals? Chances are quite high that you didn’t. To demonstrate that you are in good company, the result chart shows the distribution of the number of correct answers for all people who participated in the quiz (blue bars). The gray area shows how the distribution should look like for a confidence of 90%.
Numerous studies have shown that people tend to set the intervals much too narrow. I found studies mentioning actual confidence levels between 30% and 60% in software development projects when a 90% confidence level was required. Think about what that means for project management, estimated timelines, budgets, etc. - a nightmare for every project manager.
Many of you might argue now that their results would have been different if the questions in our quiz above would have been from a domain they are expert in (e.g. questions regarding software development projects where you are a project manager or developer in). Unfortunately, this is not true. Over-confidence is a phenomenon that is independent of intelligence or expertise, even independent of professions and cultures. Are experienced project managers better in effort estimation? Not necessarily.
“We should not expect that the level of over-confidence is reduced with more expertise” (Jorgensen, 2002).
Why do so many people fail in quizzes as the one you just did? In his excellent book Software Estimation – Demystifying the Black Art Steve McConnel tells about interviews he did with software developers and project managers. Many of them mention a kind of inner stress to set the interval as narrow as possible because of a kind of personal pride. Did you feel the same? Nassim Nicholas Taleb calls this phenomenon epistemic arrogance, our arrogance concerning the limits of our knowledge. We simply underestimate how little we know - or in effort estimation how poor we are in predicting the future.
If you are a project manager I encourage you to read N. N. Taleb's book The Black Swan: The Impact of the Highly Improbable. It really changed my entire thinking about estimation in business life in general.
6 Tips to Improve Your Estimation Skills
So what can you do if you are e.g. in project management and you have to estimate? Do you have to simply accept that all your estimations will be entirely wrong? The good news is that you can do something in order to get more successful when estimating. Here are some of my personal ways to improve:
1. Learn to Say "I Don't Know"
If you have to give an estimation for something that you really don't know, try to apply things like the Fermi method I introduced before. However, be aware of the fact that you have very limited data and try to resist the (internal or external) pressure to set estimation intervals to narrow. It is a well known fact that especially at the beginning of projects uncertainty is very high by nature (see also Cone of Uncertainty which has profound consequences on project management especially regarding effort estimation).
However, you rarely need to say "I don't know". If you are forced to estimate, answer with a range where you set the interval wide enough. Maybe even such a wide range is helpful for your counterpart (e.g. to recognize that his budget would not even be enough to cover the lower bound of your estimation).
2. Don't Do the Opposite And Always Say "I Don't Know"
Once you have learned that most people are over-confident, you might tend to setting your estimation intervals way too wide all of the time. This does not make sense, too. Even if your customer or manager really gives you the (over-estimated) time, your projects will not be successful (see also Parkinson's Law or the Student Syndrom).
3. Evaluate Your Estimation Accuracy
Training lets you get better - this is true for estimation, too. The Jorgensen, 2002 study mentioned above suggests "that software companies provide estimation training opportunities through their database of completed projects. An estimation training session should include estimation of completed projects based on the information available at the point-of-estimation applying different estimation processes". This is one of the reasons why I recommend project time tracking even in agile projects.
4. Become a Domain Expert and Benefit From Economy of Scope
In my article Six Reasons for Time Tracking in Agile Projects I refer to Stacey’s Agreement and Certainty Matrix. Try to move the majority of projects away from "Anarchy" to "Complex" or even "Simple". This gives you more data to come up with better estimations (remember Fermi). However, try to withstand the tendency to over-estimate your ability to predict the future.
5. Be Prepared
No matter how well your estimation model or domain expertise is - there will be situations in which your estimations are absolutely wrong. Prepare yourself for such situations. Here are some examples:
- Don't sign a contract with penalties for late delivery that can ruin you. Penalties for late delivery are sometimes ok if there is an acceptable cap.
- Negotiate your project prices so that you benefit from being unexpectedly productive and you don't lose too much money if you run into unexpected problems.
- Generally avoid agreements where you can win a little and lose a lot.
6. Try Things Out
You have an idea but you don't know (i.e. cannot estimate) whether it will work? Give it a try if a failure will not ruin you. Sometimes things go unexpectedly well and you become incidentally successful.
comments powered by 