This is the last article in my initial four-part series on the concept of project analysis. In the first three articles I spoke about how my company name reflects the analytical approach I bring to projects, provided a conceptual definition for project analysis, and gave my thoughts on what makes for good project analysis. The point of these first three articles was to provide a foundation for understanding project analysis and my approach to diagnosing and solving problems projects. Now we can we’ve got the basics in hand, we can dig into some more meatier discussions.
This post summarizes and distills an article written by a legend in the academic world of risk analysis, Baruch Fischhoff. I know what you’re thinking….academic works can be frustrating and boring to read. Even though Fischhoff brilliantly distills complex ideas, it is still a long and heavy piece to read. So let’s make it a deal then. I’ll do the reading for you. I’ll address the key points and give my thoughts about why they are important to capital project development.
[Side note: the original article was published in Science and is behind a paywall, but there is a similar and much shorter article available on HBR that I recommend.]
Fischhoff’s main goal in this article is to identify a few challenges with risk-cost-benefit analysis and how we can address these challenges. As always, I’ve taken some editorial liberties to spice up the subject matter.
- Your analysis might be bad, but your job is to make it less bad. Fischhoff speaks volumes to how analytical methods are imperfect and can always be improved. This is pretty obvious point, but then he goes on to discuss how most GOOD analysis is often born from BAD analysis.
What does this mean? It means that any published analyses are subject to critical (and sometimes savage) reviews. Holes are poked in the methods, assumptions are identified, and simplistic approaches and uncovered. It gets put through the wringer. But in the end, you’ve hopefully identified how to make it stronger. And guess what? This whole process of critique/improve is iterative: smarter and broader minds will continue to suggest improvements. A huge lesson here is that analysts need to thoroughly document assumptions, calculations, and thought-processes so the work can be reviewed and examined appropriately.
What does this mean for capital projects? Understand the shortcomings of your analytical models, either as a way to reduce the shortcomings, or simply to be aware of the limitations of your analysis. Consider how we use discounted cashflow (DCF) methods such as Net Present Value (NPV) or Internal Rate of Return (IRR) for project evaluation. Using static values for highly volatile parameters in these models is highly flawed and doesn’t come close to predicting the absolute value of a project, but they may perform better at predicting the relative valuation of projects that are both subject to the same volatilities. Likewise, we could assess our use of static average values for volatile parameters like commodity prices and foreign exchange rates and determine if there is a better way to model these using probabilistic methods.
- Analysis can be subjective. Get over it. Subjectivity in analysis is practically unavoidable. Why? Because subjectivity comes from relying on expert judgement. Fischhoff goes into great detail two different types of expert judgements. Ethical judgement sets the stage of the analysis, framing the scope, terms, and even the question being considered in the analysis. Scientific judgement affects the mechanics of the analysis: it provides input to the details and parameters that are seeking to answer the analytical question.
Engineers are scared of anything that doesn’t deal with hard numbers, so we tend to underappreciate subjective judgements as we perceive them to be unscientific or lacking analytical rigour. But here’s the thing: expert judgement is by its very definition subjective. Where subjectivity becomes a problem is when we hide it, ignore it, or gussy it up with a veneer of quantitative methods.
What does this mean for capital projects? Don’t let subjectivity be the elephant in the room. Get subjectivity, in all its uncomfortable mushiness, out in the open. Allow experts to discuss and disagree. Also, don’t misuse expert judgement to assume or present a higher degree of analytical objectivity. Think back to the last project risk assessment workshop you attended. In my experience, more often than not the experts in the room spend their time trying to come up with seemingly objective assessments of probability and financial consequences. Instead let’s use our expert judgement to help characterise the nature of the risk – identifying a range of possible outcomes, the uncertainty underlying the risk, strength of knowledge and potential information improvements, and risk controls that can be put in place. This is the true benefit of expert judgement.
- Analysts should be unbiased, but they can’t live in silos. How often are analyses commissioned, created, and completed only to realize the analysis doesn’t give the insights that it was intended to produce? Or the analysis gives misleading results because the framing didn’t represent the interests of the stakeholders?
Fischhoff recommends following something called the analytical-deliberative process, which is just a fancy way of saying that analysts should talk to stakeholders to make sure they understand their concerns and objectives. If not, the results of the analysis might not answer the questions or provide the insight it was intended to.
What does this mean for capital projects? Analysts should be rational and dispassionate about the process and the results of the analysis, but they need to understand the objectives, key decision criteria, and constraints. I was once involved in a project where a project controls analysts did a detailed time-cost-tradeoff to determine that spending an additional $10M to accelerate the schedule would result in a $15M increase to project NPV, only to be rebuffed by a project sponsor who told them they would not be given any additional money even with a strong economic justification, they were soon reaching the end of their credit facility and had to minimize capital spending. In addition, the permitting regulations and community impacts would not allow an increase to the construction workforce that schedule acceleration would require. The best analysis in the world will fall on its face if it’s not built on a thorough understanding of constraints and objectives.
This article introduced some topics and themes that I’ll address in more detail in future posts, but for now I’d like you to think back to a recent issue or situation you analyzed as an input to a decision. How would understanding the comments above have changed the analysis. Would it have led to an improvement?
Until next time…