Douglas Hubbard, in his book "The Failure of Risk Management", claims that risk management failed us in the lead up to the GFC because of flawed risk models, the use of qualitative risk assessment through the use of risk matrices or both. He contends that anything can be measured and that we should be measuring.
The case for quantification
There is no doubt in my mind that quantification is better than using our best judgement because our minds are at the mercy of our psychological biases. A couple of examples:
Confirmation biases - eg: If you are told a contractor is a poor performer you will have a tendency to pick up on bits of information about their poor performance and ignore the data about their good performance. This is because we all have a tendency to hear through all the noise the "evidence" to confirm our initial feeling about a subject.
Biases from the "availability heuristic" - eg: Shark attack. We tend to overestimate the likelihood of a dramatic event if we have lived it and seen it or been exposed to it in the news recently because the evidence of it has recently been available to us. This is why we underestimate the likelihood of disasters after a long period of calm.
The case against quantification
Two points only here:
1. Business is extremely complex and to do quantification justice it can be very resource intensive. The harder we make it for the business the less likely they will listen to us.
2. You can have the best risk models and risk modellers in an organisation, however, if you have a poor risk culture, calamities are not far away. By taking the less complex option and bulking-up risk management efforts with subjective risk ratings with minimal quantification, you are more likely to lead more staff to better consider risk in their decision-making.
The solution: Create datasets to provide yourself with the opportunity to quantify risks. I am always saying there is no "right way" to do risk management. ISO 31000 itself indicates it is a guidance standard providing principles and guidelines rather the "right way". So I believe both quantification and qualification have their place, however, longer term I believe we need to increase our ability to quantify risk. If I were you I would be looking to create datasets where success and failure rates can be derived. This would result in more informed analysis of risks as common as IT budget blowouts.
Comments
Risk Culture Building is the process of growth and continuous improvement in the way each and every person in an organisation will respond to a given situation of risk as to mitigate, control and optimize that risk to the benefit of the organisation.
No two people will respond the same way to a situation of risk, the way any person responds to risk is influenced by a number of factors, the main ones are:
• Nationality & culture
• Childhood experiences (and formative environment)
• Work ethics, trust & honesty
• Education (and the way it was obtained)
• Work experience
• Religion and other spiritual thinking
• Attitude towards life (and death)
Risk practitioners generally fail to address the underlying human aspect. Since the publication of the Basle accord, ISO 31000 and other standards and regulations, it has often been argued that compliance with these standards and regulations will mitigate and control risk, but this is only true if the standards and regulations are embraced in an effective Enterprise Risk Management Culture. Just like the policies, procedures and systems, these are worthless if human attitude, acceptance and desired response lack.
Addressing the aspect of people risk is the only way an organisation can improve the way their people respond to a situation of risk and the effectiveness of their risk management function. No organisation can ever have a perfect risk management culture, but organisations can achieve a level of maturity where they have an effective risk culture process and every employee is risk-minded and does something on a daily basis to mitigate, control and optimize risk
The development of Risk Culture Building is focused on awareness and training in business ethics and human behaviour, both the behaviours we want to encourage and the behaviours we ant to avoid. Organisations should frequently evaluate the progress (or regress) they are making on the path to maturity and implement action plans
Agree on trying to predict performance. That's what we do ( on an individual risk basis) all the time in our risk assessments/impact analyses. But the one area we don't do well or often is go back and check what actually happened from an overall perspective and then compare to our initial predictions. And that, I expect, is one of the only things non-risk trained managers will accept as a performance predictor/lesson learned. We should use whatever status and monitoring metrics are being gathered and place them against our initial predictions of business performance over the period of the program/project. This could also indicate the results of management decisions on control actions/non actions so we could say since you did this, this happened. So comparing our initial predictions against final results might accomplish some "believable" (trying to prove a negative is incredibly difficult) metrics for the overall ROI of risk management.
Predicting business team future performance would require a roll up of all risks and some knowledge of risk culture (behavior under pressure) but should provide better indications of risk culture and allow us to indicate how the entire program will perform against simply doing individual risks or looking at individual risks on a risk list. This is a interesting consideration in a risk management process - looking at the risk culture as well as individual risks and trying to determine how that culture will affect team performance. Don't think a lot of risk managers/risk processes have that as part of their work efforts currently.
David - I agree and have been spending quite a bit of time of late working on exactly this issue. Metrics are no doubt one part of the equation, however, since risk management is all about managing the uncertainty around our objectives, if the business achieves its objectives, it is very difficult to declare that risk management was the reason for success. Most business managers will tell you it was because of their good management or because of the team they built around them. This leads to the next question, "Which came first, the high performing business team that is good at risk management or good risk management that produced a high performing business team?"
I believe the answer is in prediction of future performance of business teams based on measuring risk culture (through a series of higher level risk metrics which would be difficult to go into here) and comparing the prediction with business outcomes. If we can predict business team poor performance (good and bad) based on measurements of risk culture then we can start to draw conclusions about the performance of our risk management program. The key here is in having the right predictive model of performance alongside metrics for assessing the quality of activity driven by the risk management program. Your thoughts?
The one area I agree with Douglas Hubbard in is that very few (if any) organizations use some type of metrics to determine if their risk management efforts are bearing fruit. There are some metrics for how successful risk mitigation efforts are and a few on how many risks are found per month, how risk levels are lowered, etc., but none that are looking at the overall "success" rate of a risk management process. Did we identify the proper risks, are we really lowering the risk level for the organization? Since you don't really do anything unless you can measure it, how do we show that risk management, from a strategic/enterprise perspective, is addressing the right risks?