[Editor Note: This letter to the editor was originally published in Volume 38 Issue 1 of Hazard Prevention (now Journal of System Safety) in 1Q 2002. The letter has been reformatted, but the text is unchanged.]

The following is a response to Jack Crawford’s article entitled “Opinion: What’s Wrong with the Numbers? A Questioning Look at Probabilistic Risk Assessment.” The article appeared in the third quarter 2001 issue of JSS.

In a recent opinion article in the third quarter 2001 issue of Journal of System Safety, Jack Crawford criticizes Probabilistic Risk Assessment (PRA) for various shortcomings. We would like to respond to those criticisms, and we have organized our responses according to the basic questions that Mr. Crawford poses, and the answers he submits.

Question 1: To what extent does PRA encompass the main causes of accidents?

Mr. Crawford presents examples in which a PRA did not identify the causes of failures or accidents occurring. The examples he presents are functional failures in which the system operated but did not function as required.

We respond that it is true that most PRAs do not model functional failures, but instead model one of two operational failures: failure to operate or failure to function. A failure to operate occurs when the system fails to run as defined by sufficient components running. A failure to run also includes the starting of the component. In contrast, a failure to function occurs when the system fails to perform as required. All the components of the system operate or run, but the system still fails its function.

An example of a functional failure is an air conditioning system that operates but fails to perform its function of cooling a control room. Another example is a spray system that operates but fails to wash the air of radioactive particles. Still another example is a booster rocket that operates but does not have sufficient thrust to lift the spacecraft to the desired orbit.

Most PRAs only model failure to operate. In particular, a PRA that uses only component failure rates models failure to operate. A well-constructed, failure-to-operate PRA is a credible and useful model if properly applied. It is applicable to a system that has been operating and functioning, or one that is a reproduction of a system that has operated and functioned. When a PRA models a new system that has not operated, the PRA needs to clearly indicate which failure is being modeled. This is not a deficiency in the basic PRA method but a deficiency in communicating the assumptions and boundaries of the analysis.

A PRA can also model functional failures. Many nuclear power plant PRAs include functional failures. For example, response times of safety systems and operators are modeled. Safety system performance is modeled as a function of temperature, flow, pressure and components that have failed. Steam binding of pumps and water-hammer effects on water lines are modeled. Another performance feature modeled is the dynamic forces on a valve when it attempts to close under accident pressures and flows. [Refs. 1-4]

When functional failures are modeled, performance must be modeled. Performance modeling involves defining output performance variables and their relation to input variables and conditions. Performance modeling also involves modeling continuous variables, not just the usual binary variables (fail or no fail) that are modeled in an operational model. When performance and functional failures are modeled, the analyst must obtain and interpret system performance requirements, and gather and model performance characteristics that have been measured in tests. The functional failure models that are developed are then incorporated into the PRA, as they are in nuclear power plant PRAs.

Thus, the fact that the PRAs examined by Mr. Crawford did not include functional failures is not an indictment of PRA methodology. It merely indicates a lack of communication of assumptions and constraints of the PRAs. When PRAs are compared using applicable failure definitions and accounting for quantified PRA uncertainties, the results are generally consistent with history. These comparisons are made by properly comparing probability predictions with experience.

When probability predictions are compared with experience, detailed causes generally cannot be compared. Instead, patterns of occurrence and categories of causes are compared. This is the standard approach in comparing any probability prediction with experience. The U.S. Nuclear Regulatory Commission (USNRC) has issued various reports on making such comparisons, including comparing precursors to accidents. [Ref. 5,6] Precursor occurrences, because of their higher probabilities, can be meaningfully compared with PRA predictions, using available data to obtain statistical conclusions with confidence.

The comparison of detailed causes is not really a statistically meaningful comparison, even if applicable system failure types are compared. Generally, when there is only one failure occurrence, no conclusions can be reached on comparisons between causes due to the sparseness of data. One needs to carry out valid statistical tests to reach valid conclusions. The author’s conclusions may be interesting conjectures, but they have no statistical basis.

Regarding observed failures, Mr. Crawford states in the beginning of his article that he rarely sees “random” failures modeled in a PRA. “Random” in PRA and in reliability modeling does not mean “having no cause.” “Random” means that the pattern of failure times can be modeled by a probability distribution. All failures have causes. However, a PRA is concerned with modeling the probability distribution of failures. Fitting a probability distribution to failure time occurrences is a standard statistical procedure.

The author goes on to say that a gap in PRA is caused by assumptions that become invalid or are invalidated. A PRA, as with any model, can only be meaningfully interpreted within the bounds of its assumptions. Using a PRA or any model with invalid assumptions is not the fault of the PRA or model. Instead, it is the fault of the application. It is important that the PRA clearly state its assumptions and bounds. If these become invalid or are invalidated, then the PRA is not applicable. The USNRC, as part of its risk management plan, reviews nuclear power plants to ensure that the assumptions of the PRA are valid. If they are not, actions are taken so that the assumptions become valid. [Ref. 7,8]

Mr. Crawford further criticizes PRAs for their lack of accounting for people’s perceptions of risk. PRAs do not account for perceptions of risk. Instead, they evaluate technical risk, including the probabilities and measurable consequences of accidents. The decision-maker must then use these results as one of the inputs in making decisions. A PRA does not make decisions. It provides input for decision-making. When changes are made, the assumptions and bounds of the PRA are no longer valid. Other risks may be incurred, as the article notes. However, if these are not in the bounds of the original PRA, the PRA is no longer applicable and needs to be modified.

Mr. Crawford notes that after accident free periods, complacency may set in. Overconfidence may breed shortcuts and underdesigns. Thus, he indicates that the PRA is no longer valid. This again reflects the fact that the PRA assumptions and bounds are no longer valid. To ensure that the risk remains low, the assumptions and bounds of the PRA need to be valid. This is part of an effective risk management plan, as is conducted by the USNRC.

Finally, the author criticizes PRAs for not modeling management effects. PRAs implicitly include modeling effects if the probabilities and failure rate data that are used are plant-specific. It is true that PRAs generally do not explicitly model management effects. However, this does not mean that the results evaluated by a PRA are erroneous. An effective risk management program includes understanding the PRA and ensuring that the assumptions and bounds are maintained to lower risk. A safety culture and risk culture need to have an understanding of the risks, and PRA can be a significant tool in developing this understanding. Within an effective risk management program, a PRA can be used to ensure low risk, if it is complemented with management actions that consider risk.

Question 2: Can statistical inference take us forward from the past to the future?

Mr. Crawford argues that to extrapolate collected data, conditions of stability must exist in the collected data. He also argues that the recorded data depend on the conditions under which they were collected. He points out that prediction therefore requires applying judgment and knowledge to the available data.

We respond that Mr. Crawford is right that data cannot be naively extrapolated. The conditions under which it was collected and the stability of data need to be considered. It is true that most databases do not ensure the stability of their data, and many do not clearly identify the conditions under which the data were collected. When data are used, information must be included that not only accounts for statistical uncertainties, but also for the variability in conditions.

When Mr. Crawford discusses numbers, he references only point values with no associated uncertainties. With PRA results, we cannot be confident in a point value for a probability, such as a failure probability of 0.0035. We can only be confident in intervals, or ranges, of probabilities. These probability intervals are often very wide because of uncertainties in the data, as well as in the modeling. In well-performed PRAs, the uncertainty range for a risk result is often a factor of 10 or larger where there are gaps in knowledge.

A critical component of PRA is the uncertainty and sensitivity analyses that accompany the calculations. A PRA that reports only a point value is inadequate. A decision-maker who uses only a point value is a misinformed decision-maker. Recorded data can be used to predict the future only if knowledge and judgment, as well as a thorough uncertainty and sensitivity analysis, accompany that prediction.

Mr. Crawford argues that most statistical analyses assume independence of failures. He also argues that the dependency approaches typically used in PRAs, such as common-cause failure beta factors, are arbitrary. Many statistical analyses do assume independence of failures. However, most good PRAs include dependency models, such as beta factor models. Beta factor models and other associated common-cause failure models are not arbitrary. Significant effort by the USNRC has resulted in databases for beta factors and other common-cause parameters that are used to estimate dependent failure probabilities. [Ref. 9,10]

Beta factors and other common-cause failure parameters are empirical parameters that are estimated from actual occurrences of dependent failures. They are used to estimate the probabilities of dependent failures occurring in similar conditions. When these dependent failure probabilities are identified as significant contributors to PRA risks, more specific analyses are carried out to identify causes and protective measures against such dependencies.

The author is correct in saying that the account accompanying a PRA result is at least as important as the number and associated uncertainty range. No PRA result should be taken at face value.

Question 3: How much force does the mathematical theory of probability add to the probability statement?

Mr. Crawford argues that the definitiveness of a PRA prediction is a delusion. He also gives references on the meaninglessness of precise probability statements.

We respond that the mathematical theory of probability determines the laws for combining probabilities. It says nothing about the precision or believability of the probabilities that are obtained. Again, we can’t be confident in any PRA result that is quoted as an exact point value — only in probability intervals. The uncertainty and sensitivity analyses that accompany the PRA results, as well as the modeling and assumption descriptions, are what give the PRA results credibility.

Question 4: If the numbers generated by a PRA do not represent probabilities of future events, are they still useful? If so, for what?

Mr. Crawford argues that the numbers are useful for identifying risk contributors and for identifying risky situations. He also says that he learns by digging for answers to the questions raised by the numbers.

We respond that PRA numbers are useful if interpreted within the bounds of the analysis and the associated uncertainties. However, PRA provides more than just the final numbers. The relative importance of the contributors is prioritized, which helps to focus attention and actions. These relative importances generally have smaller associated uncertainties than the final absolute numbers.

The risk uncertainties and risk sensitivities identified by the PRA offer information as valuable as the final numbers. These uncertainties and sensitivities identify focal points for gathering additional information, or for instituting actions to control the sensitivities. The qualitative results of the PRA are as valuable as the quantitative results. The qualitative results show the relationships among the contributors that lead to risk. They also show the nature of the contributors, and their redundancies and diversities. Viewing a PRA as providing only final numbers misses most of the useful information it contains.

References

1. U.S. Nuclear Regulatory Commission (USNRC). “Severe Accident Risks for Five U.S. Nuclear Power Plants,” NUREG-1150, June 1989.
2. Harper, F.T. et al. “Evaluation of Severe Accident Risks: Quantification of Major Input Parameters,” NUREG/ CR-4551, June 1991.
3. Kelly, D.L. et al. “Assessment of ISLOCA: Risk Methodology and Application to a Westinghouse Four- Loop Ice Condenser Plant,” NUREG/ CR-5774, April 1992.
4. USNRC. “Accident Source Terms for Light Water Nuclear Power Plants,” NUREG-1465, February 1995.
5. Belles, R.J. et al. “Precursors to Potential Severe Core Damage Accidents, A Status Report,” NUREG/CR-4674, June 1995.
6. Johnson, J.W. and Dale M. Rasmuson. “The USNRC’s Accident Sequence Program: An Overview and Development of a Bayesian Approach to Estimate Core Damage Frequency Using Precursor Information,” Reliability Engineering and System Safety, pp. 205-216, Vol. 53, No. 2, 1996.
7. Camp, A.L. et al. “The Risk Management Implications of NUREG-1150: Methods and Results,” NUREG/CR- 5263, August 1989.
8. Caruso, M.L. et al. “An Approach to Using Risk Assessment in Risk Informed Decisions on Plant Specific Changes to the Licensing Basis,” Reliability Engineering and System Safety, Special Issue on Developments in Risk-Informed Decision Making for Nuclear Power Plants, pp. 231-242, Vol. 63, No. 3, 1999.
9. Mosleh, A. et al. “Guidelines on Modeling Common Cause Failures in Probabilistic Risk Assessment,” NUREG/CR-5485, November 1998. 10. USNRC. “Common Cause Failure Data Collection and Analysis System,” NUREG/CR-6268, June 1998.

The video below provides a comprehensive introduction to modern PRA techniques – Ed.

by Jack Crawford

[Editor Note: This Opinion piece was originally published in Volume 37 Issue 3 of the Journal of System Safety) in 4Q 2001. The article has been reformatted, but the text is unchanged.]

Probabilistic Risk Assessment (PRA), or Probabilistic Safety Assessment as it is called in the nuclear power industry, has been developed over the last 30 years as a discipline heavily influenced by the mathematical theory of probability. But while its mathematical methods are endlessly extended and refined in industry literature, how confident can we be that the output numbers mean what they claim to mean — i.e., probabilities of future events? I believe that the time has come to think about this basic issue.

I recently initiated a study of the foundations of PRA. In this article, I identify some key questions that needed to be asked about PRA’s credibility, and even come up with a few (albeit provisional) answers. The factors that prompted this study are:

• The incredible magnitude of many probability numbers
• The often overly optimistic assumption that an assessment encompasses all credible failures
• The observation of gross discrepancies between predictions and outcomes
• The difficulty in finding examples of accidents caused by genuinely random component failures
• The narrow focus of PRA on measurable events, especially failure rates, and the fact that it can ignore accidents which are not caused by failures

During 17 years of involvement in risk and safety assessment in the weapon systems field in the U.K. and in Australia, I have been bombarded with numerical probabilities. Many of them have seemed incredible, or at best have ventured into the unknowable, with some of the powers of ten ascending into the high teens and even the twenties. The record in my experience was a probability of premature functioning of a mine fuzing system predicted to be 1 in 1044.

In another example, the design authority (DA) for a weapon system decided to include in it an electromechanical device which had an excellent record in another application. After pages of calculations to assess the effects of stresses in its new application, the DA predicted that its probability of mechanical failure would be 9.116 in 109 operating hours. The operating cycle time of the device was only 40 seconds at a likely rate of fewer than 10 cycles per battlefield day, so the predicted failure rate should have seen us through many times more use than the system would ever get in service. But in a system test that included four of the devices, we had two mechanical failures before they had accumulated one hour of operation. The failures happened in two different modes, neither of which had been considered in the analysis.

I have found it more difficult to find examples of accidents caused by what textbooks and safety standards describe as “random” failures. Three years ago, a dozen of us attended a meeting in the U.K. Ministry of Defence at which the contribution of random hardware failures to accidents was questioned. Between us, we could think of only one example of an accident caused by a combination of genuinely random events. Six years ago, the U.K. Health & Safety Executive published a booklet called Out of Control [Ref. 1], which contained 34 examples of control system failures. In the summary of causes at the end of the booklet, not one system failure is attributed to random hardware failure. If that kind of failure were indeed a major cause of accidents, we could surely expect it to turn up somewhere in 34 examples.

Readers may remember the disastrous first flight of the European Space Agency’s Ariane 5 rocket in June 1996, when it exploded 40 seconds after launch. According to Aviation Week [Ref. 2], the pre-launch estimate of the probability of a successful mission was 98.5%. The reality, as the report of the Board of Inquiry [Ref. 3] showed, was that the design ensured that the rocket would crash after 40 seconds. The real probability of success was zero, so the estimated probability was optimistic by a factor of infinity. This example illustrates:

• There was a gross discrepancy between prediction and outcome
• There was nothing random about any of the causes
• The accident was not caused by component failures; the inquiry did not report that any component of the rocket system failed to behave as it was designed to behave throughout the short flight
• The analysis did not consider the real causes of the accident, which in this case were errors of management.

After observing these and other examples, it seemed reasonable to look into the methodology of PRA. In the course of a few quick checks, my pocket calculator failed to find anything wrong with the mathematics of any of the assessments that were readily at hand, so my next step was to investigate the basis on which the mathematical structures were built.

For several years, I have been searching for a test of the theory that says we can draw probabilistic data on failure rates from past experience, and then synthesize a selection of the data in order to predict the failure rate of a new system. Safety and reliability literature does not help much because it generally goes no deeper than the mathematics upon which the theory is built.

My search has involved talking to many people in the U.K., including the Civil Aviation Authority, the Health & Safety Executive, and several leading engineering companies, as well as academic and engineering institutions. The only entity that attempted to test the theory was AEA Technology. They gave me a study [Ref. 4] that compared predicted and observed reliability figures for equipment used in nuclear power plants. It concluded that the correlation was reasonably good. That was useful, but the study seemed to have two shortcomings. One was that it looked at failure rates at the reliability level, rather than at the safety level, which (in the military field at least) are much harder to predict. The other was that it had been done as an afterthought, so it was not the properly designed and controlled experiment for which I had been looking.

Based on this data, I find myself being driven toward a conclusion that the scientific method may never have been applied to this particular theory — i.e, PRA. The apparent lack of science in this field threatens to become the most disappointing finding of this study. I hope to be proven wrong in this conclusion.

The Main Questions

Having observed that PRA might be questionable, it became necessary to decide what questions should be asked. There are four key questions: one practical, one theoretical, one philosophical and one contingency question that depends on the answers to the other three.

Question 1: To what extent does PRA encompass the main causes of accidents?

This is the key practical question. First, it is inevitable that any potential causes, modes and effects of failure which have not been foreseen will escape the attention of PRA. One of the effects of the ever-increasing complexity of systems is that we must expect to find some failure modes that we have failed to anticipate. We can and should do more thinking to reduce the number of missed tricks. But, when we have done our best, we still have no way of knowing whether we have thought of everything, as the example of the electromechanical device illustrated.

Second, PRA tends to lead us into a mindset which assumes that systems fail only if their critical components fail. It does not lead us to think enough about the class of accidents in which everything functions as designed.

Here are some examples:

• Turner [Ref. 5] describes a collision on an unmanned railway level crossing. The drivers of the train and the road vehicle did nothing wrong, and there was no equipment failure.
• Kletz, quoted by Leveson [Ref. 6], describes an accidental release from a computer-controlled chemical reactor. No human operator was involved. The automatic control system, in triggering the release, functioned as designed.
• From my own experience, an antitank mine design was proposed, which in certain conditions would have killed soldiers laying the mines according to the correct drill.

A third gap in the coverage of PRA is caused by invalid, or invalidated, assumptions. The assumptions made in a safety assessment are not always made explicit and may later be forgotten. When an important assumption is invalidated by changed circumstances, and nobody knows that it was made or that anything depended on it, an accident will be waiting to happen as soon as certain conditions prevail. One of the findings of the subsequent inquiry is likely to be that in those conditions, the probability of the accident was one.

A major source of uncertainty is the way people respond to their perceptions of risk. For example, Adams [Ref. 7] produces evidence that the compulsory use of seat belts has not improved road safety. He shows how the reduced risk to people in vehicles has been balanced, through small changes in drivers’ behavior, by an increased risk to those who are not in vehicles. He also provides an example of such “risk compensation” being enshrined in law: in Germany, coaches fitted with seat belts are allowed to travel faster than those without. In civil aviation, there has been concern about the frequency of near misses between aircraft queuing to land at busy airports. Yet the U.K. National Air Traffic Services, observing that aircraft have become better at station-keeping, have decided to reduce the vertical interval between aircraft “stacked” while awaiting clearance to land. Even NATO is not immune. The announcement of a forthcoming workshop on insensitive munitions [Ref. 8] specified objectives which included both “reduction in collateral damage in the event of an accidental initiation” and “reduction in safety zone for storage and transportation.” The organizers seemed unaware that the latter benefit can be gained only at the expense of the former. In these ways, potentially effective measures to improve safety, for which quantified claims are commonly made, may in practice be consumed in return for some other benefit such as improved performance.

In many fields, the fact that an accident had not happened for a long time would be seen as indicating a low, and probably diminishing, risk. As the time since the last accident increases, that view will be reinforced by conventional statistical methods indicating that the probability of an accident is being reduced because the mean time between failures is increasing. The reality may be quite different. Many of us will have come across examples of accident-free periods leading to complacency and greatly increased risk. In the civil engineering field, Petroski [Ref. 9] identifies the “design climate” as a critical factor in catastrophic failures of bridges. His argument, based on examples, is that a period of successful use of a novel design can lead a designer to become overconfident and consequently to underdesign a new structure in the interest of economy or beauty. The bridge is then liable to fail if it is subjected to extreme conditions. In situations such as these, where risks change inversely as people’s perceptions of risk change, our attempts to pin down numerical probabilities of accidents are likely to be about as successful as trying to capture a will-o’-the-wisp.

Of all the sources of risk which PRA overlooks, management must be the most prolific. Many apparent technical failures have their roots in management weaknesses. Leveson [Ref. 10] points out that “unmeasurable factors (such as …. management errors) are ignored even though they may have greater influence on safety than those that are measurable.” As she was writing those words, the European Space Agency was committing the management errors that led to the Ariane Flight 501 debacle, while using measurable data to predict a high probability of success.

An important aspect of risk management is the quality of the culture in an organization. For example, the Piper Alpha inquiry found that “Senior management … adopted a superficial response when issues of safety were raised,” and the judge in the Herald of Free Enterprise case criticized the “disease of sloppiness” which had spread down from the top of the Townsend Thoresen company. In each case, the company’s safety culture had contributed much to the disaster.

All of those sources of risk are soft, or unmeasurable, factors. They affect the frequency and scale of accidents, but PRA does not encompass them. It focuses, rather, on the measurable causes, modes and effects of failure. With so limited a view of the scene, PRA must be expected to deliver optimistic results, contrary to what we normally aim to do in risk assessments. In terms of the “As Low As Reasonably Practicable” (ALARP) principle, the consequence is that PRA can neither demonstrate that a risk is as low as reasonably practicable, nor that it is tolerable.

Question 2: Can statistical inference take us forward from the past to the future?

This question addresses the theoretical basis of PRA, for which the apparent absence of any proper justification or test was noted above. The clearest argument I have found is one developed by Deming [Ref. 11], in which he explores the limits of statistical inference. He argues that the historical results that provide input data for predictions depend on the sets of conditions in which they were produced, and that those exact conditions are unrepeatable. Furthermore, as Feynman [Ref. 12] reminds us, we cannot assume that all of the conditions that contributed to a result were recorded or even noticed. In other words, the historical record is not a reliable guide to the future. Worse still, it can be hard to tell whether it is even a reliable guide to the past.

In statistical terms, Deming concludes that there is no mathematical method from which to extrapolate past results to future conditions, and consequently, no objective way of assigning a numerical probability that a prediction will be right or wrong. Prediction therefore means applying judgment and knowledge of the subject to the available data, rather than just manipulating numbers.

A further problem is that most statistical methods assume that component failures will be independent. In reality, dependent failures contribute to many accidents. The “fudge factors” sometimes introduced to allow for dependencies, such as cut-offs and beta factors, do at least move the numbers in the right direction. On the other hand, they are arbitrary and are no substitute for an understanding of the dependencies within a system and their potential consequences.

As an aid to predicting the behavior of systems, Deming [Ref. 13] advocates the concept of stability developed by Shewhart. “Stability” in this context means that the functions of the system display a stable range of variation. He argues that stability is a prerequisite for predictable behavior, and that in a manmade system, it is not a natural state — it has to be achieved and maintained. Systems are constantly threatened by destabilizing influences, so their stability must be monitored and, whenever necessary, restored. Hence, a system will remain stable and predictable only by virtue of people’s vigilance, knowledge and effort. It is not a question of probability.

Without stability, there is no basis for prediction, but I have yet to find a safety or reliability database which assures us that its estimates of component failure rates were derived from stable systems by stable methods of measurement. Some may have been so derived, but even then, when we take those types of components and build them into a new system, we leave the stability behind because we have changed the operating environment. A new state of stability will have to be achieved and maintained, and new data generated for monitoring and predicting behavior.

Collectively, those arguments seem to falsify the theory that we can rely on historical frequency data to take us across the boundary between the past and the future. To that conclusion many would reply that our contracts and our regulators nevertheless insist that we deliver predictions in the form of numerical probabilities. What, then, should we do? Many years ago, Tukey [Ref. 14] offered some relevant advice: “It is far easier to put out a figure than to accompany it with a wise and reasoned account of its liability to systematic and fluctuating errors. Yet if the figure is to serve as the basis of an important decision, the accompanying account may be more important than the figure itself.” That seems to indicate a reasonable way to go.

Question 3: How much force does the mathematical theory of probability add to a probability statement?

This is the key philosophical question. In looking for an answer, I have used ideas put forward by Toulmin [Ref. 15]. When we make a prediction, especially a safety prediction, we want as much precision as we can manage. Toulmin distinguishes between precision in the sense of definiteness and precision in the sense of exactness. So for example, if we judge that an event is extremely unlikely to happen, we are relying on definiteness. But if we estimate a probability that the event will happen twice in a thousand rocket launches, we are relying on exactness. This leads to further questions, such as how much do we gain when we are able to add exactness to definiteness? And what should we do if we find that we have one, but not the other? Those sorts of questions may seem ethereal to some people, but the study indicates that they actually matter when it comes to making decisions such as whether a system is safe enough to be accepted for service.

PRA uses mathematical probability in an attempt to deliver precise predictions. But Toulmin, from a logician’s standpoint, argues that “Little is altered by the introduction of mathematics into the discussion of the probability of future events” and that “The development of the mathematical theory of probability accordingly leaves the force of our probability-statements unchanged; its value is that it greatly refines the standards to be appealed to.”

If we accept the arguments of Deming and Shewhart, the refinement is spurious in the context of PRA. (Deming [Ref. 11] points to areas in which numerical probability does provide a valid guide to action, but they do not relate to PRA.) The spurious refinement of the numbers is starkly illustrated by the examples given earlier, in each of which, when the definiteness of the prediction proved to be a delusion, its exactness was exposed as ridiculous.

A relevant, if irreverent, statement of philosophy comes from Feynman [Ref. 16], who preferred engineering judgment to what he regarded as meaningless numerical probabilities: “If a guy tells me the probability of failure is 1 in 105, I know he’s full of crap.”

Question 4: If the numbers generated by PRA do not represent probabilities of future events, are they still useful? If so, for what?

This is the contingency question, and it clearly needs to be answered. My view is that the numbers are still useful. For one thing, factors that are measurable do contribute to risk, and PRA has been successful in helping us see how to reduce risks from those causes (it may even have contributed to the scarcity of accidents from “random” causes). For another, its inherent optimism tells us, when it indicates a risk that is too high, that improvements are definitely needed. Also, I found, when working as a safety regulator in the weapon systems field, that I can learn much from the numbers by digging for answers to the questions they raise.

Conclusion

The study remains incomplete, partly because of the difficulty in finding a justification for PRA. If anyone can find or construct one, it would be very welcome. Meanwhile, the provisional conclusions to be drawn are:

a) The numbers delivered by PRA do not represent the probabilities of future events because:

• The PRA methodology, by focusing on measurable factors, ignores some of the most significant sources of risk.
• The theory that it is justifiable to extrapolate historical data, in order to assign a numerical probability to a future event, is false.

b) If PRA is used on its own to support an ALARP or any other safety case, it is likely to be misleading. To be complete and credible, the case should provide:

• Qualitative data and arguments on the issues not covered by PRA
• A reasoned account of the liability to cause an error in each quantified prediction

c) Quantitative probability statements have no more force than qualitative probability statements. At best, they may be more refined, but only if the numbers can be shown to be credible.

d) Our quest for reliable predictions would be better served by paying more attention to the stability of the systems from which we draw data, and to the stability of those whose behavior we need to predict.

Should PRA be scrapped? My answer is no, for the reasons given in the answer to Question 4. It remains an invaluable tool for focusing our minds on issues related to measurable factors. We do not need to believe that the numbers are probabilities in order to use them for purposes such as comparison of design options, sensitivity checks and the improvement of designs. It is only the “P” of PRA that ought to be abandoned if nobody can justify it.

By now, it is clear that there is a Question 5 to be answered: “What would be a better way, and what place should (P)RA have in it?” The investigation continues.

References

1. Health & Safety Executive. Out of Control. HSE Books, Sudbury, Suffolk, U.K., 1995.
2. Aviation Week & Space Technology. p. 33, July 29, 1996.
3. Report by the Inquiry Board. Ariane 5 Flight 501 Failure, Paris, July 19, 1996.
4. Snaith, E. R. “The Correlation between the Predicted and the Observed Reliabilities of Components, Equipment and Systems.” U.K. Atomic Energy Authority National Centre of Systems Reliability, Culcheth, U.K., 1981.
5. Turner, Barry A. Man-Made Disasters, Wykeham Publications, London, 1978.
6. Leveson, Nancy G. Safeware, p. 165. Addison-Wesley Publishing Company, Reading, Massachusetts, 1995.
7. Adams, John. Risk, Chapter 7. UCL Press, London, 1995.
8. NIMIC Newsletter, NATO Insensitive Munitions Information Center, Brussels, 1st Quarter 2000.
9. Petroski, Henry. Design Paradigms — Case Histories of Error and Judgment in Engineering, Cambridge University Press, 1994.
10. Leveson, Nancy G. Safeware, p. 59, Addison-Wesley Publishing Company, Reading, Massachusetts, 1995.
11. Deming, W. Edwards. “On Probability as a Basis for Action,” The American Statistician, Vol. 29, No. 4, pp. 146- 152, 1975.
12. Feynman, Richard P. The Meaning of It All, Addison-Wesley Longman Inc., 1998.
13. Deming, W. Edwards. The New Economics for Industry, Government, Education, Massachusetts Institute of Technology, 1993.
14. Tukey, John W. The American Statistician, Vol. 3, p. 9, 1949.
15. Toulmin, S.E. The Uses of Argument, paperback edition, Chapter 2, Cambridge University Press, 1993.
16. Feynman, Richard P. What Do You Care What Other People Think?, paperback edition, p. 216, HarperCollins, London, 1993.

Acknowledgments

The author acknowledges, with thanks, the constructive comments provided by Professors David Kerridge and Henry Neave, and by Felix Redmill, editor of Safety Systems, in which an earlier version of this article was published.

About the Author

Colonel Jack Crawford spent most of his working life in the British Army, having been commissioned into the Corps of Royal Engineers in 1949. He has served, among other places, in Korea, Norway, Germany, the Pacific and Australia. He became seriously interested in risk and safety assessment during his appointment to the Ordnance Board of the U.K. Ministry of Defence in 1978. After serving on the Ordnance Board, he became a member of its counterpart, the Australian Ordnance Council, during an interesting period of major Royal Australian Navy and Royal Australian Air Force re-equipment programs. Since leaving the Army, he has continued to work in the safety field, mostly for the Ministry of Defence, and is currently working on improvements in the methods used for safety assessment.