约翰Ridgway on safety-critical systems

卢克Muehlhauser: What is the nature of your expertise and interest in safety engineering?

约翰Ridgway: I am not an expert and I would not wish to pass myself off as one. I am, instead, a humble practitioner, and a retired one at that. Having been educated as a physicist, I started my career as a software engineer, rising eventually to a senior position within Serco Transportation Systems, UK, in which I was responsible for ensuring the establishment and implementation of processes designed to foster and demonstrate the integrity of computerised systems. The systems concerned (road traffic management systems) were not, initially, considered to be safety-related, and so lack of integrity in the delivered product was held to have little more than a commercial or political significance. However, following a change of safety policy within the procurement departments of the UK Highways Agency, I recognised that a change of culture would be required within my organisation, if it were to continue as an approved supplier.

如果有任何导致t合法性his forum, it is this: Even before safety had become an issue, I had always felt that the average practitioner’s track record in the management of risk would benefit greatly from taking a closer interest in (what some may deem to be) philosophical issues. Indeed, over the years, I became convinced that many of the factors that have hampered software engineering’s development into a mature engineering discipline (let’s say on a par with civil or mechanical engineering) have at their root, a failure to openly address such issues. I believe the same could also be said with regard to functional safety engineering. The heart of the problem lies in the conceptual framework encapsulated by the terms ‘uncertainty’, ‘chance’ and ‘risk’, all of which appear to be treated by practitioners as intuitive when, in fact, none of them are. This is not an academic concern, since failure to properly apprehend the deeper significance of this conceptual framework can, and does, lead practitioners towards errors of judgement. If I were to add to this the accusation that practitioners habitually fail to appreciate the extent to which their rationality is undermined by cognitive biases, then I feel there is more than enough justification for insisting that they pay more attention to what is going in the world of academia and research organisations, particularly in the fields of cognitive science, decision theory and, indeed, neuroscience. This, at least, became my working precept.

卢克你觉得是起源n of your concern for the relevance of philosophy and cognitive science to safety engineering? For example did you study philosophy, orKahneman and Tversky, in university?

约翰：作为一个物理学生，我提供了一点机会（并且诚实没有雄心壮志）在哲学中追求教育。当然，Quantum Mechanics就是我珍贵的信念，在不观察到的客观现实的情况下，我非常感谢I know it makes no sense whatsoever but just shut up and carry on calculating’school of practice. In fact, it wasn’t until I had become a jobbing software engineer, investigating the use of software metrics to predict development timescales, that philosophical issues started to take on an occupational importance.

The practice within my department had been to use single-point estimations for the duration of tasks, then feed these figures and the task dependencies into a Gantt chart and simply use the chart to read off the predicted project end date. Sometimes the results were very impressive: predicted project end dates often proved to be in the right decade! Then one bright spark said, ‘You’re living in the dark ages. What you need are three-point estimations for the duration of each task (i.e. estimate a most likely duration together with an upper and lower band). You then use Monte Carlo Simulation to create a project outturn curve; a curve which indicates a statistical spread of possible project durations. Everyone does this nowadays. And it just so happens I have an expensive risk management tool to do all the nasty calculations for you’. Imagine his look of disgust when I told him that I still thought the old way was better!

他跟一个自以为是的白痴吗?我想没有t. The basis of my objection was this: As a physicist, I was well aware of the benefits of using Monte Carlo Simulation in areas such as solid state and nuclear physics. Here it is used to determine the macroscopic behaviour of a physical system, where probability distribution curves can be used to model stochastic variability of the micro behaviour of the system under study. Now, however, I was being invited to use Monte Carlo methods to draw conclusions based upon the averaging of various levels of posited (and decidedly non-stochastic) ignorance regarding the future. In such circumstances, no one could provide me with a convincing argument for deciding the most appropriate form for the probability distribution curve upon which the simulation would be based. In fact, I was told this choice didn’t matter, though quite clearly it did. If, as seemed likely, a realistic distribution curve would have a fat tail, the results would be hugely influenced by the choice of curve. Furthermore, the extra two estimates (i.e. the upper and lower bands for task duration) were supposed to represent a level of uncertainty, but the uncertainty behind their selection was at least of the same order of magnitude as the uncertainty these bounds were supposed to represent. In other words, one could not be at all certain how uncertain one was! It occurred to me that no real information was being added to the risk model by using these three-point estimates, and so no amount of Monte Carlo Simulation would help matters.

这让我对natu哲学沉思re of uncertainty and the subtleties of its relationship with risk. These musings had a potentially practical value because I thought that a lot of people were spending a lot of time and money using inappropriate techniques to give themselves false confidence in the reliability of their predictions. Unfortunately, none of my colleagues seemed to share my concern and all I could do to try and persuade them was to wave my hands whilst speaking vaguely about second order uncertainty, model entropy and the important distinction between variability and incertitude. So I decided there was a gap in my education that needed filling.

发生在发生后，我的调查很快就引起了我的工作Professor Norman Fentonof Queen Mary University London, and I familiarised myself with concepts such as subjective probability and epistemic versus aleatoric uncertainty. Furthermore, once subjectivity had been placed centre stage, the relevance of cognitive sciences loomed large and although I can’t claim to have studied Tversky and Kahneman, I became familiar with ideas associated with decision theory that owe existence to their work.

Suddenly, my career seemed so much more interesting. And once I moved on to safety engineering, the same issues cropped up again in the form of over-engineered fault tree diagrams replete with probability values determined by ‘expert’ opinion. Now it seemed all the more important that practitioners should think more deeply about the philosophical and psychological basis for their confident proclamations on risk.

卢克: In many of your articles for the Safety-Critical Systems Club (SCSC) newsletter, you briefly discuss issues in philosophy and cognitive science and their relevance to safety-critical systems (e.g.1,2,3,4,5,6）。在您在安全工程项目工作期间，您的同事似乎有多感兴趣？他们中的许多人已经深入熟悉这些问题？概率和风险的哲学以及（例如）启发式和偏见的认知科学似乎是那些工作安全工程的标准培训的一部分 - 至少是你遇到的人民？

约翰: Perhaps my experience was atypical, but the sad fact is that I found it extremely difficult to persuade any of my colleagues to share an interest in such matters, and I found this doubly frustrating. Firstly, I thought it to be a missed opportunity on my colleagues’ part, as I felt certain that application of the ideas would be to their professional advantage. Their diffidence was to a certain extent understandable, however, since there was nothing in the occupational training provided for them that hinted at the importance of philosophy or psychology. However, what really frustrated me was the fact that no one appeared to be at all excited by the prospect of introducing these subjects into the workplace. How could that be? How could my colleagues fail to be anything other than utterly fascinated? In fact, their lack of interest seemed to me to represent nothing less than a wanton refusal to enjoy their job!

The key to the problem lay, of course, with the training provided by my employer, and it didn’t help that the internal department that provided such training glorified under the title of ‘The Best Practice Centre’. Clearly, anything that I might say that differed from the company endorsed view was, by definition, less then best! And I soon found that berating the centre’s risk management course for failing to explore the concept of uncertainty was, if anything, counter-productive. Upon reflection, I think that some of these frustrations led me to seek an alternative forum in which I could express my thinking. Publishing articles for the Safety Critical Systems Club newsletter provided such an outlet.

卢克：你说你“觉得肯定认为这个想法将是他们的专业优势。”你能给我一些原因和/或举例，为什么你感觉到这一点？

约翰: I think that my concerns were the product of working within a profession that appears to see the world rather too much in frequentist terms, in which the assumption of aleatoric uncertainty would be valid. In reality, it is increasingly the case that the risks a systems safety engineer has to analyse are predicated predominantly upon epistemic uncertainty. I cite, in particular, safety risks associated with complex software-based systems, adaptive systems and so-called Systems of Systems (SoS), or indeed any system that interacts with its environment in a novel or inherently unpredictable manner. Whilst it is true that analysing stochastic failure of physical components may play a significant role in predicting system failure, the probabilistic techniques involved in such analysis simply cannot address epistemic concerns, i.e. where the parameters of any posited probability distribution curve may be a matter for pure speculation. (I am aware that Monte Carlo simulation is sometimes used to probabilistically model the parametric uncertainty in probabilistic models, but this strikes me as an act of desperation reminiscent of the invention of epicycles upon epicycles to shore up the Ptolemaic cosmology).

安全分析师有许多合适的方法可以寻求适应认知不确定性（贝叶斯方法，可能性理论和Dempster-Schafer，而不是三个）。然而，虽然从业者甚至没有意识到存在一个问题，并且继续承担所有概率的客观性，但很少希望这些方法能够吸引他们应得的注意力。

Then, of course, we have to consider the pernicious effect that cognitive bias has upon the analyst’s assessment of likelihood. It is in the nature of such biases that the individual is unaware of their impact. Surely, therefore, even the most basic training in this area would be of considerable benefit to the practitioner. On a similar theme, I have become concerned that the average safety analyst is insufficiently mindful of the distinction to be made between risk aversion and ambiguity aversion. This may lead to a failure to adequately understand the rationality that lies behind a particular decision, but it may also explain why my colleagues didn’t appear to appreciate the importance of undertaking uncertainty management alongside risk management.

Finally, when one considers the interconnectivity of risks, and the complications introduced by multi-stakeholders, it becomes very difficult to think about risk management strategies without having to address ethical issues associated with risk transfer, optimisation and sufficing. But perhaps that is another story.

卢克: Yes, can you say more about the “ethical issues associated with risk transfer, optimization, and sufficing”?

约翰：在英国的健康和安全立法中，有义务减少现有的风险水平“至关重要地实际”（SFAIRP）。这导致剩余风险的概念“与合理切实可行的一样低”（ALARP）。ALARP概念假定可以定义上限，上面有风险被认为是“无法忍受”的风险。此外，有一个下限，下面的风险被认为是“广泛接受”。只要可以证明进一步的减少需要不成比例的成本和努力，可能允许风险在这两个限制之间位于这两个限制之间。除了这里使用的术语的模糊性，这个观点的主要问题是它对一个风险的管理可能符合另一个风险的可能性一无所有。实际上，人们可以设想这种敲击效应将传播的连接效果的网络，导致风险净水平增加（记住传播可能包括正极和负反馈循环）。出于这个原因，存在全球至少等同的（大风）原则，该原则认为，在修改系统时，必须评估系统构成的总体，而不是纯粹对修改意图的风险来聚焦的风险等级亚博体育苹果app官方下载地址。这一想法当然是整体层面永远不应该增加。

So far this has all been very basic risk management theory and, on the face of it, the ALARP and GALE principles appear to complement each other. But do they always? Well, in the simple case where all risks are owned and managed by a single authority, this may be the case. But what if the various risks under consideration have differing owners and stakeholders? In such circumstances, parties who own risks and seek to reduce them SFAIRP may find themselves in conflict with each other, with the various stakeholders and with any body that may exist to ensure that the global risk level is not increased.

Perhaps we are now in the province of game theory rather than decision theory. If so, it seems reasonable to insist that the game be played in accordance with ethical constraints, but has anyone declared what these might be? Some seem obvious; for example, never transfer risk to another party without their knowledge and consent. Others may not be so straightforward. I think we are all familiar with the old chestnut of the signalman who can prevent a runaway train from ploughing into a group of schoolchildren by changing the points, but only by causing the certain death of the train driver. Does the signalman have the moral authority to commit murder? Would it be murder whether or not he or she switches the points? If we find this difficult to answer, one can easily envisage similar difficulties when deciding the ethical framework associated with the management of risk collectives.

卢克：您最渴望安全关键系统行业的哪些变化？亚博体育苹果app官方下载

约翰: I think that there is a lot to be said for making membership of a suitably constructed professional body a legal imperative for undertaking key systems safety engineering roles. Membership would require demonstration of a specified level of competence, adherence to formulated codes of conduct and the adoption of appropriate ideologies. Given my responses to earlier questions, your readers will probably be unsurprised to hear that I hope that this would provide the opportunity to promote a greater understanding of the conceptual framework lying behind the terms ‘risk’ and ‘uncertainty’. In particular, I would like to see a professional promotion of the philosophical, ethical and cognitive dimensions of system safety engineering.

I appreciate that the various engineering disciplines are already well served by a number of professional societies and that, for example, an Independent Safety Assessor (ISA) in the UK would be expected to be a chartered engineer and a recognised expert in the field (whatever that means). However, the epistemic uncertainties surrounding the development and application of complex computer-based systems introduce issues that perhaps the commonly encountered engineering psyche may not be fully equipped to appreciate. It may be that the safety engineering professional may need to think more like a lawyer. Consequently, the professional body I am looking for could be modelled upon the legal profession, as much as upon the existing engineering professions. I know for some people ‘lawyer’ is a dirty word, but so is ‘subjectivity’ in some engineers’ minds. Being pragmatic, and in order to stay in the game, we may all have to become sophists.

卢克：谢谢，约翰！