Partners and Programs:
  • BARS
  • GSIP
  • SKYbrary
  • ASN
  • Contact Us
  • Members' Center
  • Login
  • Support Aviation Safety

  • Industry Updates
  • The Foundation
    • About the Foundation
    • Founders
    • Mission
    • History
    • Leadership
    • Officers and Staff
    • Media/Communications
    • Aviation Award & Scholarship Programs
    • Work with Us
    • Join Us
  • Events
  • AeroSafety World
  • Toolkits & Resources
    • COVID-19 Crisis Resources
    • Fatigue Management
    • Flight Path Monitoring
    • Global Action Plan for the Prevention of Runway Excursions (GAPPRE)
    • Go-Around Project
    • Global Safety Assessment Project
    • Learning From All Operations
    • Past Safety Initiatives
    • Pilot Training and Competency
    • Special Reports
  • Industry Updates
  • The Foundation
    • About the Foundation
    • Founders
    • Mission
    • History
    • Leadership
    • Officers and Staff
    • Media/Communications
    • Aviation Award & Scholarship Programs
    • Work with Us
    • Join Us
  • Events
  • AeroSafety World
  • Toolkits & Resources
    • COVID-19 Crisis Resources
    • Fatigue Management
    • Flight Path Monitoring
    • Global Action Plan for the Prevention of Runway Excursions (GAPPRE)
    • Go-Around Project
    • Global Safety Assessment Project
    • Learning From All Operations
    • Past Safety Initiatives
    • Pilot Training and Competency
    • Special Reports
  • Contact Us
  • Members' Center
  • Login
  • Support Aviation Safety
Partners and Programs:
  • BARS
  • GSIP
  • SKYbrary
  • ASN

FLIGHT SAFETY FOUNDATION HEADQUARTERS

701 N. Fairfax Street, Suite 250,
Alexandria, Virginia 22314

Phone: +1 703 739 6700 Fax: +1 703 739 6708

  • Aviation Safety Experts
  • AeroSafety World
  • AeroSafety World November 2015
  • Can Risk Be Managed If It Can’t Be Measured?

InSight, Risk Management

Can Risk Be Managed If It Can’t Be Measured?

by Barry C. Davis, Julia Pounds, Paul Krois, and Melissa Wishy | November 6, 2015

The U.S. Federal Aviation Administration’s (FAA’s) new compliance philosophy1 has introduced an oversight approach that proactively manages risks through the “identification and control of existing or emerging safety issues” and by concentrating resources on mitigating risk and problem solving. Traditionally, safety professionals focused on controlling known hazards while also trying to anticipate if or when unknown hazards would surface (Figure 1). For example, a special issue of Risk & Regulation in 2010 devoted eight articles2 to a discussion of risk management related to close calls, near misses and early warnings. The articles represented different industries and issues, ranging from oil refinery accidents to failures in physicians’ clinical performance to vulnerabilities in nuclear reactors.

To complicate matters, many definitions of risk exist, depending on whether the word is used in everyday conversation or by experts in different domains. In an effort to standardize the definition, the International Organization for Standardization (ISO) defined risk simply as “the effect of uncertainty on objectives.”3 However, uncertainty may exist because of either potential events or their consequences, and whether information is ambiguous or missing. Further, uncertainty can either positively or negatively impact one’s objective.

Today, an organization’s capability to accumulate information about risk and to use that information to develop mitigations can be labeled risk intelligence (or risk intel), that is, generating previously unknown knowledge or understanding from data. However, gathering risk intelligence is dependent on correctly positioning and providing resources to people, processes, systems and tools, including processes for transforming data into useful information for risk analysis, mitigation and planning purposes. The ability of an organization to gather risk intelligence increases the quality and quantity of information available for improved decision making.4,5

In the pursuit of safe outcomes, many aviation safety professionals have produced vast amounts of risk intelligence about the characteristics of known hazards. As illustrated by Figure 1, however, we need better methods to develop risk intelligence about unknown hazards. We cannot directly understand something we do not know, so are there new ways to capture characteristic signals or markers that can point out an underlying issue? This use of such indicators is very common in the medical field, where a full blood test, for example, provides an analytically valuable dashboard of chemical markers.

Figure 1 — Safety Compliance vs. Risk Management

Source: Barry C. Davis, Julia Pounds, Paul Krois and Melissa Wishy

Our operational definition of risk is an effort to clarify the relationship between safety and risk so that both can be measured. The intent is for aviation safety professionals to be able to better recognize how and when the risk occurs, what influences it, etc.

Given that a common goal of the global aviation community is to continually increase the level of safety, we offer a very simple solution for the complex problem of identifying risk:

  • We define risk as unintended variation; and,
  • We believe that today’s high level of safety can be increased by reducing unintended variation.

Aviation relies on a system of systems highly dependent on procedures to maintain flight safety and efficiency. Rules and regulations for all airspace users specify procedures containing best practices for executing safe operations. Like drivers of cars approaching head-on must know to stay on their own side of the road, safe aviation operations depend on the shared expectation that applicable procedures and best practices will be followed.

Maintaining safety requires controlling risk. However, safety is usually defined relative to absence of risk, a concept with no single agreed-upon definition. Instead, controlling risk often has involved various combinations of expectation, intent, opacity, action and outcome regarding either an entity’s purposeful engagement with an unknown or the outcome of an entity’s having encountered an unknown. This inconsistency poses problems for aviation professionals who are charged with identifying and reducing risk to improve safety levels. We suggest a common definition so that safety professionals can objectively identify, measure, study and reduce risk; identify changes in risk; and determine the effectiveness of risk mitigation strategies.

We defined risk as unintended variation because some intended variation among users of the National Airspace System must be tolerated to permit them to flexibly respond in unanticipated or uncontrollable circumstances. For example, some variation is tolerated and expected when users are allowed to exercise discretion and judgment to safely accomplish their goals. These situations are generally covered in safety controls within policies in phrases such as “the operator may discontinue the alerts if …” and “the documentation should include … .” Unintended variation means that which is outside approved and predetermined safe tolerance levels and is therefore not intended to occur.

In sum, alertness for unintended variation to identify risk provides an objective method for managers of aviation organizations in government and industry to address risk. Measuring the presence or absence of unintended variation is an objective approach to the more imprecise concepts of risk.

The scientific concept of variation is not new. To apply risk intelligence to aviation, however, involves understanding the nature of variation, whether variation is present and where none is intended. Our definition also aims to help safety professionals to take advantage of statistical methods For example, statistical process-improvement techniques have long been used to identify and remove the causes of variability in manufacturing. Processes have characteristic attributes that can be measured, analyzed, controlled and improved. Any variation during the process or in its outcomes can be identified and corrected by effective quality assurance and quality control programs. Stable and predictable results are attained by reducing variation in processes.

This insight into processes extends across many domains of human activity, including aviation safety, as illustrated by the examples below. It also can lead us to a better understanding about how, when and where unintended variation goes unnoticed in processes and systems. To appreciate the pervasive nature of the problem, we need only to begin asking questions like the following.

Do we want unintended variation in machine parts that are used in safety-critical equipment?

Do we want unintended variation in information that is used in critical safety decisions?

Is risk introduced if there is unintended variation in how we:

  • Process information through systems;
  • Format information to be used by systems;
  • Identify hazards;
  • Describe a hazard’s characteristics;
  • Describe hazards for developing mitigations;
  • Evaluate hazards for prioritizing resources;
  • Implement mitigations in systems;
  • Distribute mitigations in systems;
  • Make decisions;
  • Implement security; and,
  • Manufacture aircraft parts?

While this is not a complete list, unintended variation clearly is a marker that signals risk whether it involves a safety-related decision, mechanism, aircraft part or other element within the aviation system.

Barry C. Davis manages FAA’s Air Traffic Safety Oversight (AOV) Information Standards. Paul Krois is involved with FAA human factors research. Julia Pounds, Ph.D., manages AOV’s Research and Analysis. Melissa Wishy is a subject matter expert and senior policy analyst with AOV Information Standards.

This article represents the opinions of the authors and does not reflect FAA policy.

Notes

  1. FAA. “National Policy: Federal Aviation Administration Compliance Philosophy.” Order 8000.373. June 26, 2015.
  2. ESRC Center for Analysis of Risk and Regulation. Special issue on close calls, near misses and early warnings. London, U.K.: London School of Economics and Political Science, July 2010.
  3. ISO. ISO 31000/ISOGuide 73. Risk Management — Vocabulary, 2009.
  4. Apgar, D. Risk Intelligence: Learning to Manage What We Don’t Know. Boston: Harvard Business School Publishing, 2006.
  5. Funston, F. Surviving and Thriving in Uncertainty: Creating the Risk Intelligent Enterprise. Hoboken, New Jersey, U.S. John Wiley and Sons, 2010.
InSight is a forum for expressing personal opinions about issues of importance to aviation safety and for stimulating constructive discussion, pro and con, about the expressed opinions. Send your comments to Frank Jackman, vice president, communications, Flight Safety Foundation, 801 N. Fairfax St., Suite 400, Alexandria VA 22314-1774 USA or jackman@flightsafety.org.

Featured image: © Tifonimages | Adobe Stock; © stevanovicigor | istockphoto.com

Share:

Print:

Key Safety Issues

  • Controlled Flight Into Terrain (CFIT)
  • Loss of Control–In Flight (LOC-I)
  • Mechanical Issues
  • Runway Safety (approach and landing)
  • Sabotage/Intentional Acts
  • Midair Collisions (MAC)
  • Runway Safety (Conflicts)
  • Wildlife Issues
  • Fatigue
  • Cabin Safety
  • Emerging Safety Issues
    • Lithium Batteries
    • Safety Information Sharing and Protection
    • Unmanned Aircraft Systems

Related Content

Accident/Incident Investigation, Flight Ops, News, Safety Management

Safety News

The ATSB says 10 instances of fatigue cracks occurred in Boeing 737 flap mechanisms from 2017 to 2022.

by FSF Editorial Staff

News, Safety Management, Safety Regulation

NTSB: Slow Progress on ‘Most Wanted’ Items

A review of 19 related aviation safety recommendations says seven were met with an ‘unacceptable response.’

by Linda Werfelman

News, Safety Management

A Change in Plans

Change management programs can smooth the path for organizational changes.

by Michael Ott

Read more articles

701 N. Fairfax Street, Suite 250, Alexandria, Virginia 22314

Phone: +1 703 739 6700 Fax: +1 703 739 6708

Flight Safety Foundation on Twitter

How can we ensure #aviation is both sustainable and safe? A firebrand panel of industry leaders shares their insigh… https://t.co/7ZVY8244hI

February 6, 2023

Follow @flightsafety

Projects & Partners

  • Basic Aviation Risk Standard
  • Global Safety Information Project
  • SKYbrary
  • Aviation Safety Network
  • Donate
  • Advertise on our website
  • Sponsor & Exhibit at our Events
  • Work with Us
  • Contact Us
  • Site Map
  • Privacy

© 2023 Flight Safety Foundation

Join our group on LinkedIn