Safety performance indicators (SPIs) are essential to evaluating the success of an aviation organization’s safety management efforts, according to a speaker at Flight Safety Foundation’s 63rd annual Business Aviation Safety Summit (BASS), held in May in Chicago. And separate research shows significant variability in how individuals and organizations across the industry define and use SPIs and safety performance targets (SPTs).
In his BASS presentation, “Zooming in on Safety Culture — From Culture to Performance Indicators,” Christopher Young, director of safety solutions, Baldwin Aviation — Safety and Compliance, said, “Hope is not a method for success,” and added that SPIs can serve as a health check for organizations by showing them where they may be drifting from standards and processes that have been established to mitigate risk in operations.
Flight Safety Foundation also recognizes the importance of SPIs to safety management systems (SMS) and to monitoring an organization’s safety performance. During work on its ongoing Global Safety Information Project (GSIP), the Foundation developed an SPI survey to gain insight into how the industry uses safety data to define SPIs and SPTs. The survey also provided an opportunity to understand how industry applies these thresholds to monitor routine operations and identify potential or emerging safety risks, according to the GSIP Year Three Report, published earlier in 2018.
As Young pointed out in his BASS presentation, the International Civil Aviation Organization (ICAO) in Annex 19, Safety Management, defines SPIs as “data-based parameter[s] used for monitoring and assessing safety performance.”
The GSIP Year Three Report also notes that Annex 19 requires service providers to “develop and maintain the means to verify the safety performance of an organization and to validate the effectiveness of safety risk controls” and that a service provider’s safety performance “shall be verified” in reference to the SPIs and SPTs of the SMS. SPTs are the “planned or intended objective for safety performance indicator(s) over a given period,” according to ICAO.
In research conducted through an online survey as well as GSIP webinars and workshops, the Foundation discovered that “there is significant variability in how SPIs and SPTs are defined, interpreted, and used across industry. While ICAO Annex 19, Safety Management, and ICAO Doc 9859, Safety Management Manual, offer definitions and examples, FSF also discovered that different interpretations exist across industry and GSIP stakeholders.1 Whereas ICAO defines an SPI as a data-based parameter used for monitoring and assessing safety performance, some consider an SPI to be a broader term that refers to general risk areas of concern.”
In his presentation, Young said that the monitoring and measuring necessary in an SMS should involve the use of SPIs, as well as corresponding safety performance targets and alert levels. He said SPIs should be based on an analysis of data collected by organizations through risk assessments; various types of audits, safety surveys and reviews; and incident reporting systems, among other sources, and that SPIs generally are monitored using basic quantitative data trending tools that generate graphs or charts incorporating alert and target levels.
In its research, the Foundation identified a need to provide guidance for the development of meaningful SPTs and organizational alerting thresholds. “While setting an SPT of zero undesired events represents the most ideal outcome, simply identifying all undesired outcomes and setting SPTs of zero for each risk area is not sustainable or beneficial in achieving meaningful safety performance improvements. Rather, it is important to set SPTs that help monitor the impact of change,” the Year Three Report said.
When determining what to monitor using SPIs, Young suggested known safety risks, parameters that may help detect emerging safety risks, and data that show the effectiveness of an organization’s risk controls. SPIs need to take into account the organization’s risk tolerance, the cost and/or benefits of implementing improvements, regulatory requirements and public expectations.
Young said there are two types of indicators: leading and lagging. He said leading indicators are proactive or predictive and can be either negative, which involves measuring things with the potential to create a negative outcome, or positive, which means measuring things that contribute to safety in a positive way. Lagging indicators are reactive in that they measure events that already have occurred, particularly events with negative outcomes, as well as lower level failures or events that did not result in a serious outcome.
Generally speaking, SPIs should be well-defined, quantifiable, connected to accident probability and important to achieving safety goals, Young said, adding that SPIs should be developed for all areas of an operation. “You need to make sure problems bubble to the surface,” he said.
Once the SPIs are set, they should be periodically reviewed to determine if an organization’s safety efforts are trending in the right direction, Young said. If not, SPIs and SPTs can be used to identify potential problem areas for further study and correction. Young also warned about avoiding SPI pitfalls, such as measuring things that are easy to measure or unimportant; only focusing on one area of an operation; not communicating the SPIs to the organization; and failing to review SPIs frequently enough or to update them on a regular basis.
As a result of its GSIP findings to date, the Foundation has developed a second SPI survey to facilitate the collection of additional information on current safety performance monitoring practices. The intent of the survey is to gain a deeper understanding of how the international aviation industry monitors safety performance, specifically how organizations identify, monitor and update SPIs and SPTs, and tools and techniques employed for collecting, analyzing and sharing safety performance information. The information gathered through the survey will be used to develop SPI and SPT guidance material in the form of a handbook.
The survey is based on feedback from GSIP participants, and will include additional research questions that arose during analysis of the results of the original survey and discussions with industry. To access and complete the survey, please click here.
More information on GSIP is available on the Foundation’s website, where the GSIP Year Three Report and the previously published safety data collection, processing and analysis toolkits also can be found.
Note
- Flight Safety Foundation. Year Three Report of the Global Safety Information Project. March 2018.
Featured image: composite, Susan Reed; background, © bakhtiarzein | VectorStock
Goal metrics: © filo | iStockphoto
