Unleashing SPIs

As safety management systems (SMS) become more prevalent among aviation service providers, the focus of many safety practitioners is moving away from early forms of data collection and analysis toward mechanisms that ensure the continuing effectiveness of the entire SMS. Key elements of that effort are the development and use of safety performance indicators (SPIs).

According to the International Civil Aviation Organization’s (ICAO’s) Doc 9859, Safety Management Manual (SMM)¹ and ICAO Annex 19, Safety Management,² an SPI is “a data-based safety parameter used for monitoring and assessing safety performance.” The SMM says, “An SMS defines measurable performance outcomes to determine whether the system is truly operating in accordance with design expectations and not simply meeting regulatory requirements. The safety performance indicators are used to monitor known safety risks, detect emerging safety risks and to determine any necessary corrective actions.”

SPIs play a significant role in SMS because they enable performance-based safety management. In addition, they support the determination of safety objectives specific to an organization, the setting of alert levels, the appreciation of when alert levels are reached and the overall monitoring of safety performance.

What makes a good SPI?

While the term SPI is defined in the latest SMM, the document provides only a few examples of high-level SPIs. Aviation service providers themselves, therefore, need to become competent in developing SPIs that will yield meaningful insights.

“I had my first exposure to SPIs when I was 5 years old,” says Dai Whittingham, chief executive of the U.K. Flight Safety Committee (UKFSC). “My father, who enjoyed a joke, spent some time trying to convince me and my younger brother that the ‘special powder’ he was spreading round the garden would stop elephants (the threat) coming anywhere near us. The SPI was the absence of any elephants and so, with none sighted for many days, the powder (normal bread flour) was obviously a great success. Unfortunately, we lived in North Wales at the time, and I now know the nearest elephant would have been in the zoo 80 miles [129 km] away,” he said.

“A good SPI has to give you direct, unambiguous information on a specific item of interest; the data underpinning the SPI must have a causal link with the outcome you are seeking to measure and must not be inferred or derived from elsewhere. It does not matter whether it is quantitative or qualitative, leading or lagging, provided it gives you concrete evidence that either demands remedial action or proves that no action is required, and provided that the parameters you measure are actually relevant to the original aim of the SPI. As the elephant powder case shows, the local absence of elephants proved neither the effectiveness of the powder nor the validity and scale of the threat. What you measure is usually more important than how you measure it, and you need to take care to ensure your chosen metric means what you think it does,” said Whittingham.

The best SPIs are meaningful data points that support safety management, add value to safety performance and draw attention to possible adverse trends or events, according to Patrick Chaignon, SMS manager and deputy director of quality at FlightSafety International’s Paris Le Bourget training center.

Nancy Rockbrune, head of SMS at the International Air Transport Association (IATA), says a good SPI “focuses on the controls that have been implemented by the organization to ensure safe outputs. This includes the specific safety controls to mitigate identified risks, as well as consideration of how both external factors and internal interaction may impact said safety controls. Therefore, by creating effective SPIs, organizations measure and control the causal factors (inputs) to ensure the desired effect (output), shifting the focus to precursors of serious events, rather than the events themselves.”

Jorge Leite, vice president of quality and safety at TAP Maintenance & Engineering (M&E), says that SPIs must meet a series of specific requirements for users to ascertain their quality. SPIs must be identified as part of the SMS, aligned with the safety targets of an organization and reflect the actual safety performance of an organization. According to Leite, SPIs must be valid (measure what is intended to be measured), reliable (independent of conditions, situations and individuals), sensitive (responsive to changes and statistically significant), representative (cover all relevant aspects), resistant to bias (cannot be manipulated) and cost-effective (cost no more than they give back).

Focus of SPIs

The challenge facing developers of aviation service providers’ SPIs is determining how narrow an SPI’s focus will be.

As an approved training organization (ATO), FlightSafety International at Paris Le Bourget has been learning to develop SPIs over the past few years, and the SPIs are becoming increasingly meaningful as the SMS matures. “Across all industry segments, SPIs will probably be broad at the onset and narrow over time, as more granular safety management data are generated,” Chaignon says. “Within our SMS, we started with the goal of identifying a small number of high-level measures that were relevant to the level of maturity of our system. After a year of collecting data, it was easier to see those SPIs that added value and those that missed the mark. Data from hazard reporting and safety assurance activities have been growing and are a valuable resource that will be used to focus and narrow future SPIs on emerging safety trends and areas of concern.”

According to IATA’s Rockbrune, SPIs “should not focus on events, but more on leading indicators — metrics on current environment that could impact future performance (input focus) and lagging indicators — the metrics on previous events, or known unwanted events you are trying to prevent (output focus). They need to be relevant to the specific organization, taking into account the maturity of the SMS as well as specific areas needing attention.”

Setting the narrowness of an SPI’s focus depends on the SPI itself. “Each SPI must be created for a different purpose,” TAP M&E’s Leite says. “There are no set rules for their scope. Some SPIs have a broader scope, and their focus is larger (for example, measuring the degree to which state safety program requirements have been accomplished). Others must be very [narrowly] focused (such as measuring the number of discrepancies in accomplishment of airworthiness directives by an air operator, a continuing airworthiness organization, or an MRO [maintenance, repair and overhaul organization] contracted for the purpose.”

UKFSC’s Whittingham adds, “There is no problem with some SPIs being narrow in focus; in fact, the more critical the issue, the narrower the focus should be. That does not mean data can only be collected from a narrow source, as many SPIs will require data to be aggregated. For example, an SPI to monitor the risk from ETOPS [extended-range twin-engine operations] routes might include global and fleet-wide engine malfunction trends, destination FOD [foreign object damage] and wildlife strike information, as well as ETOPS route exposure to give a risk of failure per ETOPS hour.”

With regard to what an SPI needs to examine, Whittingham says that a high frequency of a minor occurrence might well demand an SPI, even though the relative risk indicated by a frequency/severity graph is quite low.

Whether an SPI needs to look at major or minor accident precursors, for example, depends on the way SPIs have been created by a particular organization, according to Leite. He said, “There is not a set rule for their creation. Each organization should create their own SPIs in a way that better serves what an organization wants to monitor, based on what its safety objectives and targets are. For example, at TAP M&E, we have created SPIs that monitor our safety performance related purely to typical safety issues such as, for example, the number of technical incidents per 1,000 flight hours; the number of IFSDs [in flight shutdowns], FRs [flight returns], RTs [rejected takeoffs], etc. due to technical reasons; the number of voluntary reports received; and the number of hangar and shop occurrences. In addition, we have other SPIs that monitor our safety performance in proactive terms, e.g., the number of extensions to the MEL [minimum equipment list], and even other SPIs related to the management of safety at a corporate level, namely, the measurement of safety performance — what I often designate as the ‘quality of our safety’ — of our MRO activities as delivered to and perceived by our MRO customers.”

‘Must Have’ SPIs

Apart from providing the definition and guidance on meaningful use of SPIs, the ICAO SMM only provides examples of generic SPIs, not specific ones.

As noted, it is therefore up to individual organizations to develop meaningful SPIs. FlightSafety International’s Chaignon says that “individual operators’ ‘must have’ SPIs will likely be unique to the nature of their business model.”

Nevertheless, Whittingham believes that it would be useful for a set of industry best practice SPIs to be developed, and Leite believes that there already are some “must have” SPIs in the aviation industry. “An organization managing safety in accordance with ICAO Annex 19 must have a safety policy, a set of safety objectives and their correspondent safety targets within a defined time period. So, there must be at least one SPI to measure the degree of accomplishment of each objective within the given time period,” says Leite.

Flexibility of Organizations

According to Rockbrune, SPIs, like any other aspect of safety management, must be considered “in the context of each individual organization. As such, the organization has complete flexibility to develop its own SPIs. The requisite is the ability to demonstrate, to their respective state, that they are indeed effective in monitoring and assessing safety performance.”

“IATA is currently developing guidance material for operators on SPIs with the assistance of an SPI task force. This material will include sample SPIs with supporting methodology and guidance to assist organizations in developing the means to verify their safety performance against meaningful safety objectives,” says Rockbrune.

Leite concurs that there is considerable flexibility for aviation service providers in developing SPIs, “unless the competent authority supervising an organization, through the respective state safety program (SSP), determines some ‘mandatory’ SPIs, or, e.g., an MRO customer, whenever specifically expressed in the maintenance contract, [or] determines some SPIs must be created and monitored as a tool to monitor compliance of the safety requirements in the contract,” says Leite.

Whittingham believes that the main SPIs should be “agreed [upon] by an organization with its regulator, as these SPIs will be a cornerstone of performance-based oversight. Otherwise, the industry must be free to choose.”

Flexibility in developing SPIs is important because there may not be pre-defined SPIs for specific categories of aeronautical services. “ICAO Doc 9859 [the SMM] is a good place to start for operators,” says Chaignon. “Appendix 6 to Chapter 5 provides guidance and SPI examples for air operators, aerodrome operators, ATS [air traffic services] operators, and production, design and maintenance organizations. However, not all aviation organizations fall into those categories. FlightSafety International conducts ATO [approved training organization per European Aviation Safety Agency] training in advanced full flight simulators. Accordingly, FlightSafety’s SPIs need to focus on training delivery.”

SPI Examples

At FlightSafety International at Paris Le Bourget, the SMS is more than 2 years old and maturing. “It is important that safety management be implemented, exercised and have a good foundation before adding complexities. This idea is similar to the evolution of safety risk management from reactive to proactive and predictive. You need to build a base with reactive risk management before moving to proactive,” Chaignon said.

“Our initial goal was to use SPIs as activity markers to measure SMS actions at the learning centers and with regard to training operations. At this point, SPIs provide a high-level measure of baseline targets that are used to augment SMS evaluations. SPIs were selected for each of the four SMS pillars³ and measure management commitment, hazard reporting and risk assessment activity, safety assurance inputs and communication of safety-related issues with customers, manufacturers, regulators and other stakeholders.”

Table 1 and Table 2 are examples of SPIs for use at an advanced phase of SMS implementation. The figures were provided with permission for use in this article by TAP M&E, and they are based on guidelines by a working group of the Star Alliance (the international airline alliance of which TAP is a member).

Setting Alert Levels

Safety performance management methodology expects organizations to establish a maximum number of precursors of safety events they will record — or a maximum number of actual safety events — for results to remain in the realm of acceptability in relation to their defined safety objectives. If alert levels are reached or surpassed, then further investigation or causal analysis is necessary to bring performance back to acceptable levels.

But how can the safety practitioner set alert levels?

According to UKFSC’s Whittingham, alert levels are normally subjective, but it is sound practice to use a statistician to help distinguish between the significant and insignificant.

IATA’s Rockbrune says that “through statistical analysis, the ‘threshold’ or ‘alert level’ can be determined in a quantifiable way. There is no specific formula per se, as it would depend on the type of data or information collected. There are, however, sound analytic principles that should be followed to provide these thresholds, making them, and any decisions derived from them, defensible.”

Leite adds, “At TAP M&E, we set alert levels both quantitatively and qualitatively, depending on each SPI and how it was constructed. Usually, each SPI is connected to a dashboard, where we have green, yellow and red ranges. These ranges are analyzed and redefined or reset at least once a year in our safety review board (SRB). If, for a particular SPI, the SRB decides we are OK and the prospective and technical and operational expectations remain the same, then the SPI is kept as it is; we do not change the green/yellow/red ranges. If not, the SRB decides how the ranges are tuned and set for the next monitoring time period for that SPI. Alert levels are established to a large extent based on common sense and based on what we know about our organization, people, expertise and typical time to react to a problem within the organization.

“As a general rule, and if there is no better input for a particular SPI, we set the yellow alert 10 percent off the target and the red alert 15 percent off the target. But, for specific SPIs, that decision can be completely different, and the alert levels are set in a stepped fashion and discrete format. For example, when measuring the number of IFSDs due to technical reasons, if we have set the target as a maximum of one IFSD per year [and] we had less than one, we are in the green band; if we had one already, we are in the yellow band; if we have had two or more, we are in the red band,” says Leite.

Moving on

As aviation service providers upgrade safety performance management capabilities, their practitioners stand to gain significant benefits from effective SPIs. Ideally, these will be metrics that provide meaningful information, cannot be manipulated and reflect the operational peculiarities because they have been flexibly developed by service providers in accordance with their safety objectives. 

Mario Pierobon is a safety management consultant and content producer.

Notes

1. ICAO. Safety Management Manual (SMM). Doc 9859 AN/474. Third edition, 2013.
2. ICAO. Annex 19, Safety Management. First edition, July 2013.
3. The four pillars or components of an SMS are safety policy, safety risk management, safety assurance and safety promotion.

Featured image: © Jorgen Syversen | AirTeamImages