The worldwide implementation of safety management systems (SMSs) by aviation service providers signals a shift from traditional reactive and compliance-based oversight to a new model that includes proactive and performance-based tools and methods. Such a shift, however, introduces a parallel need for civil aviation authorities (CAAs) to perform their safety oversight functions in a similar way. This means accepting performance-based oversight (PBO) as the upcoming challenge in enforcing safety regulations.
Gian Andrea Bandieri, standardization team leader in the Approvals and Standardization Directorate of the European Aviation Safety Agency (EASA), defines oversight as “the function by … which a state ensures implementation of aviation regulations, in order to ensure an adequate level of safety across the regulated industry.”
Compliance-based oversight (CBO) is built upon “the assumption that if an organization is fully compliant with the applicable safety requirements, then an adequate level of safety is achieved. Hence, CBO focuses on verifying the compliance of service providers with all applicable regulatory requirements and requires such verifications to be repeated at regular intervals, regardless of the level of compliance and maturity achieved by the organization under scrutiny,” says Bandieri.
Brooke Williams, regional communications officer at Transport Canada (TC), says that “compliance-based oversight uses a traditional audit approach methodology that looks at line-by-line compliance to a set of regulations or standards.”
Safety oversight based on compliance was the predominant characteristic of aviation safety regulation until relatively recently, and it is still in use. It was introduced at the beginning of civil aviation, when any new or amended regulation typically was written after a major or minor occurrence and focused on the technical causes. In a CBO environment, CAAs “look over the shoulder” of the aviation industry and inspect it regularly.
“This approach proved to work from the early years of aviation, and is still valid for small organizations, or when the regulatory environment is not fully mature. However, the regulatory environment in several domains has reached a level of maturity where further safety improvements cannot be achieved by following a purely compliance-based approach,” says Bandieri.
According to the French Direction Générale de l’Aviation Civile (DGAC), formal compliance with safety regulations through quality control checks alone is not sufficient to keep upgrading operational safety to the desired level. Available resource utilization is not optimized for safety in this system.1
The DGAC graphically depicts the traditional compliance-based safety oversight regime in the Deming Quality Cycle (plan, do, check, act) to emphasize that the cycle closes upon compliance with the rule —not based on safety performance — with the continuous alternation of four subsequent steps: rulemaking, compliance, compliance check and correction of findings (Figure 1).2
The idea that compliance alone may not be the proper course of action to mitigate all risks led the International Civil Aviation Organization (ICAO) to introduce the framework of SMS, an approach that requires service providers to collect risk data, classify threats according to operational exposure and define and apply appropriate mitigation actions. Under this approach, states are required to do exactly the same thing at a higher level to define and carry out their strategic safety plans.3
New Oversight Regime
Bandieri says that PBO is “an answer to the increasing size and complexity of regulated subjects. Regulators needed to find a way to better target the areas posing risks to safety, in order to ensure continuing safety improvements in a more challenging environment. Performance-based safety oversight requires an adequate and mature regulatory environment, where safety risk management is the recognized way forward to address, and possibly improve, aviation safety. The publication of ICAO Annex 19 [Safety Management] confirms this worldwide trend.”
TC’s Williams describes PBO as “an assessment of the level of compliance exhibited by an enterprise with respect to the aviation regulations. This assessment is used to determine whether the organization has effective or non-effective performance, which is one of [the] considerations in risk-based decision making.”
PBO is peculiar to the new environment of performance-based regulations, and it implies “a new approach for authorities to discharge their responsibilities, as it considers the implementation of safety management by service providers and links it to the implementation of state safety programmes (SSPs) by states, in the context of performance-based regulations. These are regulations focusing on measurable safety objectives, rather than prescribing mandatory methods of compliance to achieve the same objectives,” says Bandieri.
Performance-based safety oversight is not simply meant to monitor how service providers implement the new performance-based regulations related to safety management. It also is meant to continue ensuring service providers’ compliance with technical regulatory requirements. However, “it will do it in a completely different way. It will concentrate more on the effectiveness of SMS. It will lead to a compliance optimization of oversight activities,” says Thomas Mickler, head of the Standardisation Department at EASA.4
The French DGAC also graphically depicts the perspective of the PBO regime by using the Deming Quality Cycle to close the cycle based on actual safety performance with the continuous alternation of four subsequent steps: definition of prioritized risk-mitigation actions (plan); consideration of risk-mitigation actions by operators in their SMSs (do); check of the application of the mitigation actions through the SMS oversight program (check); and revision of action plans where necessary (act) [Figure 2].5
Measuring Safety Performance
PBO “must consider the safety performance of service providers both at an individual and aggregate level, where safety performance can be seen as the service provider’s ability to manage [its] own risks (safety management capability) in respect to its ability to comply with applicable requirements, implement and maintain effective safety management, identify and manage safety risks, achieve and maintain safe operations,” says Bandieri.
He defines the character of today’s oversight as the way a regulator collaborates with an organization to meet mutual safety objectives, and he says that in the context of performance-based safety regulations, PBO becomes the key to success. Moreover, PBO will require authorities to assess the safety management capabilities of regulated entities by developing a different oversight regime from legacy practices, that is, a framework that is more tailored to that organization’s specific identified risks.
“It will mainly focus on the service provider’s ability to identify risks within its operations and [on] mitigating them appropriately, as demonstrated through appropriate [safety] performance indicators. The performance of the service provider should be taken into account, on top of its ability to comply with requirements. This may involve more interaction, monitoring, negotiation and objective judgment, both for the service providers and the authorities’ staff involved,” says Bandieri.
Performance measurement can occur through multiple means, in particular through qualitative interpretations, actual measurements, predictions and the determination of the appropriate level of risk.6
Performance objectives and key performance indicators (KPIs) will form the basis for decision-making processes. There still is debate as to what constitutes a good KPI, and this is the area that is the least mature. More effort needs to be placed on how to define good KPIs. There should not be reliance simply on KPIs; the judgment of the aviation safety inspector is crucial.7
Redefining Inspector Functions
If technical expertise was the main skill expected from CAA aviation safety inspectors under the CBO framework, additional skills are now expected from them under the developing framework of PBO.
“Performance-based oversight,” says TC’s Williams, “includes physical inspections and looks at the effectiveness of the documentation and system that is in place while compliance-based oversight involves a review of documentation, and may include physical inspections as well, to verify that the regulatory requirements are being met and the systems are in place. Inspectors must possess technical expertise, be skilled in analytical thinking and data analysis and be able to apply risk management processes consistently.”
Regarding the additional set of skills expected from inspectors, Bandieri says they “need to acquire the ability to assess safety management systems. This requires them to also change their approach to the role … becoming more of a ‘sparring partner’ of the organization [overseen], rather than an inspector checking compliance with regulatory requirements. To this end, a less rigid and more pragmatic and listening approach would enable inspectors to better understand how risks are mitigated and to assess the effectiveness of the mitigation process.”
As CAA oversight becomes increasingly based on performance, “the ability to measure safety performance should also become part of the inspectors’ knowledge base. This means a basic understanding of safety analysis techniques and an understanding of how to work with safety indicators,” adds Bandieri.
Stephan Eder of Switzerland’s Federal Office of Civil Aviation lists some additional inspector skills required under PBO. Specifically, he cites experience in management, competence in identifying and agreeing on safety performance metrics, competence in understanding business processes, and judgment (the ability to deal with subjectivity).8
PBO not only requires different qualifications for inspectors but also requires a cultural change; safety professionals have to accept the new safety culture and eventually have to “live” it.9
Implementing PBO
In addition to appropriately training inspectors, a CAA should implement a number of additional programs to ensure that a country’s aviation industry is thoroughly overseen in a performance-based fashion.
According to TC, the starting point is the state having an SSP. “Every regulatory authority will have its own state safety program as required by [ICAO]. A solid SSP will allow the authority to promote and build awareness of performance-based programs,” says Williams.
“The answer to this challenge,” according to Bandieri, “is in the adoption of the safety management methodology at all levels — through the state safety program at state level, through the … SMS at service provider’s level.”
“The SSP provides a structure for meeting state responsibilities for safety management using a systematic, performance-based approach. It provides a framework to system safety that stresses the performance of safety-critical processes in service providers’ activities and in state oversight functions. As such, it supplies a framework for safety decision making. An important aspect of the SSP is in defining the relationship between the state, through the SSP, and the system of service providers through their [SMSs],” adds Bandieri.
“Safety improvements will depend on identification and control of hazards in a more nuanced fashion using strategies that help managers of individual aviation organizations identify and control hazards in the context of their unique operations. This is where the SMS is important. Safety measurement must, therefore, include measures that indicate the robustness of SMS design and the performance and effectiveness of the safety management capability of each organization,” he says.
It is very important to highlight that in a PBO system, prescriptive requirements and compliance with them are not replaced by safety management, they are complemented by it. In fact, this is the only way it is possible to achieve substantial safety improvements when addressing random or unique causes of occurrences, which are specific to a given aviation system or to a certain service provider.
“Prescriptive and performance-based regulations are not mutually exclusive; most regulatory structures will continue to contain both elements with different proportions. When promoting performance-based regulations, it is essential to consider the specific case and regulatory context of the area under consideration, including the capabilities of regulated persons to implement performance-based schemes, as well as of competent authorities to ensure proper oversight. Regulatory policy should set clear criteria for decision making regarding regulatory alternatives,” says Bandieri.
Regulatory Progress
ICAO recognized the need to develop a frame of reference for PBO to support implementation by CAAs and responded by publishing Annex 19, the first new ICAO annex in 30 years.
“The regulations contain a mix of prescriptive and performance-based standards. In some cases, prescription is necessary to meet the ICAO requirements. To pursue performance-based oversight, regulators need to define the intent of regulations and develop a solid interpretation of how the regulation can be met. The regulator has to make sure the industry is well informed and aware of the regulatory intent. This is generally accomplished through the SSP and, in particular, guidance material and promotions,” says Williams.
“At state level,” says Bandieri, “the key enablers for performance-based safety oversight are the mature implementation of state safety programs and the availability of less prescriptive, more performance-based requirements. Several countries have recognized such a need and are on their way to have these enablers in place. The level of achievement so far is quite different, depending on many local and cultural factors, however there is consensus that this is the way forward.”
An example of the availability of the key enablers for PBO is Europe, where EASA has been conforming to the worldwide trend for SSP implementation and to the publication of more performance-based requirements by articulating its aviation safety regulations under three broad categories: authority requirements, organization requirements, and technical requirements and standards. These three categories can be seen in recently published EASA regulations, such as Air Crew and Air Operations.
Authority and organization requirements for PBO address systems and processes, and the rights and obligations, respectively, of competent authorities and service providers required to hold an approval. Technical requirements and standards are applicable either to individuals or approved organizations.10
Since 2009, ICAO, the U.S. Federal Aviation Administration, EASA and TC Civil Aviation have convened a working group to collaborate on issues related to PBO implementation, called the Safety Management International Collaboration Group (SMICG). Later, the group was expanded to include Australia, New Zealand, Brazil, Japan and other EASA states, such as the United Kingdom, Spain, Switzerland, France and the Netherlands.
“The purpose of the group,” says Bandieri, “is to promote a common understanding of safety management principles and requirements and to facilitate their implementation across the international aviation community. In particular, SMICG members collaborate on common SMS/SSP topics of interest, share lessons learned, encourage the progression of a harmonized approach to safety management and share products with the aviation community. This cooperation indicates that, although there is still some way to go before PBO can be fully implemented, authorities are not just waiting for somebody to provide solutions, but they are active and pooling knowledge and ideas in order to proceed towards the common objective of performance-based safety oversight.
“As a matter of fact, this collaboration is already demonstrating its benefits in avoiding the duplication of efforts for states, assisting in developing robust and affordable safety management systems and increasing knowledge on SSP and SMS. Furthermore, there are also benefits for the aviation industry in terms of harmonization of SMS requirements and activities, consistent application of SMS oversight and provision of guidance material and tools.”
Mario Pierobon is a safety management consultant and content producer. He currently is involved in an airside safety Ph.D. project at Cranfield University in the U.K.
Notes
- Vernay, André; Marcou, Bernard (DGAC). Presentation on “From State Safety Programme to Risk Based Oversight.” EASA Safety Conference, October 2012.
- Ibid.
- Ibid.
- Mickler, Thomas (EASA head of Standardisation Department). “Conclusions of 2012 EASA Safety Conference.” EASA Safety Conference, October 2012.
- Vernay; Marcou.
- Kneepkens, Jules (EASA rulemaking director). Presentation on “Performance-Based Regulation.” EASA Safety Conference, October 2012.
- Mickler.
- Stephan Eder (Swiss Federal Office of Civil Aviation). Presentation on “Performance Based Oversight.” EASA Safety Conference, October 2012.
- Mickler.
- Kneepkens