When an organization has successfully implemented a safety management system (SMS), it has completed only the first stage of a much longer itinerary. Implementation establishes the policy, the processes, the documentation, and the appointments — but it does not immediately guarantee that those foundations are functioning as intended. The central question then becomes how well an SMS is working, and what needs to change for it to continuously improve.
Assessing SMS effectiveness requires looking beyond documented procedures and formal compliance to examine whether safety processes are driving decisions and improvements, and whether the organization as a whole is developing the culture and capability needed to consistently sustain safety performance.
Before engaging with assessment methodologies, it is essential to establish a distinction between compliance and effectiveness.
A compliant SMS is one that meets the requirements set out in applicable regulations and standards — principally International Civil Aviation Organization (ICAO) Annex 19, Safety Management,1 and, in European Union Aviation Safety Agency (EASA) member states, the EASA management system framework. Compliance is required, but that’s just the beginning.) An organization can document a hazard identification process, but that doesn’t guarantee that the process will identify hazards of genuine operational significance. The same organization can maintain a safety reporting system; however, personnel may not trust it sufficiently to report their own errors.
The EASA Management System Assessment Tool (MSAT) establishes a regulatory baseline for competent authorities to evaluate a management system, and it makes the distinction between compliance and effectiveness foundational to its entire methodology. MSAT guidance clearly indicates that the tool may not be used as a compliance checklist to verify that all individual elements of a management system are in place. Assessors are required to use the tool as a guide to prepare for the assessment, selecting what is relevant based on the size, complexity, and nature of the operations, and providing an overall evaluation of the management system in terms of maturity and performance.
This shift from compliance to performance has a practical consequence for how the assessment frameworks are used. Rather than verifying that each element of the management system is in place, assessors must evaluate how the organization performs and manages safety. The focus must fall on proactive safety risk management, the use of data, the organization’s resilience and agility in the face of disruptive events and an ever-changing operational environment, and its overall maturity in managing safety beyond mere regulatory adherence.
Regulatory Assessment Baseline
The MSAT tool is structured around the 12 elements of the ICAO SMS framework, corresponding to the four parts of ICAO Annex 19 (policy and objectives, risk management, assurance, and promotion). EASA has added two supplementary sections — interface management and compliance monitoring — reflecting requirements in the EASA Management System framework that are not fully captured by ICAO’s structure.
For each feature within the tool’s sections, MSAT applies the following four-level scale — present, suitable, operating, effective (PSOE) — which is a maturity model rather than a compliance progression:
- Present means there is evidence that the relevant item is documented within the organization’s management system documentation.
- Suitable means the item is appropriate given the size, nature, complexity, and inherent risk of the organization’s activities.
- Operating means there is evidence that the item is in use and an output is being produced.
- Effective means there is evidence that the item is achieving the desired outcome and has a positive safety impact.
Reaching an effective assessment, or a high score, is not the end of the journey, as the operational environment is dynamic. Occurrences, disruptive events, emerging risks, and changes to the management system can all render past risk assessments invalid.
Although it was designed for use by competent authority inspectors, organizations may use MSAT to assess their own management system for the purposes of continuous improvement or to assess the management system of contracted organizations that are required to have their own management systems in place.
Upper Reaches of Maturity
While MSAT establishes the regulatory floor, it was not designed for exploring the upper reaches of SMS maturity or for consistently integrating the human factors and organizational dimensions that increasingly characterize advanced safety management thinking. The SMS Maturity Assessment and Refinement Tool (SMART), developed by NLR–Royal Netherlands Aerospace Centre Senior Scientist Sybert Stroeve and others under the Future Sky Safety program, addresses this requirement.
SMART provides 34 assessment topics grouped across the four ICAO Annex 19 SMS parts, each scored on a five-level scale from A to E. Level A represents no formal approach or a bare minimum; Level E represents the most advanced approach, which goes well beyond regulatory requirements and incorporates Safety-II thinking and organizational learning. This granularity allows organizations to map their management system, to pinpoint where they are, and plan how to improve from there.
The five-level scale of SMART relates to the four-level PSOE scale of MSAT; while the MSAT PSOE scale assesses regulatory compliance and performance with a competent authority perspective, the SMART scale assesses organizational maturity and continuous improvement capability. The two frameworks are complementary rather than directly equivalent.
SMART Levels A and B correspond broadly to the MSAT present and suitable levels — a management system that is beginning to be established but has not yet reached operational maturity. Level C corresponds broadly to the MSAT operating level — a management system that is functioning and producing outputs. Levels D and E extend beyond what MSAT defines as effective, into territory that MSAT does not prescribe but that advanced practitioners and regulators increasingly recognize as distinguishing high-performing organizations. It is in this D-to-E range that the most significant gains in safety performance are typically found.
Considering Performance Variability
The SMART maturity framework refers, at the upper levels of its scale, to Safety-II thinking and to the Safety-II perspective on safety management. These references point to a genuine reorientation of how safety performance is understood and managed, and have direct implications on SMS assessment.
For the greater part of its history, aviation safety management has defined safety as the absence of accidents and incidents. Under this framework — which safety scientist Erik Hollnagel and colleagues have labelled Safety-I — safety management is reactive in its orientation: When something goes wrong, its causes are investigated and measures are introduced to prevent recurrence. Systems, procedures, performance indicators, and the regulatory frameworks that govern them were developed with this goal in mind. Safety is understood as a condition to be preserved by identifying and eliminating the sources of failure, and human performance is viewed primarily as a potential source of error to be constrained, standardized, and monitored.
This paradigm has produced substantial safety gains, and it remains an indispensable component of any well-functioning SMS. However, its limitations become increasingly visible as systems approach the upper boundaries of their safety performance. Commercial aviation is among the most instructive examples. As physician and cognitive psychologist René Amalberti observes, ultra-safe systems (those in which a disastrous occurrence arises less frequently than once per million production cycles) pose new questions that traditional safety paradigms struggle to answer. Systems operating at this level cannot be improved further by focusing exclusively on failure: The failures are too infrequent, so there is less to learn from mistakes.2
The systemic approach to safety, from which the Safety-II perspective emerged, begins to answer this question by reframing the object of analysis. Rather than treating human performance primarily as a source of error, it draws attention to what Hollnagel terms performance variability — the ways in which individual and collective performance are continuously adjusted to match current demands and resources to ensure that things go right. Since the conditions under which work is actually performed are never identical to the conditions that system designers imagined when writing procedures, operators at the sharp end are constantly making small adjustments, exercising judgement, and applying adaptive capacity to keep the system functioning.3
The distinction between “work as imagined” and “work as done” is a direct expression of this insight. Work as imagined indicates what should happen under nominal conditions — the idealized task as conceived by the procedure writer. Work as done stands for what actually happens: how work unfolds in a concrete situation, shaped by specific conditions of the moment, and the accumulated practical knowledge that no procedure fully captures.4
In terms of SMS assessment, this shift reorients the analytical focus from one that inquires about what the organization has failed to do or that it is missing, to one that also asks what the organization consistently does well, especially at the highest level of SMS effectiveness.
SMS Assessment in Practice
Assessors must develop the practical capability to conduct effectiveness assessments, interpret results, and translate those results into a credible improvement strategy.
MSAT and SMART can be applied in complementary ways that together provide a comprehensive assessment of where an organization stands. A practical approach is to use MSAT in the first instance as a structured self-assessment against the regulatory standard — generating a clear picture of which management system features are present, suitable, operating, and effective from a regulatory perspective — and then to apply SMART to those features that MSAT has confirmed as broadly operating or effective, to explore in greater depth where maturity is concentrated and to identify the most significant opportunities for improvement.
SMART provides three application configurations of increasing sophistication. In the single-user configuration, a safety manager or senior safety professional completes the questionnaire individually, using the process of scoring and justifying each topic to develop a structured analytical picture of the organization’s management system. The value of this configuration lies not in the numerical scores it produces but in the reflective process it demands: The requirement to provide a written justification for each topic score forces engagement with evidence and surfaces assumptions that may not previously have been made explicit.
The multiple-user, single-organization configuration extends this process into a facilitated organizational dialogue. Different participants — safety managers, compliance monitoring managers, line managers, and senior executives — each complete a SMART survey independently, and the resulting spread of scores and justifications is analyzed for patterns of agreement and disagreement. Systematic differences in how different parts of the organization perceive the management system are themselves diagnostically informative: They reveal where the system is understood and valued, where it is seen as a bureaucratic overhead, and where it is largely invisible to the people whose behavior it is intended to influence. A structured workshop brings participants together to discuss these differences, develop a shared understanding of the system’s current state, and agree on priorities for improvement.
The multiple-user, multiple-organization configuration — which is the most demanding and the most powerful — enables interfacing organizations within a service delivery chain to compare their management system maturity levels, identify misalignments in safety management approaches, and work collaboratively on shared improvements.
An SMS assessment that ends with a scored questionnaire and a list of weaknesses has not fully delivered its potential value, which is realized instead when the assessment generates a credible, prioritized, and time-bound improvement plan that connects identified weaknesses to specific actions, assigns accountability for those actions, and establishes the measures that will confirm improvement has been achieved.
One principle of the transition from assessment to improvement deserves emphasis. The objective of SMS maturity assessment, as Stroeve makes clear, is to trigger thinking and discussion to support improvement rather than to achieve maximum scores per se. EASA reinforces this position in the MSAT guidance on scoring: A high score is not an end state, because an organization that achieves effective status across all management system features has not permanently secured its safety performance — it has demonstrated current capability in a dynamic environment that will continue to generate new risks, organizational changes, and external pressures.
Frameworks such as MSAT and SMART offer structured methodologies, common terminologies, and maturity scales that help organizations perform systematic assessments. However, they should not be regarded as prescriptive templates that must be followed mechanically. Similarly to what is contemplated under EASA Part-IS in relation to the maturity model, MSAT and SMART can be adapted to suit the size, complexity, operational context, and strategic objectives of the organization.
Assessing SMS effectiveness is fundamentally different from verifying SMS compliance. The latter confirms that the elements of a management system exist; effectiveness assessment seeks to understand whether those elements actually make operations safer.
More than adherence to a particular assessment tool, what matters is the adoption of a disciplined and methodical approach. An effectiveness assessment should avoid becoming another checklist exercise focused solely on verifying the presence of procedures, documents, or organizational structures. Instead, it should maintain a broad perspective of how the management system functions in practice and how people experience it.
Particular attention should be given to the quality and usefulness of safety performance indicators, since these provide one of the most visible demonstrations of whether safety management activities are generating meaningful intelligence.
Equally important is the ability of the assessment process to capture the overall maturity of the safety management capabilities, including the extent to which safety thinking is understood, valued, and applied throughout the organization.
Ultimately, SMS effectiveness assessment requires both analytical rigor and professional judgement. Its objective is to develop an informed understanding of whether the system is genuinely working and where it can be strengthened. In this sense, effectiveness assessment is about maintaining a coherent, organization-wide view of safety performance, safety culture, and continuous improvement.
Image: @ alexsl | iStockphoto
Mario Pierobon, Ph.D., is the owner and scientific director of a safety consulting and training organization.
Notes
- ICAO (2016) Annex 19 to the Convention on International Civil Aviation — Safety Management. 2nd edn. Montreal: ICAO.
- Amalberti, R. (2001). The paradoxes of almost totally safe transportation systems. Safety Science, 37(2-3), 109-126.H
- Hollnagel, E. (2006). Resilience: The Challenge of the Unstable. In E. Hollnagel, D. D. Woods, & N. G. Leveson (Eds.), Resilience Engineering – Concepts and Precepts (pp. 9-18). Farnham: Ashgate.
- EUROCONTROL. (2013). From Safety-I to Safety-II: A White Paper. Brussels: EUROCONTROL.
