Image: © Tyler Olson | Adobe Stock
As the aviation industry grapples with increasingly complex technologies, evolving regulatory frameworks and operational challenges, traditional pilot training models are no longer sufficient. The shift toward evidence-based training (EBT) and competency-based training and assessment (CBTA) may represent a critical evolution in how pilots are trained, assessed, and prepared to handle both routine and extraordinary scenarios. Rather than simply replacing outdated syllabi with new content, EBT introduces a fundamental rethinking of pilot training that is data-driven, context-specific, and rooted in measurable behavioral competencies. Central to this transition is the role of instructors, whose ability to assess and develop pilot competencies under a standardized framework directly influences training effectiveness and, ultimately, flight safety.
The implementation of EBT entails a paradigm shift that cannot be understood as a mere replacement of an often-outdated set of critical events with a new set. EBT implementation should aim to establish more effective training programs along with operational safety increases. For this to happen, the EBT program must be closely linked to the respective operator’s environment, and it must not be too generic, the European Cockpit Association (ECA) says. “Also, before a competent authority approves an EBT program for a respective operator, it must assess the operator’s capability to support the implementation of such [a] program,” ECA says.
There is an overlap between the EBT and CBT terms, which are often – and improperly – used interchangeably. It is thus necessary to clarify both terms.
Peter Hogston, head of training at Acron Aviation, cites as a case study Qantas Flight 32, which suffered a catastrophic engine failure four minutes after takeoff from Singapore Changi Airport in November 2010. “An explosion in the core of one of the four engines caused debris to pass through the engine casing and penetrate the wings, fuselage, and several other key components and flight controls of the aircraft,” he says. “At one point, five crewmembers were in the cockpit dealing with an ever-increasing workload caused by approximately 600 critical system and function failures.”
Using the combined skills and aptitudes of the five pilots, the aircraft returned safely to Changi approximately two hours later with no injuries, explains Hogston. “The industry refers to incidents like this as “black swan” events (i.e., short, rare moments of chaos that are not covered by standard, task-based training programmes, standard operating procedures, and relevant documentation from the manufacturer and industry regulators),” he says. “As more complex systems with built-in redundancy and automation have been developed to increase the industry’s inherent safety performance, it is found that these same systems require careful management and monitoring or human intervention in the event of a catastrophic failure, usually sudden and invariably presented as an unknown problem to the frightened flight crew.”
In response to these modern problems, operators, original equipment manufacturers (OEM), and national aviation authorities have accelerated the development of, with skills, knowledge, and attitudes measured through cockpit behaviors in normal and non-normal operations, according to Hogston.
“This is with the goal of developing the human in the system to identify, manage, and safely operate the aircraft in ‘standard’ and, very rare, black swan events,” he says. “The multi-pilot licence (MPL) programme was initially developed through a CBTA ethos; this ethos has extended to the introductory training for most airlines and is widely recognised as a positive and safety-enhancing step from ab initio to the development of type ratings and standards for pilots, instructors, and examiners (assessors).” He adds that the aim is to prevent the aircraft from entering an undesirable aircraft state or to enable recovery.
EBT vs CBTA
EBT relies on the concept of CBTA. According to the EBT methodology, as the name suggests, an operator’s recurrent training program is built on evidence, using the underpinning concept of CBT, according to ECA. “In EBT, the development of the curriculum is data driven, synthesizing information from … flight data records, training data, audit observations, accidents, and incidents. The approval of the program should be focused on the process and validation of developing the curriculum rather than on the contents itself,” says ECA.
CBT is the underpinning concept, concentrating on the output of training, explains ECA. “Teaching and learning using this approach aims at developing and strengthening concrete skills rather than learning abstract concepts. In CBTA, training and assessment are oriented at performance, with an emphasis on the standards of performance and their measurement. That, on the other hand, requires properly defined competencies and competence standards,” ECA says.
Chris Ranganathan, CAE’s chief learning officer, affirms that the evolution of airline pilot training toward CBTA represents a shift from traditional, regulation-focused methods to a performance-driven approach. “CBTA prioritises mission readiness through scenario-based learning, seeking to develop the timely application of relevant observable behaviours that make up the pilot competency framework as threat and error management tools. A major driver for CBTA adoption is the reducing levels of total relevant crew experience caused by attrition and growth, requiring the acceleration in the development of competencies normally acquired through a decade or more of line operations,” he says.
In addition, under CBTA, the focus is placed on training and developing competencies rather than on pure assessments, ECA says. “Some competencies will have to develop over time and cannot be taught as such. The assessment as such must focus on the enhancement of the competency rather than just following a fail/pass concept. CBTA can exist independently from EBT. One example where that is the case – is MPL, which shifted focus from prescriptive requirements to competency training and assessment,” says ECA.
CBTA is a paradigm shift for the entire spectrum of pilot training, according to Gilad Scherpf, senior director aviation training development at Lufthansa Aviation Training. “Key drivers are competency- instead of task-driven training, outcome- vs. time-governed delivery, and the continuous adaption towards changing operational needs (e.g., EBT). All this ultimately contributes to enhanced flight safety,” he says. “Every CBTA programme requires comprehensive change management, primarily for instructors and trainees. It is of paramount importance to convey the sense and meaning of the related methodological changes and to constantly align on consistent standards and derived training data.”
Standardization Matters
EBT has been endorsed by the International Civil Aviation Organization (ICAO) and by States since 2013, says Yann Renier, head of training and licensing, OSS-flight and technical operations, at the International Air Transport Association (IATA).
Renier noted that ICAO published Amendment 7 to Doc 9868 (Procedures for Air Navigation Services Training in 2020; the document formalized the global expansion and applicability of CBTA principles to all licensing training (ICAO Annex 1) and operator training (ICAO Annex 6).
“From that moment on, CBTA is applicable to all aviation disciplines in general and to all modules and roles throughout a pilot’s career,” Renier says. The defined competencies for pilots, instructors, and assessors should, therefore, be applied consistently during pilot aptitude tests; initial (ab initio) training; type rating training and tests; command refresher, recurrent and evidence-based training; and the selection and training of instructors and examiners.”
Instructor standardization for behavioral competency assessment is crucial in EBT because it ensures consistency, fairness, and reliability in evaluating behavioral competencies. Instructor standardization helps align individual assessments with objective behavioral performance criteria, thereby reducing variability among instructors and improving training effectiveness.
EBT relies on assessing behavioral competencies such as decision-making, situational awareness, and communication. EBT instructor standardization should be a formalized approach to ensure a consistent and standardized approach to the EBT program, including practical training to reinforce application of the assessment and grading system and maximize inter-rater reliability.1
Standardized instructors provide assessments that are reliable (that is, consistent across multiple individual assessors and over time) and valid (measuring competencies as intended in training design). Without standardized assessment protocols, evaluations can be influenced by a instructor’s personal biases or subjective interpretations, leading to inconsistencies in pass/fail decisions. The U.S. Federal Aviation Administration (FAA) emphasizes that an effective assessment should be objective, highlighting that “the personal opinions, likes, dislikes, or biases of the instructor might affect an assessment.
As EBT aims to enhance pilots’ abilities to manage real-world operational challenges, standardized instructor assessments ensure that pilots receive consistent and constructive feedback, which is essential for continuous improvement. The FAA highlights that under a standardized curriculum, operators will “fly as they train” and “train as they fly,”, thus focusing on the quality of the training program rather than on its administration.
Instructor Training
Despite the benefits, the transition to CBTA presents challenges, including integrating data-driven training systems, adapting instructor training, investing in course designer training, and ensuring regulators are ready to provide the required oversight, according to Ranganathan. “Airlines must invest significant resources to restructure training programmes while ensuring consistency across training centres and providers,” he says.
EBT instructor training requirements in the European Union Aviation Safety Agency (EASA) AIR OPS regulations provide a structured approach to EBT instructor training and standardization to ensure consistency and effectiveness in CBTA.
According to EASA AIR OPS AMC1 ORO.FC.146(c), before instructors can deliver training within an operator’s EBT program, they must first complete an initial standardization program. This program consists of EBT instructor training, followed by an assessment of competence (AoC). The training course must be delivered by an experienced EBT instructor who has demonstrated proficiency in training the required elements. It includes both theoretical and practical components, ensuring that instructors acquire a comprehensive understanding of EBT principles encompassing competency-based training, learning from positive performance, building resilience, and implementing data-driven training.
Upon completion of EBT instructor training, the instructor must demonstrate knowledge of the structure of an EBT module, the appropriate methods for training delivery in each phase, and the principles of adult learning as they relate to EBT. Additionally, the instructor must be able to conduct objective observations based on a competency framework, document performance evidence, and accurately relate specific observations to competencies. Further requirements include the ability to analyze trainee performance to determine competency-based training needs, recognize strengths, evaluate performance using a competency-based grading system, and apply appropriate teaching styles in simulator training to accommodate learning needs. Facilitating trainee learning with a focus on competency-based training needs and conducting debriefs using facilitation techniques are also essential instructor requirements, according to the regulations.
EBT instructors are required to conduct minimum number of qualified trainings over a 36-month period. In addition, EBT instructors must undergo annual recurrent standardization training, including refresher EBT training and concordance training. The intent of refresher training is to provide the framework for existing instructors to develop their competence to conduct EBT. Concordance training is designed to ensure that evaluation practices are consistent within the EBT instructor community.
How to Conduct an AoC
The purpose of assessments is to thoroughly evaluate personnel for their behavioral competencies. Effective assessments are well-structured, accurately recorded, and performed by well-trained and motivated assessors, explains a publication jointly issued by the Oil Companies International Marine Forum (OCIMF) and the International Association of Independent Tanker Owners (INTERTANKO)2 — organizations in another field where AoCs are common.
Assessors must prioritize safety over assessment and intervene if a hazardous situation arises. They should also identify and report if assessed individuals are not receiving adequate support and mentoring. Several factors can impact assessment performance, including the type of task (theoretical or practical), knowledge and technical skill levels, language and cultural barriers, guessing answers, fatigue or ill health, morale and motivation, previous training, role experience, safety, and contextual factors (such as after a major incident), and personal issues outside of work, say OCIMF and INTERTANKO.
The assessment process involves preparation, pre-assessment briefing, observing performance, avoiding coaching, maintaining records, evaluating the process, and determining the assessment outcome based on performance criteria. The last step of the assessment process is the debriefing, which must provide constructive feedback, specify the results, identify training needs, and close with recommendations, according to OCIMF and INTERTANKO.
Concordance Assurance
In an EBT program, there are primarily two outputs from the EBT instructors; the first one is the actual enhancement of standards provided directly to the pilots and the second involves data output from the instructors via the grading system used to provide the training system with feedback on pilot competency, according to an EASA-commissioned research project report.3 “There are inherent risks associated with the judgment of Grade 1, (not competent) and Grade 2, (minimum acceptable level of competence), which determine whether a pilot may conduct line operations or may require tailored or additional training, respectively. Lack of concordance in determining these grades is associated with a potential safety risk and should be considered when assessing if the levels of concordance are acceptable. Concordance assurance metrics should focus on these risks,” the report says. “The purpose of instructor concordance training and the Instructor Concordance Assurance Programme (ICAP) is to ensure that instructors are properly trained and identify areas of weak concordance to drive improvements in the quality and validity of the grading system. Effectively, these activities also provide mitigation against the risks associated with invalid training data. Instructor concordance is, therefore, critical to the success of EBT and the improvement in pilot performance over time.”
According to EASA AIR OPS AMC1 ORO.FC.231(a)(4), an ICAP identifies areas of weak concordance to drive improvements in the quality and validity of the grading system. The ICAP should be adapted to the size and complexity of the instructor group and the operator’s EBT programme. Complex operators should include ICAP-specific data analysis, demonstrating instructor-group assessment homogeneity (agreement) and instructor assessment accuracy (alignment). An operator should verify instructor concordance once per cycle for a sufficient number of competency-grade combinations and establish procedures to address instructors who do not meet required standards.
Instructor concordance is a tool for continuous improvement, as data reliability leads to a more accurate and effective training system. The operator may conduct concordance assessments more frequently or even continuously, offering more opportunities for improvement. Instructor-group assessment homogeneity (agreement) may be inferred from those who observed the same content, while instructor assessment accuracy (alignment) can be determined by comparing assessments against an established standard. Operators with a small group of instructors may integrate data-driven concordance assurance into annual refresher training, while complex operators may require a more extensive assessment system, explains EASA AIR OPS GM1 ORO.FC.231(a)(4).
CAE’s Ranganathan points out that CAE is actively deploying telemetry and eye-tracking technology and constantly refining the “automatic” lesson plan features in their simulators to alleviate instructor workload; an important factor that influences standardized evaluations (concordance). The byproduct of this rich data is the potential for an objective analysis of referent rater reliability (alignment) collected in live training; otherwise, the analysis would be conducted in a setting where evaluators grade the same recorded simulator scenario, or by “standards evaluators” observing instructors conduct and grade live crew performance.
Further, this dataset provides regulators, OEMs, and industry with essential feedback on crew performance in failure situations, using diverse demographic pilot data to compare against system safety design assumptions.
Authority Oversight
The role of the competent authority should not be downplayed in relation to EBT program effectiveness sustained by effective instructor standardization. Indeed, according to the EASA-commissioned research project report, the ability of a competent authority inspector to assess the suitability of an EBT program also relies on an inspector’s understanding of instructor concordance data, especially the metrics an operator uses to determine concordance and the criteria that an operator uses to assess the acceptability of agreement and alignment of instructors’ assessments.
To support the transition to CBTA, IATA is developing a standardization program for civil aviation safety inspectors (CASI) responsible for approving and supervising CBTA programmes. “This standardisation programme will provide CASIs with the necessary expertise to approve and perform supervision of CBTA programmes by providing them with a thorough understanding of CBTA principles and how CBTA programmes should be developed, in accordance with ICAO principles and IATA best practices,” Renier says.
The effective implementation of EBT and CBTA is not just a technical or procedural shift, it is a cultural transformation that requires buy-in from operators, instructors, assessors, and regulators alike. The move from prescriptive, task-based training to a performance-driven, data-informed model imposes rigorous instructor standardization, reliable assessment methodologies, and continuous improvement through concordance assurance. Ensuring that all stakeholders — from EBT instructors to civil aviation authorities — share a common understanding of competency standards is essential to unlocking the full safety benefits of EBT.
Mario Pierobon, Ph.D., is the owner and scientific director of a safety consulting and training organization.
Notes
- International Civil Aviation Organization (ICAO), 2013, Document 9995 – Manual of Evidence-Based Training.
- Oil Companies International Marine Forum (OCIMF) & International Association of Independent Tanker Owners (INTERTANKO), Behavioural Competency Assessment and Verification for Vessel Operators, First edition 2018.
- INSTRUCTOR CONCORDANCE ASSURANCE, Digital transformation — Case studies for Aviation Safety Standards – Data Science Applications (DATAPP), 31 July 2024.