Low-time airline pilots are just as capable during normal flight operations as their more experienced colleagues, according to the findings of an Australian study designed to measure the relationship between total flight time and job performance.
The study, by researchers from the University of South Australia, was based on evaluations of data collected by trained observers during 287 normal sectors by pilots engaged in short-haul jet flights for a high-capacity regular public transport operator in Australia. A report on the study was published in the August issue of The Aerospace Medical Association’s Aviation, Space, and Environmental Medicine.1
In the study’s examination of pilots’ technical proficiency, the report said, “There were no statistically significant differences between experience groups for first officers or captains. … However, there were minor differences with regard to nontechnical measures as a function of crew composition. There was also a difference in automation use.”
The report noted that issues involving pilot experience are especially relevant in the aftermath of the approval in the United States of a law that will require airline first officers to have accumulated at least 1,500 flight hours and of the consideration of a similar requirement for airline pilots in Australia.
“The importance of flight training is widely recognized by airlines, regulators, safety investigators and the aviation industry as a primary tool in maintaining the ultra-safe system of commercial aviation,” the report said. “However, the focus on hours attained at the time of license issue appears to give little regard to individual skill development and learning requirements, which may have been somewhat overlooked in the syllabus. There is an implicit suggestion in the legislation that, if an individual has attained the required hours to be issued with a pilot license, they also possess the requisite skill. In many debates and discussions that have been held regarding low-hour pilots, the one thing that remains constant is the view that hours [equal] experience.”
The report noted that previous studies of the relationship between a pilot’s experience and his or her performance have yielded differing results, with some concluding that experience had a significant effect on performance and others finding little connection between the two. Many of these studies, however, have included general aviation pilots, some of whom had fewer than 100 flight hours, or pilots for non-major airlines, and the results do not necessarily apply to airline pilots.
The issue of pilot experience gained renewed attention in the aftermath of the fatal Feb. 12, 2009, crash of a Colgan Air Bombardier Q400 on approach to Buffalo Niagara (New York, U.S.) International Airport. The crash killed all 49 people in the airplane and one person on the ground and destroyed the airplane. The U.S. National Transportation Safety Board said the probable cause was the captain’s “inappropriate response” to the activation of the stick shaker and the airplane’s subsequent stall.
The first officer’s flight time — 2,244 hours at the time of the accident but less than 1,500 hours when she was hired by Colgan in January 2008 — was cited by members of Congress who voted into law a requirement that pilots in U.S. Federal Aviation Regulations Part 121 operations have an airline transport pilot certificate, which is issued only to pilots with at least 1,500 hours. The requirement is scheduled to take effect in 2013.
Methods
In developing the framework for the study, the researchers classified both the first officers and the captains into two groups of low-hour and high-hour pilots. Of the first officers, 17 percent were low-hour first officers, who had 1,500 or fewer flight hours; 83 percent were high-hour first officers, with more than 1,500 hours. Of the captains, 15 percent were low-hour captains, with 5,000 or fewer flight hours, and 85 percent were high-hour captains with more than 5,000 flight hours.
“These thresholds were chosen as they are used by the industry as a standard for promotion to captain and often for entry of first officers,” the report said, “and yet … there is little or no scientific evidence behind their use.”
Data for the study were gathered over six weeks during a line operations safety audit (LOSA), with trained jump-seat observers recording information on a variety of metrics, including the operational environment, crew performance and threats from weather and other sources. Some pilots may have been observed more than once, but that could not be determined because of the anonymous nature of data collection, the report said.
The observers took detailed notes throughout the flight, beginning with preflight preparations, and at the end of each flight, they completed forms that called for an analysis of all external threats and crew errors that had been observed.
The observers looked at both individual performance and, in considering non-technical performance, at the entire flight crew.
In assessing individual performance, the primary markers were stabilized approach criteria based on the parameters identified in Flight Safety Foundation’s Approach-and-Landing Accident Reduction (ALAR) Tool Kit.
“The six criteria outlined by [the Foundation] at both 1,000 ft AFE [above field elevation] and 500 ft AFE were used to benchmark crew performance,” the report said. “These [criteria] were airspeed, vertical speed, appropriate thrust set, approved landing configuration, on proper flight path and briefings and checklists completed.
“The pilot flying was assessed on whether [he or she] met each stabilized approach [criterion] at the 1,000-ft or 500-ft AFE target. In addition to this, information was collected on the altitude at which the autopilot was disconnected on approach by the pilot flying.”
The crew’s non-technical skills (NTS) were evaluated on the four primary criteria of communication, situational awareness, task management and decision making, and their subsections — a total of 16 markers. The crew was rated on a scale from 1 (poor) to 4 (outstanding) on each of the NTS performance criteria for each of five phases of flight — pre-departure; takeoff; cruise; descent, approach and land; and taxi.
The study evaluated the crews’ threat and error management (TEM) abilities according to a University of Texas model that considered the proportion of threats that were managed effectively, the proportion that were inconsequential, the number of errors made and the number that were managed effectively, and the proportion of errors that were inconsequential.
Results
During the 287 observed flight sectors, the study identified and analyzed 845 threats and 811 errors. The findings were “in line with typical observations of normal commercial flight operations,” the report said.
Among the most frequent threats were those involving adverse weather, air traffic control requirements and ground handling events, the report said. The most common errors involved aircraft handling, management of aircraft systems and instruments, and management of aircraft automation.
Results of the individual performance analysis showed no statistically significant differences among the four pilot groups.
“This is not entirely unexpected, as most airlines have developed stabilized approach criteria and procedures, which all flight crew are regularly assessed against, both as the pilot flying and as the pilot not flying,” the report said.
However, in one area — not an element of the stabilized approach criteria — a significant difference was noted. Low-hour first officers disconnected the autopilot “at a significantly lower average altitude of 655 ft, compared with 1,168 ft for high-hour first officers,” the report said.
Nevertheless, the report added, “there was no difference observed in the safety of the flight, against the stabilized approach criteria of the low-hour group.”
Researchers identified several possible explanations for the increased use of automation among low-hour first officers.
“For instance, the increased reliance on automation flying the aircraft to a lower altitude during approach may be indicative of the lower experience of the group; that is, ‘expert’ performance has not yet developed,” the report said. “As such, these first officers may be relying on automation to manage the flight for a longer period of time, thereby reducing the time spent … hand-flying, than their more experienced colleagues. This would allow the first officers more time to conduct the other necessary tasks during approach and would also free up their cognitive capacity to do so, which would be a beneficial flight management strategy.”
The earlier disconnection of autopilot for high-hour first officers “may be an indication of the automaticity” of their actions during approach, the report said. “It is equally possible that the high-hour first officers are choosing to disconnect at a higher altitude in order to build up hand-flying time as a way of gaining experience in preparation for upgrading to captain. …
“Both scenarios are equally valid and neither [presents] any detriment to safety of the flight.”
Little Variation
The study found no significant difference in TEM performance, with low-hour and high-hour captains receiving similar scores; the scores of low-hour first officers were no different than those of high-hour first officers.
In the review of NTS performance, the study found a slight difference between low-hour and high-hour first officers in two areas.
First, crews that paired a low-hour captain with a low-hour first officer scored lower on cooperation than any of the other pairings — possibly an indication of “the low-hour first officer’s being new to the flight deck and not yet familiar enough to work in close cooperation with the captain,” the report said. “It may also be due to the low-hour captains having recently upgraded and may be indicative of them adjusting to the new role.”
Second, crews that paired a high-hour captain with a low-hour first officer received low scores in the “monitoring and cross-check” category, possibly an indication that low-hour first officers “are not yet familiar enough with operations to monitor and cross-check as well as their high-hour colleagues,” the report said. The document also noted the possibility that high-hour captains were “not completing the monitoring and cross-check functions as thoroughly as their colleagues, which may in part be due to managing the low-hour first officer.”
The report said, however, that the “minor differences” between low-hour and high-hour first officers in these areas should not be overstated.
“Despite the significance, which may well be from chance, there is very little difference between the hour groups and overall no real difference in performance in the dataset,” the report said.
No Less Able
Overall, the report said that the study found no indication that low-hour pilots were “systematically different or less able in their performance when compared to their more experienced colleagues, despite the arguments often quoted in any discussion of pilot training and selection.”
The document noted, however, that there may have been some limitations to the LOSA-based study because some of the observers may have been known to the pilots they were observing and because the pilots may have engaged in “compensatory behavior” by making certain that more difficult flight sectors were flown by the more experienced crewmember.
In addition, because LOSA is designed to assess normal flight operations, the study was limited in scope, the report said.
“While the LOSA snapshot of normal operations is useful, it may be of further benefit to explore the same hypothesis but with tighter experimental control and under non-normal flight conditions,” the document added, suggesting that another study using flight operational quality assurance (FOQA) data might provide a more comprehensive view of pilot performance during stabilized approaches.
Flight simulator studies also might be useful in examining pilot performance during non-normal flight conditions that can only be created in simulators, the report said.
“This may go some way to answering the question of whether or not low-hour first officers are performing less well than their more experienced cohort and if they are performing well enough to be operating in an airline environment with minimal total flight hours,” the report said.
Nevertheless, the report said that the study provides “concrete evidence to inform legislators, regulators, safety groups, pilots and the industry in the ongoing debate surrounding pilot hours and inferred performance. There is a continued need for scientific rigor, rather than political commentary, to inform the debate on commercial pilot training and licensing, in particular the individual differences that make up the competence of a pilot instead of adherence to an arbitrary threshold that might somehow ensure performance, and, therefore, safety.”
Note
- Todd, Melanie A.; Thomas, Matthew J.W. “Flight Hours and Flight Crew Performance in Commercial Aviation.” Aviation, Space, and Environmental Medicine Volume 83 (August 2012): 776–782.