Use It or Lose It

Extensive use of automated cockpit systems causes pilots to lose proficiency in some cognitive skills required for manually flying an airplane — such as keeping track of aircraft position without using a map display — although other skills remain relatively intact over a long period of time, a new study says.

The study, led by Stephen M. Casner of the U.S. National Aeronautics and Space Administration (NASA) Ames Research Center, found that pilots’ instrument scanning skills and manual control skills remained strong, even among pilots who said they practiced them infrequently.

Casner and his research team based their conclusions on results obtained when 16 airline pilots flew routine and nonroutine flight scenarios in a Boeing 747-400 simulator. The researchers varied the level of automation in use, graded the pilots’ performance and asked questions about their thoughts during the simulator sessions.

A companion study, conducted during the same simulator sessions (ASW, 7-8/14), found that, although cockpit automation systems were designed to give pilots more time to think about and plan for upcoming portions of the flight, instead, during uneventful periods, their minds sometimes wandered.¹

The report on the new study, published in the December 2014 issue of Human Factors, noted that a research report published in 1971 said that pilots had varying degrees of success in remembering different types of skills.²

“The researchers found that when [hand-eye skills such as those used to scan instruments and manipulate flight controls] were initially well learned, they were surprisingly resistant to forgetting, even after four months of inactivity,” the 2014 report said. “Another type of skill considered in the study is the set of cognitive skills needed to recall procedural steps, keep track of which steps have been completed and which steps remain, visualize the position of the aircraft, perform mental calculations and recognize abnormal situations. Like researchers before them, [this team] found that after four months of inactivity, pilots’ cognitive skills had significantly deteriorated.”

The 1971 research was used at the time as guidance for regulators responsible for setting minimum recent experience requirements for pilots, the new report said.

“The wisdom provided by this early research is evident in the regulations we have today,” the report said. “Pilots can wait almost two years without flying and still operate under visual flight rules (with no passengers aboard). If they want to exercise the privileges of operating under the more cognitively demanding instrument flight rules, six months of inactivity is the limit.”

Today, the report added, concern about deteriorating pilot skills centers on inactivity associated with the increasing use of cockpit automation to do everything from performing fuel calculations and tracking the aircraft’s position to reconfiguring navigation equipment and monitoring and identifying instrument system failures.

Nevertheless, cockpit procedures have retained methods intended to prevent a lack of use from leading to a deterioration of pilots’ manual flying skills, by closely monitoring the work performed by automated systems and occasionally shutting off those systems to practice manual flying skills.

To determine how effectively these methods help pilots retain their manual skills, the researchers asked seven captains and nine first officers, all of whom worked for U.S. air carriers, to participate in the 747-400 simulator study. The pilots had an average of 17,844 flight hours, including an average of 623 hours in the 12 months before the simulator evaluation and 13 hours during the previous week. Participating pilots said that they had accumulated 73 percent of their total flight hours in airplanes equipped with a flight management computer (FMC) and 89 percent of their time in airplanes with flight directors.

Hand-Eye Skills

To enable the researchers to evaluate the pilots’ hand-eye skills — their instrument scanning abilities and their manual control of the airplane — the pilots flew routes that had been programmed into the simulator’s FMC with three different combinations of automation.

The autoflight phase involved use of the autopilot, flight director and autothrottle to follow the route programmed into the FMC. The manual control phase involved use of the flight director and autothrottle system along with manual manipulation of the control yoke “in response to flight director commands that directed them along the FMC-programmed route,” the report said. In the raw data and manual control phase, pilots followed the same route while manipulating the control yoke, controlling thrust levels and relying on primary flight instruments for information.

“We asked each pilot to fly during three phases of flight (i.e., arrival, approach and missed approach) in the three automation conditions,” the report said. “To save time, we did not ask pilots to fly all three flight phases using the autopilot, as we did not expect to see much variation in pilots’ performance across the three flight phases when the autopilot was used.”

Researchers scored the pilots on their ability to comply with course, altitude and speed assignments on the route.

In their responses to a research survey, the participating pilots said that they had “strong background in basic instrument flying, moderate recent experience in flying without an autopilot and very little recent experience flying with both the autopilot and flight director turned off,” the report said.

Table 1 shows how pilots performed — and how many times they committed significant³ deviations from speed, altitude or course — in the three different automation conditions and three phases of flight.

The researchers’ analysis of the results showed that during the arrival and approach phases, there was “no significant association between automation condition or recent practice on pilot performance,” the report said. In the missed approach phase, researchers found “a significantly higher likelihood of a speed deviation in the manual control condition when compared to the raw data and manual control condition. … Pilots’ scanning and manual control skills seemed to be more likely overwhelmed in the midst of this high-tempo phase of flight.”

The results supported the findings of earlier research that, as long as pilots had been formally trained in instrument scanning and manual control, those skills were “reasonably well-retained, even in the absence of regular practice.” Nevertheless, the study said, the results also showed “some atrophy [in those skills] that perhaps merits additional practice.”

Cognitive Skills

The participating pilots were unanimous in telling researchers that, although they had strong backgrounds in conventional navigation methods, they had no recent experience in that area.

Table 2 shows how pilots performed on eight navigation tasks — and how many times they committed at least one operationally significant error⁴ — while flying an arrival, approach and missed approach without using an FMC. For this portion of the study, researchers compared each pilot’s performance while using the simulator’s FMC against his or her performance using a conventional VHF omnidirectional radio (VOR) receiver.

“Aside from requiring different procedures to operate them, the two types of navigation equipment differ more strikingly in how much pilot involvement they require,” the report said. “Whereas VORs require the pilot to closely follow the progress of the flight and reconfigure the equipment as [the airplane] arrives at each waypoint, the FMC permits the pilot to program the entire route prior to departure and to think of the navigation process as a ‘once-and-done’ programming exercise.”

The process included three specific unannounced instrument system failures as part of the test of pilots’ abilities to recognize and confirm an abnormal instrument indication by cross-­checking their instruments. The failures involved the participating pilot’s heading indicator and altimeter — although heading indicators and altimeters elsewhere in the cockpit continued operating; and blocking the pitot-static system, which caused malfunctions in all airspeed indicators in the cockpit. The engine indicating and crew alerting system also was disabled.

Table 2 shows that all pilots were able to hold their airspeed within allowable limits and all but one were able to tune a VOR station and select an inbound course; in addition, only one pilot had difficulty navigating to the VOR station. But six pilots failed to fly the published heading on the missed approach, and seven incorrectly announced their arrival at the missed approach point. Only one pilot completed the entire process without errors.

“Overall, like instrument scanning skills, pilots reported that navigation skills, once initially mastered, are seldom, if ever, practiced,” the report said. “But rather unlike instrument scanning skills, which are resistant to forgetting, navigation skills that have been supplanted by the use of cockpit automation are highly susceptible to forgetting and likely require frequent practice to keep them sharp.”

In its analysis of the pilots’ responses to the three events involving instrument system failure, the study noted that 81 percent of participants told researchers that they had received “considerable training and practice with recognizing and dealing with puzzling instrument indications.” However, fewer than half said their airline recurrent training had included similar practice.

Table 3 shows that in each of the instrument system failures — altimeter lag, heading indicator skew and unreliable airspeed — all but one of the pilots verbalized the problem.

In dealing with altimeter lag and heading indicator skew, fewer pilots correctly took the next step — cross-checking instruments. In the case involving unreliable airspeed, only one pilot failed to make “an obvious attempt” to check the other instruments.

In two of the three scenarios — altimeter lag and unreliable airspeed — most of the pilots deviated from the assigned altitude and failed to prevent the approach of a stall, respectively. They were better at coping with heading indicator skew, with 38 percent deviating from the assigned heading.

Heading indicator skew was the easiest of the three problems to diagnose, the report said, noting that only one pilot failed in his diagnosis. Eighty-one percent successfully diagnosed the case of altimeter lag, and 56 percent correctly identified the heading indicator skew, the report said.

Data showed that pilots who reported that they had at least occasionally had practice during recurrent training in dealing with puzzling instrument indications performed no better than the others in the three instrument failure scenarios. The report said that one explanation might be that the recurrent training focused on “a few familiar failures” and did not include general methods of handling other types of abnormal events.

“Overall,” the report said, “the data suggest that pilots performed well at detecting failures but often neglected to cross-check other instruments, diagnose the problem and avoid the consequences of an unresolved failure. In regard to the reported frequency at which pilots receive initial and recent practice in dealing with puzzling instrument indications, our findings suggest that this sort of skill is vulnerable to forgetting and could also benefit from more emphasis during initial and recurrent training.”

This article is based on “The Retention of Manual Flying Skills in the Automated Cockpit,” by Stephen M. Casner, Richard W. Geven, Matthias P. Recker and Jonathan W. Schooler, published in Human Factors, Volume 56 (December 2014):1506–1516.

Notes

1. Casner, Stephen M.; Schooler, Jonathan W. “Thoughts in Flight: Automation Use and Pilots’ Task-Related and Task-Unrelated Thought.” Human Factors Volume 56 (May 2014): 433–442.
2. The current study cited Mengelkoch, R.F.; Adams, J.A.; Gainer, C.A. “The Forgetting of Instrument Flying Skills.” Human Factors Volume 13 (1971): 397–405.
3. Deviations were classified as “significant” if airspeed was more than 10 kt from the assigned speed, if altitude was more than 300 ft above or below the assigned altitude and if a full-scale deflection was recorded on a course deviation indicator.
4. Airspeed and altitude deviations were scored as outlined in Note 3. While navigating to a VOR or a missed approach point, deviations were recorded “when pilots missed the assigned point by more than 3 nm [6 km]” or flew more than 10 degrees off an assigned heading.