Eyeing the Eyes

Eye-tracking technology, used along with simulator training, could identify problems with pilots’ instrument scans and help them correct their techniques for future flights, experts in aviation training and the use of eye tracking say.

For example, if a session with eye-tracking technology shows that the pilot monitoring is looking outside when he or she should be monitoring instruments, “you’ve identified a problem,” Capt. Mark Cameron of Emirates said during a presentation to Flight Safety Foundation’s International Air Safety Summit, held in November in Seattle.

Co-presenter Patrick Nolan, general manager for aviation with eye-tracking equipment manufacturer Seeing Machines, added that the goal is not only to determine what a pilot is looking at but also to put that information into context.

For example, Nolan said, eye-tracking equipment can help determine whether a pilot is checking the airspeed before calling for flap extension. An instructor who is armed with that information, Cameron added, can help pilots improve.

They cited three accidents — the Aug. 23, 2013, crash of a CHC Scotia Eurocopter AS332 L2 Super Puma during approach to Sumburgh Airport in the U.K. (ASW, 6/16); the July 6, 2013, crash of an Asiana Airlines Boeing 777 just short of the landing runway at San Francisco International Airport (ASW, 10/14); and the March 24, 2008, crash of a Spanair McDonnell Douglas DC-9 on takeoff from Madrid-Barajas Airport (ASW, 12/11-1/12; see “Identifiable Patterns”).

“Wouldn’t you like to know what those pilots were scanning” during the critical moments immediately preceding each of those accidents, Cameron asked. “We can guarantee that all … of these events happened [during simulator training] first,” and that, if eye-tracking technology had been used in the simulator, it would have identified the pilot’s eye movement patterns and enabled the pilot to adjust those patterns for a more productive instrument scan.

Eye-tracking technology is still in its early stages, Nolan said, adding that one challenge is “to move from something that was pretty cool to something that will add value for pilots.”

Seeing Machines announced in December that it had collaborated with L3 Commercial Aviation to incorporate eye-tracking technology into an L3 Boeing 787 full flight simulator for an Australian airline — a development that Seeing Machines said was the first time that permanent eye-tracking equipment had been incorporated into a full flight simulator for civil aviation pilot training.

The company said its technology “provides insights into pilot scanning techniques, and supplies instructors with an objective assessment of what their trainees are looking at (e.g., speed tape, height, glide slope, heading, [flight mode annunciator]), at any given point in time during a flying sequence. Instructors can observe crew instrument scanning in real-time in the full flight simulator environment, as well as during debrief.”

Similar eye-tracking technology also is in place or being developed for use in other sectors of the transportation industry, including commercial trucking and rail operations. In those cases, the technology can be used to monitor for distractions and to identify cases of operator fatigue, the company said.

Studies Began in the ’50s

Basic eye-tracking studies were first conducted in the 1950s, according to a 2018 report in the journal safety by researchers in France who traced the beginnings of eye-tracking research and its likely future.¹

Early studies examined data collected during simulator flights by 40 pilots who were equipped with an eye-tracker system; the conclusion was that those pilots checked their flight instruments at varying frequencies and for varying lengths of time, but that more experienced pilots tended to focus for shorter periods of time on individual instruments.

“At the time, researchers were confronted by the complexity of the recording setup and elaborate manual analysis procedures,” the report said, adding that the eye-tracking systems used in today’s research are easier to work with and that the technology has potential for use not only in simulators but also in flight.

The report cited a 2016 study by French researchers that found that, during approach, the eye movements of the pilot monitoring (PM) showed that “the PM’s attentional allocation was not optimal, especially during the short final. It corresponded to a high percentage of dwell time out of the window to the detriment of the processing of the flight parameters, such as the energy of the aircraft.”² Another cited study found that the “gaze allocation of pilots who failed to stabilize the approach was not optimal compared to the pilots who succeeded.”³

The 2018 report concluded that the results of those two recent studies “suggest that there is an optimal visual scan path for a given visual problem that can be acquired with years of expertise but that can also be learned with eye-movement educational examples.”

While acknowledging the difficulties of incorporating an on-board gaze recording device into cockpit recording systems, the report nevertheless suggested that, in the future, a cockpit voice recorder could be enhanced to serve as a “human data recorder” capable of recording gaze data. Such data could be used not only in accident and incident investigation but also during post-flight debriefings of the flight crew, the report said.

Other elements of eye-tracking technology also could detect spatial disorientation, fatigue, attentional tunneling or automation surprise, the report said, adding that in studies conducted in a simulated maritime environment, eye tracking helped predict upcoming decisions.⁴ In another setting, a proposal called for eye tracking to be used to infer an upcoming decision by a truck driver to change lanes.⁵

As an example of how eye-tracking technology could be used operationally, the report cited an Air France Boeing 777-200’s approach to Charles de Gaulle Airport in Paris on Nov. 16, 2011.⁶

“During the approach, while the aircraft was stabilized on the descent path, a Master Warning alarm indicated that the automatic landing mode had changed, and this was called out by the relief pilot. … [T]he PF [pilot flying] erroneously and unintentionally pushed the autothrottle disconnection switch instead of the takeoff/go-around switch that engages the go-around modes. Therefore, his nose-up inputs were contradictory with the autopilot system that was trying to keep the airplane on the descent trajectory. Meanwhile, the PM fully concentrated his attention on monitoring the retraction of flaps, a process which takes about 10 seconds. The PM’s failure to properly monitor the aircraft’s attitude (position) and energy state had resulted in a brief loss of flight path and late adaptation of the go-around pitch. Thankfully, a third relief pilot was on board and made two deviation callouts of ‘pitch attitude’ that led the pilots to apply the nose-up input.”

In its final report, the French Bureau d’Enquêtes et d’Analyses (BEA) said the incident resulted from the flight crew’s “inadequate monitoring of flight parameters.”

Bursts of Light

HeliOffshore, a worldwide safety association for the offshore helicopter industry, and human factors specialist Jarvis Bagshaw have completed two phases of eye-tracking research designed to improve understanding of the pilot monitoring function and thereby aid helicopter flight crews in safer use of automation in their cockpits.

The research involved 39 pilots from HeliOffshore member operators in an Airbus Helicopters EC225 simulator and a Bristow Group Sikorsky S92 simulator. The pilots wore eye-tracking glasses that were connected to a combined camera and infrared light source that emitted bursts of infrared light (not harmful to the human eye). As the light bursts bounced off the eyes’ irises and corneas, the eyelids and the skin surrounding the eyes, those areas “reflected different levels of infrared light, which was picked up by the camera and then analysed to reveal which areas of the cockpit displays are monitored during which phases of flight,” HeliOffshore said.

The results enabled researchers to determine “how crews observe flight instruments and other cockpit displays together and independently, as they perform prescribed and unprescribed procedures,” the organization said.

HeliOffshore Operations Director François Lassale added that the results have been used to enhance training and standard operating procedures and “give feedback to the manufacturers about the design of automation and cockpits.” The research also will be reviewed as part of a European Aviation Safety Agency project on evidence-based training (EBT), he said.⁷ 

In early January, HeliOffshore said that it had held the first in a planned series of workshops to improve understanding of EBT and that “the case for EBT has been strongly supported” by the organization’s eye tracking research.⁸

Critical Skill

The 2018 report in the journal safety concluded that the maturation and increased use of eye-tracking technology could benefit not only flight crews but also air traffic controllers and motor vehicle operators.⁹

“If used wisely, the eye-tracking technology can facilitate and accelerate pilot training, facilitate the investigation of in-flight incidents and considerably enhance flight safety,” the report said.

Technological advances, and especially the increased reliance on automated systems, have played a primary role in achieving today’s relatively low accident rates — which were 20 to 30 times higher in the 1960s, the report said. At the same time, increased automation has made active monitoring of flight parameters into a critical skill.

“Maintaining an excellent monitoring performance is not easy,” the report said, citing earlier research that concluded that, when pilots are not performing a specific, well-defined task, lapses in monitoring may occur. The report also cited a U.S. National Transportation Safety Board (NTSB) accident review that concluded that inadequate monitoring and cross-checking occurred during more than 80 percent of major flight-crew-involved accidents from 1978 through 1990.

‘A Suitable Tool’

A 2016 study by researchers from the U.K. and Taiwan found that the eye movement patterns detected by eye-tracking equipment demonstrated that the eye tracker is “a suitable tool to investigate pilots’ cognitive process of attention and decision making on [the] flight deck.” It also can help improve pilots’ situational awareness, according to a presentation prepared for delivery to a conference of the European Association for Aviation Psychology.¹⁰

“Eye-tracking research provides scientific evidence on the underlying relationships between independent variables (such as flight experience or interface designs) and dependent variables (such as attention distributions or SA performance,” the researchers said. “In other words, eye trackers offer not only what causes the situations but also how the situations are triggered.”

Notes

1. Peysakhovich, Vsevolod; Lefrançois, Olivier; Dehais, Frédéric; Causse, Michaël. “The Neuroergonomics of Aircraft Cockpits: The Four Stages of Eye-Tracking Integration to Enhance Flight Safety.” safety Volume 4 (Issue 1). Feb. 27, 2018.
2. Reynal, M.; Colineaux, Y.; Vernay, A.; Dehais, F. “Pilot Flying vs. Pilot Monitoring During the Approach Phase: An Eye Tracking Study.” In Proceedings of the International Conference on Human-Computer Interaction in Aerospace. Paris, 2016. Cited by Peysakhovich et al.
3. Lefrançois, O.; Matton, N.; Causse, M.; Gourinat, Y. “The Role of Pilots’ Monitoring Strategies in Flight Performance.” In Proceedings of the 32nd Conference of the European Association for Aviation Psychology. Cascais, Portugal, Sept. 26–30, 2016. Cited by Peysakhovich et al.
4. Peysakhovich, V.; Vachon, F.; Vallières, R.B.; Dehais, F.; Tremblay, S. “Pupil Dilation and Eye Movements Can Reveal Upcoming Choice in Dynamic Decision-Making.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. October 2015.
5. Zhou, H.; Itoh, M.; Inagaki, T. “Eye Movement-Based Inference of Truck Driver’s Intent of Changing Lanes.” Society of Instrument and Control Engineers Journal of Control, Measurement and System Integration Volume 2 Issue 5 (2009): 291–298.
6. BEA. Serious Incident Report, “Momentary Loss of Control of the Flight Path During a Go-Around.” Nov. 16, 2011.
7. HeliOffshore. “News: HeliOffshore Consults Flight Instructors and Examiners on Next Steps for Eye-Tracking Research.” June 13, 2018. http://helioffshore.org/helioffshore-consults-flight-instructors-examiners-next-steps-eye-tracking-research/
8. HeliOffshore. “News: HeliOffshore Workshops Support Operators Preparing to Adopt the Safety Benefits of Evidence-Based Training.” Jan. 8, 2019.
9. Peysakhovich, Lefrançois, Dehais, Causse.
10. Li, W.-C.; Lin, J.; Braithwaite, G.; Greaves, M. “The Development of Eye Tracking in Aviation (ETA) Technique to Investigate Pilots’ Cognitive Processes of Attention and Decision Making.” In Proceedings of the 32nd Conference of the European Association for Aviation Psychology. Cascais, Portugal, Sept. 26–30, 2016.

---

Featured image: illustration: Susan Reed; cockpit image, Matti Blume | Wikimedia CC-BY-SA 4.0
Asiana Airlines accident: Alexander Novarro | Wikimedia CC-BY-SA 3.0
Eye-tracking glasses: SensoMotoric Instruments | Wikimedia CC-BY-SA 4.0