Sensational Development

Reports

Two-Sense Worth

An Investigation of Sensory Information, Levels of Automation, and Piloting Experience on Unmanned Aircraft Pilot Performance

Williams, Kevin W. U.S. Federal Aviation Administration (FAA) Civil Aerospace Medical Institute (CAMI). DOT/FAA/AM-12/4, March 2012. 21 pp. www.faa.gov/data_research/research/med_humanfacs/oamtechreports/2010s/media/201204.pdf.

“Research looking at UAS [unmanned aircraft system] accident causal factors has suggested that sensory deficiencies have played a role in UAS accidents,” the report says. That is, a lack of sensory information provided by the system contributed to some pilot mismanagement of UAS flight.

The experiment described in the report sought to investigate the role of sensory information, particularly as it affected pilot response to system failures, as well as other influences on UAS pilot performance.

“Other factors besides the types of sensory information available can influence the ability of a pilot to effectively manage a flight,” the report says. “UAS control, for many current systems, is highly automated. Automation-induced complacency, which is the tendency for humans to become less vigilant or focused on a task that is being performed by automation, is possible when automation replaces a task that occupies a human activity. … A pilot’s ability to respond to system failures, therefore, will be influenced not only by the sensory information available but also by the type and level of automation employed in the system and the control-interface requirements on the pilot.”

The researchers were interested in a third “unresolved question” — Is it advantageous for UAS pilots to have experience piloting manned aircraft? The FAA requires UAS pilots to have a manned aircraft pilot certificate for most operations, but the development of a UAS-only form of pilot certification has been proposed.

The experimental design involved manipulating two levels of sensory information (visual versus visual/auditory), two levels of control automation (manual versus automatic) and two levels of manned piloting experience (some versus none).

A simulated UAS control station was devised, providing three types of aircraft control. “Manual control can be accomplished through the use of [a] throttle and joystick,” the report says. “Vector control is done using the mouse and onscreen buttons for changing the altitude and heading of the aircraft. Waypoint control is accomplished by entering a series of waypoints on the moving-map display and establishing altitude settings for each leg of the flight.”

Of the 32 experiment participants, half had flown as pilot-in-command of a manned aircraft; the others had no piloting experience. None of the participants had controlled a UAS.

Participants were asked to pilot a UAS along a predetermined route while responding to various system failures. They had to monitor traffic in the area and, at set times during the flight, determine the aircraft position relative to a specific location.

“It was expected that the visual/auditory level of sensory information would be superior to the visual-only level, and that participants would respond to system failures more quickly when they received both a visual and auditory failure cue,” the report says. “For the two levels of automation, it was expected that the more automated condition would lead to a certain level of complacency for the participants, thus inducing slower responses to system failures and perhaps poorer performance at monitoring traffic. Finally, participants with manned-aircraft experience were expected to be better at determining the relative position of the aircraft and, because of a more effective scan, detecting system failures in the visual-only condition.”

While some results were as expected, there were also surprises.

“The notion that simply adding a second type of sensory information (sound) would increase the ability of pilots to identify and respond to failures was not supported in the current study,” the report says. “While the presence of sound did improve responses to engine failures, it did not improve responses to failures in heading control. One difference between the engine failure cues and heading control failure cues was the presence, in the condition where sound was used, of engine noise in addition to the auditory warning. Unfortunately, it is not possible to determine whether this additional sound cue was the cause of the difference in responding to the failures.”

The expectation that higher automation levels would lead to complacency or a slump in vigilance was not borne out. “Perhaps the relatively short flight used for the experiment (approximately 40 minutes) did not allow for an effect to occur,” the report says. The “relatively simple” nature of the task also might have confounded any decrease in vigilance, it adds.

Still, automation differences had some effects: “As expected, a higher level of automation led to lower estimates [by participants] of subjective workload. This was reflected in the flight technical-error performance findings that showed superior flight performance, in general, for participants in the high-automation condition.”

The participants with manned aircraft experience were no better than non-pilots at monitoring traffic or estimating relative direction. But the pilots flew significantly closer to the fight path than the non-pilots, which was unexpected.

“It is difficult to believe that only the pilots noticed that the aircraft was deviating from the flight path during the first flight segment, so the question is why some of the non-pilots did not attempt to correct the deviation,” the report says. “[The fact that] it occurred suggests individual differences between the pilots and some of the non-pilots could be due to either training or are innate traits that contribute to success as a pilot. If manned aircraft training and/or experience leads to more responsive flight-path control, it would be important to identify what portion of the training was responsible.”

A significant proportion of pilots responded to failures of automated heading control before the failure warning occurred, recognizing on their own that the aircraft was drifting from the commanded heading. “However, this occurred only in the no-sound condition,” the report says. “The presence of an auditory warning for pilots actually seemed to inhibit a response to a heading failure. None of the non-pilots responded early to the heading control failure, regardless of the warning condition.

“For both pilots and non-pilots, it was clear that some of the participants noticed the heading failure early but waited for the warning by positioning the cursor over the heading recovery button. Again, there are questions of whether individual differences allowed some of the pilots to respond early, why the presence of a sound cue would prevent this response, and whether training or other factors were involved in the differences between the groups.”

Under these experimental conditions, some differences appeared between those with prior manned-aircraft experience and those without. Future studies of the qualifications for UAS-only pilot certification should try to determine whether those differences resulted from pilot training and experience, or identifiable traits among people who choose to become pilots and those who do not, the report said.

Audit Trail

SMS Audit Results 2011

PRISM Solutions, a subsidiary of ARGUS International. www.argus.aero/FreeData/PRISM_SMS_Audit_Results_2011.aspx.

Each year PRISM Solutions reviews deidentified audits performed by its sister company, ARGUS PROS, on private and commercial flight operations. It then compiles the results from all of the audits into a single report, of which this is the most recent. “Although the audit reports highlight many positive trends and accomplishments within the SMS [safety management system] area, the annual SMS audit results report focuses on the recurring problem areas found in SMS implementation and execution,” the company says.

In 2011, 74 audits were analyzed. “The majority of the 2011 audit findings point to deficiencies in a general operating manual (GOM) and SMS training,” the report says. “A GOM defines policies, procedures and organizational structures to accomplish the company’s goals. It must be accurate, up-to-date and consistent with other manuals in order to prevent miscommunication and confusion. A lack of employee SMS training accounted for many of the recommendations in the area of SMS training. Employees need to be active participants and have a good understanding of safety management concepts in order for an SMS to be effective.”

The GOM was the subject of 64 percent of the SMS recommendations to the total 74 operators audited. These samples were cited in the report:

• “Recommend that the executive’s letter on safety and non-punitive reporting policy be included in the forefront of the GOM.”
• “The duties and responsibilities of the safety manager should be consistent between the operations manual and SMS manual.”

As guidelines, the audits said that the GOM should contain accurate descriptions of a safety system and contain an accurate outline of the safety officer or manager responsibilities.

The next most frequent subject mentioned in recommendations was SMS training. It was recommended, for example, that “the safety manager should receive formal training for the development and implementation of [an SMS].”

Other areas of SMS recommendations included “SMS manual” (38 percent of audits); “risk assessment” (36 percent); “internal evaluation program” (28 percent); “safety policy” (24 percent); “safety committee” (20 percent); and “hazard reporting and tracking” (9 percent).

PRISM Solutions also reviewed the audit reports of the operators’ emergency response programs. The largest share of recommendations — 30 percent — concerned emergency response plan (ERP) documentation. Auditors recommended, for example, that “on-site team members be identified in the SMS manual by official job position within the company, and all ERP documents be controlled.” Next-of-kin notification and family assistance recommendations made up 23 percent of the total. One example was: “The ERP should include guidance offering trauma counseling to company employees in the event of an accident or other devastating event.”

The report compares the 2011 audits with the total of 175 in 2008–2010. In the three-year period, the largest share of recommendations, 55 percent, concerned the internal evaluation program, versus 28 percent in 2011. The GOM, which generated 64 percent of recommendations in 2011, represented 35 percent of the 2008–2010 total.

---

Websites

Facteurs Humain

MentalPilote, www.mentalpilote.com.

This site, primarily in the French language, is subheaded “Facteurs humains, la clé du savoir agir” — human factors, the key to knowing what to do.

MentalPilote was created by Jean Gabriel Charrier, an instructor with the training branch of the French Direction Générale de l’Aviation Civile (DGAC) and then a check pilot with the DGAC. Its theme is suggested by the description of one of the three books written by Charrier, L’Intelligence du Pilote (Pilot Intelligence): “Why do certain pilots encounter fewer dangerous situations than others, and why do these same pilots commit many fewer errors? It’s because they know how to take good decisions … . But for a pilot to take good decisions is not innate; it is learned.”

MentalPilote articles and recommended books suggest that flying sports, such as soaring and hang-gliding, encourage pilots to develop cognitive skills to improve performance, and also enhance their safety consciousness.

The site currently contains 130 articles. Many are case studies from a threat-and-error management viewpoint. The reader can select from four categories of interest: private pilot, professional pilot, instructor and fundamental information. The menu offers a drop-down listing of further subdivisions — for example, accidents, good practices, culture, errors, stress, perception, risks and training.

In the professional pilot article archive, the first article is headed, “CRM [crew resource management] — the first steps for today. Test your knowledge.”

It continues: “Over to you! Before reading the rest of this article, we invite you to perform a personal reflection: What do you associate with CRM? Take a sheet of paper and pen and note your responses. Don’t cheat! This experience will be useful for the following: If you had to define CRM using only three words or key expressions, how would you define it?”

All articles are illustrated with at least one photograph, and many with several photographs and diagrams. The site is visually accented with graphic symbols.

MentalPilote links to a blog, www.formation-facteurs-humains.fr, for pilot self-training in human factors and safety.

Michel Trémaud, a retired Airbus safety specialist, contributes a “Pilot’s Whisperer” column in English, which he says is “intended to enhance the awareness of air traffic controllers on the main features and use of automation on modern business or commercial aircraft. Indeed, it is most important for air traffic controllers to understand the pilots’ working environment; this includes the fundamentals of aircraft automation (understood in this article as automatic flight guidance), how pilots interface with automated systems and how the optimum use of automation contributes to the overall management of the aircraft flight path.”

The site is intended to be useful not only in France but in French-speaking areas of North and West Africa, as well as the Middle East.