Error containment improves as the severity of operational errors (OEs) increases, according to a U.S. Federal Aviation Administration (FAA) study of events involving U.S. en route air traffic controllers.1 But looking at the OE data strictly by risk categories masks some error containment inefficiency.
The report describes two different studies of OEs. The first concerned the probability of resolution (POR); the second, the effects of the controller’s time on position (TOP).
Study 1 was described as measuring “OE containment.” The FAA Air Traffic Organization (ATO) classifies OEs into four risk categories in increasing order of severity: proximity events, in which 90 percent or greater separation is retained either horizontally or vertically; Category C, or low risk; Category B, or moderate risk; and Category A, or high risk. The ATO calculates the rates of Category B and Category A OEs to monitor progress toward safety goals.2
The report says that while existing safety metrics track error prevention, they do not measure containment of errors that occur. “The POR is a measure of the efficiency with which the NAS [National Airspace System] is able to resolve separation losses (i.e., contain the errors through the actions of controllers and pilots) before they degrade into greater risks to safety.”
For the study’s purpose, the risk categories were conceived as zones representing a series of points in time as air traffic separation is reduced, beginning when separation conformance3 is less than 100 percent. “Thus, each of the OE safety risk categories represents a potential containment field,” the report says.
In Study 1, a total of 1,293 OEs were taken from a pre-existing database for the period May 1, 2001, to May 31, 2003. “Since our primary goal in this study was to demonstrate the utility of employing a measure of OE containment for SMS [safety management system] purposes, we were not as concerned about using current OE data as we were with having a data set that we understood,” the report says.
Data examined included the OE report number, the lateral and vertical distances recorded at the time of the closest proximity of aircraft, and the required lateral and vertical separation standards. Because an OE can be attributed to more than one controller, the researchers used data only for the controller who was primarily responsible, so that no OE would show up more than once.
Category C, or low-risk, OEs were most frequent in the data set, with 719 events — 56 percent of the total. Category A — high-risk OEs — represented 15 events, or 1 percent.
The report says, “When considering the en route centers as an aggregate, we see that the NAS was 26 percent effective at resolving losses of separation within the proximity-event range, 75 percent effective at the low-risk range, 94 percent effective at the moderate-risk range and 100 percent effective at the high-risk range. … The distribution of PORs shows a continuous rise in efficiency as we progressed through the OE severity categories and ended with 100 percent resolution by the time we reached the Category A region.”
The OE categories in the original analysis did not represent equal intervals. The researchers wondered whether the same POR efficiency would be seen if the total OEs were “sliced” evenly into thinner intervals, or whether some regions of inefficiency would be evident. “Thus, we eliminated the OE severity categories and instead divided the region of separation conformance into 10 equal percentage intervals,” the report says. “We then computed the number of OEs associated with each interval and calculated the corresponding PORs.”
Looked at this way, the trend line for POR efficiency was no longer a smooth ascent. “Whereas we saw a continuous rise in efficiency of OE containment as we progressed through the OE severity categories, we [saw] a drop in efficiency of OE containment occurring between the third and fourth interval,” the report says. “A zone of relatively lower OE containment efficiency (intervals 4–6) continues until reversing at the transition between the sixth and seventh interval. This zone is primarily in the moderate OE severity (Category B) region.”
It is possible only to speculate why containment efficiency declined after the third interval until recovering at the seventh, the report says: “Perhaps OEs in this region were surprises. That is, the controller may have been unaware that an OE was occurring until the separation loss crossed the 75 percent separation conformance threshold. By the time the OE was discovered, the controller did not have sufficient time to restore separation before incurring a further loss.”
The report argues for use of POR data as a safety metric in addition to conventional OE data. “Looking just at the error prevention indicators, we see the number of separation losses that were resolved within each of the four OE severity categories,” the report says. “However, there is a risk associated with using just these kinds of numbers.
“The prevention numbers in and of themselves do not help us understand how well controllers’ actions and the pilots’ responses prevented (i.e., contained) an initial loss of separation from getting worse. This is what the POR captures; thus, we recommend that it be included as an additional metric for the ATO’s [SMS].”
Four degrees of separation may not be enough, the report suggests: “It is important for an SMS to collect data at the finest level of detail necessary to make informed decisions. While there may be valid reasons for defining the official categories of OE severity as they are, the advantages of doing so must be weighed against [their] possible obstruction of more detailed information. Methodologically speaking, equal-interval measurements are preferred over categorical assignments and, thus, it may be advantageous for the ATO to adopt equal-separation conformance intervals both for metrics of OE prevention and containment.”
Time on Position
Study 2, of time on position, found that a higher number of OEs occurred early in a controller’s TOP. The number of OEs was highest in the first 30 minutes of TOP.
“The trend is counterintuitive, given what is known about time on task and mental fatigue, in which lapses of attention become more likely as time on task increases,” the report says. “One would expect that the longer a controller is on position, the greater the chances that mental lapses in attention would occur. However, the assumption is made that controllers coming on position must not be fully prepared to manage the traffic situation due, in part, to a faulty position relief briefing.”
TOP data for U.S. en route controllers in 2006 were available to the researchers. To keep the data manageable, they were restricted to samples from the six facilities with the highest number of OEs in that year.
Researchers extracted 1,397,206 TOP records and 290 OE records. No attempt was made to account for OE severity because previous studies had failed to identify any statistically meaningful differences in OE severity based on the amount of time a controller was on position.
With data indicating the length of time controllers were on position when OEs were not occurring, “we were able to match these data with the length of TOP at the onset of an OE,” the report says. “Together, the two data sets allowed us to calculate the probability that an OE would occur, based on the length of time a controller was on position, referred to as time on position probability (TOPP). We then used the TOPP to determine whether an exposure effect was influencing the TOP distribution of OEs.”
Because the number of controllers on position varied among the 10-minute intervals, counting the numbers of OEs in each interval did not measure the probability of a controller having an OE in any given interval. Therefore, “TOPPs were calculated by dividing the number of OEs that occurred during a particular 10-minute time interval by the total number of controllers who were signed on (i.e., exposed to the possibility of having an OE) during that time interval,” the report says.
“When considering the [six en route] facilities as an aggregate, the TOPPs ranged from a low of 0.002 percent for the ninth and tenth intervals to a high of 0.006 percent for the twelfth interval, for an overall average TOPP of 0.004 percent, which is equivalent to four OEs out of every 100,000 [controller] sign-ons,” the report says. “At the level of the individual, this means that, on average, a controller has a four-in-100,000 chance of having an OE each time he or she signs on to position. At the level of the [six en route facilities], this means that for every 100,000 position changes, four OEs will likely occur.”
Thus, the TOPP results present a different picture from analyzing OE numbers by TOP. The report says, “The OE data suggest that the NAS is most vulnerable to OEs occurring early on position and that the vulnerability decreases with time. In contrast, the TOPP data suggest that a period of vulnerability may exist early on position, but that the vulnerability is greatest when a controller has worked longer on position. The latter interpretation is more consistent with the literature associated with time-on-task fatigue, in which the operator experiences greater mental fatigue the longer the time spent on task.”
Summing up the two analyses, the report says, “The probabilities associated with OE containment and TOP are two important measures to be considered for inclusion in the [ATO’s SMS]. The probability of OE containment (i.e., probability of resolution) provides a measure of effectiveness of the NAS through the actions of controllers and pilots at containing OEs at the lowest risk to safety. The time-on-position probability provides a measure of the risk of an OE occurring based on how long a controller is working on position. Both measures represent enhancements, compared to just the reporting of the frequency of OE occurrences and OE rates; thus, both measures should be considered for inclusion in the [ATO’s] system of safety metrics.”
Notes
- Bailey, Larry. “Analysis of En Route Operational Errors: Probability of Resolution and Time-on-Position.” FAA Civil Aerospace Medical Institute. February 2012. www.faa.gov/library/reports/medical/oamtechreports/2010s/media/201202.pdf.
- The report’s author told Flight Safety Foundation that since the report was written, the FAA ATO has revised the way it reports and investigates operational errors. Three relevant orders, JO 7210.632, JO 7210.633 and JO 7210.634, can be found at 1.usa.gov/GWm59e.
- The term separation conformance is used because the actual required distances vary under different flight circumstances, such as altitude and proximity to airports.
Download the PDF of this article, which includes the figures.