Opening new doors by mining aviation data linked to operating trends, a new FAA-industry program delves into safety vulnerabilities.
By Wayne Rosenkrans
AeroSafety World, May 2008
(Expanded version of article)
Twenty U.S. airlines routinely analyze parameters of airplane operation captured by their flight operational quality assurance (FOQA) programs.1 Sixty-one have processes to rectify safety deficiencies identified by narrative reports submitted by pilots through aviation safety action programs (ASAPs).2 These airlines for years have had measurable safety improvements, but none has had — until 2008 — a robust way to compare its safety performance indicators with industry benchmarks or with broader trends identified through the new Aviation Safety Information Analysis and Sharing (ASIAS) program, a collaborative effort of the U.S. Federal Aviation Administration (FAA) and private-sector partners.
The FAA and airlines now have the capability to cross-query de-identified aggregate data distributed across private network servers and corresponding data on government servers. ASIAS analysts then make sense from results of this fusion of numerical data and narrative-text records. This nascent capability is expected to enhance the safety intelligence that airlines have gained individually with FOQA and ASAP, both voluntary safety programs (ASW, 2/08, p. 34–39).
After about 10 years of discussion and rule making in partnership with airlines and pilot unions to develop legal protections, the FAA in April 2004 issued Advisory Circular 120-82, Flight Operational Quality Assurance, while conducting research on new methods of analyzing de-identified aggregate FOQA and ASAP data off the premises of airlines. Aggregate data means summary statistical indexes associated with FOQA event categories based on analysis of FOQA data from multiple aircraft operations (ASW, 8/07, p. 11–15).
The AC says, “FOQA is a voluntary safety program that is designed to make commercial aviation safer by allowing commercial airlines and pilots to share de-identified aggregate information with the FAA so that the FAA can monitor national trends in aircraft operations and target its resources to address operational risk issues (e.g., flight operations, air traffic control [ATC], airports). … A cornerstone of this new program is the understanding that aggregate data that is provided to the FAA will be kept confidential and the identity of the reporting pilots or airlines will remain anonymous as allowed by law. … [U.S. Federal Aviation Regulations (FARs) Part] 13.401 requires operators of approved FOQA programs to provide the FAA with aggregate FOQA data in a form and manner acceptable to the Administrator.”
FOQA data typically are shared with the FAA during quarterly briefings of personnel from the agency’s certificate management offices or flight standards district offices, covering trend analysis and corrective action plans without physical exchange of data, and during twice-a-year FAA meetings known as FOQA–ASAP Infoshare. ASAP reports from pilots, with some exceptions, typically are handled monthly by event review committees that include an FAA inspector, then are archived in a secure airline database and at the U.S. National Aeronautics and Space Administration (NASA) Aviation Safety Reporting System (ASRS).
Each participating air carrier has primary responsibility for identifying threats and errors, taking corrective action and monitoring program effectiveness. Less well known, however, are details of how the FAA benefits from FOQA, ASAP and the other voluntary programs.
By the end of 2006, the FAA, NASA, government contractors and data-analysis companies had made headway with several proof-of-concept demonstrations, all to enable safety analysis by the FAA and industry at the national level. A high-level architecture for timely awareness of problems had been a key missing element, according to Jay Pardee, director, FAA Office of Aviation Safety Analytical Services, and Michael Basehore, Ph.D., FAA ASIAS program manager. Some prior efforts — such as Voluntary Aviation Safety Information-Sharing (VASIS) and the U.S. Commercial Aviation Safety Team (CAST), which both laid the groundwork for launching ASIAS in October 2007 — had grappled with the challenges of managing extremely large volumes of data, they said.
Among factors that make ASIAS possible are recent advances in the suitability of text-mining tools in commercial off-the-shelf software and the long-sought capability to link weather conditions and/or air traffic control (ATC) environment to aggregated flights without compromising airline or pilot anonymity. The FAA assigned the Center for Advanced Aviation System Development at the MITRE Corp., its federally funded research and development center, to develop the high-level architecture and to synthesize new databases from a secure networked repository of remote databases.
NASA was instrumental in overcoming impediments to the fusion of databases — such as developing the Aviation Data Integration System, which enabled links between airport weather data and de-identified aggregate flight information without revealing to analysts protected information such as flight numbers, airlines or pilot identities. This solution, now accepted by ASIAS airlines and pilot unions, involves encrypting data aboard participants’ airplanes so that all links to identities remain secure on a trusted third party’s Web server.3
“The primary responsibility for safety is still at the air carrier level, but the FAA has a responsibility to advance a coalition for information sharing beyond the capability of any single carrier, tapping into the industry’s vast operational and technical expertise,” Pardee said. “The accident at Lexington [Kentucky, U.S.; a Comair Bombardier CRJ100ER in August 2006] exhibited and emphasized the very low frequency of some threats … and the inability to detect them. This was the first fatal accident on a wrong-runway departure in 20 years [ASW, 11/07, p. 38–43]. When we went back to incident data during our subsequent wrong-runway departure case study — knowing what we were looking for — the significant disappointment to us was that there were 116 prior events, yet we did not see the Lexington accident coming. We identified 22 contributing factors and nine airport geometries that contributed to wrong-runway departures. Probably no single carrier’s database would have revealed more than one event, if any. It was such a rare case.”
The FAA considered this case study a breakthrough in narrative text mining. “We were able to look at 5.4 million records — most in the ASRS, the U.S. National Transportation Safety Board Accident Database and the FAA Accident/Incident Data System — in 10 days,” Pardee said. “That would not have been possible six to eight months earlier.”
The urgency of analysis at the national level also has been driven by the forecasted neartripling of the number of U.S. airline flight operations by 2025. “From a safety perspective, that means that we need to reduce the accident rate by an equivalent order of magnitude,” Pardee said. “We are no longer looking only at repetitive/recurrent threats like controlled flight into terrain, loss of control or approach and landing accidents (ASW, 7/06, p. 26–39). Cross-querying databases did not exist at the refined level we have needed; we need to do it now automatically 24 hours a day, seven days a week.”
Part of the task of ASIAS is developing safety vulnerability–discovery applications of text–mining tools and tools that manually or automatically can find trends, atypical events, exceedances and aberrations. “We want to transfer technologies and key data sources into the FAA’s national distributed data archive,” Pardee said. “We want to cast our net around as many databases as possible — including service difficulty reports and international sources such as the European Coordination Centre for Accident and Incident Reporting Systems — leveraging them to get the earliest indication of something to which we can draw the attention of the subject matter experts working for ASIAS.”
Automated tools being developed at MITRE and elsewhere eventually will enable ASIAS to monitor databases.4 Automation is possible partly because once ASIAS analysts successfully apply a new multi-database query or analytical procedure, the query or procedure can be replicated to investigate other issues. “In the past, we would have to crunch through the data all over again,” Basehore said. “Now, if I’ve already got that procedure, I just use that technique and it runs through the data again.”
Trends to monitor include the number of stabilized approaches at various points in the approach; circumstances of minimum fuel/emergency fuel declarations; and systemic factors involved in runway incursions. “We need to know the important emerging safety threats as they are occurring, or as early as we could possibly detect them,” Pardee said. “Automated tools will flag any of the criteria that we direct them to flag. We expect to have the ability to cross-query for the presence of problems we think we already have mitigated through 70 CAST safety enhancements to make sure that, in fact, they have been declining in frequency, and they have not been exacerbated by changes to the National Airspace System or increases in the number of operations.”
The main purpose of the FAA’s distributed archive is trending, Basehore added. “We can ask, for example, ‘Prior to putting this CAST safety enhancement in place in 2006, what was the trend?’ or ‘After we changed to this reduced vertical separation minimum, did we start seeing more resolution advisories from traffic alert and collision avoidance systems?’” he said.
Directed by the government-industry ASIAS Executive Board, the ASIAS program focuses on known-risk monitoring, vulnerability discovery and directed studies.5 The board determines priorities and where to send investigation results and analyses for follow-up action. As of April 2008, ASIAS had access to de-identified FOQA and ASAP data from seven airlines that operate under U.S. Federal Aviation Regulations Part 121; MITRE’s national airspace data related to flight operations, weather, radar and air traffic; and the FAA’s data on safety and air traffic trends. “Aircraft manufacturers are likely to be added in May 2008 and maintenance and repair organizations probably will follow within a few months,” Pardee said. “ASIAS is in its stand-up phase and to a certain extent beginning to deliver safety products. The primary role of NASA has shifted to development work on vulnerability-discovery tools for ASIAS.”
The work of CAST also remains closely interrelated with ASIAS, with ASIAS newly responsible for the identification of emerging risks. “CAST will determine appropriate mitigations to those risks and help implement them around the United States and the world,” Pardee said. Plans also call for ASIAS gradually to include more manufacturers, maintenance and repair organizations and other communities such as rotorcraft, general aviation, the U.S. Department of Defense and other federal entities.
Atypical events, aberrations and exceedances may become apparent in computer graphics generated from the fusion of data. Flight paths retraced from FOQA data and FAA radar-track data, for example, may enable an analyst to notice which flights are significantly different from the rest. “A flight track displayed that has unusual characteristics is considered an atypical event,” Pardee said. “We are looking for annunciation to ASIAS of those flight tracks — not that any track in its own right is an unsafe condition, but the fact that it has changed or might indicate an unsafe condition. We want subject matter experts to delve into that and understand why it happened. Exceedances in the flight parameters derived from FOQA, radar tracks or even excerpts from flight crew narratives in ASAP also might be cited in daily alerts.”
TAWS Alert Example
The ASIAS Executive Board can order a directed study, either on its own initiative or when an issue comes to its attention. In December 2007, the ASIAS Issue Analysis Team, the board’s analytical arm, got the green light for its first directed study. Building upon the VASIS process and nearing completion as of April 2008, the study has been examining unexplained terrain awareness and warning system (TAWS) alerts at several mountainous-terrain airports in the United States. Any pattern of TAWS alerts is a red flag to the FAA because of the risk that pilots could become complacent about immediately conducting the escape maneuver. “That is a negative reinforcement of a safety warning system,” Pardee said.
FOQA data had provided the first awareness of clusters of TAWS alerts, which are designed to prevent collisions with obstacles or terrain. Alerts can be triggered when, as ATC provides tactical radar vectoring, the aircraft has an excessive rate of closure with — or reaches a predetermined distance from — a hazard identified in the TAWS terrain database. Assignment by ATC of the minimum vectoring altitude (MVA) for an ATC sector, a predetermined altitude based only on a required 1,000 ft or 2,000 ft of obstruction clearance, complicates scenarios when the MVA is lower than the minimum sector altitude that the crew sees on paper charts or electronic charts.6
“From FOQA data, we have known the event locations,” Pardee said. “We have known the height above the ground from NASA space shuttle terrain database mapping and U.S. Geological Survey data. We have known the arrival procedures for the various ATC sectors at the airport … from FAA Standard Terminal Automation Replacement System radar data for arrivals, including the MVA portion of it. We have known which MVA areas were involved. Using FOQA data and FAA National Offload Program data — ATC radar traces — we have been able to correlate the locations where TAWS alerts have been triggered, and by overlaying the approach paths on one particular MVA area of interest, we saw what the typical flight arrival procedures were for a day selected at random.”
Instrument meteorological conditions also have been suspected as a factor in how descents below MVA occurred. “Given the charted arrival and ATC vectoring procedures for arrivals from the east at one airport of interest due to weather, and actual weather at the time, we could see from radar traces airplanes penetrating one MVA on at least six arrivals for a particular date,” Pardee said. “We have known from FOQA events at those latitude/longitude coordinates how to filter by this arrival flight pattern all the flight arrival data for any MVA penetrations. This showed that these, in fact, were the arrival tracks that actually produced TAWS alerts.”
The reasons are still being sorted out, but ASIAS analysts so far have revealed several things. “For example, ATC instructed crews to make some very sharp turns to the airport and in arrival procedures, ATC set up some flights with difficult maneuvers required to round out the descent and not penetrate the MVA,” he said. “ATC also gave some of the flights studied a high descent rate.”
ASIAS also has been used to address concerns about the accuracy of specific terrain databases. “We see potential areas for improvement, such as to upgrade the algorithms in the TAWS box [equipment], to reconfigure the MVAs used by ATC for compatibility with the arrival flow from an ATC perspective,” Pardee said. “Certain flight profiles lend themselves to TAWS alerts, and aggressive flight profiles during vectoring may make it more difficult for flight crews to avoid penetrating an MVA.
“We also think it makes a difference which version of TAWS software is installed, although none have MVAs embedded in terrain data. Later versions of TAWS software are more attuned to rising terrain and high descent rates. There also needs to be fine-tuning of MVAs and fine-tuning of arrival flight tracks. We need to make sure that all of these system pieces interact properly.”
Investigation Without an Accident
Because of its fusion of databases and extensive use of subject matter experts, ASIAS research likely will begin to resemble the accident investigation process — without the accident, Basehore said. “We pull in the folks who are familiar with the avionics design, the mechanical design of the aircraft and its capabilities, as well as what ATC and pilots were doing at the time,” he said.
During the first fiscal year of ASIAS, the FAA is keeping its scope of investigation narrow and trying to set realistic expectations internally and externally, emphasizing what can be done without major new expenditures by the government or air carriers, Basehore said. “That was an eye-opener to the industry,” he said. “We want to demonstrate what we can do with tools that already exist and databases that are already out there.” Pardee added, “We have taken on four critical problems, two directed studies, three airline benchmarks and four CAST safety enhancement indicators. We will document how they are done so they can be automated.”
Striving for Harmony
To accomplish the ASIAS program’s mission, the FAA has had to address not only the technical issues but building relationships of trust, implementing governance and policies that maintain the legal protection of data for safety purposes, convincing airlines and others of the value of participation and dispersing some of the investigative responsibilities among many non-FAA specialists.
“We have found that there is really no one entity that is the best in the business in all kinds of safety databases,” Pardee said. “We use, for any type of analysis, partners that have demonstrated excellence in fields such as synthesizing new databases from other databases, manipulating FOQA data, preparing stabilized-approach algorithms, and measuring airplane energy states on arrival.”
From inside the FAA, ASIAS also leverages the expertise of the Air Traffic Organization and its resources, such as terminal radar control radar-track arrivals data, National Offload Program en route data, airport geometry data, Airport Surface Detection Equipment Model X data from more than a dozen airports and ATC national flow data.
“Some of these databases are sensitive and protected — FOQA, ASAP and the FAA’s radar-track data, for example,” Pardee said. “An agreement in place with all the contributors cites the basic principle that the information is solely used for the purposes of safety, to drive safety decisions — not punitive actions.”
Safeguards against misuse of data provided by airlines and other non-government suppliers include physically keeping source data at the suppliers, using “middleware” for one-way encrypted transmission of de-identified aggregate data only; and externally archiving all ASIAS-generated data at MITRE. For FOQA/ASAP airlines — and counterparts such as the new air traffic safety action program (ATSAP) for FAA air traffic controllers — a memorandum of understanding between MITRE and each data supplier details how data are de-identified, aggregated and digitally bundled for transmission to MITRE/ASIAS, including the conditions for its use by ASIAS.
Pardee and Basehore frequently meet with people who bring to the table a history of concerns about any voluntary program for information sharing with the FAA. Pardee said, “We have to say to them, ‘This is a new deal, a golden opportunity. We can reconstitute all of the prior information-sharing programs — taking you to a place that you cannot go by yourself.’ The suppliers of proprietary, sensitive data have knowledge of how their data is accessed and used. Various members of the ASIAS community are granted authority to generate queries based on the kind and nature of the data they submit. We make them part of the analysis teams because they have knowledge and expertise — and not just so that they can see that proper governance procedures are being followed.”
Trending by the ASIAS program will help the FAA to identify operational effects, including any threats, when traffic flow has exceeded ATC capacity; the exact ATC radar sectors in which holding has occurred; and which sectors have been combined. Pardee said that the fusion of Air Traffic Organization databases with FOQA databases, for example, will help the FAA and industry address a variety of questions. How many go-arounds occurred at the airport under given circumstances? How many flap exceedance speeds and late gear extensions occurred? Did flight crews get TAWS alerts? How many occurred on the instrument landing system glideslope? How many times did ATC procedures result in “slam dunk” arrivals and/or unstabilized approaches?
FOQA–ASAP Infoshare continues, and expanded agendas for these meetings make all the participating airlines aware of ASIAS activities, including those that have not signed a memorandum of understanding with MITRE. Infoshare complements the work of ASIAS analysts by enabling all parties to contribute early insights and concerns about possible emerging threats, Pardee said.
- FSF Editorial Staff. “Wealth of Guidance and Experience Encourage Wider Adoption of FOQA.” Flight Safety Digest Volume 23 (June–July) 2004.
- ASAPs also have been implemented for flight attendants, dispatchers and maintenance technicians.
- Chidester, Thomas R. “Voluntary Aviation Safety Information-Sharing Process: Preliminary Audit of Distributed FOQA and ASAP Archives Against Industry Statement of Requirements.” FAA Civil Aerospace Medical Institute Report no. DOT/FAA/AM-07/7. April 2007. In June 2004, NASA led airlines, airline employee organizations and the FAA in determining key components of national distributed data archives for FOQA and ASAP, the report said. In October 2004, distributed archiving andanalysis for both were developed by NASA for the FAA. In January 2006, Alaska Airlines, American Airlines, Continental Airlines, Delta Air Lines, ExpressJet, JetBlue Airways, Southwest Airlines, United Airlines and UPS Airlines began archiving their data.
- Larder, Brian; Summerhayes, Nigel. “Application of Smiths Aerospace Data Mining Algorithms to British Airways 777 and 747 FDM [Flight Data Monitoring] Data.” Report of a technology demonstration in partnership with the FAA and the Global Aviation Information Network. December 2004. The report said, “the data mining tool … unearthed many interesting patterns and relationships at what could be called a ‘second level’ down which had not been previously detected using existing analysis techniques. If they had been detected, it is likely that they could have been dismissed as noise or random groupings.”
- The co-chairs of the board are the co-chairs of CAST: Don Gunther, senior director, safety and regulatory compliance, Continental Airlines; and Margaret Gilligan, deputy associate administrator, Aviation Safety, FAA.
- FSF Editorial Staff. “Charts Raise Pilot Awareness of Minimum Vectoring Altitudes.” Flight Safety Digest Volume 23 (September) 2004, p. 1–28.