Methods for identifying unexpected risks in the Next Generation Air Transportation System (NextGen) should be enhanced without delay as part of implementing the safety management system (SMS) of the U.S. Federal Aviation Administration (FAA), says an independent review. If upgrades to safety data collection and analysis fall behind the pace of NextGen advances, says the report by the U.S. Government Accountability Office (GAO), national-level risk analyses could be based on insufficient or untrustworthy information.
“A senior FAA official [said] that although safety assessments had been conducted on individual NextGen technologies, until the agency has finalized [the National Level System Safety Assessment] modeling project, it cannot begin systemwide assessments of the safety of NextGen technologies and procedures that are already being deployed, including 700 new navigational procedures that had been deployed as of October 2009,” the report said. “Because some NextGen changes are already taking place, it is urgent that FAA move with all deliberate speed to advance its analytical capability … model the impact of NextGen changes on the National Airspace System [NAS] and manage any risks emerging from these changes.”
The International Civil Aviation Organization, U.S. agencies responsible for aviation safety and counterparts in other countries — such as the Confidential Human Factors Incident Reporting Programme (CHIRP) in the United Kingdom — now consider data-driven analysis to be indispensable in accident prevention by revealing accident/incident precursors and emerging risks. This especially includes data reported voluntarily by aviation professionals.
“As part of SMS, FAA plans to analyze data proactively … to model the impact of proposed changes in procedures and technologies on the safety of the NAS [and to identify safety vulnerabilities and mitigating measures],” the report noted.
“Currently, FAA assesses risks for specific NextGen procedures and technologies, but cannot model the risks across the NAS in a comprehensive manner. … FAA is also developing a plan for managing data under SMS, but the plan does not fully address data, analysis or staffing requirements.”
The GAO performance audit from August 2008 through May 2010 comprised a review of 13 aviation safety databases maintained by the FAA, the National Aeronautics and Space Administration (NASA), the National Transportation Safety Board (NTSB) and the Department of Agriculture (National Wildlife Strike Database), and interviews with 10 subject matter specialists in aviation safety and/or safety data collection and analysis.
Safety specialists at the FAA have contended that statistically valid samples from subsets of all airlines and industry sectors adequately reflect risks in the entire NAS and enable effective risk management. GAO reviewers disagreed, arguing for an expansion of data collection to better monitor safety trends in some sectors and urging tighter data quality standards.
“FAA has access to some voluntarily reported data, which are important for SMS, but not all [air] carriers and aviation personnel participate in FAA’s voluntary reporting programs,” the report said. “While FAA has some information on reasons for nonparticipation and has taken some steps to promote greater participation, it lacks carrier-specific information on why air carriers are not participating.”
Reliable Sources
The report found a number of appropriate controls over data quality in the 13 databases, but for several of them cited inadequate routine review of data by a manager before data are added to a database. Correcting this weakness is a critical aspect of ensuring that data are “reliable (complete and accurate) and valid (measure what is intended),” the report said. Overall, the four government agencies’ oversight of their safety data was consistent with GAO standards for identifying, reporting and correcting erroneous data. Practices examined included checks of reliability, including whether “data are complete and accurate, measure intended safety concerns, and are useful for their intended oversight purposes.”
In addition to issuing policies and quality control standards for safety data processing,
the FAA has used techniques such as cross-referencing internal and external databases to check reliability and validity, and facilitated communication among analysts from these government agencies to identify, share and correct discrepancies, the report said.
As an example of a NextGen side effect that should be identifiable through national-level data analysis, the report cited NextGen approach procedures that enable increased rates of landings — designed to reduce airspace congestion and fuel consumption — but that also could generate greater airport surface congestion and risk of taxiway collisions.
“FAA is in the process of designing tools that will allow it to model the changes,” the report said. “To do so, it has begun to develop a baseline of current conditions [from fusion of operational data] and then expects to analyze how NextGen changes will affect those conditions, according to a senior FAA official.”
SMS and ASIAS
Some organizations within the FAA have attained SMS initial operating capability or have made significant progress toward that status, but the GAO expressed concerns about the pace during the past two years. “FAA’s goal is for the Office of Aviation Safety to have initial operating capabilities in place for SMS by the end of [September] 2010 … these initial operating capabilities include training employees and defining how to apply SMS to the agency’s overall oversight activities,” the report said, noting that the FAA Air Traffic Organization issued an implementation plan, introduced a manual that guides SMS-related daily activities of its personnel and attained initial operating capability in March. The FAA Office of Airports and FAA Office of Aviation Safety, however, were at earlier stages of the implementation process.
Another GAO observation about NextGen concerned the FAA Aviation Safety Information Analysis and Sharing (ASIAS) program — a government-industry program that conducts NAS-scale data fusion and analysis (ASW, 5/08, p. 25, and 8/09, p. 32) — which had not finalized its draft plan for operations for a period ending in 2022, a time frame similar to that for NextGen implementation (ASW, 4/10, p. 30).
“While FAA has issued agencywide guidance on implementing SMS and has some efforts such as ASIAS under way, it does not have a way to measure, or specific times to indicate, full implementation,” the report said. FAA officials and GAO reviewers agreed that full SMS and ASIAS implementation will take years, but disagreed about how best to manage this process.
Both organizations also recognized that the amount of time and work required for data analysis have been difficult to project. For automated high-volume searching, coding, integration, interpretation and analysis of narrative data, the FAA had to develop an ASIAS-specific text-mining process. “FAA has efforts under way to address two key [ASIAS] challenges: … Data are not coded to permit electronic integration, analysis and sharing [and] data from two voluntary reporting programs lack identifying details needed for some types of analysis, and … do not remain available for long-term analysis,” the report said.
Coding disparities in the original definitions, event identifiers and classifications also have complicated the integration of quantitative and qualitative/narrative data.
ASAPs and FOQA
Participants in aviation safety action programs (ASAPs) and flight operational quality assurance (FOQA) programs adhere to rules for maintaining confidentiality and trust through data de-identification processes and, with a few exceptions, protections against public disclosure or disciplinary action by the FAA or an employer for operational errors. The NTSB told GAO reviewers that — in numerous investigations of serious incidents and accidents — FOQA data alone had not revealed any precursor. As opportunities emerge for data integration, however, a conflict among safety objectives can arise.
This year, the MITRE Corp., which aggregates and analyzes the data from 28 ASIAS-participant air carriers with FOQA programs and from 13 of these carriers with ASAPs, began quarterly briefings of the ASIAS Executive Board on work in progress, including provision of industry benchmarks enabling comparisons of individual airline performance to aggregate performance. In total, 73 U.S. airlines have one or more ASAPs and 36 have FOQA programs.
“Details of reported incidents are redacted from ASAP and FOQA data before an FAA contractor analyzes the data,” the report noted. “These details include the date, time and flight number, and the names of the carrier or individuals involved. … Additionally, ASAP and FOQA data are retained for only three years. Without identifying details and without maintaining data for longer periods, opportunities for some analyses are limited.”
One accepted workaround for this conflict of safety objectives is case-by-case permission from the ASIAS Executive Board, which represents industry and government, for MITRE to perform “a specific, defined analysis [directed study] and to use data with the identifying details needed for that particular analysis,” the report said.
Similar problems surface in comparing other sources with NASA’s Aviation Safety Reporting System (ASRS) reports. “While FAA’s contractor loses access to ASAP reports after three years, about 62 percent of ASAP reports appear in ASRS, along with other reports voluntarily submitted by industry personnel, according to a NASA official,” the report said.
The GAO also contrasted NASA’s reluctance to comment on individual ASRS or ASAP reports with the CHIRP practice of advisory board review and comment on lessons learned from selected reports. “NASA noted that, in the past, it had an ASRS advisory committee that had provided a forum for FAA and industry to discuss corrective action,” the GAO report said. “The agency acknowledged the need to re-establish this committee.”
Reviewers found that despite adhering to data quality standards, processes for intake of ASAP and ASRS reports have limited control over completeness or accuracy of the content. “Voluntarily reported data are subjective and are not always accompanied by supporting documentation, such as statistics, measurements or other quantifiable information related to the reported events,” the report said. Distortions, omissions and errors may not imply failures or bad intentions of the reporter, however. Factors that influence completeness and accuracy include “the reporter’s experience, visibility conditions, the duration of the event, and any trauma experienced by the reporter,” the report added.
Missing Denominator
For decades, the availability of exposure data to calculate rates of accident/incident occurrence — such as number of fatal accidents per million departures — has been a key to monitoring airline safety trends, the report said. “FAA’s ability to monitor and manage risk for certain industry sectors, such as general aviation, air ambulance operators and air cargo carriers, is limited by incomplete data,” the report said. “[FAA] does not collect actual flight activity data for smaller air carriers that provide on-demand service, such as [air cargo,] air taxis and air ambulances, and general aviation operators. …
“Without data on the number of flights or flight hours, FAA and the air ambulance industry are unable to determine whether the increased number of accidents has resulted in an increased accident rate, or whether it is a reflection of growth in the industry. … Lack of operations data for small cargo carriers makes it difficult for FAA to prioritize risks and better target safety improvements and oversight to the areas of highest risk.”
The report also cited new or reiterated NTSB proposals for FAA safety data enhancement, including needs for “new approaches to data analysis rather than simply combining existing data sources into an analysis program” and mandatory reports from airlines on a wider scope of aircraft airworthiness and maintenance-related events.
This article is based on the May 2010 GAO report no. GAO-10-414, “Improved Data Quality and Analysis Capabilities Are Needed as FAA Plans a Risk-based Approach to Safety Oversight.”