Turning numbers into pictures often happens at a late stage, or as an afterthought, within aviation safety processes, say several specialists involved in U.S. government-industry system safety efforts and accident investigation. Awareness of revolutionary data visualization techniques has accelerated, however, among analysts and investigators. The implications affect sifting through vast volumes of recorded flight data for operator-level insights, generating tables and charts from the templates built into worksheets or databases, and replaying individual aircraft events with flight data animation software, they said.
Instead of expecting aviation safety analysts or accident investigators to focus first on raw data, then graphically communicate analytical results, data visualization facilitates and guides analysis from the earliest stage. Some specialists noted that overlaying data on geospatial/terrain imagery in Google Earth Pro, ESRI ArcGIS or equivalent software offers this capability.
Geospatial software has been used since the October 2009 launch of the U.S. Federal Aviation Administration (FAA) Aviation Safety Information Analysis and Sharing (ASIAS) program. “The ability to blend all ASIAS data sources easily has greatly improved,” said James M. Reed, senior multi-discipline systems engineer, Aviation Safety Analysis, at the MITRE Corp. Center for Advanced Aviation System Development (CAASD), the not-for-profit, FAA-funded research center that has stewardship of ASIAS data.
Developers of one of the data visualization software tools used by CAASD, called Tableau, have summarized common obstacles to effective visual analysis of data: “A major stumbling block to widespread adoption is that most people are not trained in the graphical design principles needed to construct graphical presentations that support their reasoning process or communicate their analytical results to others. … They struggle with numbers that are slow to read and do not show patterns or trends. … Users typically start visual analysis with vague tasks in mind, which are refined and transformed as they see graphical views of data. … Skilled users … require a system that attends to the graphic details so that they can stay in the flow of visual analysis.”1
ASIAS and CAASD briefed AeroSafety World and provided examples of using data visualization for extracting system-level, risk-reduction insights from government-owned and airline-owned data sources. ASIAS equips and trains its participating airlines to compare their own safety metrics with industry benchmarks derived from de-identified, aggregate data contributed by the flight operational quality assurance (FOQA) programs of all participants (see, “No Turning Back”). Boeing Commercial Airplanes shared additional data visualization experiences during Flight Safety Foundation’s 64th annual International Air Safety Seminar (IASS) in Singapore in November.
To study traffic alert and collision avoidance system (TCAS) resolution advisories (RAs) near a U.S. general aviation airport located along an approach to a major airport, ASIAS and CAASD analysts conducted data fusion, integration and analyses with several types of data visualization, said Randy McGuire, program manager, Aviation Safety Analysis, CAASD. One of the final products was a heat map, a method of depicting the concentration of RAs with the densest shown as a red circle, he said. This image is a close-up view of one of several dozen concentrations published by ASIAS on a U.S. map (ASW, 8/09, p. 34).
“The highest concentration of events is at the center of the circle, and as viewers move away from that, fewer and fewer events have occurred per unit area,” Reed said. “The two-dimensional [2-D] clustering algorithm does not take into account the differences in altitude of RAs, so in reality many are invisibly stacked up on top of each other. We prefer high-fidelity, 3-D tools to look at the altitude dimension. These tools enable us to ‘fly around,’ change the perspective, and change the vertical scale and symbols/colors. We fuse a lot of data into a common picture that we can then manipulate.”
The first step en route to the heat map was visualizing the flow of aircraft into the airport using about two hours of radar track data from the FAA’s National Offload Program, McGuire said. “As it turns out, the major flow overflies … the yellow tracks of a traffic pattern with pilots doing practice touch-and-go landings in the familiar racetrack pattern.”
Another step (Visualization 3) added evidence to the analysts’ preliminary theory, said Michael Basehore, the FAA program manager for ASIAS. “Each of the yellow spheres is where the FOQA aircraft itself got a TCAS RA,” he said. “When we put these actual RAs on top of the radar tracks, we saw that they coincided with the incoming Federal Aviation Regulations Part 121 traffic, and where each RA intersects with the general aviation traffic. The reason the RAs don’t match up identically with the radar tracks is that FOQA data are de-identified but they did confirm that the RAs occurred where the typical traffic pattern was.”
The next-to-last step replayed actual radar-track data through emulation software, generating purple diamonds where TCAS RAs hypothetically could occur given how TCAS algorithms had handled the real-world encounters (see, “Appreciating Value”). “All that the FOQA data could indicate was that TCAS RAs occurred in these locations,” McGuire said. “The data did not tell us anything about the other aircraft involved in the RAs. We went back and looked at the radar data, and then we were able to develop the software emulator using the radar data as an input and, as an output, whether or not TCAS RAs would occur. We replayed all the traffic for hot spot areas in the National Airspace System to determine if we could find out why those RAs were occurring.”
McGuire noted that, to declutter this image, some yellow/blue lines (radar tracks) underlying the purple diamonds were hidden to emphasize the major flows on the approach route studied. Tracks of aircraft just passing by the area also were hidden even if the aircraft crew had received an RA.
ASIAS receives FOQA data from about 30 percent of the total commercial flights in the United States, Basehore said. “We can now run the radar tracks through this emulator, and the emulator extrapolates where we could have seen a TCAS RA if we had had FOQA data from all the flights,” he said. The results of this data visualization further confirmed what was occurring near the two airports under review.
ASIAS also performs dynamic analysis of airspace, in which real-time data visualizations show aircraft symbols moving along their radar tracks in relation to surrounding aircraft and to Google Earth Pro terrain. Dynamic analysis helps analysts to comprehend the complex geometries of aircraft interaction from any angle, altitude or direction.
“We can display one data source at one point, then bring up another source of data on another screen and see if that confirms what is happening,” Basehore said. “We’ve got the capability to do these on multiple screens using multiple databases at one time.”
Airline Data Visualization
Less visually arresting — but equally important — types of data visualization also are available to both ASIAS analysts and analysts at participating airlines. “We use Tableau quite extensively for data-manipulation analytics as our preferred method now for visualizing the aggregate data,” said Alex Alshtein, project team manager, Aviation Safety Analysis, CAASD. “Tableau is very powerful for us because we can see over time how things may change such as a different airspace or a different runway end.”
ASIAS focuses, with rare exceptions, on systemic solutions. “Tableau generates curves telling us the normal behavior for all the aircraft using this particular airport or runway end, for example,” Basehore said. “We would look at a particular aircraft if we saw something out of the norm — far off on the tail of the curve — and see why it was different from all the others.”
ASIAS researchers are working on a capability for airline analysts to use Tableau’s dashboard interface to query the FAA’s radar surveillance data, instantly and automatically generating overlays on geospatial imagery and helping users to think more clearly about spatial relationships, Alshtein said. (Dashboards are groups of possible selections and windows containing query results, all of which change to fit the context of the information sought.)
Tableau dashboards customized by ASIAS also graphically display trends and quickly reveal “rapid upticks at a particular location,” and by how many statistical standard deviations events differ from normal operations, Basehore said.
“One of the ASIAS metrics in Tableau is ‘missed approach,’ so an airline analyst can see an aggregate view of all airports, or drill down into a specific airport or a specific runway end over time,” Alshtein said, showing a map of the United States generated by this software. “The size of each circle representing an airport symbolizes the number of operations at the airport. Circles also are color-coded so that if their rate of missed approaches is higher than usual, the airport is red; if it is lower than usual, the airport is green.”
Such a data visualization may show, for example, that the root of most unstabilized approaches among all the airlines serving an airport has been a complicating geographic factor at just one runway end under certain prevailing winds, Basehore said.
Data visualization focuses analysts’ attention, at an opportune moment, on whether a significant safety issue exists, Basehore added. “Instead of having to look at everything, the tool at least ‘taps them on the shoulder’ and says, ‘You need to go over here and look at this aspect in more detail’ or ‘Something is different here than everywhere else.’”
One caveat emphasized to participating airlines is that FOQA-event definitions adopted by ASIAS — such as the one for unstable approach — may differ from those used operationally by an airline’s flight crews or by the airline’s FOQA analysts, he said. CAASD analysts visit these airlines to learn about their safety research and to train airline analysts to take full advantage of Tableau and other data visualization methods “to get them all up to the same level so that they can take advantage of the processes we are developing at ASIAS,” Basehore added.
Google Earth Pro
During aircraft accident investigations, data synthesis and visualization with Google Earth Pro extend the capabilities of older methods, said Simon Lie, associate technical fellow and senior air safety investigator, Aviation Safety, Boeing Commercial Airplanes. Other geospatial software that models and presents 3-D data also would support the methods Boeing uses in safety investigations. He said that data visualization already stands out as a proven “high-bandwidth communication tool” — that is, promoting comprehension of complex data far more quickly than older methods. The capability supports today’s investigations involving thousands of flight data parameters, messages via aircraft communications addressing and reporting systems, and data from emergency locator transmitters and non-volatile memory in aircraft systems.
Conventional time-history plots of data points from flight data recorders remain an important tool for detailed study of parameter values such as altitudes and airspeeds, and their variation over time, Lie said. With the chronological advantages of such plots comes a significant shortcoming: “The learning curve required to quickly interpret the plot shows it is not necessarily a natural way of looking at the data,” he said. “It is difficult to form a 3-D mental picture of what is happening on a flight path based solely on parameter traces.”
FOQA-data animations — highly regarded for the depiction of spatial relationships — “make some beautiful pictures and some very realistic-looking scenes, probably more realistic than some of the data that underlies them — and there is a danger in that,” according to Lie. Moreover, investigators find themselves repeatedly “fast-forwarding” and “rewinding” through the sequential images of animations when trying to explain complex causal relationships separated by time intervals.
Other animation shortcomings include limitations in the models of aircraft performance, instrument depiction and scenery. “In particular, the instruments and the functions on the photorealistic primary flight display and other instruments are very complex,” he said. The simulation may incorporate technically inappropriate content from a flight operations manual instead of from the complete, proprietary engineering description, Lie noted.
Combining chronological and spatial relationships in ways that are natural and intuitive even for non-experts, Google Earth Pro “capitalizes on the innate ability to process visual scenes that each and every one of us has,” Lie said. The visualizations have been effective at showing both types of relationships all in one scene, he said. The main limitation noticed is the lack of the precise values or parameters seen in a conventional worksheet, table or plot.
The value to aviation safety of Google Earth Pro is not so much the satellite-based terrain imagery “painted” onto geometric contour shapes. Rather, Boeing investigators are most interested in overlaying 2-D and 3-D objects such as flight paths created from data, and selecting or deselecting these objects during analysis. Most such objects are created by converting recorded data and derived data into place marks — points specified by latitude, longitude, altitude and time — via Keyhole markup language, an international geospatial standard.
Lie presented one data visualization of the flight path of a Boeing 737-300 depicting the effects of an onboard navigation problem and subsequent safe landing at an airport that the flight crew found by visual searching. The image was based on coordinates recorded by the flight data recorder. A false track generated and recorded by one of the inertial reference computers on the airplane was compared with an estimated actual track derived from heading, airspeed and altitude data determined to have been accurate.
In another example, data visualization of intersecting coverage by two radar antenna sites in relation to a 737-400 track, overlaid on Google Earth Pro terrain imagery, clarified whether an aircraft could have flown out of radar range before it was lost at sea.
The most revealing advancement so far has been overlaying non-geometric data on geospatial imagery. Lie cited Boeing’s investigation of one flight crew’s effective response to a 777 engine power rollback event. “We colored the flight path based on the difference in oil temperature between the two engines,” Lie said. This data visualization helped investigators to identify significant moments during the sequence of ice accretion on the fuel-oil heat exchanger, engine rollback, descent, time interval for flight crew intervention by retarding throttles, melting of ice and the resumption of normal flight.
To analyze an approach-phase accident, investigators included simultaneous overlays of non-geometric parameters on both terrain and a scaled approach chart. The non-geometric method supports the use of layers representing data for thrust asymmetry as a yawing force, N1 rotor speed in relation to throttle position, control wheel displacement, radar altimeter data, autothrust disconnect location, airspeed, stick shaker activation, location of a stall recovery attempt, satellite weather images, lightning strike data, and text from transcribed cockpit voice recorders and air traffic controller voices.
One product in the set included vectors — perpendicular lines representing magnitude and direction of forces — along the flight path. The purpose was to show over time the thrust asymmetry as yellow lines protruding from the flight path, and the contemporaneous control wheel inputs as red lines. For example, a large control wheel displacement was clearly visible during straight and level flight.
“When both parameters are combined, the relationship between thrust asymmetry and control wheel becomes instantly evident — as does the combined effects … just prior to the loss of control,” Lie said. “Google Earth allows this 3-D scene to be rotated and viewed from different angles.”
Once the autothrottle had been disconnected in this situation, the asymmetric thrust condition became a function of the flight crew’s selection within the available thrust range. “At low thrust settings, there was almost no asymmetry; at very high thrust settings, there was no asymmetry,” he said. “During level off, there was a lot of thrust asymmetry. … It appears that they never recognized it.”
In summary, CAASD’s Reed said, “Visualization gives us the ability … to link patterns to our expert knowledge. The insights — the various clues to improve safety — would be much more difficult to obtain if we did not have these tools.”
- Mackinlay, Jock D.; Hanrahan, Pat; Stolte, Chris. “Show Me: Automatic Presentation for Visual Analysis.” IEEE [Institute of Electrical and Electronics Engineers] Transactions on Visualizations and Computer Graphics. Volume 13 (November 2007). www.tableausoftware.com/whitepapers/show-me-automatic-presentation-visual-analysis.