Why Good Forecasts Go Bad

Accurate weather forecasts are crucial to the aviation industry. The greatest concern is, of course, the safety of flight crews, passengers and the aircraft they are in. The economic implications are also enormous. Knowing weather conditions at the departure and arrival locations and along the flight route is critical to an industry in which, literally, time is money. From the meteorological point of view, the needs of the aviation community have often driven advances in weather forecasting for everybody.

Aviation interests are mainly concerned with forecasts for the next day or so, the realm of the terminal aerodrome forecasts (TAFs). In terms of standard forecasting, this is considered a short-range forecast. Also, there are more weather elements of concern to pilots than those in the forecasts produced for the general population. A standard public forecast includes sky condition, precipitation, temperatures, and wind. TAFs include wind and precipitation forecasts, but also visibility and specific cloud and/or ceiling heights, and they have much greater detail.

Overall, aviation weather forecasts are very accurate. The most recent statistics for the United States show that critical instrument flight rules (IFR) conditions are correctly forecast 64 percent of the time, with a false alarm rate of 36 percent.

But the old meteorologist’s adage is: “The forecasts you miss are the ones they remember.”

To understand why some forecasts are incorrect, we must examine how weather forecasts are made.

To forecast tomorrow’s weather, we must know the state of the atmosphere now. The better we can depict the current weather, the more accurate the forecast will be. Surface observations of temperature, humidity, winds, pressure, current weather, etc., are taken at thousands of stations around the world. Some observations are done by automated sensors, others are done by people. Surface aviation observations — meteorological terminal aviation routine weather reports (METARs) — are taken at least every hour and more frequently — in the form of special reports (SPECIs) — if dictated by adverse or changing weather conditions. The official meteorological surface observations are taken every three hours at designated government stations. Upper-level observations are done twice a day from far fewer sites. Balloon-borne instrument packs, or radiosondes, send back information about temperature, humidity and pressure at different heights in the atmosphere. In addition, tracking of the radiosondes provides data on wind direction and speed at various levels. Data from weather radar have been available since the 1950s. The first weather satellite was launched in 1960. Today we have many weather satellites providing an abundance of data, especially at upper levels and in remote regions of the world.

The forecast tools or methods used by meteorologists vary with the time period being forecast. With forecasts going out to six hours in the future, the time period critical for many aviation purposes, meteorologists rely heavily on current observational data derived from official site observations, satellites and, when precipitation is involved, radar.

If there is little weather system movement, a simple persistence forecast may suffice. If a terminal is socked in with fog, most likely that location will have fog in the next hour, too. Often, when weather systems are moving, continuity forecasts indicate when clouds and/or precipitation will move into or out of an area. Clouds are tracked by satellite to determine speed and direction of movement.

Weather radar can provide the same information for precipitation. A simple continuity forecast just assumes the clouds or precipitation will continue to move at the same speed and in the same direction. The most difficult situation for forecasting clouds and/or precipitation via the continuity method is one in which clouds or precipitation form at a location rather than being advected — that is, being transported by the wind. Although not the norm, this does happen, particularly where there are orographic effects, or air flow disturbed by topographic features.

For forecasts beyond six hours, meteorologists rely heavily on numerical and statistical models. Numerical weather prediction has been viable since 1960. Prior to that, weather forecasting was more “seat of the pants,” with meteorologists collecting as much data about the current situation as possible and making forecasts using their own experience, knowledge and intuition. Meteorologists theorized that the atmosphere must obey the basic laws of physics. By stating these laws as mathematical equations, real observations from the atmosphere could be used to generate a mathematical model of the atmosphere. By using time derivatives, the equations could be solved for future times, thus giving weather forecasts.

However, the inability to do all the calculations required, especially in a timely manner, made numerical weather prediction just a dream until the development of computers beginning in the 1940s. These ultimate number crunchers were exactly what were needed to make the dream a reality. By 1960, some computer-generated forecasts became superior to anything that could be done by hand. In time, numerical weather prediction would become the norm, with the meteorologist’s role relegated to “tweaking” the computer guidance to allow for variations that could not be incorporated in the models.

Even though the numerical models improved with time, they were still limited in what weather elements they could actually forecast. They were very good at producing a picture of what various layers of the atmosphere would look like in the future, but they weren’t designed to predict the parameters, especially at the surface, that both the general public and the aviation community needed — elements like temperature, chances of precipitation and visibility. Realizing these model shortcomings, meteorologists turned to statistics.

By using regression analysis — establishing a relationship between variables to allow the prediction of one variable based on changes in the other — meteorologists could now relate elements not predicted by the models to ones that were.

For example, numerical models do not predict the chance of rain or snow, the probability of precipitation (PoP). But the models do forecast the amounts of moisture at the standard cloud level of 10,000 ft. One can then statistically relate the amount of moisture at this level to the occurrence in the past of precipitation at the surface. In that way, computer-generated forecasts of cloud level moisture could be used to forecast the PoP. Statistical relationships can be made with any variable as long as there is a physical cause and effect. In other words, computers could now forecast anything. These forecasts were called MOS — model output statistics — developed in the late 1970s and a staple of today’s weather forecasts.

In simple terms, MOS is just a memory system. The computer “remembers” past weather situations. It is an analog forecaster — it relates a situation it sees now to situations it has seen in the past. It assumes a similar situation will produce similar sensible weather. Interestingly, many “intuitive” meteorologists do the same thing in making a forecast. They may not even realize that they are subconsciously remembering past analog situations while making the current forecast.

But, like any statistical forecasting scheme, MOS has its limitations. The forecast is only as good as the relationship between the predicted element and the predictor. There are no perfect relationships in meteorology, no correlation coefficients of 1.

For example, a particular moisture value at 10,000 ft doesn’t always correspond with the same PoP. There are a range of values possible, with the distribution of possible variables usually being normal — that is, following the classic bell-shaped curve. In our example, the forecast PoP produced by MOS is the most likely outcome, but there are no guarantees. Like any statistical technique, the more original data you have to make the relationship, the better the forecast.

There are a variety of potential error sources for MOS forecasts. If the numerical model that creates the basic forecast is incorrect, then the MOS it produces will also be inaccurate. Unusual or rare weather events will not be forecast well since there are very few analog situations to establish the statistical relationships. In reality, the relationship between two variables can change depending on the time of year. The statistical equations used are modified several times a year, but not often enough to catch all the changes.

Overall, there are a few basic things that can be said about weather forecast accuracy. In general, short-range forecasts are more than 90 percent accurate. It is easier to forecast good weather than bad weather. Fortunately, for most locations, fair weather — visual flight rules conditions — is more common. High pressure areas which usually bring fair weather tend to be larger and are handled well by the numerical models. Situations which bring clouds and precipitation tend to be dominated by smaller-scale weather features which are difficult for the computer models to predict.

There are a number of other reasons why weather forecasts can go wrong. As stated before, to forecast the future weather, we must know the current state of the atmosphere. Anything we miss can come up and bite us later. Only North America and Europe have enough weather-reporting stations to give an accurate depiction of current weather. Much of the less developed regions of the world and the vast ocean areas are underreported.

One of the original problems with numerical weather forecasting remains today: the time constraint. Forecasts still have to be produced in a timely fashion. Compromises have to be made in the numerical models so they can be run quickly by the computer. Whether it’s in the size of the region covered, the span of the time steps used in the calculations or changes in the basic physics of the model itself, any and all of these can influence forecast accuracy.

Some of the problems with weather forecasts stem from the fact that, frankly, we don’t fully understand everything that goes on in the atmosphere. There is a wide variety of factors that influence the weather. Taken individually, most of these are straightforward cause and effect. But, in the real world, a wide variety of forces are in play at the same time. It is difficult and sometimes impossible to judge which factors will be dominant, or which factors will cancel each other out. Added to this are the myriad interactions possible. This is not like performing experiments in a lab under controlled conditions. The atmosphere is our lab, and anything goes.

And for weather forecasting, as well as most other things, we have to allow for the implications of unforeseen events. This is captured in the “chaos theory.” In the early 1960s, pioneering meteorologist Edward Lorenz applied the chaos theory to weather. Poetically, he described how a butterfly flapping its wings could set up air currents that, under the right conditions, could alter the weather many miles away. And, as we all know, you can’t forecast butterflies.

Aviation forecasts are inherently more difficult to prepare than standard public forecasts. They have to be much more precise. In terms of time periods, standard forecasts for the public work in 12-hour increments with general references to events. “Increasing cloudiness during the day with a chance of rain by the afternoon” would be a typical forecast. Aviation forecasts often need to be broken down by the hour when conditions warrant. And pilots need to know about specific cloud heights and visibilities, elements which are, by nature, very difficult to forecast. Also, public forecasts cover a wide area. TAFs are for particular sites.

Dan Miller and Jonathan Lamb, two of my former students, have years of experience as meteorologists, much of it as aviation forecasters, with the U.S. National Weather Service. They break down the standard aviation forecast into three time periods. For the first six hours, persistence and continuity are the main forecast tools.

Regarding the six-hour forecast, Lamb said, “Sometimes the best forecast tool is to put the [computer’s] distance/speed tracker on the leading edge of clouds or an area of rain.” Miller said, “We concentrate most of our effort in the short term when it matters the most and when confidence can be higher.”

For forecasts of weather 12 to 36 hours in the future, numerical guidance is routinely used. Here, the forecaster’s local knowledge and skill can improve upon the raw computer-generated forecast. However, both Miller and Lamb noted that the intermediate time frame, 6 to 12 hours, can be challenging to forecast. It’s too far out to rely on persistence or continuity, and the standard mathematical models aren’t designed for this either.

In weather and forecasting, time and size are related. Near-term weather conditions are dominated by smaller-scale weather systems. These are not handled well by the standard models. The models were designed for larger-scale systems, those measured in hundreds of miles. But Lamb says help may be on the way for forecasters in the United States. After a number of years of trial and refinement, the high resolution rapid refresh (HRRR) model will become fully operational later this year. With an interior grid of 3 km (2 mi) length and a one hour update cycle, the HRRR should provide numerical guidance that has been lacking for the intermediate time frame so critical for aviation.

The way forecast material is presented is also changing. Rather than standard text, more of the forecast information is now displayed graphically. This trend will likely continue.

Lamb and Miller say that one of their greatest challenges in aviation forecasting is dealing with summer thunderstorms. “It was common for us to predominately [forecast] TSRA (thunderstorms with rain), or include it in ‘tempo’ groups [forecasts of temporary or possible events] for long periods of time in the late afternoon and evening in the warm season when we were expecting scattered diurnal pulse thunderstorms,” Miller said. “It turned out we were way over-forecasting the occurrence of TSRA at the airports.”

At Lamb’s office, the aviation industry made its feelings clear. “We’ve heard over and over again that we should not blanket TSRA in TAFs unless confidence is high, because it requires fuel for alternates and gets very expensive for the airlines.” Lamb said that, now, “we don’t include thunderstorms in the TAF until [1200 universal coordinated time (UTC)] at the earliest and usually with the [1800 UTC] issuance when we see where stuff is developing and where the cumulus field is better developed.” But on the downside, he said, “Since thunderstorms have such a high impact on aviation users, it stinks not being able to give much of a heads-up.” It’s similar at Miller’s office: “Now, we mention [thunderstorms] and [cumulonimbus] sparingly, especially in the later time periods. We include it when we have high confidence of it actually affecting the TAF sites, typically in the near term.”

Both Lamb and Miller agree that local knowledge and experience are critical attributes of a good aviation forecaster. As for the problems, Miller sums it up this way: “Aviation forecasting tends to be quite difficult and tricky, and can be quite frustrating at times. There is still much room for improvement.” Or, as Lamb put it: “Just this morning, I was pulling my hair out when doing the aviation forecast.”

Edward Brotak, Ph.D., retired in 2007 after 25 years as a professor and program director in the Department of Atmospheric Sciences at the University of North Carolina, Asheville.

Download the PDF of this article.