NDFD Verification Publications

• Huntemann, T. L., D. E. Rudack, D. P Ruth, 2015: Forty Years of NWS Forecasts: Past Performance and Future Advances. Harry R. Glahn Symposium, Phoenix, AZ, Amer. Meteor. Soc., 449.

  Abstract:
  The Meteorological Development Laboratory (MDL) of the National Weather Service (NWS) has issued model output statistics (MOS) guidance forecasts for nearly four decades. For many years, MOS guidance was generated for observing stations and formatted in text bulletins while official NWS forecasts for stations and zones were created by forecasters typing text. Today, the National Digital Forecast Database (NDFD) contains official NWS forecasts produced by forecasters at local Weather Forecast Offices (WFOs) and national centers on a fine-resolution grid. MDL also issues gridded MOS guidance in support of NDFD. Recently, MDL has applied the MOS approach to the European Centre for Medium-Range Weather Forecasts (ECMWF) model to generate additional station-based guidance.

  The NWS and MDL routinely evaluate official forecasts at stations and compare the skill of the human forecast to the guidance for the same weather element. Improvements in NWS public weather forecasts and in statistically post-processed numerical weather prediction (NWP) can be traced by the verification of the weather element guidance.

  In this paper, we examine the skill at stations of four decades of official NWS maximum temperature, minimum temperature, and 12-h probability of precipitation forecasts compared to MOS guidance for forecast periods out to approximately 60 hours in advance. The skill of the forecast and guidance has increased significantly since the late 1960s and early 1970s. We also investigate the performance of the last two years of NDFD, Global Forecast System-based MOS (GFS MOS), and ECMWF MOS forecasts for forecast periods out to seven days in advance.
  
  

• Ruth, D. P., B. Glahn, V. Dagostaro, K. Gilbert, 2009: The performance of MOS in the digital age. Wea. Forecasting, 24, 504-519.

  Abstract:
  Model output statistics (MOS) guidance forecasts have been produced for over three decades. Until recently, MOS guidance was prepared for observing stations and formatted in text bulletins while official National Weather Service (NWS) forecasts for stations and zones were prepared by forecasters typing text. The flagship product of today’s NWS is the National Digital Forecast Database (NDFD). In support of NDFD, MOS is now also produced on grids.

  This paper compares MOS and gridded MOS (GMOS) to the forecaster-produced NDFD at approximately 1200 station locations in the conterminous United States. Results indicate that GMOS should provide good guidance for preparing the NDFD. In those areas of the country where station observations well represent the grid, GMOS features accuracy comparable to that of NDFD. In areas of complex terrain not well represented by station observations, GMOS appears similar to NDFD in its depiction. A new score is introduced to measure convergence from a long-range forecast to the final short-range forecast. This shows good GMOS forecast continuity when compared to station MOS and NDFD.
  
  

• Dagostaro, V. J., W. A. Shaffer, M. J. Schenk, J. L. Gorline, 2004: A prototype verification system for examining NDFD forecasts. 17th Conf. on Probability and Statistics in the Atmospheric Sciences, Seattle, WA, Amer. Meteor. Soc., 2.7.

  Abstract:
  Verification of the National Digital Forecast Database (NDFD) forecasts has been a high priority issue with NDFD, especially as a part of the Operational Readiness Demonstration (ORD). A prototype system for verifying maximum and minimum temperatures and probability of precipitation (PoP) was developed and used during the ORD. The verification consisted of a standard "point-to-point"comparison of forecasts taken from the NDFD grid to observations, as well as a preliminary "gridded" verification, where forecasts for the entire NDFD grid were compared to an analysis. For the prototype, RUC 20 km grid analyses of surface temperature were chosen for the analysis fields for verifying NDFD surface temperatures. Three types of displays were developed to show ORD verification statistics. For the CONUS and each NWS CONUS region, maps of mean absolute error (MAE) or bias were generated for maximum and minimum temperature and maps of Brier score were generated for verifying probability of precipitation. Plots of statistical measures (MAE and bias for maximum and minimum temperature and Brier score for PoP) vs. forecast projection constitute the second type of display generated. Finally, statistics are presented in text format, grouped by CONUS, NWS Regions, and by individual observation site. These data are stored in GIS format to enable users to import them into local GIS's. We anticipate that this prototype verification will be the basis for a monthly verification of NDFD forecasts, expanded to include other variables (3-hourly surface temperatures, dewpoint temperatures, and winds).
  
  

• Dallavalle, J. P., and V. J. Dagostaro, 2004: Objective interpretation of numerical prediction model output – A perspective based on verification of temperature and precipitation guidance. Symp. 50th Anniversary of Operational Numerical Weather Prediction, College Park, MD, NCEP, 5.8.

  Abstract:
  In this paper, we examine the skill of the official NWS max/min and PoP forecasts compared to the guidance for forecast periods out to approximately 60 hours in advance. After a brief description of the evolution of the NWS verification program for temperature and PoP, we describe changes made to the guidance system during the period of record, summarize the methodology used to verify the guidance and local forecasts, and show a number of verification time series beginning in the late 1960’s. As expected, the skill of the NWS max/min temperature and PoP forecasts, as well as that of the guidance, has increased significantly since the late 1960’s and early 1970’s. While the increase in skill of the day 1 forecast has leveled off during the last decade, the skill of the day 2 forecast provided to the public is now comparable to that of the day 1 forecast provided 20 years ago.