## NDFD Verification General Help Page

### What is this website?

This site contains some basic verification results for selected weather elements (e.g., temperature, dew point, wind speed, etc.) from the National Digital Forecast Database (NDFD).Contents: How to Use This Site, Definitions of Weather Elements, Definitions of Statistics

### How to Use This Site

- You should see a webpage with two sections: a left section and a main section.

The left section contains several categories of menu options from which the user can select. As prompted by the "Welcome" message on the right, first select a month. The projection menu appears for maps, but not plots, since each map is unique to a projection, while the plots contain all projections.

The main section displays the data and statistical products (currently, maps and plots) as selected by the menus on the left.

- The left section contains menus you can use to select the verification product in which you are interested. You will need to first select a
**month**from the top menu. Notice that as you make selections, the options available in the other menus can change.

The next category is called**Element**. The weather elements for which verification is available are displayed.

The next category is called**Ref Time**. Currently, results based on 00Z and 12Z NDFD forecasts are available.

Next, select**Product**desired. Current options are*plot*,*map*, and*text*.

Plot files are graphs of statistics for all available projections. Map files are thematic maps of the statistics. The text files contain the statistics in ASCII format, suitable for spreadsheet use The set of statistics, and thus maps/plots/files, will change according to the weather element.

Lastly, for maps, which have individual results for projection,a**Projection**menu is provided.

**Please note:**

All statistics for a given selection are presented in the main section. If Dew Point Map is selected, then maps of the MAE and Bias are given.

- Point-based verification results are available for these weather elements:
- Max Temp
- Min Temp
- PoP 12
- Dew Point
- Sfc Temp
- Wind Speed
- Wind Dir
- Sky Cover
- Wind Gust
- Relative Humidity

Please note that the statistics available are dependent on the weather element chosen. The statistics used, particular to a certain weather element, are MAE (mean absolute error), bias (forecast minus the observation), Heidke Skill Score, Fraction Correct, Brier Score, and Relative Frequency. These are defined in the Definitions section of this page.

To print a particular**image**, right-click on the image, then choose Print from your browser's File menu.

### Definitions of Weather Elements

: Daytime maximum temperature is defined as the highest temperature observed from 7 AM to 7 PM LST. Note that maximum temperature is associated with the 24-h MOS projection from 0000 UTC, 36-h MOS projection from 1200 UTC, 48-h MOS projection from 0000 UTC, and so on.*Max Temp*: Nighttime minimum temperature is defined as the lowest temperature observed from 7 PM to 7 AM LST. Note that minimum temperature is associated with the 24-h MOS projection from 1200 UTC, 36-h projection from 0000 UTC, 48-h projection from 1200 UTC, and so on.*Min Temp*: Probability of Precipitation (PoP greater than or equal to 0.01 in) for 12-h periods, specifically, 0000-1200 UTC and 1200-0000 UTC.*PoP*: The ambient temperature observed at 2 meters above ground level.*Sfc Temp*: The dew point temperature observed at 2 meters above ground level.*Dew Point*: The wind speed observed at 10 meters above ground level.*Wind Speed*: The wind direction observed at 10 meters above ground level.*Wind Dir*: The amount of sky covered by opaque clouds for the indicated hour.*Sky Cover*: The maximum wind speed observed at 10 meters above ground level. Note that if an actual wind gust is not observed at a given location, the wind gust assumes the value of the sustained wind speed.*Wind Gust*: The ratio, expressed as a percent, of the amount of atmospheric moisture present relative to the amount that would be present if the air were saturated. The verifying observation is computed from the ambient temperature and dew point 2 meters above ground level.*Relative Humidity*

### Definitions of Statistics

*Mean Absolute Error (MAE)*

The Mean Absolute Error (MAE) is a measure of forecast accuracy. A small value indicates a better score, a perfect score is zero. MAE is defined as:

( |forecast - observation| ) / N (from i = 1 to N)

where N = the total number of observations

On our page, gridded verification scores include MAE scores. Point verification scores include MAE scores for the following weather elements: Max Temp, Min Temp, Sfc Temp, Dew Point, Relative Humidity, Wind Speed, Wind Gust, and Wind Dir.

Please note that for Wind Speed and Direction, the MAE computed at stations is limited to only those cases where any forecast and/or observation was greater than or equal to 8 knots, thus eliminating the light and variable winds. No such limitations were placed on the data sample for gridded verification of wind speed (all cases were used).

Similarly for Wind Gust, the MAE computed at stations is limited to only those cases where any forecast and/or observation was greater than or equal to 14 knots.

*Bias*

The mean algebraic error (bias) indicates whether a forecast is too high or too low in predicting a certain parameter. For example, a positively biased temperature forecast indicates that forecasts were, on average, too warm. Similarly, a negatively biased temperature forecast indicates that forecasts were, on average, too cool. Using another example, a positively biased wind speed forecast indicates that forecasts were, on average, predicting wind speeds that were too high. A bias of zero is possible if a forecaster's over-forecasting and under-forecasting cancel each other or if the forecast is perfect. Bias should be looked at in conjunction with MAE to determine forecasting error. The bias is defined as:

(forecast - observation) / N (from i = 1 to N) where N = the total number of observations

On our page, gridded verification scores include bias scores. Point verification scores include bias scores for the following weather elements: Max Temp, Min Temp, Sfc Temp, Dew Point, Relative Humidity, Wind Speed, and Wind Gust.

Please note that for Wind Speed, the bias computed at stations is limited to only those cases where any forecast and/or observation was greater than or equal to 8 knots, thus eliminating the light and variable winds. No such limitations were placed on the data sample for gridded verification (all cases were used).

Similarly for Wind Gust, the bias computed at stations is limited to only those cases where any forecast and/or observation was greater than or equal to 14 knots.

*Brier Score*

The Brier Score is the mean square error applied to probability forecasts. A common form of this score, called the half-Brier score, used on these verification web pages, is defined as follows:

(forecast - observation) ^{2}/ N(from i = 1 to N) where N = the total number of observations

- The probability forecasts used in these computations have 1% precision.
- The observation is set equal to one if precipitation greater than or equal to 0.01 inches occurred, or to zero if no precipitation (or a trace) occurred.
- The score (half-Brier score) has a range of 0 to 1, with lower scores indicating better forecasts. Example: If the PoP forecast is 100% in 10 cases, and it rains in all 10 cases, then the Brier score is 0. Similarly, if the PoP forecast is 50% in 10 cases, and it rains in only 5 of them, then the Brier score is 0.25.
- Generally,
**the rarer the event, the better the Brier score**, regardless of the forecast skill. Therefore, care must be used when comparing the Brier scores for different locations or seasons. - On our page, this score is used only for point verification for the weather element PoP.

*Heidke Skill Score (HSS)*

The Heidke Skill Score (HSS) is a measure of skill in forecasts. It is defined as follows:

(NC - E) / (T - E)

where NC equals the number of correct forecasts (in other words, the number of times the forecast and the observations match), T equals the total number of forecasts, and E equals the number of forecasts expected to verify based on chance.

This can be calculated using a contingency table:

**Heidke Skill Score Table**Observed Category Forecast Category 1 2 ... m Total 1 X _{11}X _{12}... X _{1m}X _{1p}2 X _{21}X _{22}... X _{2m}X _{2p}... ... ... ... ... ... m X _{m1}X _{m2}... X _{mm}X _{mp}Total X _{p1}X _{p2}... X _{pm}X _{pp}

where m is the number of categories, the element X_{ij}indicates the number of times the forecast was in the jth category and the observation was in the ith category. The row and column totals are shown by the subscript (and category) p.

NC = X _{ii}(from i = 1 to m)

T = X _{pp}

E = (X _{ip}X_{pi}) / T(from i = 1 to m)

A negative HSS indicates that a forecast is worse than a randomly based/generated forecast. On our page, we indicate a perfect score in two ways. First, for a sample whose forecasts and observations fall into more than one category (i.e., the matched forecast and observation totals occupy more than one cell of the contingency table), the computed HSS=1.0. Second, for a sample whose forecast/observation total occupies only one cell of the contingency table (there was no reason to forecast anything but the most commonly observed condition), we set the HSS equal to 9997. This difference helps to highlight the stations that achieved a perfect score under more difficult forecast conditions.

On our page, Heidke Skill Scores are used for point verification results for the following weather elements: Wind Speed and Wind Dir.

*Forecast Convergence Score (FCS)*

The Ruth-Glahn Forecast Convergence Score (FCS) is an index that measures the number of large swings in forecasts made over a series of forecast cycles for forecasts valid at the same time. Assume there are N forecasts made over a number of days. With each subsequent forecast cycle, the forecast projection decreases until the valid time of the forecast is reached. The FCS is defined as follows:

(T _{1}+ T_{2})FCS = ----------------- (T _{3}+ T_{4})

where T_{1}= the number of forecasts that changed insignificantly (less than the threshold) from the previous forecast F_{i-1}OR moved closer to the next forecast F_{i+1}, where i varies from 2 to N. When i=N, the observation is used as the next forecast F_{i+1}.

|F _{N}- F_{1}|T _{2}= -------------------threshold

T _{3}= N-1

|F _{i}- F_{i-1}|(from i = 2 to N) T _{4}= ----------------------threshold

The T_{1}and T_{3}terms account for the actual and possible number of swings, respectively.

The T_{2}and T_{4}terms account for the magnitude of the swings.

A significance threshold specifies the minimum change necessary to count as a swing.

The FCS ranges from 0.0 (many large swings with no convergence toward the observation) to 1.0 (no large swings with the forecasts converging toward the observation.)

A "case" is an unbroken series of forecasts for the entire 7 days, all valid at the same time, ending with the observation at the valid time. Thus, the number of cases is generally smaller than the number of cases for other statistics. Any gap in the forecasts or observations causes the case to be eliminated from the sample.*Fraction Correct*

The Fraction Correct is a measure of forecast accuracy. It is defined as follows:

NC / T

where NC equals the number of correct forecasts (in other words, the number of times the forecast and the observations match), and T equals the total number of forecasts.

This can be calculated using a contingency table:

**Contingency Table**Observed Category Forecast Category 1 2 ... m Total 1 X _{11}X _{12}... X _{1m}X _{1p}2 X _{21}X _{22}... X _{2m}X _{2p}... ... ... ... ... ... m X _{m1}X _{m2}... X _{mm}X _{mp}Total X _{p1}X _{p2}... X _{pm}X _{pp}

where m is the number of categories, the element X_{ij}indicates the number of times the forecast was in the jth category and the observation was in the ith category. The row and column totals are shown by the subscript (and category) p.

NC = X _{ii}(from i = 1 to m)

T = X _{pp}

On our page, Fraction Correct is used for point verification results for the following weather elements: Sky Cover.

*Relative Frequency*Relative Frequency (RF) is a measure of the number of occurrences a forecast falls within a certain bounded error. It is defined as follows:

RF _{i}= n_{i}/ N(from i = 1 to N)

where ( n_{i}) equals the count of direction errors in a certain category and N equals the total number of forecasts/observations.

For example, if there are 10 forecasts with 8 of the forecasts having an error less than or equal to 5 degrees, the relative frequency of forecasts with error of 5 degrees or less is 0.8.

On our page, Relative Frequency is used for point verification results for the Wind Dir weather element. A plot of relative frequency is provided for wind direction errors less than 30 degrees for wind speeds of 8 knots or greater only.