RE: Re: Testing the model
This attempt at verification for the first two forecasts shown (2004-2005 and 2005-2006) is actually somewhat similar to what is done. The way we verify such forecasts will be shown in more detail in a later blog. A difference between what you describe in your comment and how it is actually done, is that the U.S. is divided into over 200 squares that are approximately equal in area. These are the squares implied in the maps in Figure 1 above (note the horizontal/vertical boundaries dividing the regions having differing categories of forecast or observation). At each of these many squares, the forecast versus the observation is noted. If they match, it is called a hit, and if they do not match, it is a miss. Then the number of hits is found. But how many hits indicates a forecast with any accuracy, or "skill"? Well, just by dumb luck, 1/3 of the time a hit will occur by chance alone, since there are 3 categories of outcome, and over the long term each of these has a 33.3% chance of occurring. So, one-third of the total number of squares is expected to be hits just by luck. If this number of hits occurs, the score for the forecast is said to be zero (no skill). If all of the squares are hits (as it would be in a dream world), the score is 100%. In between, the score is caclulated accordingly. So, for example, if the number of hits is exactly halfway between the number expected by luck and 100% hits, the score would be 50%. If the number of hits is less than the number expected by luck, the score would be a negative number. One complicating factor in all of this: How do we score the squares where "equal chances" was the forecast (the white areas on the forecast maps)? One way is to simply not count them in the calculation. Another way is to count them, and give them one-third of a hit. If a forecast map is completely covered with only the "equal chances" forecast, then if it is scored in the first manner, we cannot compute a score. If it is scored the second say, the number of hits would be 1/3 of the total number of squares, and the number of hits expected by chance is also exactly the same number, so that the score would be zero. Note that in the above scoring system, the probabilities given for the designated forecast category are not taken into account. There are some skill scores that do consider the probabilities. Again, there will be a blog on this subject of verification coming in the not too distant future. This reply is just to whet your appetite for that blog!