Lab 6: Accuracy Assessment

Background and Goal

In this lab, we learned how to evaluate the accuracy of classification maps. This knowledge is important because it is a necessary post-processing exercise that highlights the strengths and weaknesses of a classification map. The results of accuracy assessments provide an easily read quality assessment and rating of the map as a whole, as well as each class within the map.

Methods

Accuracy assessments were completed for both the unsupervised classification model and the supervised classification model that were discussed in Labs 4 and 5 respectively. The process is identical for both assessments. The first step in this process is to generate random ground reference testing samples. These samples are either collected through field work with a GPS unit, or with a high resolution image. In this lab, we used a high resolution image as our reference for these testing samples. The testing samples were generated using ERDAS’ Accuracy Assessment tool. The classification map was to be assessed, using the high resolution image as a reference (this means that points would appear on the reference map, with the classification being applied to the classification map). Normally the total amount of testing samples should be equal to 50 for each class (meaning a minimum of 250 for 5 classes), but for the sake of time we only used 125 samples for our 5 classes. We used a stratified random distribution parameter to ensure a quality sample. Going further, we applied the use of minimum points, making sure that each class had 15 samples at the very least (this makes sure classes of smaller area are still factored into the accuracy assessment). Figure 1 displays the random testing samples on the high resolution image. Once the random testing samples were generated, each sample was located and a classification was applied. This classification would then be compared to the classification applied in the classification map in the accuracy assessment, creating a matrix to display an accuracy assessment report. 

Figure 1: All of the test samples that were used in one of the Accuracy Assessments. The points were randomly generated to reduce bias, and there are 125 points total.

Results

Figure 2 displays the accuracy assessment report of the unsupervised classification map, while Figure 3 displays the accuracy assessment report of the supervised classification map. The matrix displays the overall accuracy, Kappa statistic, producer’s accuracy, and user’s accuracy. The rows in the matrix list the classes that were actually classified on the classification map, while the columns display the classes that were selected with the reference points. The matrix thus displays producer’s accuracy (also known as omission error) and user’s accuracy (also known as commission error). Overall accuracy is the report of the overall proportion of correctly classified pixels in the image based on the reference sample. The Kappa statistic is a measure of the difference between observed agreement between two maps and the agreement that might be attained by chance (it calculates to what degree chance has a role in an accurate classification). A value below 0.4 means there is poor agreement while a value above 0.8 means there is a strong agreement. The producer’s accuracy is the accuracy for a given class that examines how many pixels on the map are classified correctly. The user’s accuracy is the accuracy for a given class that examines how many of the pixels on the classified image are actually what they say they are. Overall, the unsupervised classification map is more accurate, though neither maps are accurate enough to actually be used with accuracy.

Figure 2: The accuracy assessment report for the unsupervised classification map. The map has a low overall accuracy, a fairly low Kappa Statistic, and low producer and user accuracy (though it is slightly more accurate then the supervised classification map).

Figure 3: The accuracy assessment report for the supervised classification map. The map has a very low overall accuracy, and a much lower Kappa Statistic. Except for the case with Water, the producer's and user's accuracy are both awfully low. This map was not accurate whatsoever.

Sources

Earth Resources Observation and Science Center. (2000). [Landsat image used to create a classification map]. United States Geological Survey. Provided by Cyril Wilson.

National Agriculture Imagery Program. (2005). [High resolution image used for reference in accuracy assessment]. United States Department of Agriculture. Provided by Cyril Wilson.