In a comprehensive study by S&P recording defaults of sovereigns between 1975-2009, all 14 defaults from 12 different sovereigns had a rating of BB- or lower, thus all of them were regarded as non-investment grade in the year preceding its default. The accuracy and performance of sovereign credit ratings is confirmed and justified to some extent by the fact that most of the recorded defaults are considered as junk status by CRAs and are rated as non-investment grade. (IMF 2012)This graph also demonstrates an important problem highlighting the difficulty of predicting and evaluating rare events. Because the investment-grade sovereigns rarely default, default data for investment-grade categories is very scarce and it is difficult to obtain sufficient data to be able to make statistically significant conclusions.
The accuracy, performance and discriminatory power of CRA’s sovereign ratings can be formally analysed and certified with cumulative accuracy profile (CAP) curves and accuracy ratios (ARs).
The CAP curve is obtained by plotting the cumulative proportion of sovereign borrowers by their sovereign credit rating against the cumulative proportion of sovereign defaults by their sovereign credit rating. (IMF 2012)
The “perfect performance” CAP curve (represented by the red line in the figure) should have an almost vertical slope since “perfect performance” by CRAs would mean that most sovereign defaults occurred at the sovereigns that had the lowest ratings.
With the “random” CAP curve (represented by the green line in the figure below) it is assumed that sovereign defaults occur uniformly over the sovereign rating distribution attributed by the CRAs. This essentially means that the CRAs fails to differentiate between different sovereign credit risks. (John Kiff 2012)
If the actual CAP curve is close to the “perfect performance” CAP curve, it means that the CRA’s ratings are accurate, and has good discriminatory power visa versa.
The Accuracy Ratio (AR) is the ratio between:
The area between the actual CAP curve and the “random” CAP curve; and
The area between the “perfect performance” CAP curve and the “random” CAP curve
Thus, if the Accuracy Ratio is close to 1, the CRA’s sovereign credit ratings were accurate and it shows near perfect discriminatory power of the CRA and is the Accuracy Ratio is close to 0 it indicates that CRA’s sovereign credit ratings were not representative of the credit risk underlying the sovereigns and it implies very little discriminatory power by the CRA. (John Kiff 2012)
In the study it is also noted that the discriminatory power decreases as the time horizons increase.