Probabilistic interpretation of AUC

The area under the ROC (receiver operating characteristic) curve, or AUC, is a popular and robust metric for machine learning classification. However, one issue with its use is that it is not as easy to explain to a client what it is and why you are using it. Over at 0-fold Cross-Validation, Alexej Gossmann has a useful post on how to think of the AUC probabilistically:

In other words, if the classification algorithm distinguishes “positive” and “negative” examples (e.g., disease status), then AUC is the probability of correct ranking of a random “positive”-“negative” pair.

From this, you can derive a simple formula to compute AUC on a finite sample as an alternative to the trapezoidal rule:

Among all “positive”-“negative” pairs in the dataset compute the proportion of those which are ranked correctly by the evaluated classification algorithm.

And as a reminder of why the AUC has many advantages compared to other ‘single number” performance measures:

• It has a really nice probabilistic meaning!
• Independence of the decision threshold.
• Invariance to prior class probabilities or class prevalence in the data.
• Can choose/change a decision threshold based on cost-benefit analysis after model training.
• Extensively used in machine learning, and in medical research.