Evaluation

Adding a new performance measure

If you'd like to add a new measure get mldata-utils open up mleval/evaluation.py in your editor and define a function, that gets two arguments, output and truth, write some nice short docuemntation e.g.:

def accuracy(out, lab):
   """
     Computes Accuracy.
     Expects labels to be +/-1 and predictions sign(output)
   """
   return numpy.mean(numpy.sign(out) == lab)

and add a line to 'register' that measure to the pm dictionary. Here keys are human readable names values are tuples of

(functioname, application domain ('Classification', 'Regression', ...),
description), e.g.

pm['Accuracy'] = (accuracy, 'Classification', accuracy.__doc__)

Currently supported measures

Regression

  • Mean absolute error
  • Root mean square error

Binary Classification

  • "Area under Precision Recall Curve" Computes the area under the precision recall curve. Expects labels to be +/-1 and real-valued predictions
  • "Weighted Relative Accuracy" Computes Weighted Relative Accuracy. Expects labels to be +/-1 and predictions sign(output)
  • "True Positive Rate" Computes True Positive Rate. Expects labels to be +/-1 and predictions sign(output)
  • "Cross Correlation Coefficient" Computes the cross correlation coefficient. Expects labels to be +/-1 and predictions sign(output)
  • "False Positive Rate" Computes False Positive Rate. Expects labels to be +/-1 and predictions sign(output)
  • "Balanced Error" Computes the Balanced Error. Expects labels to be +/-1 and predictions sign(output)
  • "True Negative Rate" Computes True Negative Rate. Expects labels to be +/-1 and predictions sign(output)
  • "Area under ROC Curve" Computes the area under the ROC curve. Expects labels to be +/-1 and real-valued predictions
  • "Precision Recall Curve" Computes the precision recall curve. Expects labels to be +/-1 and real-valued predictions
  • "F1 Score" Computes the F1 score. Expects labels to be +/-1 and predictions sign(output)
  • "Error" Computes Error Rate. Expects labels to be +/-1 and predictions sign(output)
  • "ROC Curve" Computes the ROC curve. Expects labels to be +/-1 and real-valued predictions
  • "False Negative Rate" Computes False Negative Rate. Expects labels to be +/-1 and predictions sign(output)
  • "Accuracy" Computes Accuracy. Expects labels to be +/-1 and predictions sign(output)

Multiclass Classification

Analogues exist for most of the binary classification measures above

Contents

Acknowledgements

This project is supported by PASCAL (Pattern Analysis, Statistical Modelling and Computational Learning)
PASCAL Logo
http://www.pascal-network.org/.