## Evaluation

### Adding a new performance measure

If you'd like to add a new measure get mldata-utils open up mleval/evaluation.py in your editor and define a function, that gets two arguments, output and truth, write some nice short docuemntation e.g.:

def accuracy(out, lab): """ Computes Accuracy. Expects labels to be +/-1 and predictions sign(output) """ return numpy.mean(numpy.sign(out) == lab)

and add a line to 'register' that measure to the pm dictionary. Here keys are human readable names values are tuples of

(functioname, application domain ('Classification', 'Regression', ...), description), e.g. pm['Accuracy'] = (accuracy, 'Classification', accuracy.__doc__)

### Currently supported measures

#### Regression

- Mean absolute error
- Root mean square error

#### Binary Classification

- "Area under Precision Recall Curve" Computes the area under the precision recall curve. Expects labels to be +/-1 and real-valued predictions
- "Weighted Relative Accuracy" Computes Weighted Relative Accuracy. Expects labels to be +/-1 and predictions sign(output)
- "True Positive Rate" Computes True Positive Rate. Expects labels to be +/-1 and predictions sign(output)
- "Cross Correlation Coefficient" Computes the cross correlation coefficient. Expects labels to be +/-1 and predictions sign(output)
- "False Positive Rate" Computes False Positive Rate. Expects labels to be +/-1 and predictions sign(output)
- "Balanced Error" Computes the Balanced Error. Expects labels to be +/-1 and predictions sign(output)
- "True Negative Rate" Computes True Negative Rate. Expects labels to be +/-1 and predictions sign(output)
- "Area under ROC Curve" Computes the area under the ROC curve. Expects labels to be +/-1 and real-valued predictions
- "Precision Recall Curve" Computes the precision recall curve. Expects labels to be +/-1 and real-valued predictions
- "F1 Score" Computes the F1 score. Expects labels to be +/-1 and predictions sign(output)
- "Error" Computes Error Rate. Expects labels to be +/-1 and predictions sign(output)
- "ROC Curve" Computes the ROC curve. Expects labels to be +/-1 and real-valued predictions
- "False Negative Rate" Computes False Negative Rate. Expects labels to be +/-1 and predictions sign(output)
- "Accuracy" Computes Accuracy. Expects labels to be +/-1 and predictions sign(output)

#### Multiclass Classification

Analogues exist for most of the binary classification measures above

#### Contents

#### Acknowledgements

This project is supported by PASCAL (Pattern Analysis, Statistical Modelling and Computational Learning)

http://www.pascal-network.org/.