cc.mallet.fst.confidence
Class ConfidenceEvaluator

java.lang.Object
  cc.mallet.fst.confidence.ConfidenceEvaluator

public class ConfidenceEvaluator
extends java.lang.Object

Nested Class Summary

static class

ConfidenceEvaluator.EntityConfidence a simple class to store a confidence score and whether or not this labeling is correct

Constructor Summary

ConfidenceEvaluator(InstanceWithConfidence[] instances, boolean sorted)

ConfidenceEvaluator(PipedInstanceWithConfidence[] instances, boolean sorted)

ConfidenceEvaluator(Segment[] segments, boolean sorted)

ConfidenceEvaluator(java.util.Vector confidences)

ConfidenceEvaluator(java.util.Vector confidences, int nBins)

Method Summary

double	accuracyAtCoverage(double cov)
java.lang.String	accuracyCoverageValuesToString()
java.lang.String	accuracyRecallValuesToString(int totalTrue)
double	correlation() Calculate pearson's R for the corellation between confidence and correct, where 1 = correct and -1 = incorrect
double[]	getAccuracyCoverageValues() get accuracy at coverage for each bin of values
double[][]	getAccuracyRecallValues(int totalTrue) get accuracy at recall for each bin of values
double	getAverageAccuracy()
double	getAverageCorrectConfidence() Average confidence score for the incorrect entities
double	getAverageIncorrectConfidence() Average confidence score for the incorrect entities
double	getAveragePrecision() IR Average precision measure.
double	getConfidenceMean()
double	getConfidenceStandardDeviation() Standard deviation of confidence scores
double	getConfidenceSum()
double	getWorstAveragePrecision() For comparison, rank segments as badly as possible (all "incorrect" before "correct").
int	numCorrect()
int	numCorrectAtCoverage(double cov)
double	pointBiserialCorrelation() Correlation when one variable (X) is binary: r = (bar(x1) - bar(x0)) * sqrt(p(1-p)) / sx , where bar(x1) = mean of X when Y is 1 bar(x0) = mean of X when Y is 0 sx = standard deviation of X p = proportion of values where Y=1
int	size()
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ConfidenceEvaluator

public ConfidenceEvaluator(java.util.Vector confidences,
                           int nBins)

ConfidenceEvaluator

public ConfidenceEvaluator(java.util.Vector confidences)

ConfidenceEvaluator

public ConfidenceEvaluator(Segment[] segments,
                           boolean sorted)

ConfidenceEvaluator

public ConfidenceEvaluator(InstanceWithConfidence[] instances,
                           boolean sorted)

ConfidenceEvaluator

public ConfidenceEvaluator(PipedInstanceWithConfidence[] instances,
                           boolean sorted)

Method Detail

pointBiserialCorrelation

public double pointBiserialCorrelation()

Correlation when one variable (X) is binary: r = (bar(x1) - bar(x0)) * sqrt(p(1-p)) / sx , where bar(x1) = mean of X when Y is 1 bar(x0) = mean of X when Y is 0 sx = standard deviation of X p = proportion of values where Y=1

getAveragePrecision

public double getAveragePrecision()

IR Average precision measure. Analogous to ranking _correct_ documents by confidence score.

getWorstAveragePrecision

public double getWorstAveragePrecision()

For comparison, rank segments as badly as possible (all "incorrect" before "correct").

getConfidenceSum

public double getConfidenceSum()

getConfidenceMean

public double getConfidenceMean()

getConfidenceStandardDeviation

public double getConfidenceStandardDeviation()

Standard deviation of confidence scores

correlation

public double correlation()

Calculate pearson's R for the corellation between confidence and correct, where 1 = correct and -1 = incorrect

getAccuracyCoverageValues

public double[] getAccuracyCoverageValues()

get accuracy at coverage for each bin of values

accuracyCoverageValuesToString

public java.lang.String accuracyCoverageValuesToString()

getAccuracyRecallValues

public double[][] getAccuracyRecallValues(int totalTrue)

get accuracy at recall for each bin of values

Parameters:

totalTrue - total number of true Segments

Returns:

2-d array where values[i][0] is coverage and values[i][1] is accuracy at position i.

accuracyRecallValuesToString

public java.lang.String accuracyRecallValuesToString(int totalTrue)

accuracyAtCoverage

public double accuracyAtCoverage(double cov)

numCorrectAtCoverage

public int numCorrectAtCoverage(double cov)

getAverageAccuracy

public double getAverageAccuracy()

numCorrect

public int numCorrect()

getAverageIncorrectConfidence

public double getAverageIncorrectConfidence()

Average confidence score for the incorrect entities

getAverageCorrectConfidence

public double getAverageCorrectConfidence()

Average confidence score for the incorrect entities

size

public int size()

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object