cc.mallet.fst.confidence
Class ConfidenceEvaluator

java.lang.Object
  extended by cc.mallet.fst.confidence.ConfidenceEvaluator

public class ConfidenceEvaluator
extends java.lang.Object


Nested Class Summary
static class ConfidenceEvaluator.EntityConfidence
          a simple class to store a confidence score and whether or not this labeling is correct
 
Constructor Summary
ConfidenceEvaluator(InstanceWithConfidence[] instances, boolean sorted)
           
ConfidenceEvaluator(PipedInstanceWithConfidence[] instances, boolean sorted)
           
ConfidenceEvaluator(Segment[] segments, boolean sorted)
           
ConfidenceEvaluator(java.util.Vector confidences)
           
ConfidenceEvaluator(java.util.Vector confidences, int nBins)
           
 
Method Summary
 double accuracyAtCoverage(double cov)
           
 java.lang.String accuracyCoverageValuesToString()
           
 java.lang.String accuracyRecallValuesToString(int totalTrue)
           
 double correlation()
          Calculate pearson's R for the corellation between confidence and correct, where 1 = correct and -1 = incorrect
 double[] getAccuracyCoverageValues()
          get accuracy at coverage for each bin of values
 double[][] getAccuracyRecallValues(int totalTrue)
          get accuracy at recall for each bin of values
 double getAverageAccuracy()
           
 double getAverageCorrectConfidence()
          Average confidence score for the incorrect entities
 double getAverageIncorrectConfidence()
          Average confidence score for the incorrect entities
 double getAveragePrecision()
          IR Average precision measure.
 double getConfidenceMean()
           
 double getConfidenceStandardDeviation()
          Standard deviation of confidence scores
 double getConfidenceSum()
           
 double getWorstAveragePrecision()
          For comparison, rank segments as badly as possible (all "incorrect" before "correct").
 int numCorrect()
           
 int numCorrectAtCoverage(double cov)
           
 double pointBiserialCorrelation()
          Correlation when one variable (X) is binary: r = (bar(x1) - bar(x0)) * sqrt(p(1-p)) / sx , where bar(x1) = mean of X when Y is 1 bar(x0) = mean of X when Y is 0 sx = standard deviation of X p = proportion of values where Y=1
 int size()
           
 java.lang.String toString()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

ConfidenceEvaluator

public ConfidenceEvaluator(java.util.Vector confidences,
                           int nBins)

ConfidenceEvaluator

public ConfidenceEvaluator(java.util.Vector confidences)

ConfidenceEvaluator

public ConfidenceEvaluator(Segment[] segments,
                           boolean sorted)

ConfidenceEvaluator

public ConfidenceEvaluator(InstanceWithConfidence[] instances,
                           boolean sorted)

ConfidenceEvaluator

public ConfidenceEvaluator(PipedInstanceWithConfidence[] instances,
                           boolean sorted)
Method Detail

pointBiserialCorrelation

public double pointBiserialCorrelation()
Correlation when one variable (X) is binary: r = (bar(x1) - bar(x0)) * sqrt(p(1-p)) / sx , where bar(x1) = mean of X when Y is 1 bar(x0) = mean of X when Y is 0 sx = standard deviation of X p = proportion of values where Y=1


getAveragePrecision

public double getAveragePrecision()
IR Average precision measure. Analogous to ranking _correct_ documents by confidence score.


getWorstAveragePrecision

public double getWorstAveragePrecision()
For comparison, rank segments as badly as possible (all "incorrect" before "correct").


getConfidenceSum

public double getConfidenceSum()

getConfidenceMean

public double getConfidenceMean()

getConfidenceStandardDeviation

public double getConfidenceStandardDeviation()
Standard deviation of confidence scores


correlation

public double correlation()
Calculate pearson's R for the corellation between confidence and correct, where 1 = correct and -1 = incorrect


getAccuracyCoverageValues

public double[] getAccuracyCoverageValues()
get accuracy at coverage for each bin of values


accuracyCoverageValuesToString

public java.lang.String accuracyCoverageValuesToString()

getAccuracyRecallValues

public double[][] getAccuracyRecallValues(int totalTrue)
get accuracy at recall for each bin of values

Parameters:
totalTrue - total number of true Segments
Returns:
2-d array where values[i][0] is coverage and values[i][1] is accuracy at position i.

accuracyRecallValuesToString

public java.lang.String accuracyRecallValuesToString(int totalTrue)

accuracyAtCoverage

public double accuracyAtCoverage(double cov)

numCorrectAtCoverage

public int numCorrectAtCoverage(double cov)

getAverageAccuracy

public double getAverageAccuracy()

numCorrect

public int numCorrect()

getAverageIncorrectConfidence

public double getAverageIncorrectConfidence()
Average confidence score for the incorrect entities


getAverageCorrectConfidence

public double getAverageCorrectConfidence()
Average confidence score for the incorrect entities


size

public int size()

toString

public java.lang.String toString()
Overrides:
toString in class java.lang.Object