org.apache.ctakes.ytex.kernel

Class ImputedFeatureEvaluatorImpl

java.lang.Object

org.apache.ctakes.ytex.kernel.ImputedFeatureEvaluatorImpl

All Implemented Interfaces:

ImputedFeatureEvaluator

public class ImputedFeatureEvaluatorImpl
extends Object
implements ImputedFeatureEvaluator

Calculate the mutual information of each concept of a corpus wrt a concept graph and classification task (label) and possibly a fold. We calculate the following:

• raw mutual information of each concept (infogain). We calculate the joint distribution of concepts (X) and document classes (Y), and compute the mutual information for each concept.
• mutual information inherited by parents (infogain-parent). For each concept in the concept graph, we merge the joint distribution of child concepts. This is done recursively.
• mutual information inherited by children from parents (infogain-child). We take the top n concepts and assign their children (entire subgraph) the mutual info of the parent.

The mutual information of each concept is stored in the feature_rank table. The related records in the feature_eval table have the following values:

• type = infogain, infogain-parent, infogain-imputed, infogain-imputed-filt
• feature_set_name = conceptSetName
• param1 = conceptGraphName

How this works in broad strokes:

• evaluateCorpus(Parameters) load instances, iterate through labels
• evaluateCorpusLabel(Parameters, ConceptGraph, InstanceData, String) load concept - set[document] map for the specified label, iterate through folds
• evaluateCorpusFold(Parameters, Map, ConceptGraph, InstanceData, String, Map, int) create raw joint distribution of each concept, compute parent joint distributions, assign children mutual info of parents
• completeJointDistroForFold(Map, Map, Set, Set, String) computes raw joint distribution of each concept
• propagateJointDistribution(Map, Parameters, String, int, ConceptGraph, Map) recursively compute parent joint distribution by merging joint distro of children.
• #storeChildConcepts(Parameters, String, int, ConceptGraph) take top ranked parent concepts, assign concepts in subtrees the mutual info of parents. Only concepts that exist in the corpus are added (depends on computing the infocontent of concepts with CorpusEvaluator)

Author:

vijay

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	ImputedFeatureEvaluatorImpl.ConceptInstanceMapExtractor fill in map of Concept Id - bin - instance ids
static class	ImputedFeatureEvaluatorImpl.JointDistribution joint distribution of concept (x) and class (y).
static class	ImputedFeatureEvaluatorImpl.Parameters We are passing around quite a few parameters.

Nested classes/interfaces inherited from interface org.apache.ctakes.ytex.kernel.ImputedFeatureEvaluator

ImputedFeatureEvaluator.MeasureType

Field Summary

Fields

Modifier and Type	Field and Description
protected ClassifierEvaluationDao	classifierEvaluationDao
protected ConceptDao	conceptDao
protected org.springframework.jdbc.core.JdbcTemplate	jdbcTemplate
protected KernelUtil	kernelUtil
protected org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate	namedParamJdbcTemplate
protected org.springframework.transaction.PlatformTransactionManager	transactionManager

Fields inherited from interface org.apache.ctakes.ytex.kernel.ImputedFeatureEvaluator

SUFFIX_IMPUTED, SUFFIX_IMPUTED_FILTERED, SUFFIX_PROP

Constructor Summary

Constructors

Constructor and Description
ImputedFeatureEvaluatorImpl()

Method Summary

Modifier and Type	Method and Description
protected static double	entropy(double[] classProbs)
protected static double	entropy(Iterable<Double> classProbs) calculate entropy from a list/array of probabilities
boolean	evaluateCorpus(ImputedFeatureEvaluatorImpl.Parameters params)
boolean	evaluateCorpus(String propFile)
ClassifierEvaluationDao	getClassifierEvaluationDao()
ConceptDao	getConceptDao()
DataSource	getDataSource(DataSource ds)
InfoContentEvaluator	getInfoContentEvaluator()
KernelUtil	getKernelUtil()
Properties	getYtexProperties()
static void	main(String[] args)
void	setClassifierEvaluationDao(ClassifierEvaluationDao classifierEvaluationDao)
void	setConceptDao(ConceptDao conceptDao)
void	setDataSource(DataSource ds)
void	setInfoContentEvaluator(InfoContentEvaluator infoContentEvaluator)
void	setKernelUtil(KernelUtil kernelUtil)
void	setYtexProperties(Properties ytexProperties)
void	storeChildConcepts(List<FeatureRank> listRawRanks, ImputedFeatureEvaluatorImpl.Parameters params, String label, int foldId, ConceptGraph cg, boolean bAll) save the children of the 'top' parent concepts.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

classifierEvaluationDao

protected ClassifierEvaluationDao classifierEvaluationDao

conceptDao

protected ConceptDao conceptDao

jdbcTemplate

protected org.springframework.jdbc.core.JdbcTemplate jdbcTemplate

kernelUtil

protected KernelUtil kernelUtil

namedParamJdbcTemplate

protected org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate namedParamJdbcTemplate

transactionManager

protected org.springframework.transaction.PlatformTransactionManager transactionManager

Constructor Detail

ImputedFeatureEvaluatorImpl

public ImputedFeatureEvaluatorImpl()

Method Detail

entropy

protected static double entropy(double[] classProbs)

entropy

protected static double entropy(Iterable<Double> classProbs)

calculate entropy from a list/array of probabilities

Parameters:

classProbs -

Returns:

main

public static void main(String[] args)
                 throws org.apache.commons.cli.ParseException,
                        IOException

Throws:

org.apache.commons.cli.ParseException

IOException

evaluateCorpus

public boolean evaluateCorpus(ImputedFeatureEvaluatorImpl.Parameters params)

Specified by:

evaluateCorpus in interface ImputedFeatureEvaluator

evaluateCorpus

public boolean evaluateCorpus(String propFile)
                       throws IOException

Specified by:

evaluateCorpus in interface ImputedFeatureEvaluator

Throws:

IOException

getClassifierEvaluationDao

public ClassifierEvaluationDao getClassifierEvaluationDao()

getConceptDao

public ConceptDao getConceptDao()

getDataSource

public DataSource getDataSource(DataSource ds)

getInfoContentEvaluator

public InfoContentEvaluator getInfoContentEvaluator()

getKernelUtil

public KernelUtil getKernelUtil()

getYtexProperties

public Properties getYtexProperties()

setClassifierEvaluationDao

public void setClassifierEvaluationDao(ClassifierEvaluationDao classifierEvaluationDao)

setConceptDao

public void setConceptDao(ConceptDao conceptDao)

setDataSource

public void setDataSource(DataSource ds)

setInfoContentEvaluator

public void setInfoContentEvaluator(InfoContentEvaluator infoContentEvaluator)

setKernelUtil

public void setKernelUtil(KernelUtil kernelUtil)

setYtexProperties

public void setYtexProperties(Properties ytexProperties)

storeChildConcepts

public void storeChildConcepts(List<FeatureRank> listRawRanks,
                               ImputedFeatureEvaluatorImpl.Parameters params,
                               String label,
                               int foldId,
                               ConceptGraph cg,
                               boolean bAll)

save the children of the 'top' parent concepts.

Parameters:

labelEval -

parentConceptTopThreshold -

parentConceptEvalThreshold -

cg -

bAll - impute to all concepts/concepts actually in corpus. if we are imputing to all concepts, filter by infocontent (this includes hypernyms of concepts in the corpus). else only impute to conrete concepts in the corpus