org.apache.ctakes.core.nlp.tokenizer

Class Tokenizer

java.lang.Object

org.apache.ctakes.core.nlp.tokenizer.Tokenizer

public class Tokenizer
extends Object

A class used to break natural text into tokens. The token markup is external to the text and is not embedded like XML. Character offset location is used to identify the boundaries of a token.

Author:

Mayo Clinic

Constructor Summary

Constructors

Constructor and Description
Tokenizer() Constructor
Tokenizer(Map<String,Integer> hyphMap, int freqCutoff) Constructor

Method Summary

Modifier and Type	Method and Description
boolean	isAlphabetLetter(char c)
static boolean	isNumber(String tokenText) Applies number rules to the given token.
List<Token>	tokenize(String text) Tokenizes a string of text and outputs a list of Token objects.
List<Token>	tokenizeAndSort(String text) Tokenizes a string of text and outputs a list of Token objects in sorted order.
static void	validateHyphenMap(Map<String,Integer> hyphMap) Validate the structure of the hyphen map.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer()

Constructor

Tokenizer

public Tokenizer(Map<String,Integer> hyphMap,
                 int freqCutoff)

Constructor

Parameters:

hyphMap - Map where key=hyphenated string (lower cased) value=freq Integer

freqCutoff - frequency cutoff

Method Detail

validateHyphenMap

public static void validateHyphenMap(Map<String,Integer> hyphMap)
                              throws Exception

Validate the structure of the hyphen map.

Throws:

Exception

tokenizeAndSort

public List<Token> tokenizeAndSort(String text)
                            throws Exception

Tokenizes a string of text and outputs a list of Token objects in sorted order.

Parameters:

text - The text to tokenize.

Returns:

A list of Token objects sorted by the order they appear in the text.

Throws:

Exception - Thrown if an error occurs while tokenizing.

tokenize

public List<Token> tokenize(String text)
                     throws Exception

Tokenizes a string of text and outputs a list of Token objects. The list is not guaranteed to be sorted.

Parameters:

text - The text to tokenize.

Returns:

A list of Token objects.

Throws:

Exception

isAlphabetLetter

public boolean isAlphabetLetter(char c)

isNumber

public static boolean isNumber(String tokenText)

Applies number rules to the given token.

Returns:

True if the token is a number, false otherwise.