Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## cc.mallet.pipe
Class SimpleTokenizer

    java.lang.Object
      cc.mallet.pipe.Pipe
          cc.mallet.pipe.SimpleTokenizer

All Implemented Interfaces:
  AlphabetCarrying, java.io.Serializable

    public class SimpleTokenizer

extends Pipe

A simple unicode tokenizer that accepts sequences of letters as tokens.

See Also:
  Serialized Form

Field Summary
`protected java.util.HashSet<java.lang.String>` | `stoplist`
`static int` | `USE_DEFAULT_ENGLISH_STOPLIST`
`static int` | `USE_EMPTY_STOPLIST`

Constructor Summary
`SimpleTokenizer(java.io.File stopfile)`
`SimpleTokenizer(java.util.HashSet<java.lang.String> stoplist)`
`SimpleTokenizer(int languageFlag)`

Method Summary
`SimpleTokenizer` | `deepClone()`
`Instance` | `pipe(Instance instance)` Really this should be 'protected', but isn't for historical reasons.
`void` | `stop(java.lang.String word)`

Methods inherited from class cc.mallet.pipe.Pipe
`alphabetsMatch, getAlphabet, getAlphabets, getDataAlphabet, getInstanceId, getTargetAlphabet, instanceFrom, instancesFrom, instancesFrom, isDataAlphabetSet, isTargetProcessing, newIteratorFrom, preceedingPipeDataAlphabetNotification, preceedingPipeTargetAlphabetNotification, precondition, readResolve, setDataAlphabet, setOrCheckDataAlphabet, setOrCheckTargetAlphabet, setTargetAlphabet, setTargetProcessing`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### USE_EMPTY_STOPLIST

    public static final int USE_EMPTY_STOPLIST

See Also:
  Constant Field Values

### USE_DEFAULT_ENGLISH_STOPLIST

    public static final int USE_DEFAULT_ENGLISH_STOPLIST

See Also:
  Constant Field Values

### stoplist

    protected java.util.HashSet<java.lang.String> stoplist

Constructor Detail

### SimpleTokenizer

    public SimpleTokenizer(int languageFlag)

### SimpleTokenizer

    public SimpleTokenizer(java.io.File stopfile)

### SimpleTokenizer

    public SimpleTokenizer(java.util.HashSet<java.lang.String> stoplist)

Method Detail

### deepClone

    public SimpleTokenizer deepClone()

### stop

    public void stop(java.lang.String word)

### pipe

    public Instance pipe(Instance instance)

  Description copied from class: `Pipe`
  Really this should be 'protected', but isn't for historical reasons.

  Overrides:
    `pipe` in class `Pipe`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD