com.bigdata.btree.raba.codec

Class TokenizeKeysGenerator

java.lang.Object

com.bigdata.btree.raba.codec.TokenizeKeysGenerator

All Implemented Interfaces:

IRabaGenerator

public class TokenizeKeysGenerator
extends Object
implements IRabaGenerator

Tokenize an input file, collect the set of distinct keywords, and encode those keywords as unsigned byte[]s.

Version:

$Id$

Author:

Bryan Thompson

Field Summary

Fields

Modifier and Type	Field and Description
static String	charset The encoding used to serialize the term (the value of each tuple).
protected static org.apache.log4j.Logger	log

Constructor Summary

Constructors

Constructor and Description
TokenizeKeysGenerator(String fileOrResource)

Method Summary

Methods

Modifier and Type	Method and Description
byte[][]	generateKeys(int size) Generate B+Tree keys (no null, no duplicates, data are fully ordered based on unsigned byte[] comparison).
byte[][]	generateValues(int size) Generate B+Tree values (may include nulls).
boolean	isKeysGenerator() Yes.
boolean	isValuesGenerator() Yes.
Set<String>	tokenize(String fileOrResource, Reader r)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.apache.log4j.Logger log

charset

public static final transient String charset

The encoding used to serialize the term (the value of each tuple).

See Also:

Constant Field Values

Constructor Detail

TokenizeKeysGenerator

public TokenizeKeysGenerator(String fileOrResource)

Method Detail

generateKeys

public byte[][] generateKeys(int size)

Description copied from interface: IRabaGenerator

Generate B+Tree keys (no null, no duplicates, data are fully ordered based on unsigned byte[] comparison).

Specified by:

generateKeys in interface IRabaGenerator

Parameters:

size - The #of keys.

Returns:

The keys.

generateValues

public byte[][] generateValues(int size)

Description copied from interface: IRabaGenerator

Generate B+Tree values (may include nulls).

Specified by:

generateValues in interface IRabaGenerator

Parameters:

size - The #of values.

Returns:

The values.

isKeysGenerator

public boolean isKeysGenerator()

Yes.

Specified by:

isKeysGenerator in interface IRabaGenerator

isValuesGenerator

public boolean isValuesGenerator()

Yes.

Specified by:

isValuesGenerator in interface IRabaGenerator

tokenize

public Set<String> tokenize(String fileOrResource,
                   Reader r)
                     throws Exception

Throws:

Exception