com.bigdata.rdf.internal.impl

Class AbstractIV<V extends BigdataValue,T>

java.lang.Object

com.bigdata.rdf.internal.impl.AbstractIV<V,T>

Type Parameters:

V - The generic type for the RDF Value implementation.

T - The generic type for the inline value.

All Implemented Interfaces:

IV<V,T>, IVCache<V,T>, Serializable, Comparable<IV>, org.openrdf.model.Value

Direct Known Subclasses:

AbstractInlineIV, AbstractNonInlineIV

public abstract class AbstractIV<V extends BigdataValue,T>
extends Object
implements IV<V,T>

Abstract base class for the inline representation of an RDF Value (the representation which is encoded in to the keys of the statement indices). This class is responsible for combining the VTE and the DTE together into the flags byte used as a common prefix for all keys formed from RDF Values regardless of whether they are based on an assigned term identifier or the inlining of the RDF Value.

Literals which are projected onto primitive data types with a natural order (int, float, double, long, etc.) are in total order within that segment of the statement index and such segments are partitioned into non-overlapping key ranges. Thus key range scans may be used for where the primitive data type allows it while filtering on the value must be used where the data type does not have a natural ordering. Unicode values do not have a "natural" ordering within the statement indices as they are modeled by a reversible compression rather than a collation sort key. Therefore, when a high level query is constrained such that a variable is known to be of a given data type you can use IV aware operations. Otherwise, the IVs must be materialized and operations performed on BigdataValues instead.

Binary record format

We currently have 14 built-in (or intrinsic) data types (see DTE). Each of those types has a natural order which we can encode and decode from the B+Tree key. In general, there is a relatively limited set of interesting intrinsic codings, which is important since we will dedicate just 4 bits for to code the natural order of the value space, which is just only 16 distinctions. Given that we have 14 intrinsic data types, that leaves room for just two more. One of those bits provides for Unicode data (see DTE.XSDString without a collation order). The other bit provides extensibility in the framework itself as described below (see DTE.Extension).

The header byte contains various bit flags which are laid out as follows:

 [valueType]    : 2 bits
 [inline]       : 1 bit
 [extension]   : 1 bit
 [dataTypeCode] : 4 bits
 

valueType

These 2 bits distinguish between URIs, Literals, Blank Nodes, and statement identifiers (SIDs). These bits are up front and therefore partition the key space first by the RDF Value type. See VTE which governs these bits.

inline

This bit indicates whether the value is inline or represented by a term identifier in the key. This bit is set based on how a given triple store or quad store instance is configured. However, because the bit is present in the flags, we know how to decode the key without reference to this configuration metadata.

extension

This bit is ignored (and should be zero) unless the RDF Value is a Literal with a data type URI which is being inlined. For data type literals, this bit is set if the actual data type is not one of those which we handle intrinsically but is one of those which has been registered (by the application) as an "extended" data type projected onto one of the intrinsic data types. Thus, this bit partitions the key space into the intrinsic data types and the extended data types.
When true, this bit signals that information about the actual RDF Value data type will follow (see below). When false, the datatype URI is directly recoverable (for a data type Literal) from the dataTypeCode.

dataTypeCode

These 4 bits indicate the intrinsic data type for the inline value and are ignored (and should be zero) unless a data type Literal is being inlined. These bits partition the key space. However, since extension bit comes first this will not interleave inline values for intrinsic and extended data types having the same dataTypeCode.
Note: The dataTypeCode 0xf (DTE#Extension) is reserved for extending the set of intrinsic data types. When the code is 0xf the next byte must be considered as well to determine the actual intrinsic data type code.

 ---------- byte boundary (IFF DTE == DTE.Extension) ----------
 

If the DTE value was DTE.Extension was, then the next byte(s) encode the DTEExtension (extended intrinsic datatype aka primitive datatype).

 ---------- byte boundary ----------
 

If extension was true, then the next byte(s) encode information about the source data type URI (its IV) and the key space will be partitioned based on the extended data type URI.

 ---------- byte boundary ----------
 

The unsigned byte[] representation of the value in the value space for one of the intrinsic types. The length of this byte[] may be directly determined from the [dataTypeCode] for most data types. However, for xsd:integer and xsd:decimal, the length is part of the representation.

 ---------- byte boundary and end of the record ----------
 

Extensibility

There are three core use cases for extensibility:

projections

A projection takes an application specific data type and maps it onto one of the intrinsic data types (int, float, double, etc). Projections provide an extensible mechanism which allows an application to benefit from inline representation of RDF Values and allows the query optimizer to chose key-range scans for application defined data types if they can be projected onto intrinsic data types. For example, if you define an application specific data type foo:milliseconds representing milliseconds since the epoch, then the value space of that data type can be projected onto an xsd:long.

enumerations

An enumeration is an application specific data type having a specific set of values. Those values are then projected onto an intrinsic data type such as byte (256 distinctions) or short (64k distinctions). Enumerations make it possible to inline application specific data types while benefiting from XSD validation of those RDF Values. When an enumeration is registered, the order in which the members of the enumeration are given may optionally specify the natural order of that enumeration. The natural order is imposed by projecting the first member of the enumeration one ZERO, the second member onto ONE, etc. An enumeration with a natural order will be sorted based on that defined order and query optimizations may perform key-range scans informed by that natural order.
Enumerations may be used in cases where you might otherwise use short character codes. For example, an enumeration could be defined for the two character abbreviations for the 50 US States. That enumeration could be mapped onto a single byte.

custom indices

The best example here is spatial data, which requires literals which represent points, rectangles, circles, arcs, clouds, etc to be inserted into special spatial indices. Queries must be aware of spatial data and must be rewritten to run against the appropriate spatial indices.
Another use case would be carrying specialized indices for bioinformatics or genomics data.

Note: Both projected and enumerated extensible data types MAY map many RDF Values onto the same internal value but each internal value MUST map onto a single RDF Value (materialization must be deterministic). This can be seen as normalization imposed by the database.

Author:

Bryan Thompson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected byte	flags Bit flags indicating the kind of RDF Value (VTE), whether the RDF Value is inline, whether this is an extension datatype, and the natural order and binary representation of the inline value (#DTE).

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractIV(byte flags) Constructor used when decoding since you already have the flags.
protected	AbstractIV(VTE vte, boolean inline, boolean extension, DTE dte) The RDF Value type (URI, BNode, Literal or Statement) and the data type are combined and stored in a single byte together with whether the RDF value has been inlined (an inline bit) and whether the RDF Value is an extended data type (the extension bit).

Method Summary

Methods

Modifier and Type	Method and Description
abstract int	_compareTo(IV o) Compare two IVs having the same intrinsic datatype.
int	compareTo(IV o) Imposes an ordering of IVs based on their natural sort ordering in the index as unsigned byte[]s.
IKeyBuilder	encode(IKeyBuilder keyBuilder) Encode the IV as an unsigned byte[].
abstract boolean	equals(Object o) Return true iff the two IVs are the same point in the same value space.
byte	flags() The value of the flags representing the VTE and the DTE.
DTE	getDTE() Return the DTE for this IV.
static DTE	getDTE(byte flags) Helper method decodes a flags byte as found in a statement index key to an DTE.
DTEExtension	getDTEX() IFF IV.getDTE() returns DTE.Extension then this method will report the DTEExtension value that specifies the intrinsic datatype for this IV.
static VTE	getInternalValueTypeEnum(byte flags) Helper method decodes a flags byte as found in a statement index key to an VTE.
V	getValue() Return a pre-materialized RDF BigdataValue which has been cached on this IV by a previous invocation of IVCache.asValue(LexiconRelation).
protected V	getValueCache() Return the cached BigdataValue or -null if it is not cached.
VTE	getVTE() Return the VTE for the IV
static VTE	getVTE(byte flags) Return the VTE encoding in a flags byte.
abstract int	hashCode() Return a hash code based on the value of the point in the value space.
boolean	hasValue() Returns true if the RDF BigdataValue has been pre-materialized and cached on this IV.
boolean	isBigNumeric() true for xsd:integer and xsd:decimal.
boolean	isBNode() Return true iff this is an RDF BlankNode.
boolean	isExtension() true iff the flags byte is followed by an IV which defines how the subsequent value (represented according to the DTE) will be interpreted.
static boolean	isExtension(byte flags) Return true if the flags byte has its extension bit set.
boolean	isFixedNumeric() This is !isBigNumeric() and is true for any of the fixed length numeric data types (xsd:byte, xsd:unsignedByte, xsd:short, xsd:unsignedShort, xsd:int, xsd:unsignedInt, xsd:long, xsd:unsignedLong, xsd:float, xsd:double).
boolean	isFloatingPointNumeric() true for xsd:float, xsd:double, and xsd:decimal
boolean	isInline() true iff the RDF value is directly represented inline.
static boolean	isInline(byte flags) Return true if the flags byte has its inline bit set.
boolean	isLiteral() Return true iff this is an RDF Literal.
boolean	isNullIV() true iff the IV represents a null IV reference.
boolean	isNumeric() true for any of the numeric data types (xsd:byte, xsd:unsignedByte, xsd:short, xsd:unsignedShort, xsd:int, xsd:unsignedInt, xsd:long, xsd:unsignedLong, xsd:float, xsd:double, xsd:integer, and xsd:decimal).
boolean	isResource() Return true iff this is a URI or a bnode.
boolean	isSignedNumeric() true for an signed numeric datatype ( xsd:byte, xsd:short, xsd:int, xsd:long, xsd:float, xsd:double, xsd:integer, and xsd:decimal).
boolean	isStatement() Return true iff this is a statement identifier (this feature is enabled with AbstractTripleStore.Options.STATEMENT_IDENTIFIERS).
boolean	isUnsignedNumeric() true for an unsigned numeric datatype ( xsd:unsignedByte, xsd:unsignedShort, xsd:unsignedInt, xsd:unsignedLong).
boolean	isURI() Return true iff this is an RDF URI.
boolean	isVocabulary() Return true iff this IV is a Vocabulary item.
V	setValue(V v) Set the BigdataValue on the cache.
static byte	toFlags(VTE vte, boolean inline, boolean extension, DTE dte) Return the flags byte that encodes the provided metadata about an internal value.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.rdf.internal.IV

byteLength, getInlineValue, needsMaterialization

Methods inherited from interface com.bigdata.rdf.internal.IVCache

asValue, clone

Methods inherited from interface org.openrdf.model.Value

stringValue

Field Detail

flags

protected final byte flags

Bit flags indicating the kind of RDF Value (VTE), whether the RDF Value is inline, whether this is an extension datatype, and the natural order and binary representation of the inline value (#DTE).

See Also:

VTE, DTE

Constructor Detail

AbstractIV

protected AbstractIV(VTE vte,
          boolean inline,
          boolean extension,
          DTE dte)

The RDF Value type (URI, BNode, Literal or Statement) and the data type are combined and stored in a single byte together with whether the RDF value has been inlined (an inline bit) and whether the RDF Value is an extended data type (the extension bit). The vte has 4 distinctions and is thus TWO (2) bits wide. The dte allows for up to 16 distinctions and is SIX (6) bits wide. The bits allocated to these sets of distinctions are combined within a single byte as follows:

 [vviedddd]
 

where v is a VTE bit, i is the inline bit, e is the extension bit, and d is a DTE bit.

Parameters:

vte - The RDF Value type (URI, BNode, Literal, or Statement).

inline - true iff the RDF value will be represented inline in the key. When false, the term identifier of the RDF Value will be represented inline instead of its actual value.

extension - When true, the actual RDF data type URI differs from the intrinsic data type (the DTE) but has been projected onto the natural order of the intrinsic data type.

dte - The internal datatype for the RDF value (termId, xsd:int, xsd:long, xsd:float, xsd:double, etc).

See Also:

VTE, DTE

AbstractIV

protected AbstractIV(byte flags)

Constructor used when decoding since you already have the flags.

Parameters:

flags - The flags.

Method Detail

toFlags

public static byte toFlags(VTE vte,
           boolean inline,
           boolean extension,
           DTE dte)

Return the flags byte that encodes the provided metadata about an internal value.

Parameters:

vte - The value type

inline - true iff the value is inline (no index access is required to materialize the value).

extension - true iff the value is an extension data type (a user specific data type defined whose underlying data type is specified by the DTE).

dte - The basic data type.

Returns:

The flags byte that encodes that information.

flags

public final byte flags()

Description copied from interface: IV

The value of the flags representing the VTE and the DTE. The upper TWO (2) bits code the VTE while the lower SIX (6) bits code the DTE.

Specified by:

flags in interface IV<V extends BigdataValue,T>

isInline

public static boolean isInline(byte flags)

Return true if the flags byte has its inline bit set.

Parameters:

flags - The flags byte.

isExtension

public static boolean isExtension(byte flags)

Return true if the flags byte has its extension bit set.

Parameters:

flags - The flags byte.

getVTE

public static final VTE getVTE(byte flags)

Return the VTE encoding in a flags byte.

Note: VTE.valueOf(byte) assumes that the VTE bits are in the TWO (2) LSB bits of the byte. However, the VTE bits are actually stored in the TWO (2) MSB bits of the flags byte. This method is responsible for shifting the VTE bits down before invoking VTE.valueOf(byte).

Parameters:

flags - A flags byte.

Returns:

The VTE encoded in the flags byte.

getVTE

public final VTE getVTE()

Description copied from interface: IV

Return the VTE for the IV

Specified by:

getVTE in interface IV<V extends BigdataValue,T>

getDTE

public final DTE getDTE()

Return the DTE for this IV.

Specified by:

getDTE in interface IV<V extends BigdataValue,T>

getDTEX

public DTEExtension getDTEX()

Description copied from interface: IV

IFF IV.getDTE() returns DTE.Extension then this method will report the DTEExtension value that specifies the intrinsic datatype for this IV.

Specified by:

getDTEX in interface IV<V extends BigdataValue,T>

See Also:

(Implement support for DTE extension types for URIs), ( DTEExtension for compressed timestamp)

getInternalValueTypeEnum

public static final VTE getInternalValueTypeEnum(byte flags)

Helper method decodes a flags byte as found in a statement index key to an VTE.

Parameters:

flags - The flags byte.

Returns:

The VTE

getDTE

public static DTE getDTE(byte flags)

Helper method decodes a flags byte as found in a statement index key to an DTE.

Parameters:

flags - The flags byte.

Returns:

The DTE

isLiteral

public final boolean isLiteral()

Description copied from interface: IV

Return true iff this is an RDF Literal. Note that some kinds of RDF Literals MAY be represented inline.

Specified by:

isLiteral in interface IV<V extends BigdataValue,T>

isBNode

public final boolean isBNode()

Description copied from interface: IV

Return true iff this is an RDF BlankNode.

Specified by:

isBNode in interface IV<V extends BigdataValue,T>

isURI

public final boolean isURI()

Description copied from interface: IV

Return true iff this is an RDF URI.

Specified by:

isURI in interface IV<V extends BigdataValue,T>

isStatement

public final boolean isStatement()

Description copied from interface: IV

Return true iff this is a statement identifier (this feature is enabled with AbstractTripleStore.Options.STATEMENT_IDENTIFIERS).

Specified by:

isStatement in interface IV<V extends BigdataValue,T>

isResource

public final boolean isResource()

Description copied from interface: IV

Return true iff this is a URI or a bnode.

Specified by:

isResource in interface IV<V extends BigdataValue,T>

isInline

public boolean isInline()

true iff the RDF value is directly represented inline. When an RDF Value is "inline" its value can be directly decoded from its representation in the keys of the statement indices.

This implementation based on the inline bit flag. This can be overridden in many derived classes which have compile time knowledge of whether the RDF value is inline or not.

Specified by:

isInline in interface IV<V extends BigdataValue,T>

isExtension

public boolean isExtension()

true iff the flags byte is followed by an IV which defines how the subsequent value (represented according to the DTE) will be interpreted. This is used to support projections of value spaces for data type literals onto the intrinsic types. It is also used to support indirect resolution of the namespace associated with a URI.

This implementation based on the extension bit flag. Since the extension flag is only used for specific kinds of IVs, this method can be overridden in many derived classes which have compile time knowledge of whether the value is an RDF Literal or not.

Specified by:

isExtension in interface IV<V extends BigdataValue,T>

isVocabulary

public boolean isVocabulary()

Return true iff this IV is a Vocabulary item.

The default implementation returns false.

Specified by:

isVocabulary in interface IV<V extends BigdataValue,T>

isNullIV

public boolean isNullIV()

true iff the IV represents a null IV reference. null IVs are somewhat special. They get used as wild cards for the keys in the justifications index and perhaps (?) in a few other locations.

This implementation returns false. It is overridden by TermId.

Specified by:

isNullIV in interface IV<V extends BigdataValue,T>

isNumeric

public boolean isNumeric()

Description copied from interface: IV

true for any of the numeric data types (xsd:byte, xsd:unsignedByte, xsd:short, xsd:unsignedShort, xsd:int, xsd:unsignedInt, xsd:long, xsd:unsignedLong, xsd:float, xsd:double, xsd:integer, and xsd:decimal).

Specified by:

isNumeric in interface IV<V extends BigdataValue,T>

isSignedNumeric

public final boolean isSignedNumeric()

Description copied from interface: IV

true for an signed numeric datatype ( xsd:byte, xsd:short, xsd:int, xsd:long, xsd:float, xsd:double, xsd:integer, and xsd:decimal).

Specified by:

isSignedNumeric in interface IV<V extends BigdataValue,T>

isUnsignedNumeric

public final boolean isUnsignedNumeric()

Description copied from interface: IV

true for an unsigned numeric datatype ( xsd:unsignedByte, xsd:unsignedShort, xsd:unsignedInt, xsd:unsignedLong).

Specified by:

isUnsignedNumeric in interface IV<V extends BigdataValue,T>

isFixedNumeric

public final boolean isFixedNumeric()

Description copied from interface: IV

This is !isBigNumeric() and is true for any of the fixed length numeric data types (xsd:byte, xsd:unsignedByte, xsd:short, xsd:unsignedShort, xsd:int, xsd:unsignedInt, xsd:long, xsd:unsignedLong, xsd:float, xsd:double).

Specified by:

isFixedNumeric in interface IV<V extends BigdataValue,T>

isBigNumeric

public final boolean isBigNumeric()

Description copied from interface: IV

true for xsd:integer and xsd:decimal.

Specified by:

isBigNumeric in interface IV<V extends BigdataValue,T>

isFloatingPointNumeric

public final boolean isFloatingPointNumeric()

Description copied from interface: IV

true for xsd:float, xsd:double, and xsd:decimal

Specified by:

isFloatingPointNumeric in interface IV<V extends BigdataValue,T>

hashCode

public abstract int hashCode()

Return a hash code based on the value of the point in the value space.

Note: The IV implementations typically DO NOT return hash codes that are consistent with BigdataValue#hashCode(). Therefore you MUST NOT mix IVs and BigdataValues in the keys of a map or the values of a set.

Overrides:

hashCode in class Object

equals

public abstract boolean equals(Object o)

Return true iff the two IVs are the same point in the same value space. Points in different value spaces (as identified by different datatype URIs) are NOT equal even if they have the same value in the corresponding primitive data type.

Note: The IV implementations typically DO NOT compare equals() with BigdataValues. Therefore you MUST NOT mix IVs and BigdataValues in the keys of a map or the values of a set.

Overrides:

equals in class Object

compareTo

public final int compareTo(IV o)

Imposes an ordering of IVs based on their natural sort ordering in the index as unsigned byte[]s.

Specified by:

compareTo in interface Comparable<IV>

_compareTo

public abstract int _compareTo(IV o)

Compare two IVs having the same intrinsic datatype.

encode

public IKeyBuilder encode(IKeyBuilder keyBuilder)

Description copied from interface: IV

Encode the IV as an unsigned byte[].

Specified by:

encode in interface IV<V extends BigdataValue,T>

Parameters:

keyBuilder - The object used to encode the IV.

Returns:

the key builder

getValue

public final V getValue()

Description copied from interface: IVCache

Return a pre-materialized RDF BigdataValue which has been cached on this IV by a previous invocation of IVCache.asValue(LexiconRelation).

Specified by:

getValue in interface IVCache<V extends BigdataValue,T>

Returns:

The BigdataValue.

setValue

public final V setValue(V v)

Description copied from interface: IVCache

Set the BigdataValue on the cache.

Note: This is normally invoked by IVCache.asValue(LexiconRelation) during a lexicon join cache a newly materialized Value on the IV.

Specified by:

setValue in interface IVCache<V extends BigdataValue,T>

Parameters:

v - The BigdataValue.

Returns:

The argument.

getValueCache

protected final V getValueCache()

Return the cached BigdataValue or -null if it is not cached.

hasValue

public final boolean hasValue()

Description copied from interface: IVCache

Returns true if the RDF BigdataValue has been pre-materialized and cached on this IV.

Specified by:

hasValue in interface IVCache<V extends BigdataValue,T>