com.bigdata.rdf.sail.webapp.client

Class RemoteRepositoryManager

java.lang.Object

com.bigdata.rdf.sail.webapp.client.RemoteRepositoryDecls

com.bigdata.rdf.sail.webapp.client.RemoteRepositoryBase

com.bigdata.rdf.sail.webapp.client.RemoteRepositoryManager

All Implemented Interfaces:

AutoCloseable

public class RemoteRepositoryManager
extends RemoteRepositoryBase
implements AutoCloseable

A manager for connections to one or more REST API / SPARQL end points for the same bigdata service.

Author:

bryan

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RemoteRepositoryManager.ComputeMode

Field Summary

Fields

Modifier and Type	Field and Description
protected Executor	executor Thread pool for processing HTTP responses in background.
protected org.eclipse.jetty.client.HttpClient	httpClient The client used for http connections.
protected boolean	useLBS When true, the REST API methods will use the load balancer aware requestURLs.

Fields inherited from class com.bigdata.rdf.sail.webapp.client.RemoteRepositoryDecls

BASE_URI, BD_NULL_GRAPH, BINDING_PREFIX, DEFAULT_GRAPH_URI, DEFAULT_MAX_REQUEST_URL_LENGTH, DEFAULT_NAMESPACE, DEFAULT_QUERY_METHOD, FORCE_INDEX_CREATE, HTTP_HEADER_BIGDATA_MAX_QUERY_MILLIS, INCLUDE_INFERRED, MAX_QUERY_TIME_MILLIS, MAX_REQUEST_URL_LENGTH, NAMED_GRAPH_URI, OPTION_CREATE_KB_NAMESPACE, QUERY_METHOD, QUERYID, USING_GRAPH_URI, USING_NAMED_GRAPH_URI, UTF8

Constructor Summary

Constructors

Constructor and Description
RemoteRepositoryManager() Create a manager that is not aware of a specific blazegraph backend.
RemoteRepositoryManager(String serviceURL) Create a manager client for the specified serviceURL.
RemoteRepositoryManager(String serviceURL, boolean useLBS) Create a remote client for the specified serviceURL that optionally use the load balanced URLs.
RemoteRepositoryManager(String serviceURL, boolean useLBS, org.eclipse.jetty.client.HttpClient httpClient, Executor executor) Create a remote client for the specified serviceURL (core impl).
RemoteRepositoryManager(String serviceURL, org.eclipse.jetty.client.HttpClient httpClient, Executor executor) Create a remote client for the specified serviceURL.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	booleanResults(ConnectOptions opts, UUID queryId, IPreparedQueryListener listener) Parse a SPARQL result set for an ASK query.
void	cancel(UUID queryId) Cancel a query running remotely on the server.
void	close()
void	createRepository(String namespace, Properties properties) Create a new KB instance.
void	createRepository(String namespace, Properties properties, UUID uuid) Create a new KB instance.
void	deleteRepository(String namespace) Destroy a KB instance.
void	deleteRepository(String namespace, UUID uuid) Destroy a KB instance.
JettyResponseListener	doConnect(ConnectOptions opts) Connect to a SPARQL end point (GET or POST query only).
void	doDataLoader(Properties properties) Initiate the data loader for a namespace within the a NSS
boolean	dropNamespaceFromMapgraph(String namespace) Drops the given namespace from the mapgraph runtime.
protected void	finalize()
String	getBaseServiceURL() The path to the root of the web application (without the trailing "/").
Executor	getExecutor() The executor for processing http and other client operations.
String	getMapgraphStatus() Returns the current status report for mapgraph.
int	getMaxRequestURLLength() Return the maximum requestURL length before the request is converted into a POST using a application/x-www-form-urlencoded request entity.
Properties	getPreparedProperties(String namespace, Properties properties) Prepare configuration properties for a new KB instance.
Properties	getPreparedProperties(String namespace, Properties properties, UUID uuid)
String	getQueryMethod() Return the HTTP verb that will be used for a QUERY (versus an UPDATE or other mutation operations) (default ).
protected String	getRepositoryBaseURLForNamespace(String namespace) Return the base URL for a remote repository (less the /sparql path component).
org.openrdf.query.GraphQueryResult	getRepositoryDescriptions() Obtain a VoID description of the configured KBs.
org.openrdf.query.GraphQueryResult	getRepositoryDescriptions(UUID uuid) Obtain a VoID description of the configured KBs.
RemoteRepository	getRepositoryForDefaultNamespace() Obtain a flyweight RemoteRepository for the default namespace associated with the remote service.
RemoteRepository	getRepositoryForNamespace(String namespace) Obtain a flyweight RemoteRepository for a data set managed by the remote service.
RemoteRepository	getRepositoryForURL(String sparqlEndpointURL) Obtain a flyweight RemoteRepository for the data set having the specified SPARQL end point.
RemoteRepository	getRepositoryForURL(String sparqlEndpointURL, IRemoteTx tx) Obtain a flyweight RemoteRepository for the data set having the specified SPARQL end point.
Properties	getRepositoryProperties(String namespace) Return the effective configuration properties for the named data set.
Properties	getRepositoryProperties(String namespace, UUID uuid)
RemoteTransactionManager	getTransactionManager() Return the remote client for the transaction manager API.
boolean	getUseLBS() Return true iff the REST API methods will use the load balancer aware requestURLs.
boolean	mapgraphRuntimeAvailable() Checks whether the mapgraph runtime is available.
boolean	namespacePublishedToMapgraph(String namespace) Checks whether the given namespace has been published.
protected ConnectOptions	newQueryConnectOptions(String sparqlEndpointURL, UUID uuid, IRemoteTx tx) Return the ConnectOptions which will be used by default for the SPARQL end point for a QUERY or other idempotent operation.
org.eclipse.jetty.client.api.Request	newRequest(String uri, String method)
protected ConnectOptions	newUpdateConnectOptions(String sparqlEndpointURL, UUID uuid, IRemoteTx tx) Return the ConnectOptions which will be used by default for the SPARQL end point for an UPDATE or other non-idempotant operation.
void	onlineBackup(String file, boolean compress, boolean block) Initiate an online backup using the BackupServlet.
boolean	publishNamespaceToMapgraph(String namespace) Publishes the given namespace to the mapgraph runtime.
void	rebuildTextIndex(String namespace, boolean forceBuildTextIndex)
void	rebuildTextIndex(String namespace, boolean forceBuildTextIndex, UUID uuid)
void	resetMapgraphRuntime(RemoteRepositoryManager.ComputeMode computeMode) Resets the mapgraph runtime for the compute mode.
void	setMaxRequestURLLength(int newVal)
void	setQueryMethod(String method) Set the default HTTP verb for QUERY and other idempotant operations.
Collection<RunningQuery>	showQueries() List the currently running queries on the server
String	toString()
org.openrdf.query.TupleQueryResult	tupleResults(ConnectOptions opts, UUID queryId, IPreparedQueryListener listener) Extracts the solutions from a SPARQL query.

Methods inherited from class com.bigdata.rdf.sail.webapp.client.RemoteRepositoryBase

asGraph, booleanResults, checkResponseCode, contextsResults, getContextPath, mutationResults, rangeCountResults, serialize, toStrings

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

useLBS

protected final boolean useLBS

When true, the REST API methods will use the load balancer aware requestURLs. The load balancer has essentially zero cost when not using HA, so it is recommended to always specify true. When false, the REST API methods will NOT use the load balancer aware requestURLs.

See Also:

HALoadBalancer

httpClient

protected final org.eclipse.jetty.client.HttpClient httpClient

The client used for http connections.

executor

protected final Executor executor

Thread pool for processing HTTP responses in background.

Constructor Detail

RemoteRepositoryManager

public RemoteRepositoryManager()

Create a manager that is not aware of a specific blazegraph backend. This constructor is intended for patterns where a sparql end point is available but the top-level serviceURL for blazegraph is either not visible or not known:

 new RemoteRepositoryManager().getRepositoryForURL(sparqlEndpointURL)
 

The same pattern MAY be used to perform SPARQL QUERY or SPARQL UPDATE operations against non-blazegraph sparql end points.

RemoteRepositoryManager

public RemoteRepositoryManager(String serviceURL)

Create a manager client for the specified serviceURL. The serviceURL has the typical form

 http://host:port/bigdata
 

The serviceURL can be used to obtain sparql end point URLs for:

The default namespace

http://host:port/bigdata/sparql

The XYZ namespace

http://host:port/bigdata/namespace/XYZ/sparql

The serviceURL can also be used to access the multi-tenancy API and the transaction management API. See the wiki for more details.

Parameters:

serviceURL - The path to the root of the web application (without the trailing "/"). /sparql will be appended to this path to obtain the SPARQL end point for the default data set.

RemoteRepositoryManager

public RemoteRepositoryManager(String serviceURL,
                       boolean useLBS)

Create a remote client for the specified serviceURL that optionally use the load balanced URLs.

Parameters:

serviceURL - The path to the root of the web application (without the trailing "/"). /sparql will be appended to this path to obtain the SPARQL end point for the default data set.

useLBS - When true, the REST API methods will use the load balancer aware requestURLs. The load balancer has essentially zero cost when not using HA, so it is recommended to always specify true. When false, the REST API methods will NOT use the load balancer aware requestURLs.

RemoteRepositoryManager

public RemoteRepositoryManager(String serviceURL,
                       org.eclipse.jetty.client.HttpClient httpClient,
                       Executor executor)

Create a remote client for the specified serviceURL.

Parameters:

serviceURL - The path to the root of the web application (without the trailing "/"). /sparql will be appended to this path to obtain the SPARQL end point for the default data set.

httpClient - The client is managed externally and, in particular, may be shared, so we don't close it in close(). When not present (null), an HttpClient will be allocated and scoped to this RemoteRepositoryManager instance, and closed in close().

executor - An executor used to service http client requests. (optional). When not present, an Executor will be allocated and scoped to this RemoteRepositoryManager instance. TODO Should this be deprecated since it does not force the caller to choose a value for useLBS?

This version does not force the caller to decide whether or not the LBS pattern will be used. In general, it should be used if the end point is bigdata. This class is generally, but not always, used with a bigdata end point. The main exception is SPARQL Basic Federated Query. For that use case we can not assume that the end point is bigdata and thus we can not use the LBS prefix.

RemoteRepositoryManager

public RemoteRepositoryManager(String serviceURL,
                       boolean useLBS,
                       org.eclipse.jetty.client.HttpClient httpClient,
                       Executor executor)

Create a remote client for the specified serviceURL (core impl).

Parameters:

serviceURL - The path to the root of the web application (without the trailing "/"). /sparql will be appended to this path to obtain the SPARQL end point for the default data set.

useLBS - When true, the REST API methods will use the load balancer aware requestURLs. The load balancer has essentially zero cost when not using HA, so it is recommended to always specify true. When false, the REST API methods will NOT use the load balancer aware requestURLs.

httpClient - The client is managed externally and, in particular, may be shared, so we don't close it in close(). When not present (null), an HttpClient will be allocated and scoped to this RemoteRepositoryManager instance, and closed in close().

executor - An executor used to service http client requests. (optional). When not present, an Executor will be allocated and scoped to this RemoteRepositoryManager instance.

Method Detail

getTransactionManager

public RemoteTransactionManager getTransactionManager()

Return the remote client for the transaction manager API.

Since:

1.5.2

See Also:

Support read/write transactions in the REST API

getExecutor

public Executor getExecutor()

The executor for processing http and other client operations.

getBaseServiceURL

public String getBaseServiceURL()

The path to the root of the web application (without the trailing "/").

Note: This SHOULD NOT be the SPARQL end point URL. The NanoSparqlServer has a wider interface. This should be the base URL of that interface. The SPARQL end point URL for the default data set is formed by appending /sparql.

getUseLBS

public boolean getUseLBS()

Return true iff the REST API methods will use the load balancer aware requestURLs. The load balancer has essentially zero cost when not using HA, so it is recommended to always specify true. When false, the REST API methods will NOT use the load balancer aware requestURLs.

getMaxRequestURLLength

public int getMaxRequestURLLength()

Return the maximum requestURL length before the request is converted into a POST using a application/x-www-form-urlencoded request entity.

See Also:

RemoteRepository class should use application/x-www-form-urlencoded for large POST requests

setMaxRequestURLLength

public void setMaxRequestURLLength(int newVal)

getQueryMethod

public String getQueryMethod()

Return the HTTP verb that will be used for a QUERY (versus an UPDATE or other mutation operations) (default ). POST can often handle larger queries than GET due to limits at the HTTP client layer and will defeat http caching and thus provide a current view of the committed state of the SPARQL end point when the end point is a read/write database. However, GET supports HTTP caching and can scale much better when the SPARQL end point is a read-only resource or a read-mostly resource where stale reads are acceptable.

See Also:

setQueryMethod(String)

setQueryMethod

public void setQueryMethod(String method)

Set the default HTTP verb for QUERY and other idempotant operations.

Parameters:

method - The method which may be "POST" or "GET".

See Also:

getQueryMethod()

finalize

protected void finalize()
                 throws Throwable

Ensure resource is closed.

Overrides:

finalize in class Object

Throws:

Throwable

See Also:

AutoCloseable, Memory leak in CI?

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

toString

public String toString()

Overrides:

toString in class Object

getRepositoryBaseURLForNamespace

protected String getRepositoryBaseURLForNamespace(String namespace)

Return the base URL for a remote repository (less the /sparql path component).

Parameters:

namespace - The namespace.

Returns:

The base URL.

See Also:

Missing URL encoding in RemoteRepositoryManager

getRepositoryForDefaultNamespace

public RemoteRepository getRepositoryForDefaultNamespace()

Obtain a flyweight RemoteRepository for the default namespace associated with the remote service.

getRepositoryForNamespace

public RemoteRepository getRepositoryForNamespace(String namespace)

Obtain a flyweight RemoteRepository for a data set managed by the remote service.

Parameters:

namespace - The name of the data set (its bigdata namespace).

Returns:

An interface which may be used to talk to that data set.

getRepositoryForURL

public RemoteRepository getRepositoryForURL(String sparqlEndpointURL)

Obtain a flyweight RemoteRepository for the data set having the specified SPARQL end point. The load balancer will be used or not as per the parameters to the RemoteRepositoryManager constructor.

Parameters:

sparqlEndpointURL - The URL of the SPARQL end point.

Returns:

An interface which may be used to talk to that data set.

getRepositoryForURL

public RemoteRepository getRepositoryForURL(String sparqlEndpointURL,
                                   IRemoteTx tx)

Obtain a flyweight RemoteRepository for the data set having the specified SPARQL end point. The load balancer will be used or not as per the parameters to the RemoteRepositoryManager constructor.

Parameters:

sparqlEndpointURL - The URL of the SPARQL end point.

timestamp - The timestamp that will be added to all requests for the sparqlEndPoint (optional).

Returns:

An interface which may be used to talk to that data set.

getRepositoryDescriptions

public org.openrdf.query.GraphQueryResult getRepositoryDescriptions()
                                                             throws Exception

Obtain a VoID description of the configured KBs. Each KB has its own namespace and corresponds to a VoID "data set".

Note: This method uses an HTTP GET and hence can be cached by the server.

Returns:

A VoID description of the configured KBs.

Throws:

Exception

getRepositoryDescriptions

public org.openrdf.query.GraphQueryResult getRepositoryDescriptions(UUID uuid)
                                                             throws Exception

Obtain a VoID description of the configured KBs. Each KB has its own namespace and corresponds to a VoID "data set".

Note: This method uses an HTTP GET and hence can be cached by the server.

Parameters:

uuid - The UUID to be associated with this request.

Returns:

A VoID description of the configured KBs.

Throws:

Exception

createRepository

public void createRepository(String namespace,
                    Properties properties)
                      throws Exception

Create a new KB instance.

Parameters:

namespace - The namespace of the KB instance.

properties - The configuration properties for that KB instance.

Throws:

Exception

See Also:

createRepository() does not set the namespace on the Properties

createRepository

public void createRepository(String namespace,
                    Properties properties,
                    UUID uuid)
                      throws Exception

Create a new KB instance.

Parameters:

namespace - The namespace of the KB instance.

properties - The configuration properties for that KB instance.

uuid - The UUID to be associated with this request.

Throws:

Exception

See Also:

createRepository() does not set the namespace on the Properties

getPreparedProperties

public Properties getPreparedProperties(String namespace,
                               Properties properties)
                                 throws Exception

Prepare configuration properties for a new KB instance.

Parameters:

namespace - The namespace of the KB instance.

properties - The configuration properties for that KB instance.

Returns:

The effective configuration properties for that named data set.

Throws:

Exception

getPreparedProperties

public Properties getPreparedProperties(String namespace,
                               Properties properties,
                               UUID uuid)
                                 throws Exception

Throws:

Exception

rebuildTextIndex

public void rebuildTextIndex(String namespace,
                    boolean forceBuildTextIndex)
                      throws Exception

Parameters:

-

-

Throws:

Exception

rebuildTextIndex

public void rebuildTextIndex(String namespace,
                    boolean forceBuildTextIndex,
                    UUID uuid)
                      throws Exception

Throws:

Exception

deleteRepository

public void deleteRepository(String namespace)
                      throws Exception

Destroy a KB instance.

Parameters:

namespace - The namespace of the KB instance.

Throws:

Exception

deleteRepository

public void deleteRepository(String namespace,
                    UUID uuid)
                      throws Exception

Destroy a KB instance.

Parameters:

namespace - The namespace of the KB instance.

uuid - The UUID to be assigned to the request.a

Throws:

Exception

publishNamespaceToMapgraph

public boolean publishNamespaceToMapgraph(String namespace)
                                   throws Exception

Publishes the given namespace to the mapgraph runtime. If the namespace if already published, no action is performed. The return value is false in the latter case, true otherwise. If the namespace that is passed in is null, the default namespace will be used.

Returns:

true if the namespace was not yet published already, false otherwise (i.e., in case no action has been taken)

Throws:

NoGPUAccelerationAvailableException

Exception

dropNamespaceFromMapgraph

public boolean dropNamespaceFromMapgraph(String namespace)
                                  throws Exception

Drops the given namespace from the mapgraph runtime. If the namespace was not registered in the runtime, no action is performed. The return value is false in the latter case, true otherwise. If the namespace that is passed in is null, the default namespace will be used.

Returns:

true if the namespace was published before and has been dropped, false otherwise (i.e., in case no action has been taken)

Throws:

NoGPUAccelerationAvailableException

Exception

namespacePublishedToMapgraph

public boolean namespacePublishedToMapgraph(String namespace)
                                     throws Exception

Checks whether the given namespace has been published. If null is passed in, the method performs a check for the default namespace.

Returns:

true if the namespace is registered for acceleration, false otherwise

Throws:

NoGPUAccelerationAvailableException

Exception

resetMapgraphRuntime

public void resetMapgraphRuntime(RemoteRepositoryManager.ComputeMode computeMode)
                          throws Exception

Resets the mapgraph runtime for the compute mode.

Parameters:

computeMode - the desired compute mode

Throws:

Exception

getMapgraphStatus

public String getMapgraphStatus()
                         throws Exception

Returns the current status report for mapgraph.

Returns:

the status report as human-readable string

Throws:

Exception

mapgraphRuntimeAvailable

public boolean mapgraphRuntimeAvailable()
                                 throws Exception

Checks whether the mapgraph runtime is available.

Returns:

Throws:

Exception

getRepositoryProperties

public Properties getRepositoryProperties(String namespace)
                                   throws Exception

Return the effective configuration properties for the named data set.

Note: While it is possible to change some configuration options are a data set has been created, many aspects of a "data set" configuration are "baked in" when the data set is created and can not be changed. For this reason, no general purpose mechanism is being offered to change the properties for a configured data set instance.

Parameters:

namespace - The name of the data set.

Returns:

The effective configuration properties for that named data set.

Throws:

Exception

getRepositoryProperties

public Properties getRepositoryProperties(String namespace,
                                 UUID uuid)
                                   throws Exception

Throws:

Exception

onlineBackup

public void onlineBackup(String file,
                boolean compress,
                boolean block)
                  throws Exception

Initiate an online backup using the BackupServlet.

Parameters:

file - -- The name of the file for the backup. (default = "backup.jnl")

compress - -- Use compression for the snapshot (default = false)

block - -- Block on the response (default = true)

Throws:

Exception

See Also:

https://jira.blazegraph.com/browse/BLZG-1727

doDataLoader

public void doDataLoader(Properties properties)
                  throws Exception

Initiate the data loader for a namespace within the a NSS

Parameters:

properties - The properties for the DataLoader Servlet

Throws:

Exception

See Also:

BLZG-1713

doConnect

public JettyResponseListener doConnect(ConnectOptions opts)
                                throws Exception

Connect to a SPARQL end point (GET or POST query only).

Parameters:

opts - The connection options.

Returns:

The connection.

Throws:

Exception

See Also:

RemoteRepository class should use application/x-www-form-urlencoded for large POST requests

newRequest

public org.eclipse.jetty.client.api.Request newRequest(String uri,
                                              String method)

newQueryConnectOptions

protected final ConnectOptions newQueryConnectOptions(String sparqlEndpointURL,
                                    UUID uuid,
                                    IRemoteTx tx)

Return the ConnectOptions which will be used by default for the SPARQL end point for a QUERY or other idempotent operation.

Parameters:

sparqlEndpointURL - The SPARQL end point.

uuid - The unique identifier for the request that may be used to CANCEL the request.

tx - A transaction that will isolate the operation (optional).

newUpdateConnectOptions

protected final ConnectOptions newUpdateConnectOptions(String sparqlEndpointURL,
                                     UUID uuid,
                                     IRemoteTx tx)

Return the ConnectOptions which will be used by default for the SPARQL end point for an UPDATE or other non-idempotant operation.

Parameters:

sparqlEndpointURL - The SPARQL end point.

uuid - The unique identifier for the request that may be used to CANCEL the request.

tx - A transaction that will isolate the operation (optional).

cancel

public void cancel(UUID queryId)
            throws Exception

Cancel a query running remotely on the server.

Parameters:

queryID - the UUID of the query to cancel

Throws:

Exception

showQueries

public Collection<RunningQuery> showQueries()
                                     throws Exception

List the currently running queries on the server

Throws:

Exception

tupleResults

public org.openrdf.query.TupleQueryResult tupleResults(ConnectOptions opts,
                                              UUID queryId,
                                              IPreparedQueryListener listener)
                                                throws Exception

Extracts the solutions from a SPARQL query.

Parameters:

response - The connection from which to read the results.

listener - The listener to notify when the query result has been closed (optional).

Returns:

The results.

Throws:

Exception - If anything goes wrong.

booleanResults

public boolean booleanResults(ConnectOptions opts,
                     UUID queryId,
                     IPreparedQueryListener listener)
                       throws Exception

Parse a SPARQL result set for an ASK query.

Parameters:

response - The connection from which to read the results.

Returns:

true or false depending on what was encoded in the SPARQL result set.

Throws:

Exception - If anything goes wrong, including if the result set does not encode a single boolean value.