We have just released XAP 11.0 and we’re excited to share with you its new features.
- Geospatial API
In today’s world geospatial data is on the rise—mobile phones and various sensors provide ever-changing location information on the one hand, while countless applications are interested in querying this data on the other (e.g. “Find me shops within X miles from my current location, which has product Y in stock”).
With the release of version 11.0, GigaSpaces XAP now supports storing and querying geospatial information. Here’s how it works:
For example, suppose we want to write an application to locate nearby gas stations. First, in order to use geospatial capabilities, we need to include the xap-spatial module in our app:
<dependency>
<groupId>com.gigaspaces</groupId>
<artifactId>xap-spatial</artifactId>
<version>11.0.0-14800-RELEASE</version>
</dependency>
Next, we need to create a model. Let’s create a GasStation class, which includes the location of the gas station:
import org.openspaces.spatial.shapes.Point;
public class GasStation {
private Point location;
@SpaceSpatialIndex
public Point getLocation() {
return location;
}
public void setLocation(Point location) {
this.location = location;
}
}
Finally, assuming we’ve written some gas station entries to the space, we can query them. Since the standard SQL comparisons (=, >, etc.) do not support the notion of spatial operations, we’ve extended SQL in XAP to support a set of new operations, such as “contains,” “within,” and “intersects,” to determine the relationships between spatial shapes. Use the “spatial:” prefix to instruct the SQL parser to use those operations. For example, the following function queries for a gas station within a given radius from a given location:
public GasStation findNearbyGasStation(Point location, int radius) {
SQLQuery<GasStation> query = new SQLQuery(GasStation.class,
“location spatial:within ?”)
.setParameter(1, ShapeFactory.circle(location, radius));
return gigaSpace.read(query);
}
While this query is technically correct, it’s not very efficient—since there’s no index, it’ll randomly scan space entries looking for a match, which could take a long time (depending on the amount of entries). In order to speed things up, you can annotate getLocation() with a @SpaceSpatialIndex. Under the hood, XAP-spatial leverages Lucene spatial capabilities to maintain the index.
Here are all the capabilities:
- Supported shapes: in addition to Point and Circle which we’ve shown here, XAP-spatial also supports Rectangle, LineString, and Polygon.
- Supported operations: in addition to “within,” XAP-spatial also supports “contain” (the inverse of “within”), and “intersects.”
- Geo-fencing: XAP-spatial functionality can be used with event containers to provide geo-fencing capabilities (i.e. if perimeter X is breached do something).
- Supported standards: XAP-spatial supports importing and exporting shapes using the WKT and GeoJSON standards.
- Fine-tuning: some applications prefer a fine-grained index which consumes more disk space, whereas other applications prefer more coarse index to save disk space. These settings (and similar) can be configured by XAP-spatial to control how the underlying Lucene manages the index.
The entire XAP-spatial source code is available at https://github.com/gigaspaces/xap-spatial, if you’re interested in taking a full look under the hood. In fact, XAP-spatial is implemented using another new feature in XAP 11.0 called “Query Extension,” which allows creating and injecting user-defined operations and indexes into the SQL query engine, which is explained below.
- Enhanced Query Capabilities
In our ongoing journey to enhance XAP query language, we’ve added support to a set of common SQL functions. For example, suppose your space contains “ActionItem” entries and you’re looking for “TODO” items, which might be spelled in various low/high case variations. You’ll probably start with something like this:
SQLQuery query = new SQLQuery(ActionItem.class, “label=’TODO’ OR label=’ToDo’ OR label=’todo'”);
This is very verbose and error prone, not to mention inefficient in terms of query execution. Instead, we could use the new “lower()” function to write a similar query:
SQLQuery query = new SQLQuery(ActionItem.class, “lower(label)=’todo'”);
The built-in functions are:
- Math: abs, mod, round, ceil, floor
- Text: char_length, lower, upper, append, concat, instr
- Conversion: to_number, to_char
In addition, if you find yourself missing a function, instead of submitting a feature request, you can simply implement it yourself! For example, suppose you need a prefix function that accepts a string and a length, and returns the prefix of the string up to the specified length. First, create a user-defined class which extends “SqlFunction,” and override the “apply” method according to your needs:
public class PrefixFunctionextends com.gigaspaces.query.sql.functions.SqlFunction {
@Override
public Object apply(SqlFunctionExecutionContext context) {
String s = (String) context.getArgument(0);
int length = (Integer)context.getArgument(1);
if (s.length() <= length)
return s;
return s.substring(0, length);
}
}
As you can see there’s not much to it—we’re extracting the input arguments from the provided context, then using our specific calculation (in this case “prefix”) to generate the result. Note that for production code you’d probably want to add validations about the number and types of arguments, which was omitted in this specific example for brevity.
Next, you need to register this function into the space so the SQL engine recognizes it when it parses and executes the query. This is done as part of the space configuration. So for example, if you’re configuring the space via pu.xml:
<bean id=“prefixFunctionBean” class=“com.demo.PrefixFunction” />
<os-core:embedded-spaceid=“space” name=“mySpace”>
<os-core:space-sql-functionname=“PREFIX”>
<os-core:sql-functionref=“prefixFunctionBean” />
</os-core:space-sql-function>
</os-core:embedded-space>
That’s it! Your code can now use the new functions when querying the space.
- Tiered Storage (RAM/SSD) and Fast Reload
XAP 11.0 offers a new MemoryXtend add-on which leverages RocksDB for storing data off heap. RocksDB is a persistent key-value store optimized for flash devices. It was created by Facebook as an open source project and has a large and active community. This combination lets XAP users enjoy near-RAM performance with larger capacity, at a fraction of the price.
When using this technology, each data node hosts more data, which may present a new problem: a failed node needs to reload its entire data from the primary node over network, and the larger the node, the longer it takes. Long recovery periods consume more resources and endanger system stability, which we want to avoid as much as possible.
The solution, therefore, is to take advantage of the persistent nature of flash devices, and use the data on the device for recovery instead of pulling all the data from the primary over the network. When using this approach, network overhead is minimal—the primary and backup exchange information which allows the primary to detect which exact data is missing from the backup, and all the rest of the data is loaded directly from the backup’s flash device.
To turn on MemoryXtend persistence, simply use the following:
<os-core:embedded-spaceid=“space“name=“mySpace”>
<os-core:blob-store-data-policyblob-store-handler=“myBlobStore”
persistent=“true”/>
</os-core:embedded-space>
Finally, you’ll also need to consider:
- Machine-to-Instance Affinity – When MemoryXtend is transient, it doesn’t matter which machine hosts a specific grid instance. However, when MemoryXtend is persistent and configured for a local storage device, it is desired that each grid instance will “stick” to the same machine so it can recover from its local storage, otherwise it’ll have to recover over the network. If central storage is used however, this is not an issue. For more info refer to http://docs.gigaspaces.com/xap110adm/memoryxtend.html#machine-instance-affinity.
- Ensuring Consistency with Last Primary – If a persistent system is restarted in an orderly manner, it doesn’t matter which instance is elected primary, since all copies are identical. However, if both primary and backup crashed unexpectedly for some reason and then restart, it is important to ensure that the last instance which was primary before the crash will be elected primary again, since it holds a more accurate version of the data. For information on how to configure XAP to preserve last primary correctly, refer to http://docs.gigaspaces.com/xap110adm/memoryxtend.html#last-primary.
