Dremio Spatial Implementation - Code Contribution/Direction

1

I am very impressed with Dremio at the moment. I am a big fan of Calcite, Parquet & Arrow. This project ticks so many of my check boxes.

Okay, I need a spatial implementation.

Having dug through the code, I am no clearer how I accomplish this.

I have successfully used the basic implemented spatial functions using a Calcite 1.16-SNAPSHOT via a “jdbc:calcite:fun:spatial” connection string.

Obviously Calcite is not being used this way.

Can any of the contributors direct me to the salient code where I can have “a stab” at implementing this?

Thanks in advance for your help.

Regards

Jonathan

1 Like
2

Hi @jonrmayer,

Thanks for your support, and for trying to take a stab at adding spatial support in Dremio. However, like you guessed, adding support is not just a matter of changing a connection string: this is because Calcite (like Arrow) is really at the core of our query engine .

In order to add spatial support, first thing would be to update to a newer Calcite version, but as we don’t currently provide the source yet, I don’t think it would be possible to do it as of now. But we are working hard on upgrading Dremio to the latest Calcite version, which would be the very first step (with more being needed to add the full support for spatial extensions).

If other people are interested in spatial support, please let us know so our product team can choose if/when to add it to our roadmap.

2 Likes
3

+1 on spatial support

1 Like
4
  • 1 we need also spatial support …
5

I work with geospatial data on a a daily basis. Spatial support would be a game changer for me.

1 Like
6

Plus for me here too

7

Hi all !

I support also this idea if Dremio cloud directly support spatial features.

For now you could try the Dremio GIS udf from Christy Haragan

Cheers

8

If anyone reaches this thread, I re-ported this functionality again to match with modern Dremio code:

Comes complete with maven central artifacts generation and Dremio CE version compatibility, caveat mapping , as well as updated dependencies.

Dremio UDF GIS

Cheers :sunglasses:

1 Like
9

Version 0.2.3 released.

ESRI API upgraded to JTS,
Modernized APIs
WKB/EWKB support for reading geom directly from PostGIS is also confirmed to work.

Dremio UDF GIS

1 Like
10

Versions 0.2.4, 0.3.1 and 0.4.0,
Bringing support for ST_Length, ST_Area and ST_Transform
Supported Dremio versions are 20.1.0, 21.1.1 and 21.2.0 accordingly

Dremio UDF GIS

11

Updates:

  • Dremio 22.1 and 23.0 support added
  • More GIS functionality added:
    • ST_Collect/ST_Union aggregation bugfixes
    • ST_Relate matrix variant,
    • ST_Buffer parameter string variant receives style parameters string to match PostGIS functionality/behavior
  • Tests added utilizing Kotest, with coverage details found in Coveralls

Versions for available either on github or in maven central

Dremio UDF GIS

3 Likes