Could I create a custom connect based on a REST API

1

Hi,

I have a system which expose a set of rest API to query data. And I want to create a Dremio connector to get data from my system.

Obviously my API does not support all SQL functions. How to implement connectors based on the ARP framework? What SQL instructions must be supported? How can I let Dremio know the capabilities of my connector and push the appropriate SQL processing statements?

Thanks

2

You can’t use ARP to do this. ARP is specifically designed and built for sitting on top of JDBC clients. It doesn’t work with other interfaces.

With that said, yes, you can do what you are trying to do, it’s just a lot of work… We have an internal plugin that combines MS SQL as an index with files in HDFS. In our instance the custom plugin exposes a few tables, and translates the queries into index lookups in MS SQL, retrieves data from HDFS, reads that data into memory, and finally returns it to the user, all while hiding this join between a SQL index and physical files (files are not in a Dremio supported format).

Unfortunately I don’t have anything I can share with you from our implementation, but we heavily used the Hive plugin as a starting point since it also reads HDFS.

1 Like
3

@WileyXu, it sounds like you are implementing a “pre-Dremio” SQL parser that validates the syntax before sending the query off to Dremio. Dremio broadly follows ANSI SQL standard. You can find a list of supported SQL functions here: https://docs.dremio.com/sql-reference/sql-functions/

1 Like
4

Thanks for the explanation

5

Thanks for sharing these documents