Plugin contribution guidance

Is there any guidance on how to author plugins? I looked around in the codebase and did some searches in the forum here, and didn’t come across anything.

I suppose I can figure it out mostly by looking at the existing ones, but a guide/readme or even a list of the important classes/interfaces involved, as well as how to bind/configure a plugin once created, would be really useful.

In the meantime, I will fork the codebase and see how far I can get.

Thanks much for any guidance!

Hi Scott, what source did you have in mind? There may be some 3.0-related changes that are worth considering.

Well, I have a couple specific ideas in mind for connecting to various APIs. But I was really hoping to find some kind of quick guide or overview.

FWIW I do think as Dremio grows there will be consistent demand for plugins to accommodate sources that probably don’t rise to the level of general/global importance to merit inclusion in the core codebase. For example, companies/organizations may have one-off internal APIs to which they want to connect. Making it relatively easy to develop plugins will benefit them.

1 Like

Completely agree and this is something we will get to. Other things have a higher priority at the moment. We do have users who have built custom plugins, so it is possible, but not easy.

In 3.0 there’s been a ton of work to revamp the relational connector codebase, so if this is the area of your needs it may be beneficial to wait until that’s out, sometime in the next six weeks or so.

Hi Kelly, and thanks for the response.

Is there a timeline (approximate or otherwise) that you can share regarding the 3.0 release?

In the meantime, is there anything you can share about how the relational connector components will work, what the API will look like, etc.?

Hi @scottcame,

We would like to release documentation and examples for plugin development. There are a couple levels of APIs that we could expose - one specifically for relational databases, and one for non-relational databases. Can you share some details on the type of data source that you want to connect to? Is it a relational database, REST API, …?

Hi @tshiran.

My immediate objective is to connect to a REST API that accepts query parameters via POST (JSON body) and returns either a csv or a JSON stream of records (based on a request parameter).

However, the ability to plug in a generic ansi-standard sql database via jdbc would be really nice, as well (understanding that push-downs and other optimization may not work as well as when you know the specific database type).

Finally… I actually think it’d be really useful to have a plugin that just points at a static url for a csv. For instance, if I wanted the data for the 2018 primary election in West Virginia to be a source, I’d just be able to configure a source to point (live, not downloaded/uploaded) to: https://raw.githubusercontent.com/openelections/openelections-data-wv/master/2018/20180508__wv__primary.csv.

Is there any documentation reference available on plugin contribution guide .?

1 Like

I’m interested in communicating with a proprietary API. Probably REST. Return format is flexible, so csv, json, or arrow flight are in the mix. I’d also like to be able to return some kind of entitlement fingerprint, so Dremio could do respect the entitlements in any internal caching it does.

Thanks,
-Jason