ARP is so restrictive, I don't even want to use it anymore

[patricker]

I’ve been doing my best to give ARP a really fair chance, and we’ve had tons of success with community provided plugins like Snowflake, but when trying to build your own custom plugin there are so many restrictions, and the playing field is uneven.

Lets dig into the latest issue, the uneven playing field. I’m working on a custom ARP connector. Dremio provided ARP connectors like MySQL, Oracle, and PostgreSQL include a custom ArpTypeMapper. This lets the plugins provide custom logic for how things like precision and scale should be calculated, and other things that are database specific. But if you decide you need this logic yourself for your custom ARP plugin, you are out of luck. This is because you need to instantiate instances of SourceTypeDescriptor, and this class has no public constructor. This means Dremio provided plugins can provide custom logic for how types are mapped, but custom plugins can’t.

On the ARP Documentation page it says:

Using the ARP framework not only allows you to create better connectors with improved push-down abilities but it also provides a way to develop connectors more efficiently.

I just want to emphasize that at least three of the Dremio provided ARP connectors are not reproducible without being a Dremio employee, with legal right to include your plugin inside the ARP binary.

I know that the ARP framework has a ton of promise, but it’s really being shackled and beaten down by the restrictive way in which Dremio is treating/licensing the framework. This is just one of my issues with the framework, I’ve included a link to a recent post I made with others at the end of this post.

Lets also discuss the irony of Dremio Hub vs the CE edition. If you want to contribute a custom plugin that you’ve developed with ARP to Dremio Hub, it must be licensed under the Apache license (Contributing to Dremio Hub). But if you were to build Dremio CE in OSS mode, you wouldn’t even be able to use these plugins, because they require the ARP JDBC jar, which isn’t available in OSS Dremio and isn’t available under an Apache license.

Post the code on GitHub with an Apache license and start accepting some PR’s so we can break through the restrictions on this library. It has so much promise, and you’ve taken the time to tout it as a great way to build new plugins in an efficient way, but right now there are small bugs that no one can fix and which block many plugins from being created.

@jacques @laurent

---

A recent post with some of my other issues concerning ARP, which also aggregates even more of my other posts:

Writing a generic JDBC connector that supports ANSI SQLs

@olumo I don’t have a definitive answer to your question, but I have spent a fair amount of time diving into the closed-source innards of the ARP JDBC adapter source code. Go grab a program like JD-GUI and start digging around the custom ARP adapter that Dremio implemented for MS SQL (which is how they are able to set all the custom character ENCODING stuff), which they defined through Java code instead of through a YAML file. Of course, you can only use the code you see as a guide, license wi…

[Kyle_Porter]

Hi,

Wanted to hopefully resolve a few of the technical aspects that you’ve brought up, will let others respond regarding the licensing.

First, everything that’s done via the built-in ARP connectors can be done via third-party connectors.

You can create a subclass of ArpTypeMapper which would allow you to customize the precision and scale as well. This would be done by the following:

1. Create a subclass of ArpDialect, the SQLite example instantiates this directly but you should create a subclass and instantiate it.
2. In your subclass of ArpDialect, override getDataTypeMapper() to return your own instance of TypeMapper, which can be a subclass of ArpTypeMapper.
3. In your subclass of ArpTypeMapper, override createTypeDescriptor() or createTableTypeDescriptor() as necessary to override any calculations for the column metadata returned from the JDBC driver.
4. The SourceTypeDescriptor constructor visibility issue is not ideal, but can be worked around by:

• Using reflection to instantiate the SourceTypeDescriptor
• Placing the code within the com.dremio.exec.store.jdbc.dialect package since the constructor is package private

I believe these steps should resolve the technical items you pointed out above, let me know if you have further questions.

[patricker]

Thanks for the guidance @Kyle_Porter, I agree it’s far from ideal, but matching the package name does indeed let me extend the class as you suggested. I don’t suppose you have a fix for JDBC drivers that don’t support getTimestamp with a Calendar object . I opened a question about that as well (I found it easier to modify the JDBC library than ARP…):

Enhancement Request - ARP JDBC Data Type Mapping

When mapping data types, date and timestamp objects need to support NOT passing in a calendar object to the getDate and getTimestamp methods. In my scenario, the JDBC driver just doesn’t implement these versions of the methods and only supports calling the non-Calendar method. Something like the enable_utc_calendar option shown below. source: name: “DATE” enable_utc_calendar: false dremio: name: “date” source: name: “TIMESTAMP” enable_utc_calendar: false dremio: name: “timestamp”

[Kyle_Porter]

Hey @patricker - unfortunately there’s no workaround for that one and indeed it would need to be a feature added to the product. No promises, but I’ve made PM aware in case they weren’t already that there is this ask.

[serra]

@patricker thank you for sharing your experiences. What is your current stance on this?

I am considering developing an ARP connector for Firebird and am wondering if it is worth the effort.

From this post I understand that APR connectors are now supported in the OSS CE of Dremio.