I’m trying to create Clickhouse connector using instructions from https://www.dremio.com/tutorials/how-to-create-an-arp-connector/ and https://github.com/ClickHouse/clickhouse-jdbc driver. I’ve managed to connect to my Clickhouse server with Dremio, but now am facing issues with wrong schema generated. Clickhouse has only one level of namespace (the database name), and the schema in Dremio is generated with multiple levels (“default” folder / database folders / tables). So when I try to query any table it sends request "select … from “default”.“database”.“table” instead of "select … from “database”.“table” which causes query to fail. Is there some way I could modify schema structure or query structure (assuming I’m not quite familiar with Java)?
Is there any update on the connector? Have you fix the problem?
are you @stifstyle and @mdq417 developing a custom connector? can you share your code of ARP yml and java config?
The issue that you have could be by a wrong connector code.
I’ve developed with success a custom connector for Ibmdb2 and for aws athena
Yes, I made it work somehow with official Clickhouse JDBC (https://github.com/ClickHouse/clickhouse-jdbc). Although I had to make some changes to make it work with Dremio:
- Used newer versions of com.fasterxml.jackson.core (2.9.9) and com.google.guava (20.0) (https://pastebin.com/Y4LJeeXC)
- Modified file /src/main/java/ru/yandex/clickhouse/ClickHouseDatabaseMetadata.java to make it ignore extra level of namespace. (https://pastebin.com/bHdUkVz0)
Then built jar with dependencies.
Considering Dremio ARP connector, I used the following ARP yml and Java config:
Thank you! I am new to Dremio and not a developer. I do not know how to build the connecter with the files you kindly provided.
Can you share the jar file that need to be put in the “[software location]/dremio/jars” folder that mentioned in the installation guide below?
To install a selected connector, simply download the source connector files and the vendor’s respective JDBC driver and install into Dremio with the following steps:
Select the connector of interest
Click the “Download Connector” link to download the connector file
Copy the connector file to the “[software location]/dremio/jars” folder in each Dremio node
Click the “Download JDBC” link to go to the database vendor’s website
Download the JDBC driver for the given source from the vendor’s web site
Copy the JDBC driver to the “[software location]/dremio/jars/3rdparty” folder in each Dremio node
After restarting, the community connector will appear in the list of available sources.
you dont need to be expert developer to build a custom conenctor, but yo need the correct tools for it, JDK, java project, and know how build…
the jar mentioned in the guide is the JAR that you must build of your custom connector using JDK
plase can you share what are you trying do?
I do not know how to build…
I try to connect to Clickhouse using Dremio and there is no connector available. That is how I found this post.
Can you share the jar file of the final connecter? Thank you!
The jar file would be different for different connectors, see below link to get started
I have read the link. The connector I need is for Clickhouse same as @stifstyle.
Try these files : https://transfer.sh/f02JK/dremio_clickhouse.zip
But I’m not sure if it would work.
clickhouse-jdbc-0.2.3-jar-with-dependencies.jar - goes into /<path_to_dremio>/jars/3rdparty
dremio-clickhouse-plugin-4.1.3-202001022113020736-53142377.jar - into /<path_to_dremio>/jars
I used it with Dremio version 4.1.3. And this connector I made was still a bit buggy (due to config), had no time to fix some of the strange issues I faced.
@stifstyle what’s the status of the your connector now? Were you able to make it work good? Thanks in advance