Incorrect datasource schema generated

I’m trying to create Clickhouse connector using instructions from https://www.dremio.com/tutorials/how-to-create-an-arp-connector/ and https://github.com/ClickHouse/clickhouse-jdbc driver. I’ve managed to connect to my Clickhouse server with Dremio, but now am facing issues with wrong schema generated. Clickhouse has only one level of namespace (the database name), and the schema in Dremio is generated with multiple levels (“default” folder / database folders / tables). So when I try to query any table it sends request "select … from “default”.“database”.“table” instead of "select … from “database”.“table” which causes query to fail. Is there some way I could modify schema structure or query structure (assuming I’m not quite familiar with Java)?

Hi,
Is there any update on the connector? Have you fix the problem?

are you @stifstyle and @mdq417 developing a custom connector? can you share your code of ARP yml and java config?
The issue that you have could be by a wrong connector code.
I’ve developed with success a custom connector for Ibmdb2 and for aws athena

Yes, I made it work somehow with official Clickhouse JDBC (https://github.com/ClickHouse/clickhouse-jdbc). Although I had to make some changes to make it work with Dremio:

  1. Used newer versions of com.fasterxml.jackson.core (2.9.9) and com.google.guava (20.0) (https://pastebin.com/Y4LJeeXC)
  2. Modified file /src/main/java/ru/yandex/clickhouse/ClickHouseDatabaseMetadata.java to make it ignore extra level of namespace. (https://pastebin.com/bHdUkVz0)
    Then built jar with dependencies.

Considering Dremio ARP connector, I used the following ARP yml and Java config:

  1. clickhouse-arp.yaml: https://pastebin.com/4xcmSpz4 (definitely requires changes to improve performance and optimize certain queries, but it works)
  2. ClickhouseConf.java: https://pastebin.com/W3NyBc8Q

Thank you! I am new to Dremio and not a developer. I do not know how to build the connecter with the files you kindly provided.

Can you share the jar file that need to be put in the “[software location]/dremio/jars” folder that mentioned in the installation guide below?

https://www.dremio.com/hub/

To install a selected connector, simply download the source connector files and the vendor’s respective JDBC driver and install into Dremio with the following steps:

  1. Select the connector of interest

  2. Click the “Download Connector” link to download the connector file

  3. Copy the connector file to the “[software location]/dremio/jars” folder in each Dremio node

  4. Click the “Download JDBC” link to go to the database vendor’s website

  5. Download the JDBC driver for the given source from the vendor’s web site

  6. Copy the JDBC driver to the “[software location]/dremio/jars/3rdparty” folder in each Dremio node

  7. Restart Dremio

After restarting, the community connector will appear in the list of available sources.

you dont need to be expert developer to build a custom conenctor, but yo need the correct tools for it, JDK, java project, and know how build…

the jar mentioned in the guide is the JAR that you must build of your custom connector using JDK

plase can you share what are you trying do?

I do not know how to build…
I try to connect to Clickhouse using Dremio and there is no connector available. That is how I found this post.

Can you share the jar file of the final connecter? Thank you!

@mdq417

The jar file would be different for different connectors, see below link to get started

I have read the link. The connector I need is for Clickhouse same as @stifstyle.

Try these files : https://transfer.sh/f02JK/dremio_clickhouse.zip
But I’m not sure if it would work.

clickhouse-jdbc-0.2.3-jar-with-dependencies.jar - goes into /<path_to_dremio>/jars/3rdparty
dremio-clickhouse-plugin-4.1.3-202001022113020736-53142377.jar - into /<path_to_dremio>/jars

I used it with Dremio version 4.1.3. And this connector I made was still a bit buggy (due to config), had no time to fix some of the strange issues I faced.

@stifstyle what’s the status of the your connector now? Were you able to make it work good? Thanks in advance