Kafka connector

[jacek]

Related to 2/3 years old questions:

KTable connector?

Hello, is it planned to use Kafka KTable as source for dremio? It could be great for building an IOT plaform with Kafka and Dremio where Kafka is used as a streaming datastore and Dremio as a query engine and data governance tool. No more need of hadoop then.

Is dremio suitable for big streaming data?

Hi all first of all congratulations for a really great piece of infrastructure. The premises are great. I was wondering if Dremio is meant/suitable for big STREAMING data, for example 1TB a day of streaming logs. would one (realistically) be able to operate and query the data as its streaming? so: are there streaming connectors? would Dremio understand when data is appended to a file on HDFS? how about if the data is e.g. in Elasticsearch and grows continuosly and switches index name every…

You mentioned there that Kafka is on your roadmap, but it is still not documented as a supported data source.
Any news here? Target release date?
Regards

Jacek

[balaji.ramaswamy]

@jacek

For building a Kafka connector, please take a look at the below link

Dremio Hub

Do you have any questions on reading logs, elastic search?

Thanks
Bali

[jacek]

Well, what happened that it is no longer on your roadmap?
Why it stopped to be your priority? Any technical reasons?

Writing connector could be quite a complex task, could not be?
I mean reading from Kafka topic could be easy, but to add full support for reflections could be quite complex, right?

[balaji.ramaswamy]

@jacek

Currently we do not have plans for an inbuilt Kafka connector

Thanks

[jacek]

I analyzed Vertica connector and now I see that it is not too complex.
Moreover I start considering an option to utilize ksqldb (JDBC driver available).
Is this out of your radar as well?

[balaji.ramaswamy]

@jacek

Currently it is not in the radar

Thanks
Bali