Time Series Streaming, Dremio and Arrow Flight

Hi All,

We have a system that store the time series using timescale. We use webscoket/mqtt to push the data into the storage. Then we access that via analytics sytem via jypyter notebook and also ffrom visualization/chart component like grafana.

Now we wrote time series into tsb using native/binary driver. We retrieve the data using pyspark directly so data scientist can easily do EDA and creating model.

So if i can start again, should i use arrow flight to replace MQTT and directly hit Flight endpoint?

It will be simpler as it’s already standard and need to couple to more appropriate one.

Any thought?


cc @balaji.ramaswamy


Will forward this to our engineer working on the Flight project and either of us will get back to you