What is the best open source BI tool for dremio

1

Right now we have a lot of data from various sources, so I want to use Dremio as the data warehouse to avoid ETL process
Can you please advise what open source BI tool should I use

2

Hello @Hai_Pham,

When you say you are trying to “avoid ETL process”, can you give an example of your use case? Dremio is a great tool for analytics (aggregate, many-to-one) queries against a diversity of sources, but it will not be as effective if you are transforming a large set of records into a similarly sized set of records (many-to-many) typical of ETL pipelines.

You can use a number of BI clients with Dremio, but which tool is best for you depends on your use case. You can browse our list of supported clients here for more information

3

Hi @ben
Thank you for your reply
Our data sources come from sap, dms, distribution management system, csv, excel file …
Right now I think there are 2 approaches

  1. Build our own data warehouse, then allow BI tool to connect to. But this way would take time since we have to write etl process to transform data.
  2. Use dremio as data warehouse so that we can load data sources quickly since dremio supports a lot of data sources. My concern is that, what the best practice is for the case we load data source from the excel file today, and tomorrow we would have a new data source file. Does dremio supports Api to automate this process: create data source, query …
    I just checked the list you mention and wonder if metabase BI tool can work with dremio. Have you ever tried it yet?
4

hi @Hai_Pham,

Concerning Metabase, There is already a feature request opened in its git repository.
On the other side, I had a call with Jeff Feierfeil jfeierfeil@dremio.com in october, and submitted the idea, with no feedback since.

We had the same idea, wiring dremio ontop our different data sources, in order to skip the datawarehouse step… but it was not that simple, because it takes several steps to bring up any conformed and cleaned information mart. DWH is also IMHO a place to materialize consolidated data, before to be presented.

Now I most see Dremio as the easiest way to assemble structured dwh data and unstructured lake or excel data before presentation, and benefit from the starflake reflection optimizations.

I will be glad to know how you use Metabase and Dremio !

Best regards from France,
Fabrice Etanchaud

5

Hello @Hai_Pham,

“My concern is that, what the best practice is for the case we load data source from the excel file today, and tomorrow we would have a new data source file.”

Dremio can connect to a variety of filesystem sources, and you can set metadata refresh intervals for these sources so that they detect new files with a frequency that matched your use case.

“Does dremio supports Api to automate this process: create data source, query …”

Dremio has a published RESTful API you can use automate the creation of data sources, query, etc., Please check outour documentation to get an idea of what’s possible.