How to create dataset for daily data report

1

In my company use the ETL workflow process(in Hive) to make the data for daily BI report，if I want to switch to dremio how to do it？
The workflow run daily and Hive tables are partitioned by day. The daily partitions of every tables are about 5-8GB.
4-5 hive tables have to join together.
Thanks

2

Hi @zhang_maosen,

If I understand it right, your current ETL workflow is on Hive. You should be able to add the Hive meta store as source to the Dremio UI and able to run queries through Dremio. I am pasting the link to adding a Hive source below

https://docs.dremio.com/data-sources/hive.html

Once you add the Hive source and run some workloads, you can also create reflections that can accelerate your query further

Kindly let me know if you have any other questions

Thanks
@balaji.ramaswamy

3

Thank u for your reply
I could desc more about my problem:
for example:
My Hive table T1 and T2, both are partitioned by column ‘day’. everyday I will insert new day partition into T1 and T2.
Then join T1 and T2 and insert into T3 like this:
insert overwrite into T3 as select T1.*, T2.* from T1 join T2 on T1.id=T2.id where T1.day= current_day() and T2.day=current_day()

then I create a dataset for T3 like this:
select * from (select T1., T2. from T1 join T2 on T1.id=T2.id where T1.day = T2.day) nest_0
But this query is very slowly and reflection create fail

4

Hi @zhang_maosen

What error do you get when you try to create a reflection?

Thanks
@balaji.ramaswamy