How join works with two different data sources

1

Hi,
I am trying to join tables from two different data sources. i want to understand the backed process like how Dremio joins data from datasource1.table1 and merge with datasource2.table2

is Dremio transferring data from one source to another?

2

@madhusudhan

Based on the query Dremio prepares the plan to read the erquired data/columns/rows from the both sources and provides the results.Dremio do not transfer any data from one source to other.

@Venugopal_Menda

3

you mean that every time when we a query Dremio fetches the data through network?

4

Yes we read data for each query. Dremio has a feature called refelctions which stores the data in parquet format, using reflections the query accelerates and finishes in quick time.

Reflections: https://docs.dremio.com/acceleration/

5

Thanks @Venugopal_Menda !
i need some more clarification on joins with filters.

when i am joining two tables from different data sources by applying filter on data, i want to understand where the data will get filtered. i mean will the dremio read complete data and then filter it or is there any mechanism, that will filter the data and then fetch it?

6

Hello @madhusudhan,

Depending on the types of sources you are joining, sometimes Dremio will push down the filter into the source (pre-filter) and sometimes it will grab many records from the source and filter in Dremio’s execution engine. Which source would you be joining across?

7

ben, i am joining data from S3 and postgresql.