Reflection on VD using subset of PD data

1

Hi Dremio Team,
I am trying the following use case

created a raw reflection PD

created a helper VD as select * from PD where <date_col> = <FIxed_date>
created raw reflection on VD

now if i do select * from PD where <date_col> = <FIxed_date> (same date used in VD def)
its not using reflection on VD,
from the profile it says reflection on VD chosen, but not matched. Is there a way to implement this ?

my PD will have years of data, so it would be beneficial if we can make use of VD reflection for daily reporting.

Thanks

2

We would need more information - can you share the query profiles and some information about the source?

3

@doron

PD parquet files on HDFS source, created raw reflection with partition on two columns (effective_date, source_id)

VD ( select * from PD)
helper VD ( select * from VD1 where effective_date = ‘2019-07-08 04:00:00.000’ ), created raw reflection

qry:
SELECT effective_date, source_id, count(1) FROM rdv_h3 group by effective_date, source_id
uses reflection on PD, profile 0649baea-2f68-47e2-86c8-5c5ed93a4bfd.zip (61.5 KB)

qry
SELECT effective_date, source_id, count(1) FROM rdv_h3 where effective_date ='2019-07-08 04:00:00.000' group by effective_date, source_id
also uses reflection on PD instead of reflection on helper VD, * Raw Reflection (raw): considered, not matched. profile ff334cee-9999-4f36-8278-568f45c11278.zip (67.1 KB)

Thanks