We found that: If a reflection(named R) has multiple underlying PDS (such as A,B and C). If we refresh PDS A,B and C. reflection R will be refreshed for 3 times. That will cost too much time.
So, what can we do to make sure that R will be refreshed for just 1 time.
Is there a better way to refresh reflections?
@Switch If you are using the API, you will have to only refresh one of them and Dremio should do all 3. Is that not happening? Assuming there is a VDS that joins the 3 PDS and the reflection is on on the VDS?
/ | \
VDSa VDSb VDSc
| | |
PDSA PDSB PDSC
No, like the structure above.
If I only refresh PDSB, the reflection on VDSb and VDSE will be refreshed. But reflections on VDSa and VDSc will not be refreshed.
So I have to refresh all the PDS, but the reflections on VDSE will be refreshed 3times. that is the problem.
@Switch, We describe our solution to this problem here: Automating Reflection Refresh
@Switch What is the file format for PDSA/B/C? If Parquet, do you really need reflections on the individual VDS’s (A/B/C)? Does the application query from them individually?
hey @Switch , I believe you are using CE. May I know which Dremio Version you are using?