How to refresh the reflection with multiple underlying PDS without repetition

1

Hi everyone,

We found that: If a reflection(named R) has multiple underlying PDS (such as A,B and C). If we refresh PDS A,B and C. reflection R will be refreshed for 3 times. That will cost too much time.
So, what can we do to make sure that R will be refreshed for just 1 time.
Is there a better way to refresh reflections?

Thank you.

2

@Switch If you are using the API, you will have to only refresh one of them and Dremio should do all 3. Is that not happening? Assuming there is a VDS that joins the 3 PDS and the reflection is on on the VDS?

3 
            VDSE
    /         |       \
VDSa     VDSb     VDSc
|            |         |
PDSA     PDSB     PDSC

No, like the structure above.
If I only refresh PDSB, the reflection on VDSb and VDSE will be refreshed. But reflections on VDSa and VDSc will not be refreshed.
So I have to refresh all the PDS, but the reflections on VDSE will be refreshed 3times. that is the problem.

4

@Switch, We describe our solution to this problem here: Automating Reflection Refresh

5

@Switch What is the file format for PDSA/B/C? If Parquet, do you really need reflections on the individual VDS’s (A/B/C)? Does the application query from them individually?

6

hey @Switch , I believe you are using CE. May I know which Dremio Version you are using?