Suggestion : choose accelerator cache location data path by reflection

1

I have same aggregation reflections which have a big footprint of many GigaBytes, and some which have small footprints (some MegaBytes).
Expected query duration of the big ones : 10 seconds
Expected query duration of the small ones : less than 1 seconds

I currently use Amazon S3 as Dremio’s distributed storage.

I would like to be able to choose the distributed storage for every acceleration when I need it, so that I can use fast storage layers when I need, and S3 when I don’t need results in sub-second delay.

Any thoughts ?

2

I like this idea and it is something we have discussed. With external reflections you have this ability today, but you would have to manage the maintenance of the reflections yourself.

Have you tried storing your reflections on local storage or EFS to see the performance differences? Curious what you have observed.

1 Like