Dremio Planning & Running Time is very high

1

Query planning & running time is very high for a dataset (Amazon S3) which is 1GB in size. We have setup a full reflection for it which is around 250MB in size. This is taking about 6 sec of planning and 6 sec of running.

615a01d0-216e-4c6d-b036-5ac6316e9618.zip (31.0 KB)

Attaching the profile of the job here. Thanks in advance!

2

@mohak.jhaveri

We have one known issue of S3 sources having high Convert To Rel, open your job profile, click on planning tab and you will see most of the planning time is spent in Convert To Rel, please upgrade to the latest version and try. The other feature you can try is default reflections 4.8 and above, Dremio cuts down significant planning time when matching reflections, its on by default, it will not help this issue though, for a reflection to qualify as a default reflection is should be

  • Raw reflection
  • On a VDS only
  • All columns need to be selected as display columns
  • The VDS needs to be used in the query or by the VDS that refers to this VDS

Also for the current planning time issue, try to give full context names in you SQL for all the datasets

Thanks
Bali