I have used latest docker images for dremio and deployed it on kubernate with cluster .I have mounted
/opt/dremio/data to the azure file share .after that i have connected SQL server as datasource.
when i am quering and create reflection it takes much time some time even a minutes to create reflection.i can’t see any error log
Hi there - Were you able to see my response on your other thread here? Not able to set reflection
Pasting my response here - “It seems like it’s still processing and may be taking awhile. It may be worth noting your executor node is really small and undersized, it is below our recommended minimum requirements - https://docs.dremio.com/deployment/system-requirements.html#server-or-instance”
It may be worth scaling up your cluster to see if performance increases.
I have put both coordinator and executer to 8 cpu ,16 gb ram best on yur last reply after that performance was happy for me but when i was mounted volume on azure file share its hunting down performance on each query
Understood. I am guessing there may be a read latency to Azure. Can you share a query profile to verify this?
Thanks anthony,here is the Profile for Job took 2s to execute
86f91d26-8aa1-41a3-97d8-8d50e5c7761a.zip (29.4 KB)
Can you clarify - is the issue that the reflection takes too long to create or the query takes too long to run?
For the query you sent the query profile for, can you test a query with less columns instead of the ~240+ columns it has and see if that is faster?
i have followed your thought .i have used less column of approx 150,I am querying though jdbc connector
Still query takes long time as expected from dremio. May i know how to optimize Query .I already have enabled Raw Reflection and saved in space .May i know where should i look how much time my query took to execute since we are going to use Dremio for production Big Data
For testing purposes, can you select 20 columns and see if that improves? What are the CPU/mem for the coordinator? 8cores/16gb?
The query itself is taking less than 1 second to execute, but Dremio’s query planning is taking ~2s.
Mine configuration 8 cpu co-ordinator,8 cpu executor,8 cpu master i have confugration of 32 cpu 128gb ram still query take long time