We are running the latest version of Dremio in Azure on K8s. Our data source is ADLS v1 and we are using ADLS v2 for reflection storage. We are running four executor nodes, each one with 7 CPU cores and 60GB of RAM. Our data set is relatively small, around 40GB. Accelerated queries are taking over 5 minutes to run.
I have checked relevant log files and all seems to be in order. Sometimes the executor nodes get lost and we have to restart the pods because the jobs hang. That is an intermittent problem. The main problem right now are the painfully slow queries. The system is performing worse than a standard SQL DB.
It seems this problem could be related to Asynchronous access to the ADLS data source. Known bug which requires this feature to be disabled. We are testing now and I will confirm.
This did not resolve the problem so we are doing a redeploy based on the update Helm charts provided by Dremio and throwing more resources at the platform.