Have set up Dremio, latest community edition, on an EKS cluster with 3 executor pods on r5d.4xlarge nodes.
Successfully onboarded an AWS Catalog.
However, each and every query takes > 10 mins due to Metadata Retrieval, compounded by failed attempts.
Here’s an excerpt from a query’s 3rd attempt:
Command Pool Wait:
Total Query Time:
I,ve read other thread where users have been complaining of extremly slow S3 metadata refresh rates. Has this issue been resolved? If not, it seems to also affect AWS Glue.
I have also attempted to prevent metata refresh by running:
ALTER PDS REFRESH METADATA AVOID PROMOTION
Which did not seem to help.
kubectl get pods
NAME READY STATUS RESTARTS AGE
dremio-executor-0 1/1 Running 0 11h
dremio-executor-1 1/1 Running 0 11h
dremio-executor-2 1/1 Running 0 11h
dremio-master-0 1/1 Running 0 11h
zk-0 1/1 Running 0 11h
zk-1 1/1 Running 0 11h
zk-2 1/1 Running 0 11h