How to Optimize Query Performance for Large Datasets in Dremio

Hey guys! :smiling_face_with_three_hearts:

I’m currently wrestling with optimizing query performance for some massive datasets (think billions of rows!). While Dremio’s been amazing so far, I’m hitting a snag with speeding up my queries. They’re taking longer than I’d like, and I have a feeling there’s room for some serious optimization.

Here is a quick rundown of my setup:

  • I am using Dremio with a bunch of Parquet data chilling on S3.
  • Most of my queries involve joining these hefty tables together.
  • I have been trying reflections to give things a boost, but I’m not too sure I am using them effectively for these large datasets.

My Dremio cluster has 5 nodes, each with 64GB of RAM. So, I have a few burning questions for the Dremio gurus out there:

  • How can I best set up and manage reflections in Dremio, especially when dealing with these big data beasts?
  • Any tips or tricks for optimizing my queries to cut down on execution time? Are there specific strategies for handling massive joins?
  • What tweaks can I make to my cluster configuration to squeeze out some extra performance?
  • Anyone else faced similar performance challenges? If so, what strategies or adjustments did you find most helpful?
    I also check this resource: But I have not found any solution. could anyone guide me about this?

Thanks in advance!

The link you provided is for streaming large query results whereas your question is about query execution performance. They don’t seem to be related. Why don’t you share or private DM me some verbose query profiles and I’ll give you some suggestions?