Delegating Reflection Refresh Jobs in Dremio OSS Kubernetes Cluster using queue control

1

Hi All,

I have successfully deployed a Dremio cluster on Kubernetes using Dremio’s OSS Helm charts. In my setup, I have created multiple engines, following the instructions provided here.

Now, I have a specific requirement: I need to delegate the responsibility of refreshing reflections to a particular engine group within my cluster. I understand that this functionality is typically managed using the “Workload Management” feature, which is available only in the Dremio Enterprise edition.

However, in dremio-oss version I noticed the existence of the “Queue Control” feature, which seems to have similarities with workload management. Is it possible to achieve my requirement of delegating reflection refresh jobs to particular engine group using the “Queue Control” feature available in Dremio OSS? If so, could you kindly guide me on how to configure and meet this requirement using “Queue Control”?

2

As of now, I am considering achieving the same requirement with two different Dremio clusters, each of which shares the same metadata storage configured via a shared filesystem (EFS). Is there any issue with this approach?

3

@irshad-pai

  • Dremio cluster 1 will have sources, PDS, VDS and its metadata
  • Dremio cluster 2 will have sources, PDS, VDS, reflections and metadata (same location as above)

Query will be fired on Dremio cluster1 and expectation is query will be accelerated?

4

@balaji.ramaswamy Yes. Intention is to offload reflection refresh from cluster1 as there is no support in dremio oss for the same or I cannot find how to do it as mentioned in my question.

5

@irshad-pai Dremio will not recognize Dremio managed reflections from a different cluster, instead use External reflections

6

Thanks @balaji.ramaswamy. External reflections are not a good option for us since we are using raw reflections only with all the fields in anchor data set selected due to specific requirements in our visualization layer.