Optimization and scaling question

1

So I have a cluster up and running Dremio on Kubernetes and we are testing the scale. Optimally we would like to process 1TB worth of data on Dremio cluster and at this point, I have been facing issues in processing just 1GB of data.

The errors I run into is mostly JVM Heap max reached.
below is the values.yaml file I have deployed at the moment, the server i am running is currently a cluster of 10 AWS m4.2xlarge (8CPU, 16 GiG RAM).
I even tried raising the Heap value env by editing _helpers.tpl

{{- else if ge $input 9216 -}}
4096

...

{{ sub $input 4096 }}
{{- else if ge $input 9216 -}}

Is there a sample/optimal dataset that i can run and test and see how the server is performing or somehow increase the performance?

image: dremio/dremio-oss
coordinator:
  memory: 1024
  cpu: 1
  count: 0
  # This value should NOT be set to more than 1.
  master_count: 1
  web:
    port: 9047
  client:
    port: 31010
  volumeSize: 80Gi
executor:
  memory: 9216
  cpu: 3
  count: 5
  volumeSize: 80Gi
storageClass: portworx-sc-high
2

Hi Rohithzr,

Can you provide some more details on your data - what type of data it is and where is it located?

We do run performance tests using TPCH data with a scale factor of 100 which would be around 100G of data.

Also, you can update the heap setting in config/dremio0-env.

  • Nirmalya
3

Another thing - please review the system requirements. It would apply for the pods in kubernetes too.

https://docs.dremio.com/deployment/system-requirements.html

Please try increasing the memory and cpu of the coordinator and try your tests.