Customizing Dremio Cluster Setup

#1

Hey guys,
Currently we are exploring Dremio Community Edition for our business use cases.
But what we realised was, this set up is using below high end machines only:

Co-ordinator: m5d.2xlarge

Elastic Engines:
m5d.8xlarge
r5d.4xlarge
c5d.18xlarge
r5ad.4xlarge
i3.4xlarge

These EC2 machines are costing us high and we want to use low end machines of our own configuration.As of now we don’t see any option for choosing different type of EC2 machines.
Can you guys help us, how can we change this in our AWS community edition.

It would be great if we can at-least downgrade the co-ordinator node from m5d.2xlarge instance type.

Build: 4.2.1-202004111451200819-0c3ecaea
Edition: AWS Edition

Thanks,
Jalandhar

#2

@jalandhar,

You could likely run the coordinator on a smaller host, but knowing nothing about your workload, it’s hard for us to say whether you will see significant performance hit or stability issues.

#3

Hey @ben,

For us the actual issue is, we are unable to chose a smaller node for the co-ordinator. We were not even given an option to chose the an instance of our choice.

And regarding the workload, we have different use cases and workloads where the data varies from MBs to GBs and is for sure not in TBs.

We want to try different options(with different types of instances) and benchmark our queries.

It would be great if we were given an option to chose between different types of EC2 instances instead of fixing them. We can definitely tune the configuration for our needs and workloads.

Thanks,
Jalandhar

#4

@jalandhar,

You could forego the AWS Community addition and deploy a cluster another way: https://www.dremio.com/on-prem/

This would give you the flexibility to choose different sizing.

#5

Thanks @ben

We will explore the other deployment options.