DistStorage and local NVMe

[MaxAmende]

Hi,

I am currently trying to setup Dremio in AWS using EKS, but I came across a question for which I cannot find an answer.
Dremio recommends in the guides (EC2) Instances with local NVMe storage. But since V21 DistStorage = Local is not supported anymore.
If I select S3 as Storage, does that mean that I can use EC2 instances without local storage?
Or is the local NVMe storage still used for cashing?

Greetings
Max

[lenoyjacob]

Hey @MaxAmende, Welcome to the Dremio Community!

Keep in mind that Distributed Storage and C3 (caching) are different.

• You’re right in saying that the Distributed Storage needs to be “distributed” as of v21. So it should be on object stores like S3, Azure Storage, GCS or any storage with an S3-compatible API.
• C3 on the other hand needs to be local (for performance reasons). So it should be on local storage (like on EBS, or on the ephemeral storages (NVMe/SSD) if your EC2 instance supports them).
  • If you’re using the default Dremio helm chart from here dremio-cloud-tools/charts/dremio_v2 at master · dremio/dremio-cloud-tools · GitHub then it would attempt to provision EBS volumes for C3 by default.
  • If instead, you need to make use of the ephemeral storages for C3, then you’ve got to use an SSD/NVMe provisioner (something like this GitHub - brunsgaard/eks-nvme-ssd-provisioner: EKS NVMe SSD provisioner for Amazon EC2 Instance Stores works well)

Hope that helps!

[MaxAmende]

@lenoyjacob Perfect! Now it makes sense

So Dremio does not need instances with NVMe storage, but instead EBS is “enough”.

I have a second question. In case an EC2 instance dies/gets shutdown, and thus the EBS cache of the instance is lost, will Dremio run into problems, or will the performance just be a bit worse, until the cache is rebuilt?

Greetings
Max

[lenoyjacob]

Hey @MaxAmende

@MaxAmende

In case an EC2 instance dies/gets shutdown, and thus the EBS cache of the instance is lost, will Dremio run into problems, or will the performance just be a bit worse, until the cache is rebuilt?

EBS volumes are (generally) persistent. They can survive an EC2 shutdown/restart. So EKS will bound the respective EBS volume whenever the EC2 instance for a Dremio Executor comes back up. If for whatever reason, the executor’s EBS volume is also destroyed then the latter applies (i.e. performance will be a little slower until the cache is rebuilt based on the queries sent to Dremio).

See here for C3 Docs: Configuring Cloud Cache | Dremio Documentation

Hope that helps!