Using external storage like S3 for storing Dremio's metadata

Hey guys,
As of now i see, Dremio is using local storage for storing metadata(metadata, catalog information) and using distributed storage for storing results, reflections, user uploads and downloads.

How can we restore metadata stored on local disk when the cluster/co-ordinator is terminated because of any failure.

Can we use external storage/data lake such as S3 to store our dremio metadata so that in case of cluster terminations we can always relaunch a new cluster without losing all the metadata Or do we have any other solution for this problem from Dremio.

Thanks,
Jalandhar

I have created some topics about storage of metadata also.


@jalandhar @koolay

Currently we do not support an external DB like MySQL

Thanks

@jalandhar good question. we recommend storing the metadata store on an EBS drive so when the EC2 either dies or is terminated, the EBS drive is still available for the next EC2 you use for Dremio. In fact, this is how we architect the deployment of our AWS Edition

1 Like

@balaji.ramaswamy

Just like that loki has removed the dependency of BoltDB in the new release. loki needs only s3 as storage now.