As of now i see, Dremio is using local storage for storing metadata(metadata, catalog information) and using distributed storage for storing results, reflections, user uploads and downloads.
How can we restore metadata stored on local disk when the cluster/co-ordinator is terminated because of any failure.
Can we use external storage/data lake such as S3 to store our dremio metadata so that in case of cluster terminations we can always relaunch a new cluster without losing all the metadata Or do we have any other solution for this problem from Dremio.
@jalandhar good question. we recommend storing the metadata store on an EBS drive so when the EC2 either dies or is terminated, the EBS drive is still available for the next EC2 you use for Dremio. In fact, this is how we architect the deployment of our AWS Edition