# Using external storage like S3 for storing Dremio's metadata **URL:** \ **Category:** Uncategorized\ **Created:** [June 19, 2020, 6:19am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763 "2020-06-19T06:19:08Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![jalandhar](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jalandhar/32/2553_2.png) [@jalandhar](https://community.dremio.com/u/jalandhar)\ **Post date:** [June 19, 2020, 6:19am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/1 "2020-06-19T06:19:08Z")
Hey guys, As of now i see, Dremio is using local storage for storing metadata(metadata, catalog information) and using distributed storage for storing results, reflections, user uploads and downloads. How can we restore metadata stored on local disk when the cluster/co-ordinator is terminated because of any failure. Can we use external storage/data lake such as S3 to store our dremio metadata so that in case of cluster terminations we can always relaunch a new cluster without losing all the metadata Or do we have any other solution for this problem from Dremio. Thanks, Jalandhar ---
**Author:** ![koolay](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/koolay/32/2407_2.png) [@koolay](https://community.dremio.com/u/koolay)\ **Post date:** [June 19, 2020, 7:38am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/2 "2020-06-19T07:38:42Z")
I have created some topics about storage of metadata also. > [@Can metadata support to storage in mysql?](https://community.dremio.com/t/can-metadata-support-to-storage-in-mysql/5290/11): > > It’s will be greater if support store metadata in database like mysql. > [@How to custom the storage provider for metadata?](https://community.dremio.com/t/how-to-custom-the-storage-provider-for-metadata/5610): > > We want to custom the storage for metadata, and how to do we can? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [June 22, 2020, 5:39am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/3 "2020-06-22T05:39:53Z")
@jalandhar @koolay Currently we do not support an external DB like MySQL Thanks ---
**Author:** ![jason](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jason/32/3129_2.png) [@jason](https://community.dremio.com/u/jason)\ **Post date:** [June 24, 2020, 8:50pm UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/4 "2020-06-24T20:50:32Z")
@jalandhar good question. we recommend storing the metadata store on an EBS drive so when the EC2 either dies or is terminated, the EBS drive is still available for the next EC2 you use for Dremio. In fact, this is how we architect the deployment of our [AWS Edition](https://aws.amazon.com/marketplace/pp/B08636P9HN) ---
**Author:** ![koolay](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/koolay/32/2407_2.png) [@koolay](https://community.dremio.com/u/koolay)\ **Post date:** [July 1, 2020, 3:23am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/5 "2020-07-01T03:23:40Z")
@balaji.ramaswamy Just like that `loki` has removed the dependency of [BoltDB](https://github.com/boltdb/bolt) in the new release. `loki` needs only s3 as storage now. > **[Loki v1.5.0 released, with no more dependency on a separate index store](https://grafana.com/blog/2020/05/20/loki-v1.5.0-released-with-no-more-dependency-on-a-separate-index-store/)** > > Check out the new features and improvements, including the BoltDB Shipper, which allows you to run Loki with one less dependency. ---
**Author:** ![jalandhar](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jalandhar/32/2553_2.png) [@jalandhar](https://community.dremio.com/u/jalandhar)\ **Post date:** [July 28, 2020, 6:04pm UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/6 "2020-07-28T18:04:30Z")
Thanks @jason, This problem is solved for us as of now as Dremio is taking backups from EBS volume and is able to read that backup when ever we launch a new cluster. Thanks, Jalandhar ---
**Author:** ![Ming](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@Ming](https://community.dremio.com/u/Ming)\ **Post date:** [November 29, 2022, 4:00am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/7 "2022-11-29T04:00:26Z")
@balaji.ramaswamy Do you have any update the supported external storage? Could it support distributed storage? Thanks! ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 29, 2022, 4:51am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/8 "2022-11-29T04:51:10Z")
@Ming With the unlimited splits feature, the metadata has to go to a distributed store ---
**Author:** ![Ming](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@Ming](https://community.dremio.com/u/Ming)\ **Post date:** [November 29, 2022, 9:12am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/9 "2022-11-29T09:12:01Z")
@balaji.ramaswamy Would you explain more in details? Does metadata mean data store in local? Does the rocksdb move to the distributed storage instead of the local driver or NAS? What is the benefit of unlimited splits feature? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 30, 2022, 5:09am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/10 "2022-11-30T05:09:54Z")
@Ming Have you gone through this to start with? [https://docs.dremio.com/software/advanced-administration/metadata-caching/#improved-metadata-refreshes-preview](https://docs.dremio.com/software/advanced-administration/metadata-caching/#improved-metadata-refreshes-preview) ---
**Author:** ![aaasif04](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aaasif04](https://community.dremio.com/u/aaasif04)\ **Post date:** [September 26, 2023, 7:26pm UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/11 "2023-09-26T19:26:50Z")
Hi guys, Can you please brief like what are the external storage sources (like S3, Postgres or MySQL etc) that can be configured as metadata store for dremio.? I understand that the EBS volume can be used, but can we use S3 as in Loki which is mentioned above.? I haven’t seen any configurations to support S3 as metadata store for dremio. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 27, 2023, 5:00pm UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/12 "2023-09-27T17:00:28Z")
@aaasif04 S3/Azure Storage (Gen2)/GCS/HDFS/NAS are all valid options > **[Configuring Distributed Storage | Dremio Documentation](https://docs.dremio.com/current/get-started/cluster-deployments/customizing-configuration/dremio-conf/dist-store-config/)** > > This topic specifies how to configure Dremio for distributed storage. ---
**Author:** ![aaasif04](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aaasif04](https://community.dremio.com/u/aaasif04)\ **Post date:** [October 5, 2023, 9:57am UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/13 "2023-10-05T09:57:35Z")
Hi, This storage only stores the accelerator, table, job result, download, and upload data, as i believe. Can we use S3 for metadata store. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 7, 2023, 6:51pm UTC](https://community.dremio.com/t/using-external-storage-like-s3-for-storing-dremios-metadata/5763/14 "2023-10-07T18:51:19Z")
@aaasif04 Once you are on 21.x and above, metadata folder should automatically get created under the same level as accelerator. Do you not see it? What version of Dremio are you running on? Can you please upload dremio.conf and screenshot of your support keys set