# How to add new Data Lake source without using the UI

**URL:** <https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748>\
**Category:** Uncategorized\
**Created:** [July 16, 2021, 8:46pm UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748 "2021-07-16T20:46:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gpiguing](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@gpiguing](https://community.dremio.com/u/gpiguing)\
**Post date:** [July 16, 2021, 8:46pm UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/1 "2021-07-16T20:46:39Z")

</div>

I can add new data lake sources using the UI and selecting table stores or file stores. I would like to somehow add new data lake sources programmatically. Is there a config file saved somewhere in the dremio server that I can copy/reproduce to add new data lake sources? Or is there a way to have ansible or terraform add these sources on a newly installed dremio instance?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 20, 2021, 7:45am UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/2 "2021-07-20T07:45:36Z")

</div>

@gpiguing Have you tried [Dremio](https://docs.dremio.com/rest-api/catalog/post-catalog/)

---

<div class="post-metadata">

**Author:** ![gpiguing](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@gpiguing](https://community.dremio.com/u/gpiguing)\
**Post date:** [July 20, 2021, 10:41pm UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/3 "2021-07-20T22:41:17Z")

</div>

That is very helpful, thanks. I was able to add some source type using the REST API. However, how do you add a table store **Amazon Glue Catalog**? I dont see any examples in the link you provided. Its not even an option under the **source types** , but can definitely add it using the GUI.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 23, 2021, 5:59am UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/4 "2021-07-23T05:59:14Z")

</div>

@gpiguing let me check, but it seems like S3, have you tried?

---

<div class="post-metadata">

**Author:** ![gpiguing](https://avatars.discourse-cdn.com/v4/letter/g/a6a055/32.png) [@gpiguing](https://community.dremio.com/u/gpiguing)\
**Post date:** [July 23, 2021, 9:50am UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/5 "2021-07-23T09:50:47Z")

</div>

@balaji.ramaswamy I have tried adding an S3 source using the REST API. It works fine. It works exactly just like when I added an S3 source using the GUI. Adding a source using the Amazon Glue Catalog (with the GUI) acts differently than adding an S3 source. For my case, adding an S3 adds directories leading to the parquet files. Where as adding an Amazon Glue Catalog source adds tables. I want to add an Amazon Glue Catalog with the REST API, and have the same result as when I use the GUI.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 23, 2021, 7:44pm UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/6 "2021-07-23T19:44:20Z")

</div>

@gpiguing Here is an example

```auto
    "entityType": "source",
    "config": {
        "regionNameSelection": "US_WEST_2",
        "accessKey": "",
        "accessSecret": "",
        "secure": true,
        "credentialType": "ACCESS_KEY",
        "enableAsync": true,
        "isCachingEnabled": true,
        "maxCacheSpacePct": 100
    },
    "type": "AWSGLUE",
    "name": "glue",
    "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 60000,
        "datasetRefreshAfterMs": 3600000,
        "datasetExpireAfterMs": 10800000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true,
        "autoPromoteDatasets": false
    },
    "accelerationGracePeriodMs": 10800000,
    "accelerationRefreshPeriodMs": 3600000,
    "accelerationNeverExpire": false,
    "accelerationNeverRefresh": false
}
```

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 23, 2021, 7:45pm UTC](https://community.dremio.com/t/how-to-add-new-data-lake-source-without-using-the-ui/7748/7 "2021-07-23T19:45:38Z")

</div>

@gpiguing Here is a github page from one of the SA’s at Dremio

> **[GitHub - dremio-brock/dremio-postman: collection of Postman objects for Dremio](https://github.com/dremio-brock/dremio-postman)**
>
> collection of Postman objects for Dremio. Contribute to dremio-brock/dremio-postman development by creating an account on GitHub.

@b-rock