How to add new Data Lake source without using the UI

1

I can add new data lake sources using the UI and selecting table stores or file stores. I would like to somehow add new data lake sources programmatically. Is there a config file saved somewhere in the dremio server that I can copy/reproduce to add new data lake sources? Or is there a way to have ansible or terraform add these sources on a newly installed dremio instance?

2

@gpiguing Have you tried Dremio

3

That is very helpful, thanks. I was able to add some source type using the REST API. However, how do you add a table store Amazon Glue Catalog? I dont see any examples in the link you provided. Its not even an option under the source types, but can definitely add it using the GUI.

4

@gpiguing let me check, but it seems like S3, have you tried?

5

@balaji.ramaswamy I have tried adding an S3 source using the REST API. It works fine. It works exactly just like when I added an S3 source using the GUI. Adding a source using the Amazon Glue Catalog (with the GUI) acts differently than adding an S3 source. For my case, adding an S3 adds directories leading to the parquet files. Where as adding an Amazon Glue Catalog source adds tables. I want to add an Amazon Glue Catalog with the REST API, and have the same result as when I use the GUI.

6

@gpiguing Here is an example

    "entityType": "source",
    "config": {
        "regionNameSelection": "US_WEST_2",
        "accessKey": "",
        "accessSecret": "",
        "secure": true,
        "credentialType": "ACCESS_KEY",
        "enableAsync": true,
        "isCachingEnabled": true,
        "maxCacheSpacePct": 100
    },
    "type": "AWSGLUE",
    "name": "glue",
    "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 60000,
        "datasetRefreshAfterMs": 3600000,
        "datasetExpireAfterMs": 10800000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true,
        "autoPromoteDatasets": false
    },
    "accelerationGracePeriodMs": 10800000,
    "accelerationRefreshPeriodMs": 3600000,
    "accelerationNeverExpire": false,
    "accelerationNeverRefresh": false
}
7

@gpiguing Here is a github page from one of the SA’s at Dremio

@b-rock