REST API - Not able to promote a file to physical dataset

1

I would like to create a dataset in Dremio via REST API doing the following. Convert file to a DataSet
I follow the documention -
Step 1. Get the id of the physical file via REST API:
id: dremio:/cuddle_dev_s3-storage/“newstoreglidepathcsv_1568879249742”

Step 2: POST /api/v3/catalog/encoded{id}
-> /api/v3/catalog/dremio%3A%2Fcuddle_dev_s3-storage%2F%22newstoreglidepathcsv_1568879249742%22

Still its throws 404 error. Please help me with this.
I am using encodeURIComponent in javascript to encode the ID

2

I built a local structure that resembles your s3 setup. I ran this to get the id:

GET http://localhost:9047/api/v3/catalog/by-path/nas/cuddle_dev_s3-storage/newstoreglidepathcsv_1568879249742

The id in the response is "dremio:/nas/cuddle_dev_s3-storage/\"newstoreglidepathcsv_1568879249742\"".

I ran this in the JS console:

encodeURIComponent("dremio:/nas/cuddle_dev_s3-storage/\"newstoreglidepathcsv_1568879249742\"")

which outputs:

"dremio%3A%2Fnas%2Fcuddle_dev_s3-storage%2F%22newstoreglidepathcsv_1568879249742%22"

I can then do a GET to get the file:

GET http://localhost:9047/api/v3/catalog/dremio%3A%2Fnas%2Fcuddle_dev_s3-storage%2F%22newstoreglidepathcsv_1568879249742%22

Which returns me the file.

I can promote it using CURL for example like this:

curl --request POST \
  --url http://localhost:9047/api/v3/catalog/dremio%3A%2Fnas%2Fcuddle_dev_s3-storage%2F%22newstoreglidepathcsv_1568879249742%22 \
  --header 'authorization: _dremio6d7sgacg5slt3ff5jh07p64sdi' \
  --header 'content-type: application/json' \
  --header 'x-ms-date:  Fri, 27 Jul 2018 16:26:53 GMT' \
  --header 'x-ms-version: 2017-02-22' \
  --data '{
  "entityType": "dataset",
  "id": "dremio:/nas/cuddle_dev_s3-storage/\"newstoreglidepathcsv_1568879249742\"",
  "path": [
    "nas",
    "cuddle_dev_s3-storage",
    "\"newstoreglidepathcsv_1568879249742\""
  ],
	   "type": "PHYSICAL_DATASET",
    "format": {
        "type": "Text",
        "fieldDelimiter": "|",
        "lineDelimiter": "\n",
        "escape": "\"",
        "skipFirstLine": false,
        "extractHeader": false,
        "trimHeader": false,
        "autoGenerateColumnNames": true
    }
}'
1 Like
3

@doron Thank you for the quick response.
I had followed exact same steps as you mentioned above. Turns out it is the issue with our nginx server, it does not forward the request as it is.

Thank you Doron!