API Generate Physical Data Set On Top of Folder

1

I am attempting to create a PDS on top of a folder in my source. This is the results from using GetCatalog by path

image

I was following these instructions here to configure the Example to Promote Folder

This is what I came up with in PostMan.

image
image732×364 9.41 KB

Error Message

{
“errorMessage”: “Something went wrong”,
“moreInfo”: “HTTP 415 Unsupported Media Type”
}

2

You are probably missing the Content-Type header, which needs to be set to application/json. Looks like you have it set to raw in postman.

3

@doron – correct. Changed that and now got this.

Missing type id when trying to resolve subtype of [simple type, class com.dremio.dac.api.EnterpriseDataset]: missing type id property ‘entityType’
at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 13, column: 1]

Let me know

4

https://docs.dremio.com/rest-api/catalog/post-catalog-id.html#request-input shows that you need "entityType": "dataset" in your JSON since the entity you are creating is a dataset.

5

@doron --> You Rock! Fixed.

So for those playing along at home here was my problem.

So I ran this catalog by Path to get the folder ID that was in my source.

API path: GET: http://myURL:9047/api/v3/catalog/by-path/cdo_tz/TPD/Refinitivv1

The first time I ran it I got this response.

image

(I am not sure why this was :expressionless:)

I ran the same API call and got this response.

image
image1074×375 17.8 KB

(Notice the ID)

Then to create the Physical Data Set (PDS)

API Path: POST: http://myURL:9047/api/v3/catalog/16689883-3b8f-4579-9ccf-e14f06815e2e

image
image852×301 22.8 KB

And that is all.

6

PS: I figured out the whole ID thing. I believe it was because I converted the folder to a PDS manually and then deleted it. Thus it returned an id when I queried it again. Another example below.

Here is response from

API path: GET: http://YourURL:9047/api/v3/catalog/by-path/cdo_tz/TPD/CapIQv3

Response
{
“id”: “dremio:/cdo_tz/TPD/CapIQv3/“ciqAddress””,
“path”: [
“cdo_tz”,
“TPD”,
“CapIQv3”,
““ciqAddress””
],
“type”: “CONTAINER”,
“containerType”: “FOLDER”
},

For the ID to work you have to remove the punctuation (you probably know this but there are others like me)

So this id

“id”: “dremio:/cdo_tz/TPD/CapIQv3/“ciqAddress””,

replace : with %3A
replace / with %2F

“id”: “dremio%3A%2Fcdo_tz%2FTPD%2FCapIQv3%2FciqAddress”,

POST: http://YourURL:9047/api/v3/catalog/dremio%3A%2Fcdo_tz%2FTPD%2FCapIQv3%2FciqAddress

To Create PDS

Raw Body:

{
“entityType”: “dataset”,
“id”: “dremio%3A%2Fcdo_tz%2FTPD%2FCapIQv3%2FciqAddress”,
“path”: [
“cdo_tz”,
“TPD”,
“CapIQv3”,
“ciqAddress”
],

"type": "PHYSICAL_DATASET",
"format": {
    "type": "Parquet"
}

}