API Catalog explanation

1

Hi,

I was playing with Dremio API and I can’t understand how to work with catalogs.
After I get the login token, I do:
GET http://localhost:9047/api/v3/catalog
and the result is something like this:
{
“data”: [
{
“id”: “0fef5580-2192-4d8c-b1c0-d41b00ea134d”,
“path”: [
“@ds-ui”
],
“tag”: “0”,
“type”: “CONTAINER”,
“containerType”: “HOME”
},
{
“id”: “f0eda547-f146-481b-babe-ad156d870a11”,
“path”: [
“Test”
],
“tag”: “0”,
“type”: “CONTAINER”,
“containerType”: “SOURCE”
}
]
}

GET http://localhost:9047/api/v3/catalog/f0eda547-f146-481b-babe-ad156d870a11
{
“entityType”: “source”,
“config”: {
“path”: “/var/data/warehouse”
},
“state”: {
“status”: “good”,
“messages”: []
},
“id”: “f0eda547-f146-481b-babe-ad156d870a11”,
“tag”: “0”,
“type”: “NAS”,
“name”: “Test”,
“createdAt”: “2018-10-16T07:36:14.892Z”,
“metadataPolicy”: {
“authTTLMs”: 86400000,
“namesRefreshMs”: 3600000,
“datasetRefreshAfterMs”: 60000,
“datasetExpireAfterMs”: 180000,
“datasetUpdateMode”: “PREFETCH_QUERIED”
},
“accelerationGracePeriodMs”: 10800000,
“accelerationRefreshPeriodMs”: 3600000,
“accelerationNeverExpire”: false,
“accelerationNeverRefresh”: false,
“children”: [
{
“id”: “dremio:/Test/“prova””,
“path”: [
“Test”,
““prova””
],
“type”: “CONTAINER”,
“containerType”: “FOLDER”
}
]
}

Now comes the problem, i want to go down and reach the entity /Test/prova/000 that is an already defined dataset over a folder that contains a parquet file. But first i want to check /Test/“prova”.
Accordingly to the API reference I try getting a folder using it’s ID:
GET http://localhost:9047/api/v3/catalog/dremio%3A/Test/“prova” (there are percent22 before and after “prova” but this forum replaces them with double quotes)
This fails… and this is exactly the same url that the tutorial tries to get. The error is “404 Not found”.
To test if there is some url encoding problem, I tried with an external tool to play with escapes but none of my attempts worked, I always get 404.

Where’s the problem?

Thanks

2

Hi,

You need to URL encode the entire id - so it would be dremio%3A%2FTest%2F%E2%80%9Cprova%E2%80%9D. If you are on Dremio 2.1.x, you can also use our by path endpoint to avoid walking the tree.

3

You’re right, I was using “quote” function from python urllib, and that function does not encode slashes by default (you have to pass safe=’’ to escape also slahes). Doing that, the code works fine. “by path” approach however seems easier, so I’ll go with it.

Thank you

4

@doron I am working on taking preview on my dataset through API i know in dremio documentaion there is no such kind of API for that, so I am taking the Internal API for that,
“/apiv2/datasets/new_untitled_sql?newVersion=0009519538063077” , When i am using this API am facing most of time either 404 or 400 bad request, I know this is coming because of the Version ID need to Be unique format, May i know how to Solve the Bad request Issue in the Preview Response from Dremio. Could you tell me how to Archive this case.

5

Hi,

The v2 APIs are considered internal and may change at any time. Currently we do not have a stable API that allows previews. You could always add a LIMIT clause to your sql to approximate it.

6

Ok,Thanks for the Information @doron, We already using limit 10 , But even though we are facing this issue.

7

You can use the SQL API with LIMIT to approximate previews. Are oyu saying you still see very slow queries?

8

Thanks @doron for your Support, Yes Right now we are doing instead of Internal API we are using in Rest sql API.