Creation of dataset failing through API

1

Hi team!

I’m having troubles creating a dataset through the API.

What I’m doing:

  • I create a new S3 source manually through the Dremio UI
  • I the want to create a number of datasets (which reside in S3) in a space through the API. When I try, I get a “table not found error”

Strangely enough, when I create the same dataset through the Dremio UI it works. If I then retry through the API, it work also. As if it “knows” the s3 object from that moment onwards. For your information, the S3 objects are json files.

Do I need to call a specific other API first?

Thanks!

2

Hi,

To create datasets in S3 you have to promote them:

  • Get the file/folder on S3 using its path.
  • Promote the file/folder on S3 using the promotion API using the ID returned by the previous call.
3

I tried this, but it does not seem to work. When I do this for a file on S3 that I never accessed through the Dremio UI, I get this back:

{
“entityType”: “file”,
“id”: “dremio:/S3/folder/“test.json””,
“path”: [
“S3”,
“folder”,
““test.json””
]
}

4

That is correct - that is the file you want to promote.

So you know want to follow [this](http://docs.dremio.com/rest-api/catalog/post-catalog-id.html):
Post to that endpoint with the id and the following body:

{
  "entityType": "dataset",
  "id": "dremio:/S3/folder/\"test.json\"",
  "path": [
    "S3",
    "folder",
    "“test.json”"
  ],
  "format": {
    "type": "JSON" 
  }
}
5

Do I need to URL encode the id in a particular fashion to supply to the post url? I tried different versions but either I get a 404 or an error that “Promoting can only create physical datasets.”

6

Got it to work! Thx!!

7

Yes you ahve to url encode the ID in the url. We are working on better API documentation for this!

8

For anyone else who hits this problem, specifically, you have to also encode any forward slashes in the id. So using the example above, POST to /catalog/dremio%3A%2FS3%2Ffolder%2F%22test.json%22

9

In python 3 you can use:

requests.utils.quote(string, safe="")

To correctly url encode ids.

1 Like
10

@doron I tried doing the same with javascript - it did not work for me. You can refer the details in the link below