Creation of dataset failing through API

Hi team!

I’m having troubles creating a dataset through the API.

What I’m doing:

  • I create a new S3 source manually through the Dremio UI
  • I the want to create a number of datasets (which reside in S3) in a space through the API. When I try, I get a “table not found error”

Strangely enough, when I create the same dataset through the Dremio UI it works. If I then retry through the API, it work also. As if it “knows” the s3 object from that moment onwards. For your information, the S3 objects are json files.

Do I need to call a specific other API first?

Thanks!

Hi,

To create datasets in S3 you have to promote them:

  • Get the file/folder on S3 using its path.
  • Promote the file/folder on S3 using the promotion API using the ID returned by the previous call.

I tried this, but it does not seem to work. When I do this for a file on S3 that I never accessed through the Dremio UI, I get this back:

{
“entityType”: “file”,
“id”: “dremio:/S3/folder/“test.json””,
“path”: [
“S3”,
“folder”,
““test.json””
]
}

That is correct - that is the file you want to promote.

So you know want to follow [this](http://docs.dremio.com/rest-api/catalog/post-catalog-id.html):
Post to that endpoint with the id and the following body:

{
  "entityType": "dataset",
  "id": "dremio:/S3/folder/\"test.json\"",
  "path": [
    "S3",
    "folder",
    "“test.json”"
  ],
  "format": {
    "type": "JSON" 
  }
}

Do I need to URL encode the id in a particular fashion to supply to the post url? I tried different versions but either I get a 404 or an error that “Promoting can only create physical datasets.”

Got it to work! Thx!!

Yes you ahve to url encode the ID in the url. We are working on better API documentation for this!

For anyone else who hits this problem, specifically, you have to also encode any forward slashes in the id. So using the example above, POST to /catalog/dremio%3A%2FS3%2Ffolder%2F%22test.json%22

In python 3 you can use:

requests.utils.quote(string, safe="")

To correctly url encode ids.

1 Like

@doron I tried doing the same with javascript - it did not work for me. You can refer the details in the link below