Dremio PDS Promotion takes much longer via API than UI

1

Hi,

We’ve been using the V3 API to promote folders to physical datasets and for the most part it has worked great.

However, promoting larger datasets takes a really long time and can make the API unusable. For example, when the amount of data being promoted is hundreds of GB or 1-2 TB in size, the dataset promotion can take hours.

I’ve noticed that when I promote a folder to a dataset in the UI, the operation appears to finish within seconds no matter the size. Is there some way to get this kind of speed when promoting datasets via the API?

Thanks!

Edit to add some more context - the large folders we’re promoting to PDS’s are mostly Delta Lake format - when promoting via the API we specify format type = Parquet, when promoting in the UI the Delta Lake format is auto detected.

2

Solved - specifying Delta lake format in the API request (as the UI does) increases the performance to the same level as the UI. I think this documentation can be updated to include Delta: Dremio

3

Great find @pbelanger we will add this as either an external article or a doc update