Creating a PDS from a source with a fraction of the columns

Dremio Cloud
1

I am trying to promote a source to a PDS, it brings in all of the available columns, I tried limiting it with features but it still brings in all the columns. The data is in parquet and I am using the REST API for promotion.
Does anyone else have a similar use case and/or know how to do this?

2

Hi @Ismail ,

Are you promoting a dataset for the very first time? It will promote the entire dataset if it’s the first time, and after the metadata retrieval has been done you can run other SQL commands to limit the return. We have more documentation on dataset promotion here:

https://docs.dremio.com/cloud/sources/format/

3

So you’re saying, Promoting to a PDS would retrieve the entire metadata and we should and can only apply transformations in Tables and VDS?

4

Hi Ismail,

That’s correct. You can then create VDS from the source PDS, by selecting the columns you’d like in the new VDS

Cindy

5

Sure, Thanks a lot!

Syed Ismail