# How to create VDS on top of parquet file folder(has partition columns) **URL:** \ **Category:** Uncategorized\ **Created:** [August 20, 2023, 2:21pm UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905 "2023-08-20T14:21:46Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [August 20, 2023, 2:21pm UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/1 "2023-08-20T14:21:46Z")
I want to run some curl command to create a VDS on top of a HDFS parquet folder path, can you please let me know the curl command. input might needed: hdfs parquet folder path: /my/src/path VDS path and view name: DEV/FRM/TEST\_VDS Dremio UI username/password: username/pwd Dremio UI link: [http://mcdevdremio:9766/](http://mcdevdremio:9766/) Can you please provide the command that can make VDS DEV.FRM.TEST\_VDS. the parquet folder file has partition columns(date\_yyyymmdd and catg), so the folder structure like below: ├── date\_yyyymmdd=20120710 │ ├── catg=DIV │ │ └── part-00108-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet │ ├── catg=IP │ │ └── part-00191-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet │ └── catg=XT │ └── part-00081-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet ├── date\_yyyymmdd=20120711 │ ├── catg=DIV │ │ └── part-00091-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet │ ├── catg=IP │ │ └── part-00150-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet │ └── catg=XT │ └── part-00064-7340bbf6-43a4-4574-a9bd-f2eb8c0c929c.c000.snappy.parquet ---
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [August 20, 2023, 2:31pm UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/2 "2023-08-20T14:31:52Z")
should the step be: 1.get token for auth, 2.generate PDS on top of parquet path by curl command 3.generate VDS on top of PDS by curl command? If yes, Can you please send the command for each? Thanks. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [August 22, 2023, 3:47am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/3 "2023-08-22T03:47:29Z")
@dolphinlei Yes you need token - First promote the PDS using API or enable “Automatically format files into physical datasets when users issue queries.” in the source metadata properties, API will be [Table | Dremio Documentation](https://docs.dremio.com/24.0.x/reference/api/catalog/table#formatting-a-file-or-folder-as-a-table) - Then you can use the POST SQL API to create the VDS > **[SQL | Dremio Documentation](https://docs.dremio.com/24.0.x/reference/api/sql/)** > > Use the SQL API to submit SQL queries. The response contains the ID for the job associated with the SQL query. Use the job ID in Job API requests to get more information about the job, including results. > **[SQL | Dremio Documentation](https://docs.dremio.com/24.0.x/reference/api/sql/)** > > Use the SQL API to submit SQL queries. The response contains the ID for the job associated with the SQL query. Use the job ID in Job API requests to get more information about the job, including results. ---
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [August 29, 2023, 3:52am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/4 "2023-08-29T03:52:35Z")
Hi Balaji, can you please share the steps how to generate token to run those commands? ---
**Author:** ![AndyH](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/andyh/32/4510_2.png) [@AndyH](https://community.dremio.com/u/AndyH)\ **Post date:** [August 29, 2023, 7:50am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/5 "2023-08-29T07:50:07Z")
To generate a token, you need to use the Login API: > **[API Reference | Dremio Documentation](https://docs.dremio.com/current/reference/api/#authentication-token)** > > The Dremio REST API is organized by resource types such as sources and is designed around RESTful principles. HTTP GET is used to retrieve existing resources, So, with cUrl, that would be : `curl http://mcdevdremio:9766/apiv2/login -X POST -H 'Content-Type: application/json' -d"{\"userName\":\"username\",\"password\":\"pwd\"}" ` You then just extract the value of the `token` field from the response and concatenate it with `_dremio` So your token ends up looking something like : `_dremiogrd44us5gkbuo01m2s61i80b7o` ---
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [September 17, 2023, 9:22am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/6 "2023-09-17T09:22:54Z")
@AndyH thanks for the response, I can generate token: and get the token like “psk4dfrghtyu7f6n2tqfgctj”, and retrieved my Dremio token “\_dremiopsk4dfrghtyu7f6n2tqfgctj” Is the below command to generate PDS in Dremio? curl -X POST [http://mydremiohost:9700/api/v3/catalog/DEV/dl/STORE/PARQUET/SAMPLE\_TEMP](http://mydremiohost:9700/api/v3/catalog/DEV/dl/STORE/PARQUET/SAMPLE_TEMP) -H ‘Content-Type: application/json’ -H ‘Authorization: Bearer \_dremiopsk4dfrghtyu7f6n2tqfgctj’ -d ’ { “entityType”: “dataset”, “id”: “dremio%3A%2FDEV%2Fdl%2FSTORE%2FPARQUET%2FSAMPLE\_TEMP”, “path”: [ “DEV”, “dl”, “STORE”, “PARQUET”, “SAMPLE\_TEMP” ], “type”:“PHYSICAL\_DATASET”, “format”: { “type”: “Parquet” } }’ I ran an got the “HTTP 404 Not Found” error. I am sure the parquet path is correct and there is parquet file there. please advise. ---
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [September 22, 2023, 3:11am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/8 "2023-09-22T03:11:19Z")
Hi team, can you please advise? ---
**Author:** ![AndyH](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/andyh/32/4510_2.png) [@AndyH](https://community.dremio.com/u/AndyH)\ **Post date:** [September 26, 2023, 11:27am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/9 "2023-09-26T11:27:32Z")
Hi, The 404 means that the URL you are POSTing to is invalid. When you POST a dataset to /api/v3/catalog the rest of the URL needs to be the ID of the dataset. You are currently putting in the path elements as the URL. If you look at the API docs here: [Table | Dremio Documentation](https://docs.dremio.com/24.0.x/reference/api/catalog/table) and scroll down to the “Example Request for Excel format type” section, you can see that to specify a dataset you need to transform the path to the dataset into a URL encoded string. The example in the docs is : `dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University%2Foracle-departments.xlsx` Where you have the “id” as the body, you should be putting that as the URL. E.g.: `curl -X POST http://mydremiohost:9700/api/v3/catalog/dremio%3A%2FDEV%2Fdl%2FSTORE%2FPARQUET%2FSAMPLE_TEMP` Don’t include the id in the body, follow the body example in the docs, so your body should be : ```auto { "entityType": "dataset", "path": [ "DEV", "dl", "STORE", "PARQUET", "SAMPLE_TEMP" ], "type": "PHYSICAL_DATASET", "format": { "type": "Parquet" } } ``` Obviously, I’ve not tested the above as I don’t have the same data structure, but let me know if that works ---
**Author:** ![dolphinlei](https://avatars.discourse-cdn.com/v4/letter/d/e47c2d/32.png) [@dolphinlei](https://community.dremio.com/u/dolphinlei)\ **Post date:** [October 9, 2023, 10:15am UTC](https://community.dremio.com/t/how-to-create-vds-on-top-of-parquet-file-folder-has-partition-columns/10905/10 "2023-10-09T10:15:23Z")
thanks, have tried, it works. But one point is when run the curl command , need to remove “Bearer” from Authorization part, so the command is below: curl -X POST ‘http://mydremio:96600/api/v3/catalog/dremio%3A%2FDEV%2Fdl%2Ffin%2FSTORE%2FPARQUET%2F%SAMPLE\_TEST\_DREMIO\_CREATE33’ -H ‘Content-Type: application/json’ -H ‘Authorization: \_dremio5ret45peugqwerftgtyuuh6788’ -d ’ { “entityType”: “dataset”, “id”: “dremio%3A%2FDEV%2Fdl%2Ffin%2FSTORE%2FPARQUET%2F%SAMPLE\_TEST\_DREMIO\_CREATE33”, “path”: [ “DEV”, “dl”, “fin”, “STORE”, “PARQUET”, “SAMPLE\_TEST\_DREMIO\_CREATE33” ], “type”:“PHYSICAL\_DATASET”, “format”: { “type”: “Parquet” } }’ -vvv