How to Create PDS and VDS on Top of Minio Storage

Dremio University
1

Hi Team, our Dremio is on top of Minio, do we have document how to create of Dremio VDS and PDS on top of Minio Storage Dremio?

I understood we need a token to run POST curl command, and that works on our dremio (on top of MInio), same as we did hadoop Dremio.(run like this: curl http://ourdremiohost:9325/apiv2/login -X POST -H ‘Content-Type: application/json’ -d"{"userName":"myname","password":"mypwd"}")
Previoulsy when we create PDS and VDS on Hadoop, what we ran is like below, can you please guide the corresponding curl commands for the Minio Dremio.

our s3 parquet file path it is like this : s3a://TEST/testfolder1

  1. create PDS:
    curl -X POST ‘http://ourhost:9325/api/v3/catalog/dremio%3A%2FTEST_DEV%2FTESTDEV%2FPARQUET%2FFOLDER1%2FTEST’ -H ‘Content-Type: application/json’ -H ‘Authorization: _dremio4de0123456brfv1nc2’ -d ’
    {
    “entityType”: “dataset”,
    “id”: “dremio%3A%2FTEST_DEV%2FPARQUET%2FFOLDER1%2FTEST”,
    “path”: [
    “TEST_DEV”,
    “PARQUET”,
    “FOLDER1”,
    “TEST”
    ],
    “type”:“PHYSICAL_DATASET”,
    “format”: {
    “type”: “Parquet”
    }
    }’ -vvv

  2. create VDS:
    curl -X POST ‘http://ourhost:9325/api/v3/catalog/’ -H ‘Content-Type: application/json’ -H ‘Authorization: _dremio4de0123456brfv1nc2’ -d ’
    {
    “entityType”: “dataset”,
    “path”: [
    “FDL”,
    “FRM”,
    “TEST_VDS”
    ],
    “type”:“VIRTUAL_DATASET”,
    “sql”: "select * from "TEST" ",
    “sqlContext”: [“TEST_DEV”,“PARQUET”,“FOLDER1”]
    }’ -vvv

2

@dolphinlei The distributed storage should not matter. When you say create PDS, I assume you want to promote a folder that has PARQUET files as a PDS (purple grid icon)

The

For VDS, you simply use a SQL to create VIEW and if only want REST, you can use the POST SQL REST API