Refreshing physical datasets in external s3 source failing

Dremio University
1

I am trying to refresh metadata policy of physical datasets in external s3 sources in dremio using dremio_client and its throwing me the following error

DremioBadRequestException: Requested object does not exist on entity : 400 Client Error: Bad Request for url: https://dremio.whoop.com/api/v3/catalog_id

I am using dremio simple client update_catalog method to update the metadata policy of a PDS in s3

import dremio_client.dremio_simple_client as sc
import json

 payload = {
                "entityType": "dataset",
                "id": "dc0a1a67-aa5e-4c86-8c8f-8a7f245013cd",
                "path": ['dremios3', 'com.xxxx.dremio', 'metrics'],
                "type": "DATASET",
                "accelerationRefreshPolicy": {
                    "method": "FULL",
                    # "neverRefresh": True,
                    # "neverExpire": True,
                },
            }
    resp = sc.update_catalog(auth_token, _DREMIO_HOST, 'dc0a1a67-aa5e-4c86-8c8f-8a7f245013cd', json=json.dumps(payload))

This is the configuration I am using to update the metadata policy but not sure what am I doing wrong. Any help would be appreciated

2

@abachu The policy on a PDS is the reflection refresh schedule and not metadata refresh. To refresh metadata for a PDS you have to use the ALTER PDS <PDS_NAME> REFRESH METADATA SQL command

https://docs.dremio.com/sql-reference/sql-commands/datasets/#refreshing-physical-dataset-metadata