HDFS source using Rest API

1

H Team,

Can you please help me to understand, how can I set MetaData Caching & Reflection Refresh properties using Rest API.

Basically looking for a way to set below properties using Rest API.

  • accelerationNeverExpire
  • accelerationRefreshPeriodMs
  • accelerationRefreshPeriodMs
  • accelerationNeverRefresh
  • datasetExpireAfterMs
  • datasetRefreshAfterMs
  • datasetUpdateMode

Thanks you in advance.

2

Hi,

You can use the catalog API to set these. Get the HDFS source:

GET localhost:9047/api/v3/catalog/0405f7bd-64ce-471c-88ff-2dca4e420492

Returns:

{
	"entityType": "source",
	...
	"metadataPolicy": {
		"authTTLMs": 86400000,
		"namesRefreshMs": 3600000,
		"datasetRefreshAfterMs": 3600000,
		"datasetExpireAfterMs": 10800000,
		"datasetUpdateMode": "PREFETCH_QUERIED",
		"deleteUnavailableDatasets": true,
		"autoPromoteDatasets": false
	},
	"accelerationGracePeriodMs": 10800000,
	"accelerationRefreshPeriodMs": 3600000,
	"accelerationNeverExpire": false,
	"accelerationNeverRefresh": false
}

You can then PUT to the same endpoint and update the metadata policy settings to your desired settings. Note that you will need to send the entire payload - our API currently does not support partial updates.

1 Like