How are the reflection times controlled?

1

The documentation has numbers something like below. Wondering in which order these are affected?

  "metadataPolicy": {
    "namesRefreshMs": 3600000,
    "datasetRefreshAfterMs": 3600000,
    "datasetExpireAfterMs": 10800000,
    "datasetUpdateMode": "PREFETCH",
    "deleteUnavailableDatasets": true,
    "autoPromoteDatasets": false
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false

For example, when refresh period is set (datasetRefreshAfterMs, accelerationRefreshPeriodMs) what is the need for expire and grace period? (datasetExpireAfterMs, accelerationGracePeriodMs)

There seems to be some underlying assumption here that the refresh will (some how) fail at that periodic interval - is that so?

What happens even if the grace period and expire also fail, in that case?

2

@KrishnaPG Expire after is when the metadata would be marked as expired if the total time since the last metadata refresh > expire after, if there are datasets with expired metadata and a query comes in, the planner would need metadata to be fetched during query planning. Datasets should never have expired metadata unless the metadata refreshes have been failing or the total time taken to refresh the source > expire after setting, will get back on the grace setting