Set reflections to not refresh automatically using API

1

When creating physical datasets via the API, we inject a default accelerationPolicy and specify that the method should be INCREMENTAL vs the default FULL.

We would like to also specify that reflections should never refresh or expire via the API.

image

Is this possible?

We will be triggering the refresh for our reflections when new files have been added to the datalake using an external script, so queries should consider whatever reflections are there to be the relevant and up to date ones.

2

You can set never expire/never refresh on a source - http://docs.dremio.com/rest-api/catalog/container-source.html

3

Hi Doron

Thank you, this will work for my intended use in this case.

For future reference: I assume that this means that no dataset within that source would be able to auto refresh/expire?

4

Dataset level settings will override Source settings, but if you don’t change the settings on the dataset it will default to the source settings.

5

Perfect, thank you for the insight.

6

@doron Is there any API which lets us update reflection refresh setting at the dataset level?

7

@nehal,

You can do a PUT catalog/{dataset ID} and specify a new/updated object for the “accelerationRefreshPolicy” field.

1 Like
8

@ben Works, Thank you

9

@doron How we override never expire/never refresh for a dataset using REST API?

I only found expire/never refresh property at datasource level in the documentation

10

@nehal, you should be able to set the same “accelerationRefreshPolicy” for an individual physical dataset

11

@ben but accelerationRefreshPolicy at dataset level only have these 4 properties

{
  "refreshPeriodMs": Number,
  "gracePeriodMs": Number,
  "method": String ["FULL", "INCREMENTAL"],
  "refreshField": String [optional]
}

How do I update the following at dataset level

  1. accelerationNeverExpire: true
  2. accelerationNeverRefresh: true
12

@nehal May I know if you found an answer to this? I was looking for a resolution for the same too.

13

@reshma.cs,

There is an issue with the API that needs to be addressed, which prevents you from doing this.

See comments here for the workaround