Reflections refresh

Reflections can be updated every 30 minutes ?

In the tool, the minimum I can set is hours.

I need to refresh my data every 30 minutes.

I don’t want to go straight to the database, that’s why I need to use reflections.

I think it’s fully managed in the Enterprise Edition.

If you are using OSS version, you may pass through API in order to refresh your reflection, then call the job through your scheduler.

If it can help you, there some Python script to refresh a reflection.
you’ll have to change user/password,dremioServer,port and pathReflection

import json
import requests 
import config

username = "user"
password = "psw"

headers = {'content-type':'application/json'}
dremioServer = 'http://yourServer:9047'

loginData = {'userName': username, 'password': password}
response = requests.post(dremioServer+'/apiv2/login', headers=headers, data=json.dumps(loginData))
data = json.loads(response.text)
token = data['token']
headers =  {'content-type':'application/json', 'authorization':'_dremio{authToken}'.format(authToken=token)}
def apiGet(endpoint):
    return json.loads(requests.get('{server}/api/v3/{endpoint}'.format(server=dremioServer, endpoint=endpoint), headers=headers).text)

def apiPost(endpoint, body=None):
    text = requests.post('{server}/api/v3/{endpoint}'.format(server=dremioServer, endpoint=endpoint), headers=headers, data=json.dumps(body)).text
    if (text):
        return json.loads(text)
    else:
        return None

def apiPut(endpoint, body=None):
    return requests.put('{server}/api/v3/{endpoint}'.format(server=dremioServer, endpoint=endpoint), headers=headers, data=json.dumps(body)).text

def apiDelete(endpoint):
    return requests.delete('{server}/api/v3/{endpoint}'.format(server=dremioServer, endpoint=endpoint), headers=headers)

def refreshReflection(pathReflection):
    # Should be call like 
    # refreshReflection('/DOCKER_VOLUMS/5M')
    try:
        idToRefresh = apiGet('catalog/by-path'+pathReflection)['id']
        apiPost('catalog/{id}/refresh'.format(id=idToRefresh))
    except Exception as e:
        print(f'Error reflection: {pathReflection}  ')

pathReflection = '/DOCKER_VOLUMS/TEST/RefreshReflection'
idToRefresh = apiGet('catalog/by-path'+pathReflection)['id']
print(f"ID to refresh is: {idToRefresh}")
apiPost('catalog/{id}/refresh'.format(id=idToRefresh))

Script will return you the ID
image

And you can check the reflection history:

if you want to schedule it, just run that through schedule every 30min.

Hope it helps.
Y.

1 Like

@jlcornejo On the PDS on which the VDS reflection is build you can change the frequency

I can’t find the option

it only allows me to set hours as a minimum level

thanks! :grinning_face_with_smiling_eyes:

I will try this option

There’s a support key to enable subhour refresh policies:

accelerator.enable.subhour.policies

Go to Admin Settings → Support … Add the key, and enable it.

After that, you will be able to select minutes as unit