Refresh Reflection using API call

1

Hi!

I was wondering if there is a way to trigger the refresh on certain Reflections via the Dremio API. Same concept as the “Refresh Now” button, just programatically?

When I finish loading data to the underlying Hive table, I would like to refresh immediately without waiting for the automatic hourly…

2

My bad… Sorry I was looking at the Jobs and Reflection endpoints and not in Catalog.

3

Hi dataedgehungary,

I have requested this enhancement and hope they add it soon. In the meantime, the only way I know to refresh a reflection via the API is to disable the reflection and then re-enable it. Like this (python) –

    reflection = apiGet(f'reflection/{reflection_id}')
    reflection['enabled'] = False
    response = apiPut(f'reflection/{reflection_id}', reflection)

    reflection = apiGet(f'reflection/{reflection_id}')
    reflection['enabled'] = True
    response = apiPut(f'reflection/{reflection_id}', reflection)
1 Like
4

as a simple workaround - change the name of reflection will force refresh
I simply add timestamp to the name

5

Hi,
if i disable and enable reflections via API, is possible to use the incremental refresh option or the refresh will be full?

thanks

6

when disable/enable the reflection is recreated, so no incremntal use.

7

@dataedgehungary
when I tried your method to disable then reenable, it works. It keeps the same id but changes the tag and from the looks of it does a complete refresh, and it deletes the previous reflection. When the reflection is disabled the UI shows a gray disk icon and size of 0 bytes reflecting a deletion has occurred. I’m not sure if the deletion occurs immediate when disabling, if so it should be called “truncate” instead of “delete” I would suggest they change the name. From what I can tell Dremio uses the tag as a version indicator, omitting it will return Invalid reflection: reflection version required.

I tried to use PUT update a specific raw reflection as described on Dremio API Docs. I am able to get 200 OK but it does not actually refresh it, according to the jobs view, the log, and the UI nothing is triggered.

I have been able to get reflections to fail then refresh them and get the same size footprint, which suggests they did not fail except from the standpoint where they were queued for the maximum wait time then marked as failed, after which the previous successful reflection is lost when I have to reenable them to remove the failure indicator. I think I am also having trouble with the failure count settings propagating from my admin console at the reflections tab (in AWS marketplace version, not visible in Docker version). If I set it low = 1 then it fails once, the reflection stays in failed mode forever, cannot be refreshed via the UI.

My recommendation is Dremio needs to:

  1. change wording for disable reflection to truncate
  2. get the PUT reflection API call to perform the same as the refresh now in the API
  3. explain why the version tag should change when disabling / enabling a reflection whose design has not changed.
  4. add to the UI a few things:
  • refresh all reflections
  • retry all failed reflections
  • force refresh of multiple reflections (checkboxes, run)
8

there seem to be an issue with latest ver 4.8 of dremio.
refresh reflection on VD used to work with a single put api by simply changing the name or two put api’s by setting enabled to False&True.
with ver 4.8 put request gives 200 OK, but nothing happens. it simply updates an entry in sys.reflections table without actually refreshing.

disable/enable from UI is working though.

9

@smora

Was there a reflection job created?

10

@balaji.ramaswamy

No job created,
get reflection API shows the change to name/enabled columns, but refresh is not triggered.

11

@smora

Thanks,

We will test this and get back to you

12

@balaji.ramaswamy

We have the same issue with v4.8. Do you already have any progress on this?

13

@joelhansen

Were you changing the name the reflection ? and also method 2 was to disable/enable reflection?

14

@balaji.ramaswamy
I tried both options, none of them are working.
upgraded to v4.9.0 today, same issue in this version also.

Thanks

15

@balaji.ramaswamy
I also tried both options and none working.
Later I found out that it mostly works, but it takes a long time to enable tha save button. I make any kind of changes to the reflection, and only after about 2 minutes the save button is enabled.

16

looks like 2 min wait is doing the magic.
disable + sleep(120) + enable is triggering the refresh correctly through API.

17

Hey folks, just seeing if there has been any update on this? We’re in the middle of an enterprise deployment and are looking to use this functionality…it does seem the disable/enable works for me, just seems clunky…thanks in advance

18

@RMartin

Disable drops the reflection while enable creates a new reflection

Thanks
Bali

19

Hi @balaji.ramaswamy ,

Is there any way to update the same reflection via api, without disabling and enabling?
We provide data from the Dremio table for company APIs, and we cannot run out of accelerating queries while another reflection is created.
We are trying to use the call PUT /api/v3/reflection/{id}, but the reflection is not updated.

Thanks,
@joelhansen

20

@joelhansen

To change the reflection definition you need to do PUT /reflection · Dremio but if the reflection is running as you said the refresh job would get cancelled

To refresh a reflection use POST /catalog/{id}/refresh · Dremio