Reflection are deleted mysterious

1

Hello, currently we have a strange error with dremio, we create raw reflection in some physical dataset, approximate 6 months ago, but now this reflections magically not exist

2

@dacopan,

What do you mean by “magically not exist”?

Can you find any record of the reflection or the dataset that it is configured on in the sys.reflections table?

Can you locate the last job that used the reflection?

3

the reflecton disappeared in the ui config of PDS show as not active, and in sys.reflections not exist any reference for it.

we have a airflow DAG that consume dremio api for check if all reflection are ok, but this reflection not exist from one day to next day disappeared without any modification.

Can you locate the last job that used the reflection?

I can’t because I dont have the old reflection id

The original datasource is PostgreSql

4

To find the last time the reflection was used, you could search the jobs for the the dataset on which the reflection is based.

How many admin users have access to Dremio? Is it possible that someone simply disabled the reflection at some point in the recent past?

5

@ben only one user (me), I find the last job and run sucess, but apparently during the next refresh the datasource was offline (during database restore), can be it the cause that the reflection was deleted by dremio itself, currently we have 70% of reflection with this problem

6

@dacopan

This can be caused due to two reasons

  1. If the reflection is on the PDS and the source settings have a metadata impacting change, then all reflection on all the PDS’s on that source will be deleted. Workaround is to create a VDS as select * from PDS and create the reflection on the VDS

  2. If it is a file system source like S3/ADLS and you have “Remove dataset definitions if underlying data is unavailable.” checked, which is the default (See metadata tab under the source settings) then if the source is offline and metadata refresh runs, there is a possibility that reflections would get dropped, again only on PDS, so workaround is to create reflection on the VDS

I have also seen a bug where source is offline and we do not show the reflections on the PDS during that time. Does it not come back after the source is online?