Hi, we have Dremio 25.0.0 deployed on k8s. I’m not able to create any reflections. Creation fails with this error “The option ‘reflections.planning.exclude.file_based_incremental.iceberg.accelerations’ does not exist.”
Can you please help me to resolve this issue, thanks.
Jaro
@jaroslav_marko Looks like reflections.planning.exclude.file_based_incremental.iceberg.accelerations has been set that Dremio is complaining about. Are you able to send the profile as I would like to see the non-default values for support keys
@balaji.ramaswamy thank you for your reply. can you please navigate me what exactly should I send you? i’m not quite sure what profile you mean. thank you. jaro
@jaroslav_marko Please see link below to tutorial that explains how to download a Dremio query profile
sorry for late reply. in the meantime I have managed to fix the original error (by adjusting the Dremio source code, compiling it and replacing couple of .jar files in our running Dremio deployment).
So i was able to create a simple reflection on one field, then on another one. (attaching the query profiles for both).
the jobs look like they have finished successfully, but I don’t see any acceleration
effect and there is a following error in table section:
Reflection cannot accelerate, 2 refresh jobs attempts failed, will reattempt.
Thank you for you help.
Jaro
409d0f06-e02e-4e28-b98c-c1488ee0579b.zip (67.5 KB)
0300a095-0a38-4516-93b6-36d93d28c4f4.zip (47.3 KB)
@jaroslav_marko I see both refresh jobs you have attached is successful. What failed? Can you send me the profile of the failed job?
Hi @balaji.ramaswamy, indeed both jobs looks like successfully finished, but the Reflections is reporting an error.
This is my confusion.
Thanks for support, Jaro
@jaroslav_marko The erro could be on server.log on the coord, it it possible to share that when this reflection ran? Date is 2024-08-19
Hi @balaji.ramaswamy, we tried to redeploy the server to have clear logs, created a test table and tried to create reflection on this table. the job suceeded, but the reflection was not created as before.
unfortunately we have not found any error in server.log file.
btw. we are using bitnami image, are you perhaps aware of any issues with this image?
thanks, Jaro
Hi @balaji.ramaswamy,
we redeployed the entire instance using official oss 25 image, deleted all storages and nessie catalogues to avoid some potential conflicts with old data. i have loaded some tables back and tried to create reflection and have the same problem.
can you please help me with finding the issue?
reflections in dremio are key element for performance and without them i’m blocked with the project.
thanks a lot for help.
Jaro
8a46aa5c-6df2-45e7-84cf-c0d149a77097.zip (55.3 KB)
we have found this in logs:
WARN c.d.s.reflection.ReflectionManager - Successful REFLECTION REFRESH but already expired for materialization 401a2235-97f9-4f55-a586-715696150917/146881e2-4ca7-4142-b711-07d7a27f4c05
@jaroslav_marko Can you please share your source reflection tab settings and also the reflection setting on PDS nessie.AIQ.aiq_b2b_global?
Hi @balaji.ramaswamy, i have created the reflection using this command (but did the same in UI as well with the same error).
this is the Reflections tab from the aiq_b2b_global table:
can you please help me what do you mean by PDS reflection setting?
this?
@jaroslav_marko Did you try searching on your error in the forums? I answered this question before: Error while creating reflections in Dremio OSS 25.0 - #6 by Benny_Chow
Hi @Benny_Chow, yes I have found your solution, we have corrected the code, recompiled and included in the image. the original error is not longer appearing, but as i describe above the reflection job looks like finishing successfully, the reflection files appear to be in the storage, but the reflection itself shows error. there are no errors in server.log. i’m really struggling to resolve that and would very much appreciate any help.
thanks
Jaro
Can you respond to Bali’s comment above about:
Somehow you have a setting that is causing the materialization to expire very quickly.
Hi,
this is the reflections tab (at least i think that that’s what you mean)
but i’m not sure what are PDS settings, can you please guide me where i can find it? thanks
Jaro
I have changed the refresh settings to 1 hour and Expire setting to 3 hours and i was able to create one reflection successfully. however afterwards any other reflection, even table update ends with “Failure getting source [nessie].”.
attaching the profile:
097b4031-5128-41ff-b6e2-8f9de39ecf18.zip (24.1 KB)
How did you create the source “nessie”? Was it through the REST APIs? I’m aware of an issue where if the source is created through REST APIs the Refresh Settings show zero. That issue is being fixed right now. Do you have some background process updating the “nessie” source? That’s what the error in the query profile says.
Yes during re-deployment (as a last CICD job) we are uploading the sources and also users using API to keep the setup. (is there more consistent way to keep the setup after redeployment?)
But there are no other processing running other that this one.
The strange behavior is that after I have created my first reflection successfully the entire connection to nessie is not working. The setup is ok, i can see all tables and metadata, but whatever I run fails. i have restarted the deployment and it did not help.
any ideas, please?
Hi @Benny_Chow, just want to provide you with update. Originally I have simultaneously running 2 nessie instances on two separated storages. So I have tried to keep only one and remove the second one. It works for now. Thanks.
