I am trying out Dremio cloud and am incurring excess AWS costs due to the preview engine intermittently starting up in the background.
Although the documentation appears to state that you are able to manually disable the preview engine, or any other engine under Project Settings → Engines, this appears to not be the case. When I attempt to toggle the referenced button, I receive a message from the UI indicating that I cannot turn off the preview engine because it is used to “run metadata refresh queries.”
Is there any way for me to actually disable the preview engine so that I stop incurring excess AWS compute costs due to Dremio?
Hi @jlbar, the preview engine should have a minimum replica of 0 by default so even though the engine can’t be disabled, there should be no instances that are running constantly if no jobs are running. For more information, see the first few paragraphs of Managing Engines | Dremio Documentation.
Do you have frequent metadata refreshes or automatic optimizations enabled that would trigger the preview engine to spin up instances frequently? This can be determined by going to the Jobs page and filtering down to the Query Type: Internal. If this is the case, then reducing the frequency at which these jobs run will decrease the number of times the preview engine is being spun up and allow you to manage compute costs.
You were correct, there seems to be a pair of automated jobs that run every hour. The SQL statements begin with “REFRESH REFLECTION” and “VACUUM TABLE”.
As these appear to be related to the reflection process, I disabled the associated reflection under the Reflections tab in the UI. Hopefully this settles it.
Great! Hopefully that does the trick. If you continue to see these jobs running, let me know and we can help determine if there’s additional configuration that you need to update.
I think the issue is that any source you add will by default have a reflection refresh policy of “Refresh every 1 hour”. So, if you have reflections defined on tables from this source, this will cause the preview engine to wake up hourly and refresh those reflections. You can go to source settings to change the refresh settings to “Never refresh”. You can also override the refresh policy on the source tables directly.