Appreciations to dremio team for developing such a solution. I have experimented with dremio and I must admit, it is a game changer!
coming to subject issue, I created reflections, used them to accelerate queries and switched off machines. Later I brought dremio up again and re-run a few queries. This time I realised the queries were not being accelerated. Job Acceleration tab said “Query was not accelerated. No reflections were available.”. Inspecting the accelerations I get:
All jobs are running well as expected. The failing part is loading of reflections after a restart of dremio instance. Every time I shutdown dremio instance and spin it up, I have to manually refresh reflections, otherwise jobs will not be accelerated. I trust you saw the How to reproduce in my above post.
If your docker serves as both coordinator and executor, the above behavior should not occur as the coordinator is down and reflection would not try to refresh, if your coordinator and executor are on different nodes and you are shutting down only the executor docker, the above failure is expected. If you are running both C and E on the same container then I would like to see why the reflections are turning red when you shutdown the docker. I am not asking you profiles of queries. If you see your screenshot to the right of footprint there is a history link, if you click that history link it will open only the failed reflection jobs, Can you please send the profile from one of those failed jobs for this particular reflection?
Luckily I have had to wait for the refreshing job be completed and now my reflections are reachable and usable. I think this is sorted out although I feel there is no need for reflections refresh every time dremio is restarted. Otherwise these reflections will be unusable till they are all refreshed (this is unnecessary workload on machine, users have to wait for refresh to complete otherwise they get served without acceleration).
@balaji.ramaswamy
Looks like I have to spend more days observing behavior of these reflections. Today I have started my single-node dremio cluster, all reflections are unreachable and strangely there is no refresh job queues. I had refresh settings at 2 days, now I have adjusted it to 1 week and triggered refresh manually.
Waited for refresh job to be completed then restarted dremio. On restart reflections became unreachable with no refresh jobs automatically triggered. This makes make conclude the earlier refresh was triggered probably because configured time for refresh (2 days) was due.
The profiles look ok, the next step is to investigate server.log. Under log/archive folder, look at the server.log for “2021-03-13” and search for materialization ID “7065e75c-e3c9-4fc0-9e58-0ed15cbc9884” and also reflection id “afb2a4af-675f-44a1-a391-79fa2b8ae8b0” and see if you find any errors?
Hi @balaji.ramaswamy
Thank you very much for the guidelines and your assistance. I followed logs and couldn’t find any meaningful error. So I decided to test with builds from various points as follows:
build dremio from github master branch, run distribution/server/... locally without shipping to docker. Follow steps in how to reproduce. Reflections were available after restart.
run dremio/dremio-oss docker image as in the how to reproduce. Reflections were available after restart (stupid of me I didn’t verify with this before opening the issue).
