Appreciations to dremio team for developing such a solution. I have experimented with dremio and I must admit, it is a game changer!
coming to subject issue, I created reflections, used them to accelerate queries and switched off machines. Later I brought dremio up again and re-run a few queries. This time I realised the queries were not being accelerated. Job Acceleration tab said “Query was not accelerated. No reflections were available.”. Inspecting the accelerations I get:
How to reproduce
- Pull and spin dremio/dremio-oss docker container with a volume mounted to /opt/dremio/data for persistence metadata storage.
- Add sample data source
- Grab any dataset from sample data source, format to create a PDS
- Create a VDS from PDS above
- Create a reflection for the VDS (you may also create a reflection for the PDS), wait until reflection creation job is completed
- shutdown the container, spin it up again
- check reflection status, now it is unavailable
Can you please click on the small history link and send us the profile of the failed job?
All jobs are running well as expected. The failing part is loading of reflections after a restart of dremio instance. Every time I shutdown dremio instance and spin it up, I have to manually refresh reflections, otherwise jobs will not be accelerated. I trust you saw the How to reproduce in my above post.
If your docker serves as both coordinator and executor, the above behavior should not occur as the coordinator is down and reflection would not try to refresh, if your coordinator and executor are on different nodes and you are shutting down only the executor docker, the above failure is expected. If you are running both C and E on the same container then I would like to see why the reflections are turning red when you shutdown the docker. I am not asking you profiles of queries. If you see your screenshot to the right of footprint there is a history link, if you click that history link it will open only the failed reflection jobs, Can you please send the profile from one of those failed jobs for this particular reflection?
Thanks for quick response balaji
Please find attachedcea6f722-7517-416a-aa24-beef43e36b55.zip (36.5 KB)
The reflection accessed from data set it is reported as unreachable, on jobs I see below screen
Luckily I have had to wait for the refreshing job be completed and now my reflections are reachable and usable. I think this is sorted out although I feel there is no need for reflections refresh every time dremio is restarted. Otherwise these reflections will be unusable till they are all refreshed (this is unnecessary workload on machine, users have to wait for refresh to complete otherwise they get served without acceleration).
Thanks for the tip balaji
You can change the refresh interval on the PDS
Looks like I have to spend more days observing behavior of these reflections. Today I have started my single-node dremio cluster, all reflections are unreachable and strangely there is no refresh job queues. I had refresh settings at 2 days, now I have adjusted it to 1 week and triggered refresh manually.
Waited for refresh job to be completed then restarted dremio. On restart reflections became unreachable with no refresh jobs automatically triggered. This makes make conclude the earlier refresh was triggered probably because configured time for refresh (2 days) was due.
Here is reflection load materialization job profile a6b63f05-31b7-4275-98d8-f0ab26164176.zip
(4.5 KB) and refresh reflection profile 8d0328b3-b4b7-4ea1-b97d-441882ba3a83.zip
This is how job profiles look
The profiles look ok, the next step is to investigate server.log. Under log/archive folder, look at the server.log for “2021-03-13” and search for materialization ID “7065e75c-e3c9-4fc0-9e58-0ed15cbc9884” and also reflection id “afb2a4af-675f-44a1-a391-79fa2b8ae8b0” and see if you find any errors?
Thank you very much for the guidelines and your assistance. I followed logs and couldn’t find any meaningful error. So I decided to test with builds from various points as follows:
- build dremio from github master branch, run
distribution/server/... locally without shipping to docker. Follow steps in how to reproduce. Reflections were available after restart.
- run dremio/dremio-oss docker image as in the how to reproduce. Reflections were available after restart (stupid of me I didn’t verify with this before opening the issue).
- Downloaded dremio-community-server from https://download.dremio.com/community-server/dremio-community-LATEST.tar.gz. Decompress the tar and run locally. Tests fail! I should not have trusted this build source.
All this long I was using compiled source from https://download.dremio.com/community-server and this is where I was going wrong.
Either use binaries from github source build or run dremio/dremio-oss docker image. There is less known about builds in https://download.dremio.com/community-server