Dremio 3.1 is out!
We just released Dremio 3.1, including lots of new features and performance enhancements.
Grab it from our downloads page, or register for our webinar to learn more.
Dremio 3.1 is out!
We just released Dremio 3.1, including lots of new features and performance enhancements.
Grab it from our downloads page, or register for our webinar to learn more.
Thanks for this new version.
I followed the RPM upgrade instructions,
unfortunately after 1h30 after starting Dremio, the “Dremio Started as master” message never appears, and I have no error in logs, low CPU usage, as if nothing was happening.
Last lines of server.out file in master/coordinator :
KVStore version is 3.1.0-201901172111160703-dc6f6e5
Upgrade Tasks Status before Upgrade
Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
Status: COMPLETED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526635(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
Status: OUTDATED StartTime : 1548500526347(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526347(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Last lines of server.log file
2019-01-26 12:50:12,485 [main] INFO c.d.service.jobs.LocalJobsService - Starting JobsService
2019-01-26 12:50:12,611 [main] INFO c.d.service.jobs.LocalJobsService - JobsService is up
Here is the server.log when 3.1 was running.
I can see in it error logs due to external requests that could not be fulfilled as Dremio was not “really” started.
Don’t you need the server.out file ?
server.2019-01-26_3.1.log.zip (54.9 KB)
Thanks
David
HI @dfleckinger
Have you configured systemctl? The logs might be going to standard out or journalctl. Have you tried to hit the coordinator url and you do not see the log in page? Yeah kindly send us server.out
Thanks
@balaji.ramaswamy
I did not configure anything in particular, AFAIK I use the standard configuration.
I usually monitor server.log and server.out file at the same time.
Unfortunately I don’t know where to find server.out log information 3 days later, any idea ?
should be under /log/archive
Hi, I’ve tested the RPM to 3.1.1 Community version and this time it worked like a charm.
However I have plenty of queries not accelerated by the existing reflections.
Hi @dfleckinger
Are they matching and not getting chosen? Do you have a profile?
Thanks
@balaji.ramaswamy
Hi @balaji.ramaswamy,
I’ve sent the query profile here : https://community.dremio.com/t/3-1-matching-algorithm-reflection-not-used/2661/6
Yesterday some reflections were matched but not chosen, fortunately today I have no issue.
Regards, David
I’ve posted sample profiles for not matching aggregation reflections in a separate thread: 3.1 matching algorithm : Reflection not used
BR,
Alex