Dremio 3.1 is out!

Dremio 3.1 is out! :ocean:

We just released Dremio 3.1, including lots of new features and performance enhancements.

Grab it from our downloads page, or register for our webinar to learn more.

3 Likes

Thanks for this new version.
I followed the RPM upgrade instructions,
unfortunately after 1h30 after starting Dremio, the “Dremio Started as master” message never appears, and I have no error in logs, low CPU usage, as if nothing was happening.

Last lines of server.out file in master/coordinator :

KVStore version is 3.1.0-201901172111160703-dc6f6e5

Upgrade Tasks Status before Upgrade

Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
Status: COMPLETED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526635(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
Status: OUTDATED StartTime : 1548500526347(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526347(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
Status: OUTDATED StartTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526351(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
Status: OUTDATED StartTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526352(Sat Jan 26 11:02:06 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
Status: OUTDATED StartTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019) EndTime : 1548500526350(Sat Jan 26 11:02:06 UTC 2019)

Last lines of server.log file

2019-01-26 12:50:12,485 [main] INFO c.d.service.jobs.LocalJobsService - Starting JobsService
2019-01-26 12:50:12,611 [main] INFO c.d.service.jobs.LocalJobsService - JobsService is up

Hi @dfleckinger

Can you please send us the entire server.log?

Thanks
@balaji.ramaswamy

Hi @balaji.ramaswamy,

Here is the server.log when 3.1 was running.
I can see in it error logs due to external requests that could not be fulfilled as Dremio was not “really” started.
Don’t you need the server.out file ?

server.2019-01-26_3.1.log.zip (54.9 KB)

Thanks
David

HI @dfleckinger

Have you configured systemctl? The logs might be going to standard out or journalctl. Have you tried to hit the coordinator url and you do not see the log in page? Yeah kindly send us server.out

Thanks
@balaji.ramaswamy

I did not configure anything in particular, AFAIK I use the standard configuration.
I usually monitor server.log and server.out file at the same time.
Unfortunately I don’t know where to find server.out log information 3 days later, any idea ?

should be under /log/archive

Here it is :
server.out.zip (2.2 KB)

Hi, I’ve tested the RPM to 3.1.1 Community version and this time it worked like a charm.

However I have plenty of queries not accelerated by the existing reflections.

Hi @dfleckinger

Are they matching and not getting chosen? Do you have a profile?

Thanks
@balaji.ramaswamy

Hi @balaji.ramaswamy,
I’ve sent the query profile here : https://community.dremio.com/t/3-1-matching-algorithm-reflection-not-used/2661/6

Yesterday some reflections were matched but not chosen, fortunately today I have no issue.

Regards, David

Hi @balaji.ramaswamy,

I’ve posted sample profiles for not matching aggregation reflections in a separate thread: 3.1 matching algorithm : Reflection not used

BR,
Alex