Dremio error 502 - Bad Gateway

Paulo_Vasconcellos · May 3, 2019, 1:33pm

Hi!
When I try to start dremio service on my coordinator, the service is running, but the message “Dremio has started as Master” doesn’t show up. When I try to connect to my coordinator via browser, I get an error “502 Bad Gateway”. In my server.out file, I see these lines:

Fri May  3 13:22:31 UTC 2019 Starting dremio on ip-10-0-0-28.ec2.internal
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30369
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
KVStore version is 3.1.0-201901172111160703-dc6f6e5

 Upgrade Tasks Status before Upgrade

Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
         Status: COMPLETED       StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706388045(Mon Jan 28 20:13:08 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
         Status: OUTDATED        StartTime : 1548706387775(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387775(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
         Status: OUTDATED        StartTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
         Status: OUTDATED        StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
         Status: OUTDATED        StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
         Status: OUTDATED        StartTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)


 Upgrade Tasks Status after Upgrade

Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
         Status: COMPLETED       StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706388045(Mon Jan 28 20:13:08 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
         Status: OUTDATED        StartTime : 1548706387775(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387775(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
         Status: OUTDATED        StartTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
         Status: OUTDATED        StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
         Status: OUTDATED        StartTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387779(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
         Status: OUTDATED        StartTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387781(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
         Status: OUTDATED        StartTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387780(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
         Status: OUTDATED        StartTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387782(Mon Jan 28 20:13:07 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
         Status: OUTDATED        StartTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)         EndTime : 1548706387778(Mon Jan 28 20:13:07 UTC 2019)

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

ben · May 3, 2019, 3:50pm

Hi @Paulo_Vasconcellos,

A couple of questions:

When did you last stop and restart the service?
If you wait 10-20 minutes, does Dremio start?

ben · May 3, 2019, 4:08pm

Hello again @Paulo_Vasconcellos,

Some more questions:

Is there anything printed to Dremio’s server.log (not server.out which I believe you already posted) during the pause on startup?
What infrastructure are you using? AWS, Azure, GCE?
Are there any load balancers, proxy servers in front of Dremio?

Paulo_Vasconcellos · May 4, 2019, 3:56pm

Hi @ben. After a couple of minutes, Dremio started again. The service was stoped due to an out of memory problem.

Topic		Replies	Views
Dremio 3.1 is out!	13	2381	February 6, 2019
Dremio error root dremio exiting- 502 Bad Gateway Nginx	1	55	June 24, 2024
[main] INFO c.d.s.c.TaskLeaderStatusListener - Waiting for TaskLeader for MASTER	4	1128	April 5, 2019
Dremio not running	4	2656	October 30, 2018
Failed to start services, daemon exiting	14	3073	April 2, 2018

Dremio error 502 - Bad Gateway

Related topics