Can't start after upgrade to 4.2.2

Berni · April 28, 2020, 8:42pm

Hi,
Just upgraded my stand alone server (ubuntu) and no longer able to start dremio service.
Logs tells:
2020-04-28 19:19:27,368 [main] INFO c.dremio.exec.catalog.PluginsManager - Result of storage plugin startup:
__jobResultsStore: success (575ms). Healthy
INFORMATION_SCHEMA: success (0ms). Healthy
__logs: success (273ms). Healthy
PRH: failed (629ms). Unavailable: Unavailable: IO Error: The Network Adapter could not establish the connection
__support: success (436ms). Healthy
__datasetDownload: success (394ms). Healthy
sys: success (0ms). Healthy
$scratch: success (539ms). Healthy
Gestion: success (198ms). Healthy
PRH-SNE: success (198ms). Healthy
Tickets Support: success (1675ms). Unhealthy: Elastic cluster health is yellow.
__home: success (270ms). Healthy
BeeTechLogs: success (168ms). Healthy
__accelerator: success (335ms). Healthy

2020-04-28 19:19:27,953 [start-__home] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __home
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “null”
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
at com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0(FileSystemPlugin.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:207)
at com.dremio.exec.store.dfs.FileSystemPlugin$1.load(FileSystemPlugin.java:184)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4960)
at com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem(FileSystemPlugin.java:316)
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:304)
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:300)
at com.dremio.exec.store.dfs.FileSystemPlugin.createFS(FileSystemPlugin.java:296)
at com.dremio.exec.store.dfs.FileSystemPlugin.start(FileSystemPlugin.java:607)
at com.dremio.dac.homefiles.HomeFileSystemStoragePlugin.start(HomeFileSystemStoragePlugin.java:100)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:490)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:554)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-04-28 19:19:27,972 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.RuntimeException: Unable to initialize Initializer SystemStoragePluginInitializer

And syslog mentions dremio exiting on cause 4/PERMISSION, but no mention where the lack of permission is.

Thanks for your help.

Cheers
Beeni

ben · April 28, 2020, 8:55pm

@Berni,

What were the upgrade steps you followed?

Also, where are you storing your Dremio coordinator db and where is your distributed storage configured? Sharing dremio.conf may be useful here.

Berni · April 28, 2020, 9:02pm

Hi Ben,
Thanks for looking after my issue.
Upgrade is based on the tarball, so just followed the standard procedure.
rename the installation, untar the distrib in new directory, created the missing folders like logs, archive, …, change the owner to dremio:dremio on various directories.
db is out of the dremio root folder under a local LVGroup so no copy.
Copy back the prevous conf files in the new conf directory then ran the dremio-admin upgrade and try to start.

below is my dremio.conf:

#
# Copyright (C) 2017-2018 Dremio Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

paths: {
  # the local path for dremio to store data.
  #local: ${DREMIO_HOME}"/data"
  local: "/mnt/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
  dist: ${paths.local}"/pdfs"

  # stores uploaded data associated with user home directories
  #uploads: "/mnt/data/uploads"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true
}

Thanks,

ben · April 28, 2020, 9:29pm

Did you run the upgrade as the user that runs the Dremio process?

Berni · April 29, 2020, 3:35am

As I wasn’t sure just did it again as dremio user and still the same error when I start the service.
Below is the result and I have the feeling that the upgrade process is not doing anything:
KVStore version is 4.2.2-202004211133290458-b550b6fa

Upgrade Tasks Status before upgrade

Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
Status: COMPLETED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023943(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
Status: OUTDATED StartTime : 1548533023062(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023062(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
Status: OUTDATED StartTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
Status: OUTDATED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
Status: OUTDATED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
Status: OUTDATED StartTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019)

Task: ‘Upgrade Arrow Schema’ up to Version; 2.1.0) completed. Skipping

Task: ‘Re index all stores’ up to Version; 2.1.0) completed. Skipping

Task: ‘Fix dataset split ids with invalid id’ up to Version; 3.0.0) completed. Skipping

Task: ‘Delete history of renamed datasets’ up to Version; 2.1.0) completed. Skipping

Task: ‘Deleting Hive 1.2.1 based InputSplits’ up to Version; 2.0.10) completed. Skipping

Task: ‘Delete schema stored in arrow footers of job results in KV Store’ up to Version; 2.1.2) completed. Skipping

Task: ‘Update S3 credential type’ up to Version; 2.1.7) completed. Skipping

Task: ‘Update External Reflections’ up to Version; 2.1.0) completed. Skipping

Task: ‘Deleting System table metadata’ completed. Skipping

Upgrade Tasks Status after upgrade

Task: ID (0f5c698a-55c9-45e7-84fe-c5f737e30432), Name (DeleteSysMaterializationsMetadata)
Status: COMPLETED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023943(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (11e22843-8b8e-43b1-8c46-ba8fd3418e6c), Name (MoveFromAccelerationsToReflections)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (149b8d09-9099-4eba-8902-0edf103a441c), Name (DeleteHistoryOfRenamedDatasets)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (18df9bdf-9186-4780-b6bb-91bcb14a7a8b), Name (DatasetConfigUpgrade)
Status: OUTDATED StartTime : 1548533023062(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023062(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (2153deff-8117-4edd-bf36-876fb2c61bb5), Name (MigrateAccelerationMeasures)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (285abdab-19b9-493b-8c2e-f2759c6bbd04), Name (CompressHiveTableAttrs)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (3ea6ba77-3367-43ca-8e63-727b565b544b), Name (SetAccelerationRefreshGrace)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (518bbc49-37be-474f-938d-c4960dfbdab8), Name (ConvertJoinInfo)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (5522c3bc-195f-41ba-8bfd-a33f91b1219a), Name (ReIndexAllStores)
Status: OUTDATED StartTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (69d0202a-ad32-4128-82d6-e0341b0f0151), Name (DeleteInternalSources)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (7512f256-fb80-4517-9a50-55a126fd93d5), Name (UpdateS3CredentialType)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (79312f25-49d6-40e7-8096-7e132e1b64c4), Name (UpdateExternalReflectionHash)
Status: OUTDATED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (7ce8ae11-5b8a-4ffe-96a6-413391764db5), Name (SetDatasetExpiry)
Status: OUTDATED StartTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023068(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (a5d23112-f354-42fe-bdeb-b024d8d5fb1b), Name (DeleteHive121BasedInputSplits)
Status: OUTDATED StartTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023071(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (b2bda6ef-6a7a-4f3e-bf69-7e88ada3fa3c), Name (MarkOldMaterializationsAsDeprecated)
Status: OUTDATED StartTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023069(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (bf38c3f3-68dc-41f1-b5e1-40e76f29d54d), Name (MoveFromAccelerationSettingsToReflectionSettings)
Status: OUTDATED StartTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023070(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (c33400d9-fa65-47e2-b99a-5c3db12d8f84), Name (MinimizeJobResultsMetadata)
Status: OUTDATED StartTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023072(Sat Jan 26 20:03:43 UTC 2019)
Task: ID (d7cb2438-bc97-4c76-8a7a-ff5493e48e5e), Name (UpdateDatasetSplitIdTask)
Status: OUTDATED StartTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019) EndTime : 1548533023067(Sat Jan 26 20:03:43 UTC 2019)

Cheers,

Berni · April 29, 2020, 10:55am

Hi,

Found the solution.
Had to configure all the paths in the dremio.conf file.
Previously only the local: dist: and uploads: were defined and it was enough to run the 4.0.2 version.
Added, db: spilling: accelerator: downloads: results: and scratch:, and now the server starts.

Have to dig into all the deprecated warning that hive is sending now.

Cheers,

Topic		Replies	Views
Dremio doesn't start after upgrade 2.0.1	1	1092	May 17, 2018
Dremio failed to start	5	1631	October 31, 2020
Dremio would not start	5	1691	April 25, 2018
Error Starting Dremio	2	2517	July 30, 2018
New Ubuntu install will not start	2	1783	April 23, 2018

Can't start after upgrade to 4.2.2

Related topics