Dremio not starting syddenly

1

hi ,

i have strucked in production dremio

starting dremio
2020-02-24 07:52:38,606 [main] INFO com.dremio.exec.util.GuavaPatcher - Google’s Stopwatch patched for old HBase Guava version.
2020-02-24 07:52:38,616 [main] INFO com.dremio.exec.util.GuavaPatcher - Google’s Closeables patched for old HBase Guava version.
2020-02-24 07:52:38,757 [main] INFO com.dremio.common.config.SabotConfig - Configuration and plugin file(s) identified in 127ms.
Base Configuration:
- jar:file:/opt/dremio/jars/dremio-common-4.1.3-202001022113020736-53142377.jar!/sabot-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/dremio/jars/dremio-services-datastore-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-provision-common-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-jobs-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-coordinator-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-yarn-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-pdfs-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-common-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-sabot-joust-java-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-namespace-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-backend-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-daemon-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-execselector-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-adls-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-configuration-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-common-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-telemetry-api-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-options-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-telemetry-impl-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-accelerator-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-logical-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-services-cachemanager-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-sabot-scheduler-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive-function-registry-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-s3-plugin-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive-plugin-common-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-users-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/3rdparty/dremio-hive2-exec-shaded-4.1.3-202001022113020736-53142377.jar!/sabot-module.conf

2020-02-24 07:52:38,839 [main] INFO com.dremio.config.DremioConfig - Applying provided system property to config: -Dzookeeper=zk-hs:2181
2020-02-24 07:52:38,839 [main] INFO com.dremio.config.DremioConfig - Applying provided system property to config: -Dservices.coordinator.master.embedded-zookeeper.enabled=false
2020-02-24 07:52:38,839 [main] INFO com.dremio.config.DremioConfig - Applying provided system property to config: -Dservices.executor.enabled=false
2020-02-24 07:52:39,360 [main] INFO c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.storage, com.dremio.datastore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.telemetry.api, com.dremio.service.jobs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.datastore, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.options, com.dremio.telemetry.api, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.provision, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.provision, com.dremio.options, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.datastore, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.resource, com.dremio.resource.basic, com.dremio.exec.store, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.telemetry.api, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.datastore, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.options, com.dremio.telemetry.api, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.provision, com.dremio.exec.store, com.dremio.storage, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.telemetry.api, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.provision, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.telemetry.api, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.provision, com.dremio.storage, com.dremio.exec.fn.hive, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.options, com.dremio.telemetry.api, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.provision, org.apache.hadoop.hive] from locations [jar:file:/opt/dremio/jars/dremio-services-datastore-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-jobs-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-common-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-backend-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-common-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-accelerator-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-logical-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-ce-sabot-scheduler-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-hive-function-registry-4.1.3-202001022113020736-53142377.jar!/META-INF/dremio-module-scan/registry.json]
2020-02-24T07:52:39.729+0000: [GC (Allocation Failure) [PSYoungGen: 516096K->31948K(602112K)] 516096K->32036K(1978368K), 0.0206736 secs] [Times: user=0.14 sys=0.00, real=0.02 secs]
2020-02-24T07:52:40.583+0000: [GC (Allocation Failure) [PSYoungGen: 548044K->52293K(602112K)] 548132K->52389K(1978368K), 0.0307408 secs] [Times: user=0.06 sys=0.05, real=0.04 secs]
2020-02-24 07:52:40,825 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.datastore, com.dremio.provision, com.dremio.service.jobs, com.dremio.provision.yarn.service, com.dremio.exec.store, com.dremio.joust.geo, com.dremio.service.namespace, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac.cmd, com.dremio.dac.cmd.upgrade, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.plugins.adl.store, com.dremio.plugins.elastic, com.dremio.plugins.azure, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.telemetry.api, com.dremio.exec.store.jdbc, com.dremio.options, com.dremio.telemetry.impl.config, com.dremio.plugins.mongo, com.dremio.exec.store.jdbc, com.dremio.exec.store.hive, com.dremio.extras.plugins.elastic, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.service.cachemanager, com.dremio.exec.store.hive, com.dremio.plugins.mongo, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.fn.hive, com.dremio.plugins.s3.store, com.dremio.exec.store.hive, com.dremio.exec.store.hive.exec, com.dremio.service.users, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-provision-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-joust-java-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-dac-daemon-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-execselector-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-adls-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-configuration-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-api-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-options-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-impl-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-services-cachemanager-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-s3-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive-plugin-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive2-exec-shaded-4.1.3-202001022113020736-53142377.jar!/] took 1462ms
2020-02-24 07:52:41,038 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider

2

its strting from last 1 hour but not even start

3

@balaji.ramaswamy can you please help on above issue i have production dremio down for more then 10 hour not able to get it up running

4

@Vikash_Singh

Can you attach server.log and server.out. Do you know the reason for the crash?. Or is it a maintenance restart ?

@Venugopal_Menda

5

@Vikash_Singh

Please check once on the coordinator node do you have any stale NFS mount points.

@Venugopal_Menda

6

@Venugopal_Menda no i dont have any extra nfs mount i have used helm template to launch dremio it was working fine earlier

.dremio-master-0.zip (2.3 KB)

7

i have changed drmeio heap just now and getting this error now

Dremio is exiting. Failure while starting services.
org.rocksdb.RocksDBException: While opening a file for sequentially reading: /opt/dremio/data/db/catalog/MANIFEST-246396: No such file or directory
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:286)
at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:279)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:227)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:192)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:157)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:159)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:99)
Suppressed: java.lang.IllegalStateException: #start was not invoked, so metadataManager is not available
at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.dremio.datastore.ByteStoreManager.getMetadataManager(ByteStoreManager.java:422)
at com.dremio.datastore.ByteStoreManager.close(ByteStoreManager.java:429)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:92)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:71)
at com.dremio.datastore.CoreStoreProviderImpl.close(CoreStoreProviderImpl.java:286)
at com.dremio.datastore.LocalKVStoreProvider.close(LocalKVStoreProvider.java:198)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:162)
… 1 more

8

any update on my issues

9

The error states that a file is missing for our database, do you have any backups?

10

@doron just now not able to start dremio here is logs

2020-03-06T19:08:57.481+0000: [GC (Allocation Failure) [PSYoungGen: 516096K->31314K(602112K)] 516096K->31410K(1978368K), 0.0139613 secs] [Times: user=0.08 sys=0.00, real=0.02 secs]
2020-03-06T19:08:58.174+0000: [GC (Allocation Failure) [PSYoungGen: 547410K->53656K(602112K)] 547506K->53760K(1978368K), 0.0259470 secs] [Times: user=0.04 sys=0.06, real=0.02 secs]
2020-03-06 19:08:58,502 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.datastore, com.dremio.provision, com.dremio.service.jobs, com.dremio.provision.yarn.service, com.dremio.exec.store, com.dremio.joust.geo, com.dremio.service.namespace, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac.cmd, com.dremio.dac.cmd.upgrade, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.plugins.adl.store, com.dremio.plugins.elastic, com.dremio.plugins.azure, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.telemetry.api, com.dremio.exec.store.jdbc, com.dremio.options, com.dremio.telemetry.impl.config, com.dremio.plugins.mongo, com.dremio.exec.store.jdbc, com.dremio.exec.store.hive, com.dremio.extras.plugins.elastic, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.service.cachemanager, com.dremio.exec.store.hive, com.dremio.plugins.mongo, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.fn.hive, com.dremio.plugins.s3.store, com.dremio.exec.store.hive, com.dremio.exec.store.hive.exec, com.dremio.service.users, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-provision-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-joust-java-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-dac-daemon-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-execselector-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-adls-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-configuration-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-api-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-options-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-impl-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-services-cachemanager-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-s3-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive-plugin-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive2-exec-shaded-4.1.3-202001022113020736-53142377.jar!/] took 1256ms
2020-03-06 19:08:58,752 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2020-03-06T19:09:00.104+0000: [GC (Allocation Failure) [PSYoungGen: 569752K->27095K(602112K)] 569856K->27207K(1978368K), 0.0094067 secs] [Times: user=0.04 sys=0.00, real=0.01 secs]

i have put 70GB gc

11

i have also triyed with dremio env but value not relecting plz help

12

Can you share your conf and env files?

13

Hello @doron,

here is config file
ConfigVar.zip (2.5 KB)

14

here is attached logs file while starting dremio

dremio-master-0.zip (2.1 KB)

15

here is container ps

Screenshot_1
Screenshot_11896×104 12.6 KB

16

http://docs.dremio.com/deployment/system-requirements.html#best-practices notes that 8 gigs for heap is recommended, 70 gigs is much too large.

17

sorry @doron, i have changed like this in env:

DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=16384,
but still not able to start

18

@doron thanks for your help now dremio have been started :slight_smile:

19

hello @doron ,

just now i have faced issue dremio running slow after that starting kv store struck ,i have used recommended heap

DREMIO_MAX_HEAP_MEMORY_SIZE_MB=8192

20

here is full log

2020-03-17T19:22:09.373+0000: [GC (Allocation Failure) [PSYoungGen: 516096K->31512K(602112K)] 516096K->31544K(1978368K), 0.0169553 secs] [Times: user=0.08 sys=0.02, real=0.02 secs]
2020-03-17T19:22:10.112+0000: [GC (Allocation Failure) [PSYoungGen: 547608K->52394K(602112K)] 547640K->52498K(1978368K), 0.0296154 secs] [Times: user=0.06 sys=0.06, real=0.04 secs]
2020-03-17 19:22:10,544 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.datastore, com.dremio.provision, com.dremio.service.jobs, com.dremio.provision.yarn.service, com.dremio.exec.store, com.dremio.joust.geo, com.dremio.service.namespace, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.dac.cmd, com.dremio.dac.cmd.upgrade, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.service.execselector.ExecutorSelectionService, com.dremio.plugins.adl.store, com.dremio.plugins.elastic, com.dremio.plugins.azure, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.telemetry.api, com.dremio.exec.store.jdbc, com.dremio.options, com.dremio.telemetry.impl.config, com.dremio.plugins.mongo, com.dremio.exec.store.jdbc, com.dremio.exec.store.hive, com.dremio.extras.plugins.elastic, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.sabot.rpc.user, com.dremio.service.cachemanager, com.dremio.exec.store.hive, com.dremio.plugins.mongo, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.fn.hive, com.dremio.plugins.s3.store, com.dremio.exec.store.hive, com.dremio.exec.store.hive.exec, com.dremio.service.users, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-provision-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-joust-java-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-dac-daemon-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-execselector-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-adls-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-configuration-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-api-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-options-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-telemetry-impl-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-services-cachemanager-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive3-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-s3-plugin-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive2-plugin-launcher-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-hive-plugin-common-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-4.1.3-202001022113020736-53142377.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive2-exec-shaded-4.1.3-202001022113020736-53142377.jar!/] took 1418ms
2020-03-17T19:22:10.669+0000: [GC (Allocation Failure) [PSYoungGen: 568490K->27965K(602112K)] 568594K->28077K(1978368K), 0.0121113 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2020-03-17 19:22:10,706 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider