08:20:42,925 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [admin] to INFO
08:20:42,925 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [admin] to true
08:20:42,925 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[admin]
08:20:42,925 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to OFF
08:20:42,927 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ADMINLOG] to Logger[ROOT]
08:20:42,927 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:20:42,928 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@306cf3ea - Registering current configuration as safe fallback point
Upgrade failed
com.dremio.datastore.DatastoreException : Process user (?) doesn’t match local catalog db owner (1000). Please run process as 1000.
Suppressed: java.lang.IllegalStateException : #start was not invoked, so metadataManager is not available
Error : Could not find or load main class dremio_2024-03-19_08:35:04.log
chown: changing ownership of ‘/opt/dremio/data/lost+found’: Operation not permitted
chown: changing ownership of ‘/opt/dremio/data/db/search/jobs/core/write.lock’: Operation not permitted
chown: changing ownership of ‘/opt/dremio/data/db/search/jobs/core/segments_1’: Operation not permitted
@praveenshetg Who owns the “db” folder?
@balaji.ramaswamy ‘db’ folder owned by different user. Is it mandatory to run helm install by the same user?
@balaji.ramaswamy I tried installing in local Kubernetes with kind cluster. all pods are running but master pod is restarting with below 2 errors.
error 1:
java.lang.IllegalArgumentException: resource dremio-telemetry.yaml not found.
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:218)
at com.google.common.io.Resources.getResource(Resources.java:196)
at com.dremio.telemetry.api.Telemetry$1.get(Telemetry.java:104)
at com.dremio.telemetry.api.Telemetry$1.get(Telemetry.java:86)
at com.dremio.telemetry.api.config.AutoRefreshConfigurator.refresh(AutoRefreshConfigurator.java:64)
at com.dremio.telemetry.api.config.AutoRefreshConfigurator.lambda$new$0(AutoRefreshConfigurator.java:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
error 2:
ava.lang.NullPointerException: Master coordinator is down
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:921)
at com.dremio.service.conduit.client.ConduitProviderImpl.getOrCreateChannelToMaster(ConduitProviderImpl.java:180)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightClient(SysFlightStoragePlugin.java:122)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightTableList(SysFlightStoragePlugin.java:256)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.listDatasetHandles(SysFlightStoragePlugin.java:208)
at com.dremio.exec.catalog.MetadataSynchronizer.getDatasetHandleListing(MetadataSynchronizer.java:172)
at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:186)
at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:136)
at com.dremio.exec.catalog.SourceMetadataManager$RefreshRunner.refreshFull(SourceMetadataManager.java:466)
at com.dremio.exec.catalog.SourceMetadataManager$AdhocRefresh.run(SourceMetadataManager.java:532)
at com.dremio.exec.catalog.SourceMetadataManager.refresh(SourceMetadataManager.java:212)
at com.dremio.exec.catalog.ManagedStoragePlugin.refresh(ManagedStoragePlugin.java:1242)
at com.dremio.exec.catalog.CatalogServiceImpl.refreshSource(CatalogServiceImpl.java:400)
at com.dremio.exec.catalog.CatalogServiceImpl.lambda$start$0(CatalogServiceImpl.java:287)
at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Blocked with dremio-telemetry.yaml related error. Found some document and I have created this file in conf folder with below content.
auto-reload:
enabled: True
period: 30
unit: SECONDS
metrics:
- name: reporter185_jmx
reporter:
type: jmx```
getting error related to reporter185_jmx
com.dremio.telemetry.api.Telemetry - Failure reading telemetry configuration. Leaving telemetry as is.
com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `com.dremio.telemetry.api.config.MetricsConfigurator`, problem: Invalid definition for reporter with name reporter185_jmx. Must define a reporter block.
at [Source: (URL); line: 20, column: 1] (through reference chain: com.dremio.telemetry.api.config.TelemetryConfigurator["metrics"]->java.util.ArrayList[0])
at com.fasterxml.jackson.databind.exc.ValueInstantiationException.from(ValueInstantiationException.java:47)
at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:2058)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:598)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:621)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:293)
at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:301)
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:202)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:525)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1409)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1409)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1690)
at com.dremio.telemetry.api.Telemetry$1.get(Telemetry.java:107)
at com.dremio.telemetry.api.Telemetry$1.get(Telemetry.java:86)
at com.dremio.telemetry.api.config.AutoRefreshConfigurator.refresh(AutoRefreshConfigurator.java:64)
at com.dremio.telemetry.api.config.AutoRefreshConfigurator.lambda$new$0(AutoRefreshConfigurator.java:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException: Invalid definition for reporter with name reporter185_jmx. Must define a reporter block.
at java.util.Objects.requireNonNull(Objects.java:290)
at com.dremio.telemetry.api.config.MetricsConfigurator.<init>(MetricsConfigurator.java:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:128)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:291)
... 29 common frames omitted
@praveenshetg
resource dremio-telemetry.yaml not found can be ignored so revert the changes you did for that. For the second error looks like a conduit port issue. Can you please revert the telemetry change you did, restart Dremio and if it fails again, send server.log and server.out
Another option is to revert the telemetry change and give a port for Conduit that is open in your FW rules
Search for “conduit” in the above doc link on how to define in dremio.conf