Assistance Required – Error Accessing Internal Source __accelerator on EC2 Dremio Instance

Dremio Cloud
1

Hi, we are using Dremio Cloud and we used the AWS CloudFormation template to configure the setup: EC2 instance and S3 bucket. We are able to query our database through dremio without issue and for a short period we also had reflections working.

Now after a 2 minute timeout we get the error message below every time we try to refresh or create a reflection. We’ve tried rebooting our dremio managed EC2 instance multiple times and we’re still having issues. A few weeks ago we were experiencing a similar issue with [__resultsCacheStore] that seemed to resolve after a reboot, but now reboots dont seem to be helping. We’re not sure how to access server.log within our setup either. Can someone please help with possible fixes or troubleshooting for this issue? Happy to provide more details if necessary.

      VALIDATION ERROR: Failure getting source [__accelerator].

ErrorOrigin: EXECUTOR
[Error Id: 670ea8f5-7645-426d-b363-51a05e8e8859 on redacted-ip]

  (java.util.concurrent.TimeoutException) null
    java.util.concurrent.CompletableFuture.timedGet():1892
    java.util.concurrent.CompletableFuture.get():2027
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1906
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1836
    com.dremio.exec.catalog.PluginsManager.getSynchronized():563
    com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():788
    com.dremio.exec.catalog.CatalogServiceImpl.getSource():849
    com.dremio.exec.planner.PhysicalPlanReader$1.getSource():242
    com.dremio.exec.store.parquet.ParquetWriter.<init>():53
    jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0():-2
    jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance():62
    jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
    java.lang.reflect.Constructor.newInstance():490
    com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call():128
    com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():291
    com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith():301
    com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():202
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():525
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():74
    com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue():323
    com.fasterxml.jackson.databind.ObjectReader._bindAndClose():2105
    com.fasterxml.jackson.databind.ObjectReader.readValue():1481
    com.dremio.common.serde.ProtobufByteStringSerDe.readValue():130
    com.dremio.exec.planner.PhysicalPlanReader.readValue():201
    com.dremio.exec.planner.PhysicalPlanReader.readFragmentOperator():186
    com.dremio.exec.planner.fragment.CachedFragmentReader.lambda$readFragment$1():81
    java.util.concurrent.ConcurrentHashMap.computeIfAbsent():1705
    com.dremio.exec.planner.fragment.CachedFragmentReader.readFragment():77
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():797
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():574
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1331
    com.dremio.sabot.task.AsyncTaskWrapper.run():162
    com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():283
    com.dremio.sabot.task.slicing.SlicingThread.run():188
2

@skylineit The error means the Dremio internal plug in __accelerator is not accessible, the reason should be stated during the executor startup in server.log. Can you try to disable and enable the engine via UI? If this is the preview engine the job is going to then try to roue the job to a different engine