Validation Error: Failure getting source [__resultsCacheStore]

Dremio Cloud
1

We’re testing reflections and we sucessfully created a raw reflection on a view that queries the NYC-weather table from the SampleDB. However, when we query the view, it says that it is being accelerated by the reflection but then it takes 2 minutes to run and at the 2 minute mark we get the following message:

VALIDATION ERROR: Failure getting source [__resultsCacheStore].

ErrorOrigin: EXECUTOR
[Error Id: 520c92f9-a6ef-4132-8093-2227db3de4fb on redacted-ip-address]

  (java.util.concurrent.TimeoutException) null
    java.util.concurrent.CompletableFuture.timedGet():1892
    java.util.concurrent.CompletableFuture.get():2027
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1906
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1836
    com.dremio.exec.catalog.PluginsManager.getSynchronized():563
    com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():788
    com.dremio.exec.catalog.CatalogServiceImpl.getSource():849
    com.dremio.exec.planner.PhysicalPlanReader$1.getSource():242
    com.dremio.exec.store.dfs.easy.ResultsCacheWriter.<init>():53
    jdk.internal.reflect.GeneratedConstructorAccessor233.newInstance():-1
    jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
    java.lang.reflect.Constructor.newInstance():490
    com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call():128
    com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():291
    com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith():301
    com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():202
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():525
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():542
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():570
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():439
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1409
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1375
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():218
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():187
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():170
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():136
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():74
    com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue():323
    com.fasterxml.jackson.databind.ObjectReader._bindAndClose():2105
    com.fasterxml.jackson.databind.ObjectReader.readValue():1481
    com.dremio.common.serde.ProtobufByteStringSerDe.readValue():130
    com.dremio.exec.planner.PhysicalPlanReader.readValue():201
    com.dremio.exec.planner.PhysicalPlanReader.readFragmentOperator():186
    com.dremio.exec.planner.fragment.CachedFragmentReader.lambda$readFragment$1():81
    java.util.concurrent.ConcurrentHashMap.computeIfAbsent():1705
    com.dremio.exec.planner.fragment.CachedFragmentReader.readFragment():77
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():797
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():574
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1331
    com.dremio.sabot.task.AsyncTaskWrapper.run():162
    com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():283
    com.dremio.sabot.task.slicing.SlicingThread.run():188

When rerunning the same query immediately after, it works. But then if we test again at a later time we get that same error message at the 2 minute mark. The view’s sql code is very simple and just find average temperature by day. We couldn’t find any online sources of this same error message. Any ideas what could be causing this?

2

@skylineit

If you are using K8’s for deployment, check your values.yaml (if standalone then check dremio.conf) if you have declared results separately to local or just see dist:/// , either way check server.log on executor when it last started and there will be a section like below where it prints every source name and its status (expected to be “Healthy”), All names starting with an underscore are Dremio internal sources, see if you can fine one named __jobResultsStore: success (430ms). Healthy, if it says something like Pending or Unavailable make sure to restart the executor, when the executor is starting up, tail the server.log to make sure this time the plugin says “Healthy”

1 Like
3

Thanks for your response @balaji.ramaswamy. We’re using AWS for our deployment, we used a CloudFormation Template to set up our cloud infrastructure. Our engine is running on an EC2 and we have an S3 bucket. Should we reboot our EC2 instance? I’m not sure how to access the server.log or dremio.conf if we’re using Dremio Cloud.

4

@skylineit You should be able to use the disable option in the UI (settings-engines) and use the disable option. Give it few minutes and enable again and see if this time the issue persists

5

Thank you @balaji.ramaswamy. When trying to disable our engine it says “You cannot disable the Preview engine because it is used to run metadata refresh queries for relational sources and other background activities. Disabling the engine would cause refresh and other queries to fail” and we are unable to click disable

6

@skylineit Can you please try running the query to a non-preview engine and see if it completes?