Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com in Dremio Cloud

Hello,

I just set up Dremio Cloud succesfully in my AWS account in eu-central-1. I managed to set up the connection to the Glue Data catalog available in the AWS account in which also Dremio runs - by using an IAM user that has all the necessary permissions.

When opening the data source to explore, I can properly see all the databases and tables listed. However, as soon as I try to preview the tables in the data source, I get the error Failure getting source [__metadata]. While digging deeper, I can see the following in the stacktrace:

Caused By (com.dremio.common.util.Retryer.OperationFailedAfterRetriesException) software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443

Dremio Cloud is running in a public VPC in eu-central-1.

I made sure that eu-central-1 is selected as AWS region for the data source. I don’t understand why Dremio Cloud still tries to connect to the us-east-1 STS, even if accessing the EU control plane.

Hello!

Welcome to the Dremio community!

Dremio automatically contacts the us-east-1 STS by default if no region is set for the S3 source. You can change this by following these steps:

  1. Right click your S3 source and select Edit Details
  2. Select Advanced Options on the left side menu
  3. Under connection properties, select Add a Property and name it “dremio.s3.property”. For the value, it would be “eu-central-1”.

This will allow Dremio to connect to your specified STS endpoint.

Hope this helps!

Cindy

Hello,

I tried it as you suggested, as you can see in the attached image.

I tried accessing a table with both “Enable asynchronous access when possible” turned on and off.

In both cases Dremio Cloud still tries to connect to the US sts…

Thank you for having a look…hope there’s something that can be done.

Hi,

Can you add another connection property and name it, “fs.s3a.endpoint”, with the value as “s3.eu-central-1.amazonaws.com”? If this still doesn’t work please attach the full error logs so I can take a further look.

Cindy

Hello Cindy,

thanks for the quick reply. Here is the stack of the failure:

 VALIDATION ERROR: Failure getting source [__metadata].

Fragment 1:0

[Error Id: 43e438d2-fe57-4fed-98f2-2481923f2d4a on 10.0.25.127:0]

  (java.util.concurrent.ExecutionException) com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed.
    java.util.concurrent.CompletableFuture.reportGet():357
    java.util.concurrent.CompletableFuture.get():1928
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1046
    com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1004
    com.dremio.exec.catalog.PluginsManager.getSynchronized():359
    com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():549
    com.dremio.exec.catalog.CatalogServiceImpl.getSource():592
    com.dremio.exec.planner.PhysicalPlanReader$1.getSource():118
    com.dremio.exec.store.iceberg.IcebergManifestWriterPOP.<init>():50
    sun.reflect.GeneratedConstructorAccessor263.newInstance():-1
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
    java.lang.reflect.Constructor.newInstance():423
    com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call():128
    com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():291
    com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith():288
    com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():202
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():518
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1405
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1371
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():217
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():186
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():144
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():110
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():539
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():563
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():438
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1405
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
    com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1371
    com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():217
    com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():186
    com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():144
    com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():110
    com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
    com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():74
    com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue():322
    com.fasterxml.jackson.databind.ObjectReader._bindAndClose():2051
    com.fasterxml.jackson.databind.ObjectReader.readValue():1427
    com.dremio.common.serde.ProtobufByteStringSerDe.readValue():135
    com.dremio.exec.planner.PhysicalPlanReader.readValue():235
    com.dremio.exec.planner.PhysicalPlanReader.readFragmentOperator():224
    com.dremio.exec.planner.fragment.CachedFragmentReader.lambda$readFragment$1():77
    java.util.concurrent.ConcurrentHashMap.computeIfAbsent():1660
    com.dremio.exec.planner.fragment.CachedFragmentReader.readFragment():75
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():470
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():336
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600():100
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():834
    com.dremio.sabot.task.AsyncTaskWrapper.run():120
    com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():247
    com.dremio.sabot.task.slicing.SlicingThread.run():171
  Caused By (com.google.common.util.concurrent.UncheckedExecutionException) java.lang.RuntimeException: Credential Verification failed.
    com.google.common.cache.LocalCache$Segment.get():2055
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (java.lang.RuntimeException) Credential Verification failed.
    com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():237
    com.dremio.plugins.s3.store.S3FileSystem.setup():217
    com.dremio.plugins.util.ContainerFileSystem.initialize():167
    com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1730
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
    com.google.common.cache.LocalCache$Segment.loadSync():2282
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
    com.google.common.cache.LocalCache$Segment.get():2049
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (com.dremio.common.util.Retryer.OperationFailedAfterRetriesException) software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
    com.dremio.common.util.Retryer.call():62
    com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
    com.dremio.plugins.s3.store.S3FileSystem.setup():217
    com.dremio.plugins.util.ContainerFileSystem.initialize():167
    com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1730
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
    com.google.common.cache.LocalCache$Segment.loadSync():2282
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
    com.google.common.cache.LocalCache$Segment.get():2049
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (software.amazon.awssdk.core.exception.SdkClientException) Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
    software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build():98
    software.amazon.awssdk.core.exception.SdkClientException.create():43
    software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException():198
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():66
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
    software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
    software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
    software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
    software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
    com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
    com.dremio.common.util.Retryer.call():58
    com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
    com.dremio.plugins.s3.store.S3FileSystem.setup():217
    com.dremio.plugins.util.ContainerFileSystem.initialize():167
    com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1730
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
    com.google.common.cache.LocalCache$Segment.loadSync():2282
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
    com.google.common.cache.LocalCache$Segment.get():2049
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (org.apache.http.conn.ConnectTimeoutException) Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
    org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect():151
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect():376
    sun.reflect.GeneratedMethodAccessor10.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke():80
    com.sun.proxy.$Proxy83.connect():-1
    org.apache.http.impl.execchain.MainClientExec.establishRoute():393
    org.apache.http.impl.execchain.MainClientExec.execute():236
    org.apache.http.impl.execchain.ProtocolExec.execute():186
    org.apache.http.impl.client.InternalHttpClient.doExecute():185
    org.apache.http.impl.client.CloseableHttpClient.execute():83
    org.apache.http.impl.client.CloseableHttpClient.execute():56
    software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute():72
    software.amazon.awssdk.http.apache.ApacheHttpClient.execute():253
    software.amazon.awssdk.http.apache.ApacheHttpClient.access$500():106
    software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():232
    software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():229
    software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe():64
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest():76
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():55
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():39
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():73
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():42
    software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():78
    software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():40
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():50
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():36
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():64
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
    software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
    software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
    software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
    software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
    com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
    com.dremio.common.util.Retryer.call():58
    com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
    com.dremio.plugins.s3.store.S3FileSystem.setup():217
    com.dremio.plugins.util.ContainerFileSystem.initialize():167
    com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1730
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
    com.google.common.cache.LocalCache$Segment.loadSync():2282
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
    com.google.common.cache.LocalCache$Segment.get():2049
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (java.net.SocketTimeoutException) connect timed out
    java.net.PlainSocketImpl.socketConnect():-2
    java.net.AbstractPlainSocketImpl.doConnect():350
    java.net.AbstractPlainSocketImpl.connectToAddress():206
    java.net.AbstractPlainSocketImpl.connect():188
    java.net.SocksSocketImpl.connect():392
    java.net.Socket.connect():607
    org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket():368
    software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket():113
    org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect():142
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect():376
    sun.reflect.GeneratedMethodAccessor10.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke():80
    com.sun.proxy.$Proxy83.connect():-1
    org.apache.http.impl.execchain.MainClientExec.establishRoute():393
    org.apache.http.impl.execchain.MainClientExec.execute():236
    org.apache.http.impl.execchain.ProtocolExec.execute():186
    org.apache.http.impl.client.InternalHttpClient.doExecute():185
    org.apache.http.impl.client.CloseableHttpClient.execute():83
    org.apache.http.impl.client.CloseableHttpClient.execute():56
    software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute():72
    software.amazon.awssdk.http.apache.ApacheHttpClient.execute():253
    software.amazon.awssdk.http.apache.ApacheHttpClient.access$500():106
    software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():232
    software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():229
    software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe():64
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest():76
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():55
    software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():39
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():73
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():42
    software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():78
    software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():40
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():50
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():36
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():64
    software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
    software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
    software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
    software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
    software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
    software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
    software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
    software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
    software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
    com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
    com.dremio.common.util.Retryer.call():58
    com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
    com.dremio.plugins.s3.store.S3FileSystem.setup():217
    com.dremio.plugins.util.ContainerFileSystem.initialize():167
    com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1730
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
    com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
    com.google.common.cache.LocalCache$Segment.loadSync():2282
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
    com.google.common.cache.LocalCache$Segment.get():2049
    com.google.common.cache.LocalCache.get():3966
    com.google.common.cache.LocalCache.getOrLoad():3989
    com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
    com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
    com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
    com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
    com.dremio.exec.store.dfs.FileSystemPlugin.start():801
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
    com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
    java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748

Thanks for the help!

Thank you! I’ll take a look into this. While you wait, can you try to run a query on one of the Dremio Sample Sources and let me know if it is successful, and if not, what is the error?

Add Source > Sample Source

Hello Cindy,

Also in the sample data I get a timeout, but in this case from s3:

DATA_READ ERROR: Failure reading JSON file - com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to samples.dremio.com.s3.amazonaws.com:80 [samples.dremio.com.s3.amazonaws.com/52.92.147.1] failed: connect timed out

Thank you, this definitely helps narrow down the main issue if the sample source is also not working. This last error shows that the outbound connectivity to S3 missing.

On your public subnet’s settings, is “Enable auto-assign public IPv4 address” enabled and do you have a route table routing to an internet gateway? If not, can you set those up and retry?

Awesome! The IPv4 setting was the problem - it was not enabled. I still had to respawn a new instance of the preview engine, but it worked after that. Thanks a lot!

Awesome! I’m glad it’s working now. Thanks for reaching out!