Hello,
I just set up Dremio Cloud succesfully in my AWS account in eu-central-1. I managed to set up the connection to the Glue Data catalog available in the AWS account in which also Dremio runs - by using an IAM user that has all the necessary permissions.
When opening the data source to explore, I can properly see all the databases and tables listed. However, as soon as I try to preview the tables in the data source, I get the error Failure getting source [__metadata]. While digging deeper, I can see the following in the stacktrace:
Caused By (com.dremio.common.util.Retryer.OperationFailedAfterRetriesException) software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443
Dremio Cloud is running in a public VPC in eu-central-1.
I made sure that eu-central-1 is selected as AWS region for the data source. I don’t understand why Dremio Cloud still tries to connect to the us-east-1 STS, even if accessing the EU control plane.
Hello!
Welcome to the Dremio community!
Dremio automatically contacts the us-east-1 STS by default if no region is set for the S3 source. You can change this by following these steps:
- Right click your S3 source and select Edit Details
- Select Advanced Options on the left side menu
- Under connection properties, select Add a Property and name it “dremio.s3.property”. For the value, it would be “eu-central-1”.
This will allow Dremio to connect to your specified STS endpoint.
Hope this helps!
Cindy
Hello,
I tried it as you suggested, as you can see in the attached image.
I tried accessing a table with both “Enable asynchronous access when possible” turned on and off.
In both cases Dremio Cloud still tries to connect to the US sts…
Thank you for having a look…hope there’s something that can be done.
Hi,
Can you add another connection property and name it, “fs.s3a.endpoint”, with the value as “s3.eu-central-1.amazonaws.com”? If this still doesn’t work please attach the full error logs so I can take a further look.
Cindy
Hello Cindy,
thanks for the quick reply. Here is the stack of the failure:
VALIDATION ERROR: Failure getting source [__metadata].
Fragment 1:0
[Error Id: 43e438d2-fe57-4fed-98f2-2481923f2d4a on 10.0.25.127:0]
(java.util.concurrent.ExecutionException) com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Credential Verification failed.
java.util.concurrent.CompletableFuture.reportGet():357
java.util.concurrent.CompletableFuture.get():1928
com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin():1046
com.dremio.exec.catalog.ManagedStoragePlugin.replacePluginWithLock():1004
com.dremio.exec.catalog.PluginsManager.getSynchronized():359
com.dremio.exec.catalog.CatalogServiceImpl.getPlugin():549
com.dremio.exec.catalog.CatalogServiceImpl.getSource():592
com.dremio.exec.planner.PhysicalPlanReader$1.getSource():118
com.dremio.exec.store.iceberg.IcebergManifestWriterPOP.<init>():50
sun.reflect.GeneratedConstructorAccessor263.newInstance():-1
sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():423
com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call():128
com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():291
com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith():288
com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():202
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():518
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1405
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1371
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():217
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():186
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():144
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():110
com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():539
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():563
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():438
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1405
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject():247
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1371
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():217
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():186
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize():116
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():144
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():110
com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():263
com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():74
com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue():322
com.fasterxml.jackson.databind.ObjectReader._bindAndClose():2051
com.fasterxml.jackson.databind.ObjectReader.readValue():1427
com.dremio.common.serde.ProtobufByteStringSerDe.readValue():135
com.dremio.exec.planner.PhysicalPlanReader.readValue():235
com.dremio.exec.planner.PhysicalPlanReader.readFragmentOperator():224
com.dremio.exec.planner.fragment.CachedFragmentReader.lambda$readFragment$1():77
java.util.concurrent.ConcurrentHashMap.computeIfAbsent():1660
com.dremio.exec.planner.fragment.CachedFragmentReader.readFragment():75
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():470
com.dremio.sabot.exec.fragment.FragmentExecutor.run():336
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600():100
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():834
com.dremio.sabot.task.AsyncTaskWrapper.run():120
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():247
com.dremio.sabot.task.slicing.SlicingThread.run():171
Caused By (com.google.common.util.concurrent.UncheckedExecutionException) java.lang.RuntimeException: Credential Verification failed.
com.google.common.cache.LocalCache$Segment.get():2055
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (java.lang.RuntimeException) Credential Verification failed.
com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():237
com.dremio.plugins.s3.store.S3FileSystem.setup():217
com.dremio.plugins.util.ContainerFileSystem.initialize():167
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1730
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
com.google.common.cache.LocalCache$Segment.loadSync():2282
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
com.google.common.cache.LocalCache$Segment.get():2049
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (com.dremio.common.util.Retryer.OperationFailedAfterRetriesException) software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
com.dremio.common.util.Retryer.call():62
com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
com.dremio.plugins.s3.store.S3FileSystem.setup():217
com.dremio.plugins.util.ContainerFileSystem.initialize():167
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1730
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
com.google.common.cache.LocalCache$Segment.loadSync():2282
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
com.google.common.cache.LocalCache$Segment.get():2049
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (software.amazon.awssdk.core.exception.SdkClientException) Unable to execute HTTP request: Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build():98
software.amazon.awssdk.core.exception.SdkClientException.create():43
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException():198
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():66
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
com.dremio.common.util.Retryer.call():58
com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
com.dremio.plugins.s3.store.S3FileSystem.setup():217
com.dremio.plugins.util.ContainerFileSystem.initialize():167
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1730
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
com.google.common.cache.LocalCache$Segment.loadSync():2282
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
com.google.common.cache.LocalCache$Segment.get():2049
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (org.apache.http.conn.ConnectTimeoutException) Connect to sts.us-east-1.amazonaws.com:443 [sts.us-east-1.amazonaws.com/72.21.206.96] failed: connect timed out
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect():151
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect():376
sun.reflect.GeneratedMethodAccessor10.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():498
software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke():80
com.sun.proxy.$Proxy83.connect():-1
org.apache.http.impl.execchain.MainClientExec.establishRoute():393
org.apache.http.impl.execchain.MainClientExec.execute():236
org.apache.http.impl.execchain.ProtocolExec.execute():186
org.apache.http.impl.client.InternalHttpClient.doExecute():185
org.apache.http.impl.client.CloseableHttpClient.execute():83
org.apache.http.impl.client.CloseableHttpClient.execute():56
software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute():72
software.amazon.awssdk.http.apache.ApacheHttpClient.execute():253
software.amazon.awssdk.http.apache.ApacheHttpClient.access$500():106
software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():232
software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():229
software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe():64
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest():76
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():55
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():39
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():73
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():78
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():40
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():50
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():64
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
com.dremio.common.util.Retryer.call():58
com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
com.dremio.plugins.s3.store.S3FileSystem.setup():217
com.dremio.plugins.util.ContainerFileSystem.initialize():167
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1730
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
com.google.common.cache.LocalCache$Segment.loadSync():2282
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
com.google.common.cache.LocalCache$Segment.get():2049
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (java.net.SocketTimeoutException) connect timed out
java.net.PlainSocketImpl.socketConnect():-2
java.net.AbstractPlainSocketImpl.doConnect():350
java.net.AbstractPlainSocketImpl.connectToAddress():206
java.net.AbstractPlainSocketImpl.connect():188
java.net.SocksSocketImpl.connect():392
java.net.Socket.connect():607
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket():368
software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket():113
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect():142
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect():376
sun.reflect.GeneratedMethodAccessor10.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():498
software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke():80
com.sun.proxy.$Proxy83.connect():-1
org.apache.http.impl.execchain.MainClientExec.establishRoute():393
org.apache.http.impl.execchain.MainClientExec.execute():236
org.apache.http.impl.execchain.ProtocolExec.execute():186
org.apache.http.impl.client.InternalHttpClient.doExecute():185
org.apache.http.impl.client.CloseableHttpClient.execute():83
org.apache.http.impl.client.CloseableHttpClient.execute():56
software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute():72
software.amazon.awssdk.http.apache.ApacheHttpClient.execute():253
software.amazon.awssdk.http.apache.ApacheHttpClient.access$500():106
software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():232
software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():229
software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe():64
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest():76
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():55
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():39
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():73
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():78
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():40
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():50
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():64
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():34
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():56
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():48
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute():31
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():193
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke():135
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute():161
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1():114
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess():169
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute():95
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():45
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():55
software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity():972
com.dremio.plugins.s3.store.S3FileSystem.lambda$verifyCredentials$7():233
com.dremio.common.util.Retryer.call():58
com.dremio.plugins.s3.store.S3FileSystem.verifyCredentials():231
com.dremio.plugins.s3.store.S3FileSystem.setup():217
com.dremio.plugins.util.ContainerFileSystem.initialize():167
com.dremio.exec.store.dfs.FileSystemPlugin$1.lambda$load$0():247
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1730
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():252
com.dremio.exec.store.dfs.FileSystemPlugin$1.load():229
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3533
com.google.common.cache.LocalCache$Segment.loadSync():2282
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2159
com.google.common.cache.LocalCache$Segment.get():2049
com.google.common.cache.LocalCache.get():3966
com.google.common.cache.LocalCache.getOrLoad():3989
com.google.common.cache.LocalCache$LocalLoadingCache.get():4950
com.dremio.exec.store.dfs.FileSystemPlugin.newFileSystem():486
com.dremio.exec.store.dfs.MayBeDistFileSystemPlugin.newFileSystem():75
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():474
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():470
com.dremio.exec.store.dfs.FileSystemPlugin.createFS():451
com.dremio.exec.store.dfs.FileSystemPlugin.start():801
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2():571
com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4():639
java.util.concurrent.CompletableFuture$AsyncSupply.run():1604
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Thanks for the help!
Thank you! I’ll take a look into this. While you wait, can you try to run a query on one of the Dremio Sample Sources and let me know if it is successful, and if not, what is the error?
Add Source > Sample Source
Hello Cindy,
Also in the sample data I get a timeout, but in this case from s3:
DATA_READ ERROR: Failure reading JSON file - com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to samples.dremio.com.s3.amazonaws.com:80 [samples.dremio.com.s3.amazonaws.com/52.92.147.1] failed: connect timed out
Thank you, this definitely helps narrow down the main issue if the sample source is also not working. This last error shows that the outbound connectivity to S3 missing.
On your public subnet’s settings, is “Enable auto-assign public IPv4 address” enabled and do you have a route table routing to an internet gateway? If not, can you set those up and retry?
Awesome! The IPv4 setting was the problem - it was not enabled. I still had to respawn a new instance of the preview engine, but it worked after that. Thanks a lot!
Awesome! I’m glad it’s working now. Thanks for reaching out!
