Dremio docker 4.1.8-202003120636020140-9c2a6b13, configured to use external S3 storage for reflections. Our S3 storage is more like Minio and can not support doimain style access as AWS do.
I’ve got (java.net.UnknownHostException) dremio.s3.gagalabs.ru exception.
Bucket name is dremio and it is used to access files as part of server name: dremio.h.o.s.t/path. More about different access styles: https://github.com/minio/minio/issues/4681
I configured Dremio according to docs as Minio compatible and have these lines in core-site.xml
<property>
<name>fs.s3a.path.style.access</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<description>Value can either be true or false, set to true to use SSL with a secure Minio server.</description>
<value>SSL_ENABLED</value>
</property>
Here is Java stack trace (domain name is changed):
DATA_READ ERROR: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: dremio.s3.gagalabs.ru
Failure opening parquet file
File /dremio/distributed-storage/accelerator/389f0055-d57a-432b-94c0-4746601fc57c/41907aaf-118d-4aeb-8583-e80e4220852b_0/0_0_11.parquet
Fragment 2:0
[Error Id: a08b89d2-6749-4132-8bbe-69112938bd8f on dremio:31010]
(java.io.IOException) software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: dremio.s3.gagalabs.ru
com.dremio.parquet.pages.FooterReader$WrapException.asIO():151
com.dremio.parquet.pages.async.RowGroupReader.getFooter():336
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator.lambda$getFooter$0():258
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator.handleEx():263
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator.getFooter():258
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$PrefetchingIterator.next():199
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$PrefetchingIterator.next():175
com.dremio.sabot.op.scan.ScanOperator.<init>():159
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator.createScan():158
com.dremio.exec.store.parquet.ParquetOperatorCreator.create():78
com.dremio.exec.store.parquet.ParquetOperatorCreator.create():66
com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():94
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():210
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():115
com.dremio.exec.physical.base.AbstractSubScan.accept():85
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitHashAggregate():111
com.dremio.exec.physical.config.HashAggregate.accept():99
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():81
com.dremio.exec.physical.config.Project.accept():55
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():189
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSender():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitHashPartitionSender():169
com.dremio.exec.physical.config.HashPartitionSender.accept():78
com.dremio.sabot.driver.PipelineCreator.get():107
com.dremio.sabot.driver.PipelineCreator.get():101
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():377
com.dremio.sabot.exec.fragment.FragmentExecutor.run():270
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():92
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():674
com.dremio.sabot.task.AsyncTaskWrapper.run():104
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226
com.dremio.sabot.task.slicing.SlicingThread.run():156
Caused By (software.amazon.awssdk.core.exception.SdkClientException) Unable to execute HTTP request: dremio.s3.gagalabs.ru
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build():97
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException():136
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute():94
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():62
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():57
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():240
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke():96
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute():120
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute():61
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():51
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():62
software.amazon.awssdk.services.s3.DefaultS3Client.getObject():1597
software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes():2433
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$S3SyncReadObject.lambda$run$0():134
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$RetryableInvoker.invoke():88
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$S3SyncReadObject.run():134
java.util.concurrent.Executors$RunnableAdapter.call():511
java.util.concurrent.FutureTask.run():266
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Caused By (java.net.UnknownHostException) dremio.s3.gagalabs.ru
java.net.InetAddress.getAllByName0():1281
java.net.InetAddress.getAllByName():1193
java.net.InetAddress.getAllByName():1127
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve():45
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect():112
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect():373
sun.reflect.GeneratedMethodAccessor540.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():498
software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke():80
com.sun.proxy.$Proxy146.connect():-1
org.apache.http.impl.execchain.MainClientExec.establishRoute():381
org.apache.http.impl.execchain.MainClientExec.execute():237
org.apache.http.impl.execchain.ProtocolExec.execute():185
org.apache.http.impl.client.InternalHttpClient.doExecute():185
org.apache.http.impl.client.CloseableHttpClient.execute():83
org.apache.http.impl.client.CloseableHttpClient.execute():56
software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute():72
software.amazon.awssdk.http.apache.ApacheHttpClient.execute():236
software.amazon.awssdk.http.apache.ApacheHttpClient.access$500():102
software.amazon.awssdk.http.apache.ApacheHttpClient$1.call():217
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest():66
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():51
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute():35
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():64
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute():36
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():77
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute():39
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute():113
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute():86
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():62
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():57
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer():80
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():60
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute():42
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute():206
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():37
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute():26
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute():240
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke():96
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute():120
software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute():61
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute():51
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute():62
software.amazon.awssdk.services.s3.DefaultS3Client.getObject():1597
software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes():2433
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$S3SyncReadObject.lambda$run$0():134
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$RetryableInvoker.invoke():88
com.dremio.plugins.s3.store.S3AsyncByteReaderUsingSyncClient$S3SyncReadObject.run():134
java.util.concurrent.Executors$RunnableAdapter.call():511
java.util.concurrent.FutureTask.run():266
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
Failure opening parquet file
File /dremio/distributed-storage/accelerator/389f0055-d57a-432b-94c0-4746601fc57c/41907aaf-118d-4aeb-8583-e80e4220852b_0/0_0_11.parquet
Fragment 2:0
com.dremio.parquet.pages.FooterReader$WrapException(FooterReader.java:151)
com.dremio.parquet.pages.async.RowGroupReader(RowGroupReader.java:336)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator(ParquetOperatorCreator.java:258)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator(ParquetOperatorCreator.java:263)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$SplitReaderCreator(ParquetOperatorCreator.java:258)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$PrefetchingIterator(ParquetOperatorCreator.java:199)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator$PrefetchingIterator(ParquetOperatorCreator.java:175)
com.dremio.sabot.op.scan.ScanOperator(ScanOperator.java:159)
com.dremio.exec.store.parquet.ParquetOperatorCreator$Creator(ParquetOperatorCreator.java:158)
com.dremio.exec.store.parquet.ParquetOperatorCreator(ParquetOperatorCreator.java:78)
com.dremio.exec.store.parquet.ParquetOperatorCreator(ParquetOperatorCreator.java:66)
com.dremio.sabot.driver.OperatorCreatorRegistry(OperatorCreatorRegistry.java:94)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:210)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractSubScan(AbstractSubScan.java:85)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:111)
com.dremio.exec.physical.config.HashAggregate(HashAggregate.java:99)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:81)
com.dremio.exec.physical.config.Project(Project.java:55)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:189)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:169)
com.dremio.exec.physical.config.HashPartitionSender(HashPartitionSender.java:78)
com.dremio.sabot.driver.PipelineCreator(PipelineCreator.java:107)
com.dremio.sabot.driver.PipelineCreator(PipelineCreator.java:101)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:377)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:270)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:92)
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl(FragmentExecutor.java:674)
com.dremio.sabot.task.AsyncTaskWrapper(AsyncTaskWrapper.java:104)
com.dremio.sabot.task.slicing.SlicingThread(SlicingThread.java:226)
com.dremio.sabot.task.slicing.SlicingThread(SlicingThread.java:156)
Is it bug in Dremio or wrong config?