We are in the process of evaluating Dremio Enterprise Edition. We have one coordinator and two executors cluster setup.
After adding our Data Lake ( Delta table format on PureStorage S3 storage), we are able to create PDS and query it. However, when we try to enable Raw Reflections on the same PDS and try to query it then getting the below exception:
SYSTEM ERROR: UnknownHostException: s3-example-bucket.s3-flashblade-host
SqlOperatorImpl ICEBERG_SUB_SCAN
Location 2:0:3
SqlOperatorImpl ICEBERG_SUB_SCAN
Location 2:0:3
Fragment 2:0[Error Id: ed86a3b4-616b-4385-ac27-172e7aec5d53 on exec002.example.com:0]
(org.apache.iceberg.exceptions.RuntimeIOException) Failed to read file: com.dremio.exec.store.iceberg.DremioInputFile@7a942289
org.apache.iceberg.TableMetadataParser.read():253
org.apache.iceberg.TableMetadataParser.read():245
com.dremio.exec.store.iceberg.IcebergManifestListRecordReader.setup():147
com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():311
com.dremio.sabot.op.scan.ScanOperator.setupReader():302
com.dremio.sabot.op.scan.ScanOperator.setup():266
com.dremio.sabot.driver.SmartOp$SmartProducer.setup():563
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():79
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():63
com.dremio.sabot.driver.SmartOp$SmartProducer.accept():533
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.Pipeline.setup():69
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():431
com.dremio.sabot.exec.fragment.FragmentExecutor.run():283
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600():95
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():773
com.dremio.sabot.task.AsyncTaskWrapper.run():120
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():243
com.dremio.sabot.task.slicing.SlicingThread.run():171
Caused By (java.io.IOException) software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
com.dremio.parquet.pages.async.SlidingWindowReader.read():253
com.dremio.extra.exec.store.dfs.iceberg.AsyncSeekableInputStreamFactory$1.read():89
com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded():524
com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding():129
Please note below:
- Created file core-site.xml at /etc/dremio/ for S3 Host configs
- Config Updated in dremio.conf (located at /etc/dremio/dremio.conf) as:
paths: {
the local path for dremio to store data.
local: “/var/lib/dremio”the distributed path Dremio data including job results, downloads, uploads, etc
#dist: “pdfs://”${paths.local}“/pdfs”
dist: “dremioS3:///qa-dremio-storage/reflections”
}
- At this dist: “dremioS3:///qa-dremio-storage/reflections” location, Dremio service is able to create internal directory structures.
- Reflection creation job is getting completed successfully.
Can somebody please point out where to look for?
Thanks,
Vikram.