Getting an error decoding a reflection

nafeger · February 15, 2018, 2:25pm

So i have a dataset that queries a mysql db and it returns data both from a run and a preview. I have a further dataset that queries that dataset to filter the data down to this year, and it is crashing due to it being unable to decode a date. I’m not really sure how to debug this, but here is the stack trace from the job log

DATA_READ ERROR: Failed to decode column created_at::timestamp

Total records decoded and sent upstream 0
PLAIN encoded pages read 0
DICTIONARY encoded pages read 0
Total records decoded in current page and sent upstream after passing filter 0
File path s3a://schoologydevelopment-dremio/storage_cache/accelerator/cd957ac9-a1cf-4718-9a90-c755907704de/774843fe-13a4-40a3-950c-4b3e1dff4292/0_0_0.parquet
Rowgroup index 0
SqlOperatorImpl PARQUET_ROW_GROUP_SCAN
Location 0:0:11
Fragment 0:0

[Error Id: 78f1a336-952e-495f-82c2-78a705528067 on ip-172-31-53-224.ec2.internal:-1]

  (java.io.InterruptedIOException) Reopen at position 99002348 on s3a://schoologydevelopment-dremio/storage_cache/accelerator/cd957ac9-a1cf-4718-9a90-c755907704de/774843fe-13a4-40a3-950c-4b3e1dff4292/0_0_0.parquet: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
    org.apache.hadoop.fs.s3a.S3AUtils.translateException():125
    org.apache.hadoop.fs.s3a.S3AInputStream.reopen():155
    org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek():281
    org.apache.hadoop.fs.s3a.S3AInputStream.read():364
    java.io.DataInputStream.read():149
    com.dremio.exec.store.dfs.FSDataInputStreamWrapper$WrappedInputStream.read():247
    com.dremio.exec.store.dfs.FSDataInputStreamWithStatsWrapper$WrappedInputStream.read():127
    java.io.DataInputStream.read():100
    org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf():109
    com.dremio.parquet.pages.BaseReaderIterator.readFully():157
    com.dremio.parquet.pages.SmartPRI.getPage():100
    com.dremio.parquet.pages.MemoizingPageIterator.getPage():41
    com.dremio.parquet.pages.PageIterator.nextPage():118
    com.dremio.parquet.pages.PageIterator.hasNextPage():63
    com.dremio.parquet.reader.column.generics.TimeStampMilliSimpleReader.evalNextBatch():85
    com.dremio.parquet.reader.SimpleRowGroupReader.eval():39
    com.dremio.extra.exec.store.dfs.parquet.ParquetVectorizedReader.next():293
    com.dremio.exec.store.parquet.UnifiedParquetReader.next():225
    com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.next():80
    com.dremio.sabot.op.scan.ScanOperator.outputData():207
    com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():510
    com.dremio.sabot.driver.StraightPipe.pump():56
    com.dremio.sabot.driver.Pipeline.doPump():82
    com.dremio.sabot.driver.Pipeline.pumpOnce():72
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():288
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():284
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():243
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71
  Caused By (com.amazonaws.SdkClientException) Unable to execute HTTP request: Timeout waiting for connection from pool
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException():1069
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper():1035
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute():742
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer():716
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute():699
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500():667
    com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute():649
    com.amazonaws.http.AmazonHttpClient.execute():513
    com.amazonaws.services.s3.AmazonS3Client.invoke():4221
    com.amazonaws.services.s3.AmazonS3Client.invoke():4168
    com.amazonaws.services.s3.AmazonS3Client.getObject():1378
    org.apache.hadoop.fs.s3a.S3AInputStream.reopen():148
    org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek():281
    org.apache.hadoop.fs.s3a.S3AInputStream.read():364
    java.io.DataInputStream.read():149
    com.dremio.exec.store.dfs.FSDataInputStreamWrapper$WrappedInputStream.read():247
    com.dremio.exec.store.dfs.FSDataInputStreamWithStatsWrapper$WrappedInputStream.read():127
    java.io.DataInputStream.read():100
    org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf():109
    com.dremio.parquet.pages.BaseReaderIterator.readFully():157
    com.dremio.parquet.pages.SmartPRI.getPage():100
    com.dremio.parquet.pages.MemoizingPageIterator.getPage():41
    com.dremio.parquet.pages.PageIterator.nextPage():118
    com.dremio.parquet.pages.PageIterator.hasNextPage():63
    com.dremio.parquet.reader.column.generics.TimeStampMilliSimpleReader.evalNextBatch():85
    com.dremio.parquet.reader.SimpleRowGroupReader.eval():39
    com.dremio.extra.exec.store.dfs.parquet.ParquetVectorizedReader.next():293
    com.dremio.exec.store.parquet.UnifiedParquetReader.next():225
    com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.next():80
    com.dremio.sabot.op.scan.ScanOperator.outputData():207
    com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():510
    com.dremio.sabot.driver.StraightPipe.pump():56
    com.dremio.sabot.driver.Pipeline.doPump():82
    com.dremio.sabot.driver.Pipeline.pumpOnce():72
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():288
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():284
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():243
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71
  Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection():286
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get():263
    sun.reflect.GeneratedMethodAccessor13.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70
    com.amazonaws.http.conn.$Proxy39.get():-1
    org.apache.http.impl.execchain.MainClientExec.execute():190
    org.apache.http.impl.execchain.ProtocolExec.execute():184
    org.apache.http.impl.client.InternalHttpClient.doExecute():184
    org.apache.http.impl.client.CloseableHttpClient.execute():82
    org.apache.http.impl.client.CloseableHttpClient.execute():55
    com.amazonaws.http.apache.client.impl.SdkHttpClient.execute():72
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest():1190
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper():1030
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute():742
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer():716
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute():699
    com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500():667
    com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute():649
    com.amazonaws.http.AmazonHttpClient.execute():513
    com.amazonaws.services.s3.AmazonS3Client.invoke():4221
    com.amazonaws.services.s3.AmazonS3Client.invoke():4168
    com.amazonaws.services.s3.AmazonS3Client.getObject():1378
    org.apache.hadoop.fs.s3a.S3AInputStream.reopen():148
    org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek():281
    org.apache.hadoop.fs.s3a.S3AInputStream.read():364
    java.io.DataInputStream.read():149
    com.dremio.exec.store.dfs.FSDataInputStreamWrapper$WrappedInputStream.read():247
    com.dremio.exec.store.dfs.FSDataInputStreamWithStatsWrapper$WrappedInputStream.read():127
    java.io.DataInputStream.read():100
    org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf():109
    com.dremio.parquet.pages.BaseReaderIterator.readFully():157
    com.dremio.parquet.pages.SmartPRI.getPage():100
    com.dremio.parquet.pages.MemoizingPageIterator.getPage():41
    com.dremio.parquet.pages.PageIterator.nextPage():118
    com.dremio.parquet.pages.PageIterator.hasNextPage():63
    com.dremio.parquet.reader.column.generics.TimeStampMilliSimpleReader.evalNextBatch():85
    com.dremio.parquet.reader.SimpleRowGroupReader.eval():39
    com.dremio.extra.exec.store.dfs.parquet.ParquetVectorizedReader.next():293
    com.dremio.exec.store.parquet.UnifiedParquetReader.next():225
    com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.next():80
    com.dremio.sabot.op.scan.ScanOperator.outputData():207
    com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():510
    com.dremio.sabot.driver.StraightPipe.pump():56
    com.dremio.sabot.driver.Pipeline.doPump():82
    com.dremio.sabot.driver.Pipeline.pumpOnce():72
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():288
    com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():284
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():243
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71

balaji.ramaswamy · February 15, 2018, 5:20pm

Hi @nafeger,

It looks like “s.s3a.connection.maximum” did not fix the issue this thread too, as we have not implemented that feature. Sorry about the confusion. We are working on these features and will keep you posted

It could be when you tried again it worked. Can you please try the same, re-trying again when there is not much activity on the system?

Thanks,
@balaji.ramaswamy

nafeger · February 15, 2018, 6:09pm

When you say “that feature” what do you mean?

balaji.ramaswamy · February 15, 2018, 6:19pm

It is for “fs.s3a.connection.maximum” to take effect …

nafeger · February 15, 2018, 6:20pm

got it, so the issue I’m seeing with the timeout and with the data that I can’t compute are both tied to a break in the stream of data from s3?

Topic		Replies	Views
Error Failed to decode column when refresh VDS reflection	1	2365	October 31, 2019
DATA_READ ERROR: AsyncTimeoutException	4	61	August 12, 2024
DATA_READ ERROR: Failed to decode column <column name>::<column type>	0	1334	August 11, 2022
DATA_READ_ERROR while trying to accelerate reflections	1	1017	September 2, 2022
Error creating Reflection	13	3092	July 3, 2018

Getting an error decoding a reflection

Related topics