Hi,
How can I debug what is causing exceptions on my queries?
I constantly get failed queries with the following description:
Error processing input: , line=1361220, char=260046848. Content parsed: [ ]
How can I get more information?
Thank you
I happened to have found a verbose description on the reflections exception:
DATA_READ ERROR: Error processing input: , line=1317843, char=251658240. Content parsed: [ ]
Failure while reading file adl://tbdemosadls.azuredatalakestore.net/taxis/yellow/yellow_trip_data_0000000.csv. Happened at or shortly before byte position 2131755008.
SqlOperatorImpl TEXT_SUB_SCAN
Location 1:1:4
Fragment 1:1
[Error Id: 7367a6ed-f59b-4995-82e5-5b74355a5c77 on 10.0.0.6:31010]
(com.univocity.parsers.common.TextParsingException) Error processing input: , line=1317843, char=251658240. Content parsed: [ ]
com.dremio.exec.store.easy.text.compliant.TextReader.handleException():476
com.dremio.exec.store.easy.text.compliant.TextReader.parseNext():385
com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.next():278
com.dremio.sabot.op.scan.ScanOperator.outputData():209
com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():518
com.dremio.sabot.driver.StraightPipe.pump():56
com.dremio.sabot.driver.Pipeline.doPump():82
com.dremio.sabot.driver.Pipeline.pumpOnce():72
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():297
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():293
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1836
com.dremio.sabot.exec.fragment.FragmentExecutor.run():250
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():86
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():591
com.dremio.sabot.task.AsyncTaskWrapper.run():107
com.dremio.sabot.task.slicing.SlicingThread.run():102
Caused By (com.microsoft.azure.datalake.store.ADLException) Error reading data from response stream in positioned read() for file /taxis/yellow/yellow_trip_data_0000000.csv
com.microsoft.azure.datalake.store.ADLFileInputStream.readInternal():221
com.microsoft.azure.datalake.store.ADLFileInputStream.readFromService():127
com.microsoft.azure.datalake.store.ADLFileInputStream.read():96
org.apache.hadoop.fs.adl.AdlFsInputStream.read():106
java.io.DataInputStream.read():149
com.dremio.exec.store.dfs.FSDataInputStreamWrapper$WrappedInputStream.read():250
com.dremio.exec.store.dfs.FSDataInputStreamWithStatsWrapper$WrappedInputStream.read():127
java.io.DataInputStream.read():100
com.dremio.exec.store.easy.text.compliant.TextInput.read():222
com.dremio.exec.store.easy.text.compliant.TextInput.updateBuffer():242
com.dremio.exec.store.easy.text.compliant.TextInput.nextCharNoNewLineCheck():381
com.dremio.exec.store.easy.text.compliant.TextInput.nextChar():291
com.dremio.exec.store.easy.text.compliant.TextReader.parseValueAll():199
com.dremio.exec.store.easy.text.compliant.TextReader.parseValue():212
com.dremio.exec.store.easy.text.compliant.TextReader.parseField():318
com.dremio.exec.store.easy.text.compliant.TextReader.parseRecord():137
com.dremio.exec.store.easy.text.compliant.TextReader.parseNext():366
com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.next():278
com.dremio.sabot.op.scan.ScanOperator.outputData():209
com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():518
com.dremio.sabot.driver.StraightPipe.pump():56
com.dremio.sabot.driver.Pipeline.doPump():82
com.dremio.sabot.driver.Pipeline.pumpOnce():72
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():297
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():293
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1836
com.dremio.sabot.exec.fragment.FragmentExecutor.run():250
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():86
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():591
com.dremio.sabot.task.AsyncTaskWrapper.run():107
com.dremio.sabot.task.slicing.SlicingThread.run():102
Caused By (java.net.SocketException) Connection reset
java.net.SocketInputStream.read():210
java.net.SocketInputStream.read():141
sun.security.ssl.InputRecord.readFully():465
sun.security.ssl.InputRecord.readV3Record():593
sun.security.ssl.InputRecord.read():532
sun.security.ssl.SSLSocketImpl.readRecord():983
sun.security.ssl.SSLSocketImpl.readDataRecord():940
sun.security.ssl.AppInputStream.read():105
java.io.BufferedInputStream.read1():284
java.io.BufferedInputStream.read():345
sun.net.www.http.ChunkedInputStream.fastRead():244
sun.net.www.http.ChunkedInputStream.read():689
java.io.FilterInputStream.read():133
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read():3393
com.microsoft.azure.datalake.store.ADLFileInputStream.readInternal():212
com.microsoft.azure.datalake.store.ADLFileInputStream.readFromService():127
com.microsoft.azure.datalake.store.ADLFileInputStream.read():96
org.apache.hadoop.fs.adl.AdlFsInputStream.read():106
java.io.DataInputStream.read():149
com.dremio.exec.store.dfs.FSDataInputStreamWrapper$WrappedInputStream.read():250
com.dremio.exec.store.dfs.FSDataInputStreamWithStatsWrapper$WrappedInputStream.read():127
java.io.DataInputStream.read():100
com.dremio.exec.store.easy.text.compliant.TextInput.read():222
com.dremio.exec.store.easy.text.compliant.TextInput.updateBuffer():242
com.dremio.exec.store.easy.text.compliant.TextInput.nextCharNoNewLineCheck():381
com.dremio.exec.store.easy.text.compliant.TextInput.nextChar():291
com.dremio.exec.store.easy.text.compliant.TextReader.parseValueAll():199
com.dremio.exec.store.easy.text.compliant.TextReader.parseValue():212
com.dremio.exec.store.easy.text.compliant.TextReader.parseField():318
com.dremio.exec.store.easy.text.compliant.TextReader.parseRecord():137
com.dremio.exec.store.easy.text.compliant.TextReader.parseNext():366
com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.next():278
com.dremio.sabot.op.scan.ScanOperator.outputData():209
com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():518
com.dremio.sabot.driver.StraightPipe.pump():56
com.dremio.sabot.driver.Pipeline.doPump():82
com.dremio.sabot.driver.Pipeline.pumpOnce():72
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():297
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():293
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1836
com.dremio.sabot.exec.fragment.FragmentExecutor.run():250
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():86
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():591
com.dremio.sabot.task.AsyncTaskWrapper.run():107
com.dremio.sabot.task.slicing.SlicingThread.run():102
Failure while reading file adl://tbdemosadls.azuredatalakestore.net/taxis/yellow/yellow_trip_data_0000000.csv. Happened at or shortly before byte position 2131755008.
SqlOperatorImpl TEXT_SUB_SCAN
Location 1:1:4
Fragment 1:1
com.dremio.exec.store.easy.text.compliant.TextReader(TextReader.java:476)
com.dremio.exec.store.easy.text.compliant.TextReader(TextReader.java:385)
com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader(CompliantTextRecordReader.java:278)
com.dremio.sabot.op.scan.ScanOperator(ScanOperator.java:209)
com.dremio.sabot.driver.SmartOp$SmartProducer(SmartOp.java:518)
com.dremio.sabot.driver.StraightPipe(StraightPipe.java:56)
com.dremio.sabot.driver.Pipeline(Pipeline.java:82)
com.dremio.sabot.driver.Pipeline(Pipeline.java:72)
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper(FragmentExecutor.java:297)
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper(FragmentExecutor.java:293)
…(:0)
org.apache.hadoop.security.UserGroupInformation(UserGroupInformation.java:1836)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:250)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:86)
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl(FragmentExecutor.java:591)
com.dremio.sabot.task.AsyncTaskWrapper(AsyncTaskWrapper.java:107)
com.dremio.sabot.task.slicing.SlicingThread(SlicingThread.java:102)
Hi @djpirra
This is caused by Azure SDK shipped with the version of Dremio you have. I guess it is 2.1.6. We will be releasing a newer version of Dremio today/tomorrow which has the upgraded version Azure SDK that does not have this issue
Thanks
@balaji.ramaswamy
Where can I see it this was released?
And what will be the steps to update the version?
Thank you Balaji.
Hi @djpirra
Sorry I made an error in my statement. The fix is coming up in few weeks and I will keep you updated. Meanwhile if it is possible can you please try to break down the text files into smaller sized files (through your ETL job) and see if this problem goes away?
Thanks
@balaji.ramaswamy
So version 3.0 just came out… does it fix this problem?
By the way I did the upgrade but suddenly I am getting this exception:
dremio@dremio1:/$ systemctl status dremio
● dremio.service - Dremio Daemon Server
Loaded: loaded (/etc/systemd/system/dremio.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-10-31 06:00:18 UTC; 4min 38s ago
Docs: https://docs.dremio.com
Main PID: 3728 (code=exited, status=4)
Oct 31 06:00:12 dremio1 dremio[3728]: at java.nio.file.Files.createDirectory(Files.java:674)
Oct 31 06:00:12 dremio1 dremio[3728]: at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
Oct 31 06:00:12 dremio1 dremio[3728]: at java.nio.file.Files.createDirectories(Files.java:767)
Oct 31 06:00:12 dremio1 dremio[3728]: at com.dremio.dac.daemon.PathUtils.checkWritePath(PathUtils.java:45)
Oct 31 06:00:12 dremio1 dremio[3728]: at com.dremio.dac.daemon.DACDaemon.(DACDaemon.java:141)
Oct 31 06:00:12 dremio1 dremio[3728]: at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:267)
Oct 31 06:00:12 dremio1 dremio[3728]: at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:108)
Oct 31 06:00:17 dremio1 systemd[1]: dremio.service: Main process exited, code=exited, status=4/NOPERMISSION
Oct 31 06:00:18 dremio1 systemd[1]: dremio.service: Unit entered failed state.
Oct 31 06:00:18 dremio1 systemd[1]: dremio.service: Failed with result ‘exit-code’.
Did all steps in the docs. What is the issue?
Ok I could use the data on the previous version so since this is a PoC I just cleaned and restarted again.
What I can conclude is that the error on Azure Data Lake is still happening.
In what version this will be fixed?
Thank you
Regards
Luis Simoes