I am working on the “D102Data Reflections” course and ran into an issues when creating a raw reflection. I am receiving a JSON parsing error occurs at TIMES. I starting playing around and noticed that these JSON parsing errors correspond at the same time GC allocation errors. When the reflections are successful, no GC Allocation errors are occurring. Note that I am running Dremio in a docker container locally. I believe I can reproduce this issue by just limiting the containers memory using the first experiences in the D102 course.
Has anyone else experienced this and BTW, when this does occur, the container needs to be restarted…
Below is a sample of the logs…
172.17.0.1 - - [10/Mar/2020:21:40:37 +0000] “GET /apiv2/datasets/summary/T!02%2FTrips HTTP/1.1” 200 2109 “http://192.168.1.169:9047/jobs?filters={“qt”%3A[“UI”%2C"EXTERNAL"%2C"ACCELERATION"]}&order=DESCENDING&sort=st” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36”
2020-03-10T21:40:38.323+0000: [GC (Allocation Failure) [PSYoungGen: 748223K->2286K(780800K)] 825669K->79760K(896512K), 0.0107004 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2020-03-10T21:40:40.922+0000: [GC (Allocation Failure) [PSYoungGen: 779502K->2271K(809984K)] 856976K->79877K(925696K), 0.0093943 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10T21:40:43.637+0000: [GC (Allocation Failure) [PSYoungGen: 808671K->2399K(809472K)] 886277K->80021K(925184K), 0.0100235 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10T21:40:46.386+0000: [GC (Allocation Failure) [PSYoungGen: 808799K->2495K(867328K)] 886421K->80133K(983040K), 0.0096203 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2020-03-10T21:40:49.578+0000: [GC (Allocation Failure) [PSYoungGen: 867263K->2558K(869376K)] 944901K->80213K(985088K), 0.0094482 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
2020-03-10T21:40:52.483+0000: [GC (Allocation Failure) [PSYoungGen: 868862K->2703K(927232K)] 946517K->80374K(1042944K), 0.0125619 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10T21:40:55.619+0000: [GC (Allocation Failure) [PSYoungGen: 926863K->2782K(927744K)] 1004534K->80469K(1043456K), 0.0103858 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2020-03-10T21:40:58.705+0000: [GC (Allocation Failure) [PSYoungGen: 926942K->2878K(989696K)] 1004629K->80581K(1105408K), 0.0107541 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10T21:41:02.035+0000: [GC (Allocation Failure) [PSYoungGen: 989502K->2654K(990720K)] 1067205K->80730K(1106432K), 0.0097766 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10T21:41:05.377+0000: [GC (Allocation Failure) [PSYoungGen: 989790K->1712K(1051648K)] 1067866K->80870K(1167360K), 0.0096435 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2020-03-10 21:41:05,557 [e1 - 2197f6b8-19df-8076-44ec-999daae0e400:frag:1:0] INFO c.d.e.s.easy.json.JSONRecordReader - User Error Occurred [ErrorId: 12d8f171-6280-4c45-926e-003377e62f0c]
com.dremio.common.exceptions.UserException: Error parsing JSON - Unexpected character (‘n’ (code 110)) in numeric value: Exponent indicator not followed by a digit
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776) ~[dremio-common-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.store.easy.json.JSONRecordReader.handleAndRaise(JSONRecordReader.java:191) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:236) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.op.scan.ScanOperator.outputData(ScanOperator.java:236) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.driver.SmartOp$SmartProducer.outputData(SmartOp.java:521) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.driver.StraightPipe.pump(StraightPipe.java:56) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.driver.Pipeline.doPump(Pipeline.java:109) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.driver.Pipeline.pumpOnce(Pipeline.java:99) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run(FragmentExecutor.java:320) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:273) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200(FragmentExecutor.java:87) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:658) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:104) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:226) [dremio-ce-sabot-scheduler-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:156) [dremio-ce-sabot-scheduler-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character (‘n’ (code 110)) in numeric value: Exponent indicator not followed by a digit
at [Source: (com.dremio.exec.hadoop.CompressionInputStreamWrapper); line: 739776, column: 4500]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) ~[jackson-core-2.10.2.jar:2.10.2]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) ~[jackson-core-2.10.2.jar:2.10.2]
at com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) ~[jackson-core-2.10.2.jar:2.10.2]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseFloat(UTF8StreamJsonParser.java:1617) ~[jackson-core-2.10.2.jar:2.10.2]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parsePosNumber(UTF8StreamJsonParser.java:1393) ~[jackson-core-2.10.2.jar:2.10.2]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:789) ~[jackson-core-2.10.2.jar:2.10.2]
at com.dremio.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:336) ~[dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.vector.complex.fn.JsonReader.writeDataSwitch(JsonReader.java:293) ~[dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.vector.complex.fn.JsonReader.writeToVector(JsonReader.java:231) ~[dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.vector.complex.fn.JsonReader.write(JsonReader.java:197) ~[dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
at com.dremio.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:210) [dremio-sabot-kernel-4.1.7-202002220604270180-54f278ec.jar:4.1.7-202002220604270180-54f278ec]
… 12 common frames omitted