Intermittent error reading JSON from S3

1

Hi, guys!

I’m getting this error sometimes when I try to read a JSON file from S3:

DATA_READ ERROR: Failure reading JSON file - null

Query profile when goes wrong:

de210f87-7772-4478-bb34-fbd25401ced4.zip (5.0 KB)

Query profile when goes fine:

52d1e87c-879d-4fb1-994a-3c46a35d7cbd.zip (5.4 KB)

2

Hi @allan.sene

Let me validate this on our end and get back to you

Thanks
@balaji.ramaswamy

3

Hi @allan.sene

Quick question,

The JSON files that you are reading, are they the same or different?

Thanks,
@balaji.ramaswamy

4

Thanks for replying, @balaji.ramaswamy!

Exactly the same file. :thinking:

But this error happens with any file on my bucket

5

Thanks,

So if I understand it right,

You have added these json files as dremio datasets,
When you try to query them you get the above error?

6

Actually, the error occurs when I try to create some dataset from these files:

I have this S3 bucket with lots of json files:

45
Captura de Tela 2017-10-19 às 13.27.45.png3360×2100 887 KB

Sometimes, when I click on a file, I get the expected screen:

33
Captura de Tela 2017-10-19 às 13.28.33.png3360×2100 1.18 MB

Another time, I get an error, (for the same file):

24
Captura de Tela 2017-10-19 às 13.29.24.jpg3360×2100 444 KB

Here goes one file example:
sample.zip (159.6 KB)

7

Hi @allan.sene

I was able to download the JSON you have sent and query it multiple times with out any issues. Wondering if something to do with S3? Can you try and download the JSON to your local drive (just to test), then add it into Dremio and then try querying multiple times? Want to see if this is something specific to S3

Thanks,
@balaji.ramaswamy

1 Like
8

Good idea, @balaji.ramaswamy! Just tried out this and the errors didn’t happen. Almost sure that the problem is on S3 connector. :ok_hand:

9

Have you reproduced this behavior on your environment, @balaji.ramaswamy? Any idea of what could be occasioning this?

10

Hi @allan.sene,

This could be anywhere from network, server or the actual S3 file system issue. Would you be able to see the S3 logs and see if you can find any errors? Can you also please upload the server.log when the problem happened? If you need an older log see under /log/archive/server..log.gz

Thanks,
@balaji.ramaswamy