The existing file download options for JSON doesn’t create valid json files, but jsonline files which creates one json record per line in the file…
A. Can we clearly make this distinction by changing the file extension from .json to .jsonl
B. Can we add Dremio support to read/query jsonline formatted files?
Thanks!
@david.lee thanks for bringing this to our attention. I’ve created an internal ticket to clarify the current download option behavior.
Are you having issues reading/querying jsonline files? I did a quick test and things worked as expected. Would be good to understand what is not working for you specifically.
I’ll do my own testing with JSONLINE formatted files and give an update. I’m assuming when picking the file type for JSONLINEs I would also pick JSON or maybe TEXT with custom formatting to get the metadata correct…
Ok I just tested this with a couple million jsonline formatted records and it worked fine by just choosing JSON as the file format. Pretty surprised it worked considering the data isn’t really JSON formatted.
See my reply to a similar question: Feature requests + Bug reporting
Dremio (as other bigdata tools) uses jsonlines files because they are easier to split/parallelize. I agree that it might be a bit misleading…