Dremio cannot interpret JSON layer in detail from ElasticSearch data source

1

Hi Guru’s,

Im running my own research on utilization of dremio into our production data analytics. During my research, I found an issue that not sure how to solve it.

I’ve manage to connect dremio with my data source from ElasticSearch Cluster, but notice that dremio only manage to translate only 1st layer of JSON tree to columns, not for 2nd and 3rd layer onwards. See below as example

dremio.issue_1.png1336×365 29.8 KB

So far now I’m keen to use dremio as middle man between ElasticSearch as main data lake and client BI tools. Therefore I would highly appreciate if your end have any suggestion or recommendation on above issue if it’s possible to be fix.

2

Hi @Carillpower

If you hover your mouse to the right of a field value, there should be 3 dots, you should be able to click on that, extract out another nested field and again unnest, like screenshot below

image

image
image.png1038×360 23.6 KB

image
image.png578×524 38.7 KB

Thanks
@balaji.ramaswamy

3

Hi Balaji,

Thanks for your quick response, yes I’ve test it and it works like a charm!

Just a quick check, does this feature could be made as defaut or at least extract all in bulk instead of every documentSet ?

4

Good point @Carillpower. Let me check on that internally and get back to you

Thanks
@balaji.ramaswamy

5

Cool, thanks Balaji … will wait for their response :+1:

6

Hi @Carillpower

Currently we unnest only one level, further un-nesting needs to be done via the UI or SQL as mentioned above

Thanks
@balaji.ramaswamy

7

Ok then, will try to absorb this limitation for current time being. Hope there’s a roadmap for this in future releases.

Thanks for the clarification, really appreciate the effort.

8

There are lots of examples of different things you can do in this tutorial on data curation:

Kelly

9

Hi Kelly,

Cool, trying it.

Thumbs up!