Elasticsearch index pattern appears as a physical dataset

Hi. Just trying to query elasticsearch with index pattern like “filebeat*” for example, I’ve noticed that after the query appears a new physical dataset with its name. It may be convenient for reflections settings and so on, but It seems that the dataset periodically disappears. Can someone explain these behaviour please?


It look like your dataset gets un-promoted. It could be possible that the ES index in the backend gets dropped and Dremio removes dataset definitions. Try this, open the elasticsearch source on Dremio-click the metadata tab-uncheck “Remove dataset definitions if underlying data is unavailable.” and save. See screenshot below

Thanks, but it didn’t help. Dataset disappears after some time.

@nzlysenko, where are you querying Elasticsearch with the wildcard(*)? From Dremio?

Yes. I do a simple query like this:
select * from es.“filebeat*”.doc.
And then I can find a corresponding physical dataset in the “sources”.
It would be ideal if it is possible to use this dataset, because if new es index appears every day I can’t setup incremental reflections update on it automatically (or I don’t find it yet in api).

Maybe I create misconception talking about “elasticsearch index pattern” like “index pattern” in Kibana, but I meant just ability to use wildcard in dremio’s sql query.

@nzlysenko, which version of Elasticsearch are you using?

6.8.6, running on elastic cloud with default config