Hi Dremio Team,
elastic query push down is not happening when using flatten() function on a nested array column in elastic search json data, for both preview and run dremio is sending “match_all” to elastic search.
query profiles and examples below
no-query push-down case:
VD1 - with flatten on a nested data column
select * from VD1 ( preview/run)
select * from VD1 where somecolumn = somevalue (preview/run)
execution plan from profile:
[{
“from” : 0,
“size” : 1000,
“query” : {
“match_all” : {
“boost” : 1.0
}
}
}])
query push-down case:
VD2 - do not use flatten on nested data column
select * from VD2 ( preview/run)
select * from VD2 where somecolumn = somevalue (preview/run)
execution plan from profile:
[{
“size” : 0,
“query” : {
“bool” : {
“must” : [
{
“range” : {
“pnlData.pnlHeader.businessDate” : {
“from” : “2019-12-24T00:00:00.000Z”,
“to” : null,
“include_lower” : true,
“include_upper” : true,
“format” : “date_time”,
“boost” : 1.0
}
}
},
{
“range” : {
“pnlData.pnlHeader.businessDate” : {
“from” : null,
“to” : “2019-12-24T00:00:00.000Z”,
“include_lower” : true,
“include_upper” : true,
“format” : “date_time”,
“boost” : 1.0
}
}
}
],
“disable_coord” : false,
“adjust_pure_negative” : true,
“boost” : 1.0
}
}
}])
no-pushdown 0f5078e5-4204-43b5-afb0-efca585319c6.zip (16.5 KB) no-pushdown d5da19ab-d95f-4e30-8ab5-b392b2b52d8f.zip (20.6 KB) pushdown 58179f38-f106-411f-962c-7f9297864398.zip (17.1 KB) pushdown e1c1938a-81b7-4ada-ab3f-ce012448ca20.zip (13.7 KB)