I’m testing Dremio 3.2 reflection feature. According to the documentation, now it’s possible to use new data types as reference fields. During my tests, I had the following problem:
- First, I’ve created a reflection on my PDS. I set one DateTime field as my reference field:
Dremio executed successfully the following query:
SELECT `id`, `ultima_atualizacao` AS `$_dremio_$_update_$` FROM `compra`
- However, when an incremental update is running, Dremio executes the following query:
SELECT `id`, `transacao`, `$_dremio_$_update_$` FROM (SELECT **ALL FIELDS IN THE TABLE** FROM `compra`) AS `compra` WHERE `$_dremio_$_update_$` > TIMESTAMP '2019-05-17 16:31:20.000'
Is that right? Was Dremio supposed to do this? This is a poor performance query. It performs a full table scan and, only then, Dremio filter new data. For this query, a better approach would be:
SELECT **REFLECTION FIELDS** FROM `compra` WHERE `reference_field_here` > TIMESTAMP '2019-05-17 16:31:20.000'
Is there a way to make Dremio execute a query like this one above?