Reflection doesn't seem to honor DISTINCT in Virtual Dataset


We have a Physical Dataset (PDS) that are Parquet files that may contain duplicate rows. We have defined a Virtual Dataset (VDS) that references only this PDS and performs (essentially) a SELECT DISTINCT * FROM PDS. We expected that the Raw Reflection built from this VDS would have filtered out the duplicate data.

The behavior we see is that queries against the VDS need to scan the entire dataset, even for simple queries like SELECT * FROM VDS LIMIT 10. I believe that this is due to the fact that the query needs to process the DISTINCT filter every time, but if the Raw Reflection already had filtered out the duplicate data then it wouldn’t need to perform an entire table scan for each query.

Is my theory correct that the DISTINCT is not considered when building the Raw Reflection?

@phillip I kind of remember something like this, would you be able to pass on the profile for me to validate?