We have a dataset in parquet, and we’ve used Dremio to add some new columns here and change some others there, without the need of an ETL, so Dremio has been pretty useful for this task.
We’ve saved the results as a Dataset (I guess that’s a Virtual Dataset), and we want to create an Aggregation Reflection from that, with 8 dimensions and just one 1 measure. The parquet has been created with three partition levels, by year, month and day, and those three fields are used as dimensions as well.
The dataset is not that big big, with 10 000 million rows, and even if the machine where Dremio is running has a good deal of RAM, the reflection is never finished, it’s always throwing a “Query was cancelled because it exceeded the memory limits set by the administrator” error. We’ve set the dremio_conf like:
which seems to be too much. The question would be, is there any known limit to the amount of rows/dimensions/cardinality that can be used for an Aggregation Reflection? Also, a second question would be, how can we force the creation of an Aggregation Reflection on a Virtual Dataset? So far, we are doing it but removing all reflections, and creating it again,
We are using Dremio 3.3 on CentOS server, we’ll be glad with any help you can provide!