I have been trying to create Raw reflection on datasets which contain over 60 million records. The PDS is created out of CSV file. But the reflection “refresh” is taking over 36 minutes.
I am working on an application which requires the user to start applying aggregations on such huge datasets in very less time. Reflections seem like a solution to this, but over half an hour of wait time for an end user every time a new file is processed would be difficult.
I tried referring to multiple posts on dremio community, but could not find the right solution. I have also tried applying “Minimum refresh time” while creating reflection. Still no luck.
Could you please suggest what can I do to reduce the reflection creation/refresh time