So I have setup Dremio v26 CE on a EC2 machine m6gd.4xlarge. Previously we are using Dremio v24 deployed using cloud formation template.
We tested out a reflection and it took about 14 hrs to complete on dremio v26 CE where as it takes only 7 mins to run on the older setup of v24. I have tried the following, but still the behavior does not change:
- Tried upgrading the instance m6gd.8xlarge
- Enabled disk spill to Nvme
- Enabled distributed storage on s3
- Also explored if there any difference in sys.options of v26 and v24 cant see any major difference, there are some new options and some are deprecated .
What is the reason behind this behavior on PARQUET_WRITER step. The reflection takes most of the time here and peak memory it uses on v26 is 2MB while on v24 it is 88 MB as you can see in the profiles attached. Is this due to some diff in sys.options of these versions or due to any restrictions in v26 CE.
The reflections are same and uses same source db.
v26_reflection_profile.zip (41.2 KB)
v24_reflection_profile.zip (45.1 KB)