We are testing with about 16TB of data. The reflection we are attempting to create is on two columns. The columns each have two or three distinct values. The job to create the reflection has been running for about 2hours now with only one executor at about 10% cpu utilization. Is this typical?
@balaji.ramaswamy here is the profile for one (this one failed)75d80de0-b401-4a48-b4fa-9952f1ec35ad.zip (98.5 KB)
All the time is spent in spilling, are you spilling to SSD’s or different type of disk?