I’ve noticed that typically S3 parallelism is always
MAX(number of files being queried, number of nodes available)
Repartitioning the files using reflections typically solves the issue but building that reflection STILL has the above issue so building the reflection then becomes slow and the problem has just been shifting into reflection building.
Is there anyway to get greater parallelism from S3 based queries aside from manually repartitioning the files so the number of files > number nodes always.