Iceberg LOCALSORT and PARTITION can't speed up Dremio Queries

Dremio University
1

Hi all,

I am using version 23.1.50. I have created an Icerberg file using Dremio CTAS with LOCALSORT and PARTITION. However, when I try to run a query against that files, seems that the localsort and partition can’t benefit the queries as similar setting in Reflection. Do anyone have similar experience? Thanks.

2

Hi @alaw, Can you post the query / queries profiles? I am specifically curious about the SELECT query on the localsort + partitioned table.

3

I created the iceberg file using CTAS, you might refer to CREATE | Dremio Documentation

for select, just normal SQL and found that the localsort and partition seems not recognized nor used by Dremio.

4

@alaw, could you provide the query statement or the profile of the query? In your dremio, you can download each query’s profile under the ‘jobs’. Then Select the query your executed, and hit ‘download profile’ at the bottom of the ‘summary’ section.

Are you including a WHERE clause specifically on the column(s) that you perform your Sort Order / Partition on?