I’m using dremio-community-24.2.2. My data is in Iceberg table on S3 and the data load from spark not dremio.
The first issue is that the metadata refresh sql always refresh whole table even though I specify “FOR PARTITIONS” in sql. Anyone could tell me the reason for it? Is it the same issue mentioned in 24.2.3’s release note.?
The second issue is that in the 24.2.0’s release note I can see Dremio begin to support incremental update reflection for Iceberg table. But still not able to enable the feature button on settings.
In regards to your 2nd issue, “Incremental updating is not available for Iceberg datasets”, that’s expected to be returned for now in the UI. Dremio knows when it’s best to choose the type of refresh for a certain dataset for Iceberg (and a few other formats), and most/all of the times Dremio will do that in the background, thus the user will not have the option to configure it, but will let the user know what is actually happening in the background in a newer release. Please keep an eye on this enhancement in the upcoming release, as additional messaging is being added for clarity.
