Can you remove the history automatically from Iceberg Tables?

1

We currently have a process where we are running a DELETE job then an INSERT job on a table with a few billion records. We will often delete/insert 10-20 million records which we found (because of Iceberg’s snapshots) that this will often cause a large increase in disk usage until we are able to use VACUUM to remove the deleted records from disk.

Is there any way to turn off snapshots completely? We don’t plan on using snapshots to look at historical data and it will just be extra work to have to vacuum it every day so it would be nice if we could just skip that step completely and never retain the deleted data in the first place.

2

Not at the moment however something that is currently on the backlog

3

Have this become a feature yet?

4

Can’t you just run a VACUUM TABLE <table_name> EXPIRE SNAPSHOTS retain_last 1 after you modify the table?

The amount of work for the system will be the same, as the old parquet files would have to be deleted anyways

5

Well, VACUUM seems to require a lot of permissions to execute.
So it would be a lot easier if it was more gracefully handled.