Hello,
with the latest release of Dremio it should be possible to enable iceberg support for near realtime metadata refresh. Documentation states
To activate this functionality, use the dremio.iceberg.enabled and dremio.execution.support_unlimited_splits flags.
it is unclear where to set those flags. I found another entry in community, that this can be done using sql
ALTER SYSTEM SET dremio.iceberg.enabled = true;
ALTER SYSTEM SET reflection.manager.use_iceberg_dataset.enabled = true;
but ideally we would like to activate this feature in automated way. Any hints would be appreciated.
@igorsechyn You can do it via SQL or REST using POST SQL or via UI-admin-support menu
Hi @balaji.ramaswamy , thank you for your reply. That means we cannot do it in automated way when we deploy new clusters? Is there a way to do it through a config file, so that we do not have to write manual scripts to execute these steps, when provisioning new clusters?
@igorsechyn There is a way to enable via dremio-env file, let me get back this
That would be great, thank you
Hey @balaji.ramaswamy , any update on this?
@igorsechyn Sorry, I was OOO. Will find out this week and get back to you
hi @balaji.ramaswamy , sorry to annoy you, but is there any update on this. This prevents us from upgrading to dremio 18 atm. Thank you, Igor
@igorsechyn Sorry for the delay, here you go
prepend the “-Ddremio.debug.sysopt.” to the actual sys option
Couple of examples
DREMIO_JAVA_EXTRA_OPTS="-Ddremio.debug.sysopt.store.plugin.wait_millis=120000"
DREMIO_JAVA_EXTRA_OPTS="-Ddremio.debug.sysopt.exec.preferred.codegenerator=java"
