I was using Dremio and nessies – initially nessie was setup with in-memory and i created huge iceberg table [iceberg_tbl1] when nessies rebooted obvious metadata gone and now dremio is not able to access
I configure nessies with postgre now metadata is consistent after reboot
I wanna access or register old iceberg table [iceberg_tbl1] tried all way not working ,
any idea on it.
data and metadata all available on S3 i cross verified too .
@rajnish Are you saying the Dremio UI after a reboot is empty? It looks like your KVstore was on a non-persistent disk. Is this a docker image? Can you send your dremio.conf if not a docker image?
Hi @balaji.ramaswamy
Thanks for reply,
Dremio was running on my local machine [not on docker]
cat dremio.conf
paths: {
the local path for dremio to store data.
local: ${DREMIO_HOME}“/data”
the distributed path Dremio data including job results, downloads, uploads, etc
#dist: “pdfs://”${paths.local}“/pdfs”
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
dremio is fine , all table is persist after reboot too.
when i configure Nessies [ with default config – in memory DB] and create a iceberg table – this was is not visible after reboot.
for this table, all metadata and data file saved at s3, i verified all is availble at S3, but how i reenable it on Nessie do not find a way.
Hi
Updating here,
I think there is no way to re-register table, way around options is there, which I applied and it’s working .
access data folder [contain all .parquet files ] and create table format using Dremio
now recreate table using nessies , using CTAS
it working new iceberg table created.
Thanks,