Hi ,
I am using local docker setup with minio and nessie. When i restart the containers, nessie catalog does not see the tables . This is what i see after restart:
I tried to add the table folders but i can’t. How can i solve this?
thanks
tolga
If you are running the Docker Container for Nessie, you have to configure where it will store entries, if not it will by default use the memory store which would be cleared when the server shuts off.
You can find Nessie Configuration for Storage here: Configuration - Project Nessie: Transactional Catalog for Data Lakes with Git-like semantics
Hi Alex , thanks for your help. After setting the db configs, now my metadata is persisted.
Tolga
HI, can u please elaborate the config setup
Facing the same issue here
Hi Shifas,
This is my docker-compose.yml file (only nessie part:)
I am using a local postgre db for catalog data.I created a database called nessie in postgres. ( You can give any name)
nessie:
image: projectnessie/nessie:0.67.0
container_name: nessie
extra_hosts:
- “host.docker.internal:host-gateway”
environment:
NESSIE_VERSION_STORE_TYPE: JDBC
QUARKUS_DATASOURCE_JDBC_URL: “jdbc:postgresql://192.168.0.10:5432/nessie”
quarkus.datasource.username: “postgres”
quarkus.datasource.password: “changeme”
networks:
network:
ipv4_address: 192.168.0.19
ports:
- 19120:19120
thanks
tolga
Have you checked the changes in postgresql, like how is it written on the database for persistence?
Yes i checked it. It creates 2 tables , and for each dml op, it updates these tables.Did you? If you have any insights pls share.
@tolgaevren Are you also losing your tables after a Dremio docker restart
No , table’s data (as parquet files) is in mino objects storage , catalog info is in database . It must be persistent. What happens at your site?
@tolgaevren Sorry the question was directed to @shifas
@shifas When you say you have the same issue, does that mean after a restart your tables no longer exist?
Yes, as mentioned by @tolgaevren, table’s data file exists in the MinIO storage. But catalog info (ie, tables) do not seem to appear on restart
@shifas got it, we nay have to look at the logfile and see what is going on. Can you provide the table name and the server.log when Dremio comes up. Also after it comes up, add the table again and query it and send us the profile and server.log for that time also