Dear all,
We are currently running Dremio in a Kubernetes cluster with Dremio version 24.1. The PVC for our data is configured as follows:
volume2/dremio 50G 19G 32G 38% /opt/dremio/data
Upon attempting to upgrade to the next version, the pod enters a crash loop, and upon inspecting the logs, we have identified a database compatibility issue.
We seek your suggestions or recommendations on resolving this matter.
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: dremio-deployment
namespace: dremio
spec:
replicas: 1
selector:
matchLabels:
app: dremio
template:
metadata:
labels:
app: dremio
spec:
containers:
- name: dremio
image: dremio/dremio-oss:24.1
ports:
- containerPort: 9047
volumeMounts:
- name: dremio-data
mountPath: /opt/dremio/data
volumes:
- name: dremio-data
persistentVolumeClaim:
claimName: dremio-pvc
logs
2023-11-16 10:00:57,853 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2023-11-16 10:00:58,360 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2023-11-16 10:00:58,406 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider
2023-11-16 10:00:58,409 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
java.lang.RuntimeException: org.rocksdb.RocksDBException: Unknown Footer version. Maybe this file was created with newer version of RocksDB?
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:375)
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:89)
at com.dremio.datastore.ByteStoreManager$StoreMetadataManagerImpl.createEntry(ByteStoreManager.java:522)
at com.dremio.datastore.ByteStoreManager$StoreMetadataManagerImpl.access$100(ByteStoreManager.java:492)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:268)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:159)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:191)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:184)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:100)
Caused by: org.rocksdb.RocksDBException: Unknown Footer version. Maybe this file was created with newer version of RocksDB?
at org.rocksdb.RocksDB.get(Native Method)
at org.rocksdb.RocksDB.get(RocksDB.java:810)
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:368)
... 8 common frames omitted
Dremio is exiting. Failure while starting services.
java.lang.RuntimeException: org.rocksdb.RocksDBException: Unknown Footer version. Maybe this file was created with newer version of RocksDB?
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:375)
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:89)
at com.dremio.datastore.ByteStoreManager$StoreMetadataManagerImpl.createEntry(ByteStoreManager.java:522)
at com.dremio.datastore.ByteStoreManager$StoreMetadataManagerImpl.access$100(ByteStoreManager.java:492)
at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:268)
at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:159)
at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:191)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:184)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:100)
Caused by: org.rocksdb.RocksDBException: Unknown Footer version. Maybe this file was created with newer version of RocksDB?
at org.rocksdb.RocksDB.get(Native Method)
at org.rocksdb.RocksDB.get(RocksDB.java:810)
at com.dremio.datastore.RocksDBStore.get(RocksDBStore.java:368)
... 8 more
Heap
garbage-first heap total 2576384K, used 55295K [0x00000006c0000000, 0x00000006c0104ea0, 0x00000007c0000000)
region size 1024K, 47 young (48128K), 11 survivors (11264K)
Metaspace used 23587K, capacity 24084K, committed 24192K, reserved 1071104K
class space used 2795K, capacity 2930K, committed 2944K, reserved 1048576K