Dremio 2.0.x to 2.1 upgrade takes forever

Hello people,

30 minutes in and the new version of Dremio is stuck at:

KVStore version is 2.0.5-201806021755080191-767cfb5
Upgrade Arrow Schema [1.0.6, 2.1.0)
Re index all stores [1.0.6, 2.1.0)

The server.log says:
2018-09-13 11:57:24,816 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-09-13 11:57:28,136 [main] INFO c.d.datastore.LocalKVStoreProvider - LocalKVStoreProvider is up

From looking at disk space, we had some 16GB of “db” (rocksdb). Which is not, I guess, re-indexed. But it seems to take forever.

We have a fairly big HDFS :slight_smile: with lots of partitions (800k). Dremio here seems to have a scalability issue when using this Rocksdb (whenever it needs to convert it from an old to a new version).

Any ideas on why it takes sooooo long?

Nevermind. It just finished. After 31+ minutes.