Linux RPM upgrade issue to 2.0.1 rc

1

Hi,

i’ve tried to upgrade from 1.4.9 to 2.0.1rc community version
After doing yum install successfully, the upgrade script fails with ConcurrentModificationException.

sudo -u dremio /opt/dremio/bin/dremio-admin upgrade

2018-04-18 11:49:00,439 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2018-04-18 11:49:01,827 [main] INFO c.d.datastore.LocalKVStoreProvider - LocalKVStoreProvider is up
KVStore version is 1.4.9-201802191836310213-7195059
Skipping ReIndex stores [1.0.9, 1.3.0)
Skipping Setting dataset expiry [1.0.9, 1.1.1)
Skipping Setting acceleration refresh and grace policy [1.0.9, 1.2.0)
Mark materializations created before 1.5 as deprecated [1.0.6, 1.5.0)
Marking materialization with id [MaterializationId{id=81104efb-ac01-4f01-94f3-d3cdefa96867}] (for reflection [ReflectionId{id=00f6430f-6e15-4087-bbb8-51dc7d60ad30}], series [0]) as deprecated
Marking materialization with id [MaterializationId{id=e9374b86-6589-4594-86e3-2450ae3f20e8}] (for reflection [ReflectionId{id=07360c46-b83b-4fa9-a68f-3584785bf6d6}], series [0]) as deprecated
Marking materialization with id [MaterializationId{id=ba1048cb-720e-4a41-b073-8ac7097246f7}] (for reflection [ReflectionId{id=07360c46-b83b-4fa9-a68f-3584785bf6d6}], series [1]) as deprecated
Marking materialization with id [MaterializationId{id=59e40c57-eab3-4dac-b5ec-00fef5b7087d}] (for reflection [ReflectionId{id=08bd1f56-193b-4514-8c6f-697a048baad7}], series [0]) as deprecated
Marking materialization with id [MaterializationId{id=59e40c57-eab3-4dac-b5ec-00fef5b7087d}] (for reflection [ReflectionId{id=08bd1f56-193b-4514-8c6f-697a048baad7}], series [1]) as deprecated
2018-04-18 11:49:01,996 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2018-04-18 11:49:02,091 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider
java.util.ConcurrentModificationException: tried to create, found previous version 0
at com.dremio.datastore.OCCStore.put(OCCStore.java:93)
at com.dremio.datastore.OCCStore.put(OCCStore.java:30)
at com.dremio.datastore.indexed.CoreIndexedStoreImpl.put(CoreIndexedStoreImpl.java:105)
at com.dremio.datastore.indexed.CoreIndexedStoreImpl.put(CoreIndexedStoreImpl.java:51)
at com.dremio.datastore.CoreBaseTimedStore.put(CoreBaseTimedStore.java:77)
at com.dremio.datastore.CoreBaseTimedStore$TimedIndexedStoreImplCore.put(CoreBaseTimedStore.java:143)
at com.dremio.datastore.indexed.LocalIndexedStore.put(LocalIndexedStore.java:103)
at com.dremio.service.reflection.store.MaterializationStore.save(MaterializationStore.java:378)
at com.dremio.dac.cmd.upgrade.MarkOldMaterializationsAsDeprecated.upgrade(MarkOldMaterializationsAsDeprecated.java:121)
at com.dremio.dac.cmd.upgrade.Upgrade.upgrade(Upgrade.java:166)
at com.dremio.dac.cmd.upgrade.Upgrade.upgrade(Upgrade.java:141)
at com.dremio.dac.cmd.upgrade.Upgrade.main(Upgrade.java:181)
Upgrade failed java.util.ConcurrentModificationException: tried to create, found previous version 0

and if starting dremio, following error appears

Exception in thread “main” > java.lang.IllegalStateException: KVStore has an older version than the server, please run the upgrade tool first
at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:95)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:125)

Thanks.

2

Is it possible that you’ve ran the upgrade tool more than once ? If it’s the case and you had to do it because the first attempt failed, you need to run it again from the pre-upgrade backup

3

I take that back. Looks like your kvstore has an edge case not handled by the upgrade tool. Investigating if there is a way to fix the issue on 1.4.9 before doing the upgrade. I will get back to you asap.

4

Thanks @hakim. Before today I’ve ran the 2.0.0rc upgrade tool last week but it failed with same error. I’ve restored the 1.4.9 backup, then tried to migrate again today with 2.0.1 rc today

5

I looked at the code and looks like we had a bug in 1.4 that caused the creation of multiple materializations with the same Id (should not be allowed).
One workaround is to start again Dremio 1.4.9, disable all reflections, wait for the acceleration drop materialization jobs to finish (they shouldn’t take too long), then shutdown the cluster and run the upgrade tool again (don’t forget to take a backup first).

6

Is there a command to do that (disable all reflections), or do i have to do that one by one ?

7

You’ll have to do it one by one

8

ok thanks a lot @hakim, i will give it a try

9

I have disabled all reflections (both raw and aggregate)
Unfortunately when I select sys.materializations,
only 2 rows remain, identical:

image
image.png2792×140 22.7 KB

The corresponding acceleration does not exist (I’ve checked in sys.accelerations)
Any idea how to remove such materialization ?

10

how about sys.layouts ? do you see an entry with the same layout_id as those duplicate materializations ?

11

Correct. The layout_id given by the materialization provides a layout that links to the following acceleration, that seems to have an empty table name …

image
image.png2814×110 12.9 KB

12

just to be clear: you do see a corresponding entry in sys.layouts ? if no, then the upgrade task may work

13

Yes I do see an entry in sys.layouts : here it is

image
image.png2810×120 17.4 KB

14

I could manage to make it disappear, by running dremio-admin --delete-orphans and dremio-admin --reindex-data. Thanks again for your help

15

I’m glad you got it working

16

I confirm the upgrade has run successfully and I now run 2.0.1 rc :slightly_smiling_face:

Thanks

1 Like