Unable to remove/edit source

1

I have a postgres source on my dremio that is invalid now and need to be removed, but throws the following error.

2019-12-16 16:23:06,568 [catalog-source-synchronization] WARN  c.dremio.exec.catalog.PluginsManager - Failure updating source [SourceConfig{id=EntityId{id=ceb7f389-c16b-4b5a-bd50-944c375b0198}, legacySourceTypeEnum=null, name=ks-com, ctime=1570651189221, img=null, description=null, config=<ByteString@544d737f size=126>, version=null, accelerationTTL=null, metadataPolicy=MetadataPolicy{namesRefreshMs=3600000, datasetUpdateMode=PREFETCH_QUERIED, datasetDefinitionTtlMs=null, authTtlMs=86400000, datasetDefinitionRefreshAfterMs=3600000, datasetDefinitionExpireAfterMs=10800000, deleteUnavailableDatasets=true, autoPromoteDatasets=false}, lastRefreshDate=null, accelerationRefreshPeriod=3600000, accelerationGracePeriod=10800000, type=REDSHIFT, accelerationNeverExpire=true, accelerationNeverRefresh=true, tag=1, configOrdinal=1}] during scheduled updates.
java.util.ConcurrentModificationException: Source [ks-com] was updated, and the given configuration has older ctime (current: 1576513009824, given: 1570651189221)
	at com.dremio.exec.catalog.ManagedStoragePlugin.synchronizeSource(ManagedStoragePlugin.java:220) ~[dremio-sabot-kernel-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.exec.catalog.PluginsManager.getSynchronized(PluginsManager.java:318) ~[dremio-sabot-kernel-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.exec.catalog.PluginsManager.synchronizeSources(PluginsManager.java:401) ~[dremio-sabot-kernel-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.exec.catalog.PluginsManager$Refresher.run(PluginsManager.java:120) [dremio-sabot-kernel-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) [dremio-common-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) [dremio-common-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) [dremio-common-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:187) [dremio-services-scheduler-4.0.5-201911202046080257-19b10938.jar:4.0.5-201911202046080257-19b10938]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_232]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_232]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
2019-12-16 16:23:06,786 [UserServer-1] INFO  c.d.sabot.rpc.user.UserRPCServer - [USER]: Channel closed /173.16.165.225:31010 <--> /173.16.100.71:41169 (user client)

attached screenshot.

Screenshot 2019-12-16 at 9.49.45 PM
Screenshot 2019-12-16 at 9.49.45 PM2880×1602 222 KB

some more info:

  • The whole data was migrated from dremio 3 to 4.
  • The source is no longer valid/running.
2

@Rohithzr

Have you tried this?

#1 Add a new Postgres source with a different name
#2 Restart Dremio
#3 Remove old source

Thanks
@balaji.ramaswamy

3

@balaji.ramaswamy

Just to confirm, here is what I did

  1. created a new postgres source on RDS
  2. added a new source on dremio
  3. restarted the master and executor pods
  4. tried to delete the older resource, shows error but the resource got removed
  5. tried to delete the new resource, shows error but the resource got removed.

So thanks. is this issue specific to postgres?

4

@Rohithzr

This is nothing to do with Postgres. It looks like something went wrong in the internal metadata store. Were you able to add the Postgres source finally one more time?

5

yes I was able to add a new source. thanks.