Hello,
We have a problem creating VDS using the SQL query ( CREATE VDS v1 AS SELECT * FROM pds1). Dremio will put this job in the enqueued state but never run. The subsequent queries to Dremio will hang unless restarting the Dremio instance. This only happens after upgrading to 3.2.0 and issue go away if we downgrade to 3.1.x
@dli16, does this happen with every CREATE VDS statement or just for a particular one? Any extra information you could supply (query profiles, logs) would be helpful in diagnosing the issue.
I also facing the same problem
docker hub 3.2.2. I use it in docker env
Any update for this issue ?
@Jue_Lin_Zhuang_Jack, please collect the profile for this job as it is enqueued along with thread dumps (using the java jstack tool) of the Dremio coordinator.
To do this second part, on the Dremio coordinator pod, find the PID.
$ ps -ef | grep Dremio
-- note the PID
$ jstack <Dremio PID> > coordinator_thread_dump.txt2019-07-07 13:34:07
Full thread dump OpenJDK 64-Bit Server VM (25.212-b04 mixed mode):
“Attach Listener” #120 daemon prio=9 os_prio=0 tid=0x00007f211424f000 nid=0xe0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
“qtp299830215-119” #119 prio=5 os_prio=0 tid=0x00007f211d0fc800 nid=0xb8 waiting on condition [0x00007f210051a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007165c7f40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.lang.Thread.run(Thread.java:748)
More detail please refer to :
session.zip (6.9 KB)
“22de0d62-218a-6a9c-4a9a-d953738bae00/0:foreman-planning” #106 daemon prio=10 os_prio=0 tid=0x00007f211c343000 nid=0xa7 waiting on condition [0x00007f210577f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006d2de0ac8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142)
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1309)
at com.dremio.dac.explore.QueryExecutor.runQueryWithListener(QueryExecutor.java:121)
at com.dremio.dac.explore.DatasetTool.newUntitled(DatasetTool.java:405)
at com.dremio.dac.service.datasets.DACViewCreatorFactory$DACViewCreator.createView(DACViewCreatorFactory.java:107)
at com.dremio.exec.catalog.CatalogImpl.createView(CatalogImpl.java:348)
at com.dremio.exec.catalog.DelegatingCatalog.createView(DelegatingCatalog.java:141)
at com.dremio.exec.planner.sql.handlers.direct.CreateViewHandler.toResult(CreateViewHandler.java:94)
at com.dremio.exec.planner.sql.handlers.commands.DirectWriterCommand.plan(DirectWriterCommand.java:99)
at com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:389)
at com.dremio.exec.work.foreman.AttemptManager.lambda$run$0(AttemptManager.java:292)
at com.dremio.exec.work.foreman.AttemptManager$$Lambda$201/1773892842.get(Unknown Source)
at com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
@pmo3, as a workaround, can you try running the query that defines the VDS in the Dremio UI and then saving the results with desired name?
@pmo3, are you submitting multiple CREATE VDS statements in quick succession? For example, do you have a script that might trigger this?
Got the same problem here. Apparently the problem show up after a few hours of running. Oddly the very same statement running without problem when triggered via API
If you are issuing many CREATE VDS calls at once then you are running into an issue that we are currently tracking. The workaround for now is to use the REST API to create the VDS using the Catalog API instead of using SQL.