Dremio Service stops on queries

1

When running queries, the Dremio service stops and restarts. The Dremio instance is hosted on AWS EC2. The query datasource is to a PostgrSQL instance hosted on AWS.

Here is the warning I see before the service is restarted.

    2018-10-01 04:13:47,283 [metadata-refresh-CGLogs] WARN  c.d.e.store.jdbc.JdbcSchemaFetcher - Took longer than 5 seconds to query row count for agrian_logs.labelcenter, Using default value of 1000000000.
java.sql.SQLTimeoutException: (conn:6474) Query timed out
	at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:245) ~[mariadb-java-client-1.6.2.jar:na]
	at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:322) ~[mariadb-java-client-1.6.2.jar:na]
	at org.mariadb.jdbc.MariaDbStatement.executeQuery(MariaDbStatement.java:492) ~[mariadb-java-client-1.6.2.jar:na]
	at com.dremio.exec.store.jdbc.JdbcSchemaFetcher.getRowCount(JdbcSchemaFetcher.java:427) [dremio-extra-plugin-jdbc-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.store.jdbc.JdbcSchemaFetcher.getTableDetails(JdbcSchemaFetcher.java:392) [dremio-extra-plugin-jdbc-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.store.jdbc.JdbcTableBuilder.buildIfNeeded(JdbcTableBuilder.java:103) [dremio-extra-plugin-jdbc-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.store.jdbc.JdbcTableBuilder.getDataset(JdbcTableBuilder.java:58) [dremio-extra-plugin-jdbc-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.store.jdbc.JdbcStoragePlugin.checkReadSignature(JdbcStoragePlugin.java:248) [dremio-extra-plugin-jdbc-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.catalog.SourceMetadataManager.refreshFull(SourceMetadataManager.java:388) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.catalog.SourceMetadataManager.doNextRefresh(SourceMetadataManager.java:275) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.catalog.SourceMetadataManager.access$300(SourceMetadataManager.java:63) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.exec.catalog.SourceMetadataManager$RefreshTask.run(SourceMetadataManager.java:574) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) [dremio-common-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) [dremio-common-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) [dremio-common-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:95) [dremio-services-scheduler-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.sql.SQLException: (conn:6474) Query execution was interrupted
Query is: select count(*) from agrian_logs.labelcenter
	at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:113) ~[mariadb-java-client-1.6.2.jar:na]
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:167) ~[mariadb-java-client-1.6.2.jar:na]
	at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:316) ~[mariadb-java-client-1.6.2.jar:na]
	... 21 common frames omitted

We are running Dremio Community Edition build 2.1.4-201808302048550610-0981242.

2

Hi @cyang023

If you run the same query using psql, does it work? Also can you please send me the query profile?

Share Query Profile

Thanks
@balaji.ramaswamy

3

Hi @balaji.ramaswamy

Yes the query does work in psql.

Here’s the query profile https://s3-us-west-1.amazonaws.com/testchong/a333d269-caf6-457e-97a8-c1225ea23b52.zip

Thanks,
@cyang023

4

Hi @cyang023

Kindly run the query one more time and let it fail and then send me the server.log

Thanks
@balaji.ramaswamy