Logs flooding with errors with Postgres Source

1

We have a Postgres external data source connected to Dremio.

In the Postgres (docker) logs we are seeing numerous errors continuously repeated as below:

2023-03-01 00:25:59.820 UTC [1555809] ERROR:  relation "public.gradesfinalized" does not exist at character 71
2023-03-01 00:25:59.820 UTC [1555809] STATEMENT:  SELECT reltuples::bigint AS EstimatedCount
        FROM pg_class
        WHERE  oid = 'public.gradesFinalized'::regclass
2023-03-01 01:16:59.821 UTC [1556116] ERROR:  relation "public.gradesfinalized" does not exist at character 71
2023-03-01 01:16:59.821 UTC [1556116] STATEMENT:  SELECT reltuples::bigint AS EstimatedCount
        FROM pg_class
        WHERE  oid = 'public.gradesFinalized'::regclass

Apparently Dremio is looking for “public.gradesfinalized” (all lower case) while the actual table name is ‘public.gradesFinalized’ (note the Capial F).

In the Dremio (docker) logs we are seeing many errors as below:

127.0.0.1 - - [01/Mar/2023:05:30:57 +0000] "GET /vs/basic-languages/src/sql.js HTTP/1.1" 304 0 "http://127.0.0.1:9047/source/Postgres123/public.gradesFinalized?tipVersion=0005811749022867&version=0005811749022867" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.42"
2023-03-01 05:31:59,820 [modifiable-scheduler-11] INFO  c.d.e.store.jdbc.JdbcDatasetMetadata - Took 19 ms to get column metadata for [public, gradesFinalized]
2023-03-01 05:31:59,822 [modifiable-scheduler-11] WARN  c.d.e.s.jdbc.JdbcSchemaFetcherImpl - Took longer than 5 seconds to execute row count query `SELECT reltuples::bigint AS EstimatedCount
FROM pg_class
WHERE  oid = 'public.gradesFinalized'::regclass`.
org.postgresql.util.PSQLException: ERROR: relation "public.gradesfinalized" does not exist
  Position: 71
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:473)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:393)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
        at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:236)
        at org.apache.commons.dbcp2.DelegatingStatement.executeQuery(DelegatingStatement.java:206)
        at org.apache.commons.dbcp2.DelegatingStatement.executeQuery(DelegatingStatement.java:206)
        at com.dremio.exec.store.jdbc.JdbcSchemaFetcherImpl.executeQueryAndGetFirstLong(JdbcSchemaFetcherImpl.java:360)
        at com.dremio.exec.store.jdbc.dialect.PostgreSQLDialect$PGSchemaFetcher.getRowCount(PostgreSQLDialect.java:54)
        at com.dremio.exec.store.jdbc.JdbcDatasetMetadata.build(JdbcDatasetMetadata.java:88)
        at com.dremio.exec.store.jdbc.JdbcSchemaFetcherImpl.getTableMetadata(JdbcSchemaFetcherImpl.java:650)
        at com.dremio.exec.store.jdbc.JdbcStoragePlugin$JdbcDatasetHandle.getTableMetadataResponse(JdbcStoragePlugin.java:473)
        at com.dremio.exec.store.jdbc.JdbcStoragePlugin.listPartitionChunks(JdbcStoragePlugin.java:262)
        at com.dremio.exec.catalog.DatasetSaverImpl.saveUsingV1Flow(DatasetSaverImpl.java:245)
        at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:121)
        at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:137)
        at com.dremio.exec.catalog.MetadataSynchronizer.tryHandleExistingDataset(MetadataSynchronizer.java:307)
        at com.dremio.exec.catalog.MetadataSynchronizer.handleExistingDataset(MetadataSynchronizer.java:225)
        at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:197)
        at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:130)
        at com.dremio.exec.catalog.SourceMetadataManager$RefreshRunner.refreshFull(SourceMetadataManager.java:442)
        at com.dremio.exec.catalog.SourceMetadataManager$BackgroundRefresh.run(SourceMetadataManager.java:556)
        at com.dremio.exec.catalog.SourceMetadataManager.wakeup(SourceMetadataManager.java:259)
        at com.dremio.exec.catalog.SourceMetadataManager.access$100(SourceMetadataManager.java:96)
        at com.dremio.exec.catalog.SourceMetadataManager$WakeupWorker.run(SourceMetadataManager.java:198)
        at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:209)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-03-01 05:31:59,823 [modifiable-scheduler-11] INFO  c.d.e.store.jdbc.JdbcDatasetMetadata - Took 2 ms to get row count for [public, gradesFinalized]
2023-03-01 05:35:02,508 [scheduler-18] INFO  c.d.e.w.p.ActiveQueryListService - Starting activeQueryListTask on this coordinator.
2023-03-01 05:35:02,509 [FABRIC-rpc-event-queue] INFO  com.dremio.sabot.exec.MaestroProxy - There are no active queries on this executor. So nothing to reconcile.

The log file has grown to multiple GBs in size pretty quickly due to this vast number of logs.

However, the queries and other things seem to work fine while accessing the Postgres data from Dremio.

Not sure what is causing these error logs - hope it can be fixed. Thank you.

2

@KrishnaPG

Can you please run the below command and see if the issue goes away?

alter pds <postgres-source-name-in-Dremio>.public.gradesfinalized FORGET METADATA
3

Thank you @balaji.ramaswamy Unfortunately, it did not work.

The alter PDS query completed successfully on Dremio. But the logs did not stop.

image
image1397×1140 89.7 KB

4

@KrishnaPG Can we try something? Add a new PG source with the same credentials, settings, just a new name like “testPG” and then see if you get these messages for that source too

Meanshile, can you send the server.log and metadata_refresh.log for the dates you see this

5

Thank you @balaji.ramaswamy Tried with a new PG source as suggested, and unfortunately the same errors are happening.

Attached the dremio and postgres docker logs for this session. logs.zip (28.1 KB)

(There are no server.log or metadata_refresh.log in the dremio docker container or other locations. only server.out file is present)

6

@KrishnaPG Can you check if there is only one table by name gradesfinalized in information_schema.“tables”, if unable to see, send us the PARQUET download of the below query

select * from information_schema."tables"
8

Here is the query results:
queryResults.zip (2.4 KB)

9

@KrishnaPG Dremio’s metadata only shows gradesFinalized

Can you please check pg_views?