The source catalog is currently unavailable. Metadata is not accessible; please Check node health (Or external storage) and permissions. I Could not connect to AWS Glue source catalog, Check your credentials and network settings

Dremio Cloud
1

On Friday and still today, Dremio Cloud is having problems loading the source into the Glue catalog on AWS.
This strange behavior suggests credential problems, but when I reload the pages and query the VDSs, I am successful in the query, but it oscillates between the error in the source and normality.

Some prints:

image
image1875×500 40 KB

2

@somativa-jackson-fre Is there a chance the Glue source had issues? If you look at the server.log on the coordinator, is there an error message with a reason?

3

In the log_attempt_0.json file

I had the error:

{“message”: "Log searching failed with exception VALIDATION ERROR: Object ‘json’ not found. Please check that it exists in the selected context.

startLine 7
startColumn 3
endLine 7
endColumn 6
SQL Query CREATE TABLE “__support”.“e5ce7098-cee6-47ca-9867-a3fc4562a2b0”
STORE AS (type => ‘json’, prettyPrint => false)
WITH SINGLE WRITER
AS
SELECT *
FROM
json
WHERE
(
cast(
substr(“timestamp”,
0,
length(“timestamp”) - 4
) as timestamp
) between timestamp’2024-08-30 18:25:05.552’ and timestamp’2024-08-30 18:25:21.141’
AND
levelValue >= 30000
)
OR
thread like ‘%192df179-2b5c-e537-8ed5-1c67fd4f9500%’

ErrorOrigin: COORDINATOR
[Error Id: b9a641cd-2404-4d1c-9d07-94b29bcd066d on 10.0.145.31:31010]

(org.apache.calcite.runtime.CalciteContextException) From line 7, column 3 to line 7, column 6: Object ‘json’ not found
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0():-2
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance():62
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():490
org.apache.calcite.runtime.Resources$ExInstWithCause.ex():463
org.apache.calcite.sql.SqlUtil.newContextException():790
org.apache.calcite.sql.SqlUtil.newContextException():775
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():5098
org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl():192
org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl():199
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():1045
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():1005
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3299
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3281
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3560
org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():1045
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():1005
org.apache.calcite.sql.SqlSelect.validate():247
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():980
com.dremio.exec.planner.sql.SqlValidatorImpl.validate():126
com.dremio.exec.planner.sql.SqlValidatorAndToRelContext.validate():104
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateNode():206
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateThenSqlToRel():148
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateAndConvert():176
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateAndConvert():92
com.dremio.exec.planner.sql.handlers.query.DataAdditionCmdHandler.getPlan():166
com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.doCtas():110
com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.getPlan():89
com.dremio.exec.planner.sql.handlers.EnterpriseCreateTableHandler.getPlan():41
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():56
com.dremio.exec.work.foreman.AttemptManager.plan():738
com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():576
com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$1():156
com.dremio.service.commandpool.CommandWrapper.run():73
com.dremio.context.RequestContext.run():103
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():246
com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():222
java.util.concurrent.Executors$RunnableAdapter.call():515
java.util.concurrent.FutureTask.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
Caused By (org.apache.calcite.sql.validate.SqlValidatorException) Object ‘json’ not found
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0():-2
jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance():62
jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():490
org.apache.calcite.runtime.Resources$ExInstWithCause.ex():463
org.apache.calcite.runtime.Resources$ExInst.ex():572
org.apache.calcite.sql.SqlUtil.newContextException():790
org.apache.calcite.sql.SqlUtil.newContextException():775
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():5098
org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl():192
org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl():199
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():1045
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():1005
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3299
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():3281
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3560
org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():1045
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():1005
org.apache.calcite.sql.SqlSelect.validate():247
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():980
com.dremio.exec.planner.sql.SqlValidatorImpl.validate():126
com.dremio.exec.planner.sql.SqlValidatorAndToRelContext.validate():104
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateNode():206
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateThenSqlToRel():148
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateAndConvert():176
com.dremio.exec.planner.sql.handlers.SqlToRelTransformer.validateAndConvert():92
com.dremio.exec.planner.sql.handlers.query.DataAdditionCmdHandler.getPlan():166
com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.doCtas():110
com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.getPlan():89
com.dremio.exec.planner.sql.handlers.EnterpriseCreateTableHandler.getPlan():41
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():56
com.dremio.exec.work.foreman.AttemptManager.plan():738
com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():576
com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$1():156
com.dremio.service.commandpool.CommandWrapper.run():73
com.dremio.context.RequestContext.run():103
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():246
com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():222
java.util.concurrent.Executors$RunnableAdapter.call():515
java.util.concurrent.FutureTask.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
."}

4

I don’t believe the errors are on Glue’s part since I query the catalog normally, but the error makes it take a long time to load the query and keeps running the job infinitely with the message “waiting job submitting” when my machine is already running. When I reload and run it again it works.

image
image1216×638 37.7 KB

5

@somativa-jackson-fre This looks like a wait for a slot in the command pool. How many cores do you have on your coordinator. Can you download one of your job profiles and send it across