ERROR: ExecutionSetupException: One or more nodes lost connectivity during query

Dremio Cloud
1

I am using Dremio Cloud in AWS with the default CF template. When I use Dagster to run multiple jobs using Dremio, I often get the ERROR: ExecutionSetupException: One or more nodes lost connectivity during query. It happens intermittently with different queries in different jobs.

This is the engine configuration
SIZE M
MIN REPLICAS 1
MAX REPLICAS 10
MAX CONCURRENCY 10

How can I debug this to find what is causing the error? Is there a way to activate more detailed logs?

2

@ricardolg That error means, the executor listed did not respond. Are there other queries running in the system at the same time. To narrow down the issue, is it possible to try two things

  • Can we run just this query?
  • IF the above succeeds, try to run 2 or 3 at the same time

It does looks like as you rightly pointed out happens during load. You can also create multiple engines and route different queries to different engines