Select * queries failing with IllegalArgumentException: You attempted to send while inside the rpc event thread

1

Hello,

Context:
Build 4.6.1-202007220122450047-62e084d0
Edition Community Edition
Build Time 07/22/2020 06:59:37
Change Hash 62e084d0518f722aee1af6a38502b0f14a5f1508
Change Time 07/22/2020 01:55:01

Running on kerberized YARN cluster,

Our simple select * queries are intermittently failing with

IllegalArgumentException: You attempted to send while inside the rpc event thread. This isn't allowed because sending will block if the channel is backed up.

dremio.conf

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  dist: "hdfs://10.66.53.27:8020/app/blueshift/dremio"

}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true
}

zookeeper: "cmmt-53-27.mmt.com:2181,cmmt-53-28.mmt.com:2181,cmmt-53-29.mmt.com:2181,cmmt-53-30.mmt.com:2181,cmmt-53-31.mmt.com:2181"
services.coordinator.master.embedded-zookeeper.enabled: false

services.kerberos: {
      principal: "blueshift@MMT.COM", # principal name must be generic and not tied to any host.
      keytab.file.path: "/etc/keytab/blueshift.keytab"
}

I am able to create elastic engine instances and create datasets from HDFS.

thanks,
Saurabh