Query ES errors, but execute again to get the correct result

1

DATA_READ ERROR: Failure executing Elastic request get next search result.

Request: http://192.168.61.143:9201/_search/scroll
Response Status 500
Response Reason Internal Server Error
Response Body {“error”:{“root_cause”:[{“type”:“null_pointer_exception”,“reason”:null}],“type”:“search_phase_execution_exception”,“reason”:“all shards failed”,“phase”:“query”,“grouped”:true,“failed_shards”:[{“shard”:-1,“index”:null,“reason”:{“type”:“null_pointer_exception”,“reason”:null}}],“caused_by”:{“type”:“null_pointer_exception”,“reason”:null}},“status”:500}
SqlOperatorImpl ELASTICSEARCH_SUB_SCAN
Location 0:0:5
Fragment 0:0

[Error Id: 82ed7584-7299-4397-8d95-6fbe2e5b82e0 on scq02-304j08u3233-dp-app-48-37-.host:-1]

(javax.ws.rs.InternalServerErrorException) HTTP 500 Internal Server Error
org.glassfish.jersey.client.JerseyInvocation.convertToException():1032
org.glassfish.jersey.client.JerseyInvocation.translate():819
org.glassfish.jersey.client.JerseyInvocation.access$700():92
org.glassfish.jersey.client.JerseyInvocation$2.call():701
org.glassfish.jersey.internal.Errors.process():315
org.glassfish.jersey.internal.Errors.process():297
org.glassfish.jersey.internal.Errors.process():228
org.glassfish.jersey.process.internal.RequestScope.runInScope():444
org.glassfish.jersey.client.JerseyInvocation.invoke():697
com.dremio.plugins.elastic.ElasticConnectionPool$ElasticConnection.execute():561
com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.getNextPage():275
com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.next():322
com.dremio.sabot.op.scan.ScanOperator.outputData():197
com.dremio.sabot.driver.SmartOp$SmartProducer.outputData():510
com.dremio.sabot.driver.StraightPipe.pump():56
com.dremio.sabot.driver.Pipeline.doPump():82
com.dremio.sabot.driver.Pipeline.pumpOnce():72
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():288
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():284
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1807
com.dremio.sabot.exec.fragment.FragmentExecutor.run():243
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
com.dremio.sabot.task.AsyncTaskWrapper.run():92
com.dremio.sabot.task.slicing.SlicingThread.run():71

2

@JoyJava could you enable ES request logging on all Dremio nodes and share the output for both the failed and the successful runs?

You can do this by adding the following to /<DREMIO_CONFIG_DIRECTORY>/logback.xml on all nodes:

  <appender name="elasticoutput" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${dremio.log.path}/elastic.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>${dremio.log.path}/archive/elastic.%d{yyyy-MM-dd}.log.gz
      </fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder>
      <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n
      </pattern>
    </encoder>
  </appender>
  <logger name="elastic.requests" additivity="false">
    <level value="info"/>
    <appender-ref ref="elasticoutput"/>
  </logger>

Also, are you seeing anything relevant on the Elasticsearch logs at the time you issue this query? Please share if so.