Opening elasticsearch table fails (tested on 5.5 and 6.0)

1

I am using dremio-community-1.3.1-201712020438070881

I can connect to AWS elastic search successfully (by selected querying white listed hosts only)

And I can see the indices listed.
I can click on an index and see the schema.

However when I click on one of the objects to enter the SQL Editor it always fails with the message:
“Failure while retrieving metadata for table “.kibana”.dashboard.”

I initially thought it was my schema, but the default .kibana index created by AWS elastic search also fails.
I have tried both version 5.5 and 6.0 of elastic search.

This is the dremio stack trace:
PLAN ERROR: Failure while retrieving metadata for table test.".kibana".dashboard.

Sql Query SELECT *
FROM test.".kibana".dashboard

(java.lang.NullPointerException) null
com.dremio.plugins.elastic.ElasticTableBuilder.populate():239
com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary():161
com.dremio.plugins.elastic.ElasticTableBuilder.getDataset():142
com.dremio.exec.store.SimpleSchema.getTableFromSource():377
com.dremio.exec.store.SimpleSchema.getTableWithRegistry():297
com.dremio.exec.store.SimpleSchema.getTable():415
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable():67
org.apache.calcite.jdbc.CalciteSchema.getTable():219
org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom():117
org.apache.calcite.prepare.CalciteCatalogReader.getTable():106
org.apache.calcite.prepare.CalciteCatalogReader.getTable():73
org.apache.calcite.sql.validate.EmptyScope.getTableNamespace():71
org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace():189
org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl():104
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():910
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():891
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2859
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2844
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3077
org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
org.apache.calcite.sql.validate.AbstractNamespace.validate():84
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():910
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():891
org.apache.calcite.sql.SqlSelect.validate():208
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():866
org.apache.calcite.sql.validate.SqlValidatorImpl.validate():577
com.dremio.exec.planner.sql.SqlConverter.validate():188
com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode():167
com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert():155
com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan():43
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():66
com.dremio.exec.work.foreman.AttemptManager.run():290
java.util.concurrent.ThreadPoolExecutor.runWorker():-1
java.util.concurrent.ThreadPoolExecutor$Worker.run():-1
java.lang.Thread.run():-1

2

Hi,

Could you request the nodes info from elasicsearch by running this command, and post the results here:

curl -XGET ‘elastic-hostname:9200/_nodes?pretty’

or browsing that address in your browser. (Replace elastic-hostname with the hostname or IP address of your elastic server)

The error you posted suggests there is something missing from that response which Dremio is expecting.

3

Thanks Steven,

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "699310932760:test",
  "nodes" : {
    "EMaaPhbTSWO1ASLcyhhcZQ" : {
      "name" : "EMaaPhb",
      "version" : "6.0.1",
      "build_hash" : "d951bbf",
      "total_indexing_buffer" : 855506944,
      "roles" : [ "master", "data", "ingest" ],
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "available_processors" : 4,
        "allocated_processors" : 4
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 5352,
        "mlockall" : true
      },
      "jvm" : {
        "pid" : 5352,
        "start_time_in_millis" : 1514866864168,
        "mem" : {
          "heap_init_in_bytes" : 8589934592,
          "heap_max_in_bytes" : 8555069440,
          "non_heap_init_in_bytes" : 2555904,
          "non_heap_max_in_bytes" : 0,
          "direct_max_in_bytes" : 8555069440
        },
        "using_compressed_ordinary_object_pointers" : "true"
      },
      "thread_pool" : {
        "force_merge" : {
          "type" : "fixed",
          "min" : 1,
          "max" : 1,
          "queue_size" : -1
        },
        "fetch_shard_started" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 8,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "listener" : {
          "type" : "fixed",
          "min" : 2,
          "max" : 2,
          "queue_size" : -1
        },
        "index" : {
          "type" : "fixed",
          "min" : 4,
          "max" : 4,
          "queue_size" : 200
        },
        "refresh" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 2,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "generic" : {
          "type" : "scaling",
          "min" : 4,
          "max" : 128,
          "keep_alive" : "30s",
          "queue_size" : -1
        },
        "warmer" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 2,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "search" : {
          "type" : "fixed_auto_queue_size",
          "min" : 7,
          "max" : 7,
          "queue_size" : 1000
        },
        "flush" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 2,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "fetch_shard_store" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 8,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "management" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 5,
          "keep_alive" : "5m",
          "queue_size" : -1
        },
        "get" : {
          "type" : "fixed",
          "min" : 4,
          "max" : 4,
          "queue_size" : 1000
        },
        "bulk" : {
          "type" : "fixed",
          "min" : 4,
          "max" : 4,
          "queue_size" : 200
        },
        "snapshot" : {
          "type" : "scaling",
          "min" : 1,
          "max" : 2,
          "keep_alive" : "5m",
          "queue_size" : -1
        }
      },
      "modules" : [ {
        "name" : "aggs-matrix-stats",
        "version" : "6.0.1",
        "description" : "Adds aggregations whose input are a list of numeric fields and output includes a matrix.",
        "classname" : "org.elasticsearch.search.aggregations.matrix.MatrixAggregationPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "analysis-common",
        "version" : "6.0.1",
        "description" : "Adds \"built in\" analyzers to Elasticsearch.",
        "classname" : "org.elasticsearch.analysis.common.CommonAnalysisPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "ingest-common",
        "version" : "6.0.1",
        "description" : "Module for ingest processors that do not require additional security permissions or have large dependencies and resources",
        "classname" : "org.elasticsearch.ingest.common.IngestCommonPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "lang-expression",
        "version" : "6.0.1",
        "description" : "Lucene expressions integration for Elasticsearch",
        "classname" : "org.elasticsearch.script.expression.ExpressionPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "lang-mustache",
        "version" : "6.0.1",
        "description" : "Mustache scripting integration for Elasticsearch",
        "classname" : "org.elasticsearch.script.mustache.MustachePlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "lang-painless",
        "version" : "6.0.1",
        "description" : "An easy, safe and fast scripting language for Elasticsearch",
        "classname" : "org.elasticsearch.painless.PainlessPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "percolator",
        "version" : "6.0.1",
        "description" : "Percolator module adds capability to index queries and query these queries by specifying documents",
        "classname" : "org.elasticsearch.percolator.PercolatorPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "reindex",
        "version" : "6.0.1",
        "description" : "The Reindex module adds APIs to reindex from one index to another or update documents in place.",
        "classname" : "org.elasticsearch.index.reindex.ReindexPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "repository-url",
        "version" : "6.0.1",
        "description" : "Module for URL repository",
        "classname" : "org.elasticsearch.plugin.repository.url.URLRepositoryPlugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      }, {
        "name" : "transport-netty4",
        "version" : "6.0.1",
        "description" : "Netty 4 based transport implementation",
        "classname" : "org.elasticsearch.transport.Netty4Plugin",
        "has_native_controller" : false,
        "requires_keystore" : false
      } ],
      "ingest" : {
        "processors" : [ {
          "type" : "append"
        }, {
          "type" : "attachment"
        }, {
          "type" : "convert"
        }, {
          "type" : "date"
        }, {
          "type" : "date_index_name"
        }, {
          "type" : "dot_expander"
        }, {
          "type" : "fail"
        }, {
          "type" : "foreach"
        }, {
          "type" : "grok"
        }, {
          "type" : "gsub"
        }, {
          "type" : "join"
        }, {
          "type" : "json"
        }, {
          "type" : "kv"
        }, {
          "type" : "lowercase"
        }, {
          "type" : "remove"
        }, {
          "type" : "rename"
        }, {
          "type" : "script"
        }, {
          "type" : "set"
        }, {
          "type" : "sort"
        }, {
          "type" : "split"
        }, {
          "type" : "trim"
        }, {
          "type" : "uppercase"
        }, {
          "type" : "user_agent"
        } ]
      }
    }
  }
}