Dremio does not support Elasticsearch geo_point field

1

Hi
I did an experiment on two different indices.
Index 1: Has geo_point field.
Index 2: Dont have geo_point field. Intead, use float array in the field

Dremio failed to list the schema for the first one and producing the following logs
2017-09-19 16:33:28,752 [263f29a7-7ef0-89e7-a72f-519332e3d200:foreman] INFO c.d.p.e.e.ElasticsearchJsonReader - User Error Occurred [ErrorId: 9e4505a5-0d87-4fcf-818f-89035b72b1dc]
com.dremio.common.exceptions.UserException: IllegalStateException
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:622) ~[dremio-common-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredList(ElasticsearchJsonReader.java:360) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredMap(ElasticsearchJsonReader.java:312) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeToVector(ElasticsearchJsonReader.java:215) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.write(ElasticsearchJsonReader.java:183) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.next(ElasticsearchRecordReader.java:300) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.getSampledSchema(ElasticTableBuilder.java:317) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:183) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:154) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:135) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTableFromDataset(SimpleSchema.java:302) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTableWithRegistry(SimpleSchema.java:265) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTable(SimpleSchema.java:377) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:67) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:219) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:106) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:73) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:187) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:167) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:155) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:285) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.lang.IllegalStateException: null
at com.dremio.plugins.elastic.execution.WriteHolders$InvalidWriteHolder.writeList(WriteHolders.java:58) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.FieldReadDefinition.writeList(FieldReadDefinition.java:126) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredList(ElasticsearchJsonReader.java:357) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
… 41 common frames omitted
2017-09-19 16:33:28,799 [263f29a7-7ef0-89e7-a72f-519332e3d200:foreman] INFO query.logger - {“queryId”:“263f29a7-7ef0-89e7-a72f-519332e3d200”,“schema”:"",“queryText”:“SELECT *\nFROM test2.universal.universal”,“start”:1505810008639,“finish”:1505810008799,“outcome”:“FAILED”,“username”:“weibin.wu”,“commandDescription”:""}

2

Same here, I’ve this mapping on Elasticsearch:

"location" : {
    "type" : "geo_point"
},

And I got this error:

2017-09-26 15:38:38,160 [26359993-fcf5-3ca4-591f-6568c4147b00:foreman] INFO  com.dremio.exec.store.AbstractSchema - User Error Occurred [ErrorId: 3966eb8e-8294-47b5-97e7-277117c9bbf2]
com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for localES.company.Company.
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:622) ~[dremio-common-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.store.SimpleSchema.getTableFromDataset(SimpleSchema.java:317) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.store.SimpleSchema.getTableWithRegistry(SimpleSchema.java:265) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.store.SimpleSchema.getTable(SimpleSchema.java:377) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:67) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
	at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:219) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
	at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:106) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:73) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
	at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:187) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:167) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:155) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:285) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.lang.IllegalStateException: null
	at com.dremio.plugins.elastic.execution.WriteHolders$InvalidWriteHolder.writeMap(WriteHolders.java:63) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.execution.FieldReadDefinition.writeMap(FieldReadDefinition.java:132) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredMap(ElasticsearchJsonReader.java:329) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeToVector(ElasticsearchJsonReader.java:215) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.write(ElasticsearchJsonReader.java:183) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.next(ElasticsearchRecordReader.java:300) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.ElasticTableBuilder.getSampledSchema(ElasticTableBuilder.java:317) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:183) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:154) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:135) ~[dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	at com.dremio.exec.store.SimpleSchema.getTableFromDataset(SimpleSchema.java:302) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
	... 32 common frames omitted

Also, base on Spatial Queries, I’m wondering if Dremio elasticsearch geo fields are or will be supported in the near future.

Thanks!