When we tried to query in the data set. We got this error:
REFRESH METADATA" before rerunning query. SqlOperatorImpl ELASTICSEARCH_SUB_SCAN Location 2:0:2 Fragment 2:0 [Error Id: 45b8e403-65d0-4d8e-be46-baabdeb048ef on daserver:31010] ]
Btw, we’re using elasticsearch as our datasource.
Also got this error, with the following details -
DATA_READ ERROR: Failed to parse date time value in field timestamp.
SqlOperatorImpl ELASTICSEARCH_SUB_SCAN
Location 0:0:6
Fragment 0:0
[Error Id: on :
I’m not sure I understand what is the error, or how it is connected to elastic-search, as queries through REST do work fine.
Hi @ravenr,
Would you be able to share some information about the query?
Here is how to share the query profile: https://www.dremio.com/tutorials/share-query-profile/
From looking at the error it seems like it might also be helpful if you could share some sample data from the “timestamp” field.
Thanks,
Rohan
the data is in unattached network, and cannot be shared as is.
However, the query is the result of the preview query generated by Denodo automatically.
there is a field called @timestamp, but it doesn’t seem to do any problems in other index/type (mapping), what should I be looking for to find the reason for this ?
the @timestamp values are in the default format of elasticsearch which is yyyy-MM-ddTHH:mm:ss.SSSZ
for example, 2016-03-03T12:34:12.000Z
Could see nothing special within those values.
I’ve seen I am not the only one who gets this error.
Hi @ravenr,
From the limited information provided it seems like the issue is happening when Dremio is parsing one of the values in the timestamp field. One way to clarify which field causes the issue would be to set the logging levels in logback.xml (<DREMIO_HOME>/conf/logback.xml) to “debug” and then share the debug messages from server.log (<DREMIO_HOME>/log/server.log).
Thanks,
Rohan
This limited information is all there is out there unfortunately.
Nothing more in the verbose detail errors.
I copied all the data.
It seems to happen in almost all my indexes so I guess it should have happened a lot, and all over.
@ravenr Did you get a chance to look at the debug messages in the log file? Comparing the log output for errors from different indices might help narrow the root cause of the issue down to a specific column or mapping.
It seems like using the debug level in the way you offer cause the whole server to almost hang, and things are not working. It takes almost an hour till something is up, and too many errors.
I am trying to see if the new version solves some of those issues.
till now the 1.0.8 version was used, as I was not able to reproduce those errors on the cloud with the newer version.
The only thing I can get, is this Dremio is complaining about the value ‘2017-04-28 19:59:59’ in the timestamp field. Which is the default elasticsearch field @timestamp
I am also getting the error message (when trying to read from Elasticsearch):
Failed to parse date time value in field
Was this error ever resolved?
I turned on debugging and this is in the server.log file:
2018-02-23 16:37:23,822 [256f4c3b-f706-2eaa-a74e-a387e50cf800:foreman] INFO c.d.p.elastic.execution.WriteHolders - User Error Occurred [ErrorId: bedb558f-28b0-4f41-8376-9c2261fa3258]
com.dremio.common.exceptions.UserException: Failed to parse date time value 2017/01/01 00:33:04.006 in field source_data.scantime.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.WriteHolders.getMillis(WriteHolders.java:368) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.WriteHolders.access$100(WriteHolders.java:45) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.WriteHolders$TimestampWriteHolder.write(WriteHolders.java:356) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.WriteHolders$TimestampWriteHolder.writeMap(WriteHolders.java:347) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.FieldReadDefinition.writeMap(FieldReadDefinition.java:132) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredMap(ElasticsearchJsonReader.java:328) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeDeclaredMap(ElasticsearchJsonReader.java:324) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.writeToVector(ElasticsearchJsonReader.java:214) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.write(ElasticsearchJsonReader.java:182) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.next(ElasticsearchRecordReader.java:328) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.ElasticTableBuilder.getSampledSchema(ElasticTableBuilder.java:322) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:190) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:161) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:142) [dremio-elasticsearch-plugin-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.store.SimpleSchema.getTableFromSource(SimpleSchema.java:366) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.store.SimpleSchema.getTableWithRegistry(SimpleSchema.java:284) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.store.SimpleSchema.getTable(SimpleSchema.java:406) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:67) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.4.4-201801230630490666-6d69d32]
at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:219) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.4.4-201801230630490666-6d69d32]
at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:106) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:73) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201712102340200054-6e60422.jar:1.12.0-201712102340200054-6e60422]
at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:188) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:165) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:153) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-1.4.4-201801230630490666-6d69d32.jar:1.4.4-201801230630490666-6d69d32]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]Was this error ever resolved?
Was this error ever resolved? This error still shows up even after Dremio v4.7.3
If you do not mind, would you be able to send us the profile from the failed job?
I already solve it here. In my case, my timestamp field has DOUBLE type value and not INTEGER or LONG. Maybe fix it next update so every type of number can be parsed? Because elasticsearch could parse it to datetime but dremio couldn’t.