Dremio truncating floating point values

1

I’ve connected Dremio to an Elasticsearch index where I’m storing time series data.

It looks like Dremio is truncating my time values (which are floating point values):

select _uid, datumTime, ceil(datumTime) as ceil_dt, floor(datumTime) as floor_dt from elasticsearch."timedata-*".logs limit 10

image

But I can verify in Kibana that these data are in fact supposed to have decimal values. Here’s one of the documents:

{
  "_index": "timedata-2018.01.17",
  "_type": "logs",
  "_id": "AWEBq9hVt2-twxpHRhEQ",
  "_version": 1,
  "_score": null,
  "_source": {
    "datumTime": 3598996283.886573,
    "@timestamp": "2018-01-17T01:11:34.709Z",
    "port": 5596,
    "@version": "1",
    "host": "10.40.24.12",
    "time": "2018-01-16T17:11:23.886-08:00",
    "value": -75.048318,
  },
  "fields": {
    "@timestamp": [
      1516151494709
    ],
    "time": [
      1516151483886
    ]
  },
  "sort": [
    1516151494709
  ]
}

Why is Dremio truncating the datumTime value? (Interestingly, it does not truncate the value, but it does round slightly: reports -75.04832)

EDIT: Upon further examination, it doesn’t look like Dremio actually even pulls in the right datumTime value at all? Will investigate then edit this post… But regardless, Dremio seems to be truncating.

2

Hi chuchurocket27,

I believe what is happening here is that the mapping for this index shows that the “datumTime” field is a float type, not a double. Can you confirm this is the case. Here is the elasticsearch documentation to see the mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-get-mapping.html

It seems Elasticsearch is very permissive and will allow storing double values even when the mapping indicates float.

However, Dremio assumes the mapping is correct, and coerces the values into float.

If possible, you may want to fix the mapping on the index:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-put-mapping.html

We may consider treating all floating point types (float or double) as double, since elasticsearch does not enforce this.

3

Ah, that looks to be the problem. Will fix the mapping and report back. Thank you!